Support per-turn Runner sessions on ephemeral web servers (#18932)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The web server controls native Runner sessions in remote sandboxes.
> - An ephemeral web server must recover without its previous local
filesystem.
> - The existing admission rule accepts warm sessions. Managed Claude,
Grok, and OpenCode sessions resolve to per-turn execution.
> - Per-turn execution also needs durable provider-session bytes after
the sandbox is retired.
> - This pull request adds encrypted backup publication and removes the
warm-only admission rule.
> - Both lifecycle modes can use the existing native ownership and
checkpoint contracts during a web deployment.

## Linked Issues or Issue Description

Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887.

**What existing behavior does this improve?**

Native Runner execution on web servers with
`PAPERCLIP_EPHEMERAL_SERVER=true`.

**Current behavior**

The admission rule requires a warm Runner lifecycle. Managed provider
credentials can resolve an agent to per-turn execution, so enabling the
ephemeral deployment profile blocks those turns. A database controller
checkpoint does not preserve provider files after the old web home and
per-turn sandbox are removed.

**Proposed behavior**

Admit native execution in either resolved lifecycle. Keep provider and
credential choices unchanged. Publish encrypted provider state to shared
storage before sandbox cleanup. Restore it from a fresh web home for the
next turn. Use the existing controller lease and generation checks for
active takeover.

**Reason and benefit**

Operators can remove the web server volume without excluding per-turn
providers or forcing managed accounts into warm sessions. New native
Daytona profiles automatically enter both rolling-deployment
qualification lanes.

**Breaking changes**

The additive migration creates a backup publication ledger. Existing
volume-backed local-disk installations retain their local backup path.
S3 installations publish encrypted backups and require a stable secrets
master key. Existing-volume enrollment must import surviving session
data before detaching the volume.

## What Changed

- Admit native Runner execution independently of its warm or per-turn
lifecycle. Keep host-owned legacy execution excluded from ephemeral
deployments.
- Store private provider state as encrypted objects. Journal uploads
before I/O, read them back, and fence publication against the current
native owner. Keep the previous published backup when storage or
ownership checks fail.
- Restore exact provider bytes into a private cache. Require an exact
durable backup before sandbox cleanup. Extend the existing bounded
retention pass to retire stored objects without adding idle polling.
- Import selected legacy controller and provider state in the existing
offline maintenance process. Preserve queued work, task pointers,
pending recovery, and original files. Prefer canonical database
checkpoints.
- Preserve already-failed legacy controllers with missing original
artifact proof as encrypted archives. These archives do not become
execution authority. Successful retained sessions still require proof.
Report archival and executable imports separately.
- Verify unmanaged Grok API keys through the existing xAI account
validator. The live test exposed a readiness gap that kept remote Grok
agents paused after platform checks. Preserve failed credentials and the
separate runtime warning.
- Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the
receipt to the exact outbound request/session; reject replay, conflicts,
and incomplete authority. Separate turn cost from cumulative session
cost in the sidecar schema and Rust normalization. Preserve managed
billing identity and identify direct xAI API credentials.
- Preserve published workspace directory names across cold allocations,
including destroyed-allocation cleanup. Keep unpublished epoch/lease
directories isolated and refuse to overwrite an occupied preparation.
This fixes the cold-session identity regression exposed by #18922.
- Increase bounded checkpoint-header inspection to 1 MiB. Retain the
aggregate inspection limit.
- Register all nine native Daytona profiles in per-turn and warm
rolling-deployment suites. Delete the entire departing server home,
exercise ordinary reads and writes, require active ownership transfer,
then check a later turn on another fresh server. Permit only exact
notification replay after reconnect.

Ownership stays with the existing modules: Runner validates its protocol
state; Core validates run scope and controller ownership; storage owns
object I/O; the maintenance process owns offline conversion; the E2E
wrapper owns process replacement and probes. No provider protocol is
added to the shared admission rule. Grok receipt parsing stays in its
provider module. The shared sidecar gains an explicit cost basis because
its existing price field means session-cumulative spend, while Grok
reports prompt totals. Existing consumers retain cumulative behavior
when the new field is absent.

## Verification

- Focused native admission, archive validation, publication fencing,
cleanup, readiness, legacy import, and E2E observer tests pass. Real
PostgreSQL cases verify that queued intent and historical recovery rows
stay unchanged.
- Full typecheck and build pass on the rebased branch. The review
follow-up passes server typecheck, 29 focused
PostgreSQL/archive/readiness tests, and module-boundary checks. Revision
fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no
unresolved threads. The Grok follow-up passes Runner/server typechecks,
238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the
Rust cost regression, and Core billing regression. The e9aae9cf revision
passed all 57 successful checks, with 5 skips and 2 neutral checks, and
exact-head Greptile 5/5. The latest 9cbec30a4f follow-up changes only
E2E catalog/admission controls; its 291 affected tests pass, all 57
check contexts succeeded (5 skips, 2 neutral), and exact-head Greptile
is 5/5 with no open review threads.
- Migration numbering and safety checks pass.
- A local real PostgreSQL/S3 roundtrip preserved 2,870,101,591 payload
bytes across 89,100 archive entries after deleting the source directory.
Restored SHA-256 matched; upload/readback took 56.5 seconds, restore
74.8 seconds, process peak RSS was 1.40 GB, and all fixture resources
were removed.
- Archives preserve contained relative links, reject link
chains/traversal, and enforce independent byte/entry/object limits.
Offline verification streams hashes. Orphaned deleted-run backups stay
on the original volume. Obsolete object retention preserves the selected
backup, live lease references, and pending uploads with owners.
- Isolated API-key Codex deployment: original active runner/session
transferred to the replacement and committed. All 567 reads, 57 writes,
and 57 readbacks succeeded. A second replacement completed a
same-session continuation with all 287 reads and 29 writes succeeding.
The strict cell failed two agent-authored shell commands; this result is
retained as failed.
- Isolated API-key Claude deployment: original active runner/session
transferred and committed. All 584 reads, 59 writes, and 59 readbacks
succeeded. A second replacement completed a same-provider-session
continuation, with 211 reads and 22 writes succeeding. The strict cell
failed an initial completion call that omitted a registered attachment
citation; the agent corrected it. This result is retained as failed.
- Isolated API-key OpenCode deployment: the same active runner/session
transferred and committed. All 481 reads, 49 writes, and 49 readbacks
succeeded. A second replacement retained the provider session and
completed useful work, with 156 reads and 16 writes succeeding. The
strict cell failed agent tool calls: a denied external-directory write,
malformed artifact hashes, and a second completion after the first
result was committed. There was one submitted/completed turn and one
committed result; reconnect notification replay was identical. This
result is retained as failed.
- Grok's real first turn now completes with settled usage and provider
cost ($0.41397); the next turn is admitted normally. That attempt then
exposed a shared workspace-path identity regression before deployment.
Three cold-restore cases reproduced it before the fix; 106
workspace/transport tests and four focused cold-restore cases pass
afterward, including destroyed allocation and occupied-directory
preservation. Server typecheck and build pass. At b5b0dbb03f both
controller switches passed the read/write probe without failure, and the
active runner retained its process while the controller generation
advanced. The third turn exposed an invalid reasoning-token subset
check; ad07e681ba fixes it with the actual receipt regression. Two later
live attempts failed with an upstream Grok rate limit before deployment.
A tiny preflight prompt succeeded before the second attempt, but the
full workload was still rate-limited. The final live gate remains
incomplete. No accounting or identity guard was bypassed. Earlier failed
attempts and their exact resource-cleanup receipts remain recorded.
- Earlier Codex/Claude/OpenCode receipts predate #18922; they establish
their tested revisions. Current-head live qualification remains pending;
#18943 separately preserves pre-enrollment logical/provider workspace
addresses and adds actual legacy migration proof. Its first Codex
migration attempt stopped safely at skill-version verification.
- Earlier attempts remain recorded: missing local provider-pack setup
blocked one Claude launch, and the observer rejected identical reconnect
replay in another. Setup and the observer were corrected before the
latest run.
- Named staging enrollment has not started. Its serving revision and
post-deployment journeys remain separate delivery gates.

- Broader E2E harness validation exposed deployment-catalog and
installed-lane coverage drift; the affected 291 controls now pass.
Twelve other failures reproduce on unchanged base 1a39dea9aa. They
concern cleanup call expectations, dated operating-overhead fixtures, a
Pi profile-version expectation and workflow-security expectations. No
baseline failure is reported as passing.

## Risks

- Provider archives can contain credentials. Objects use authenticated
encryption with scoped associated data and a wrapped data key. The
original stable master key remains required.
- Storage failure prevents publication and sandbox cleanup. Operators
must keep shared storage available and preserve the previous volume
during enrollment.
- Legacy state without verified identity cannot become executable
recovery authority. Archival preserves evidence and existing failures;
it does not recover already-lost provider files or clear blocked tasks.
- Local overlap probes establish the tested application behavior. They
do not establish Railway routing behavior or live managed-subscription
qualification.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and live integration tests. The exact runtime variant and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-04 02:12:51 -06:00
1 parent 1a39dea9aa
commit 612d9931a6
71 files changed
+2799 -207

No files matched your search

@@ -1,6 +1,6 @@
import { afterEach, expect, it, vi } from "vitest";
import { assertEphemeralServerConfig } from "../services/ephemeral-server-contract.js";
import { assertEphemeralWorkspaceExecution, assertEphemeralNativeLifecycle } from "../services/remote-workspaces.js";
import { assertEphemeralWorkspaceExecution, assertEphemeralNativeRuntime } from "../services/remote-workspaces.js";
import { assertEphemeralAgentInstructions } from "../services/agent-instructions.js";
const config = { databaseUrl: "postgresql://runtime:fixture@ep-example-pooler.neon.test/tenant", storageProvider: "s3" as const };
@@ -24,20 +24,20 @@ it("never connects the listener with migration privileges, another tenant, or th
expect(() => assertEphemeralServerConfig(config, { ...env, PAPERCLIP_DATABASE_LISTEN_URL: url })).toThrow("runtime role");
}
});
it("rejects disk-dependent execution and instructions while retaining remote warm execution", () => {
it("requires remote workspace authority and native execution independently of provider lifecycle", () => {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
const remote = { externalConversation: false, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true };
expect(() => assertEphemeralWorkspaceExecution(remote)).not.toThrow();
expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false, pluginTaskWorkspace: true })).not.toThrow();
for (const patch of [{ remoteAuthority: false }, { adapterType: "process" }]) {
expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("warm Paperclip Runner");
expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("native Paperclip Runner");
}
expect(() => assertEphemeralNativeLifecycle("warm")).not.toThrow();
expect(() => assertEphemeralNativeLifecycle("per_turn")).toThrow("warm Paperclip Runner");
expect(() => assertEphemeralNativeLifecycle(undefined)).toThrow("warm Paperclip Runner");
expect(() => assertEphemeralNativeRuntime(true)).not.toThrow();
expect(() => assertEphemeralNativeRuntime(false)).toThrow("native Paperclip Runner");
const agent = { id: "agent", companyId: "company", name: "Agent", adapterConfig: {} };
expect(() => assertEphemeralAgentInstructions(agent)).not.toThrow();
expect(() => assertEphemeralAgentInstructions({ ...agent, adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/old-volume" } })).toThrow("managed agent instructions");
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "false");
expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false })).not.toThrow();
expect(() => assertEphemeralNativeRuntime(false)).not.toThrow();
});
+19
View File
@@ -4,6 +4,7 @@ import { listServerAdapters, requireServerAdapter } from "./registry.js";
import * as executionTarget from "@paperclipai/adapter-utils/execution-target";
import { BUILTIN_ADAPTER_TYPES } from "./builtin-adapter-types.js";
import { QUALIFIED_ACPX_PROFILES, KIMI_DISTRIBUTION_CLOSURE_SHA256 } from "../vendor/paperclip-runner/index.js";
import { assertHarnessTestPassed } from "../services/agent-harness-verification.js";
const { probeInstallation, probeGrokInstallation, probePiInstallation, probeKimiInstallation, probeKimiModels, probeKimiRemote, verifyApiKey } = vi.hoisted(() => ({
verifyApiKey: vi.fn(),
@@ -222,6 +223,24 @@ describe("native ACPX environment checks", () => {
expect(result.checks.map(check => check.code)).toEqual(["acpx_runtime_ready", "ai_connection_api_key_reverified"]);
});
it.each(["verified", "rejected", "unavailable", "missing", "managed"])("keeps remote Grok adoption tied to account verification: %s", async state => {
if (state === "rejected") verifyApiKey.mockRejectedValue({ details: { code: "ai_connection_api_key_rejected" } });
if (state === "unavailable") verifyApiKey.mockRejectedValue(new Error("Provider unavailable"));
const result = await requireServerAdapter("paperclip_runner").testEnvironment!({ ...context,
config: { provider: "acpx", acpxAgent: "grok", model: "grok-4.7",
env: state === "missing" ? {} : { XAI_API_KEY: "fixture-xai-key" },
...(state === "managed" ? { managedAiConnection: { method: "api_key" } } : {}) },
executionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", providerKey: "test",
runner: { execute: vi.fn().mockResolvedValue({ exitCode: 0, timedOut: false, stdout: "Linux\nx86_64\n" }) } },
});
if (["missing", "managed"].includes(state)) expect(verifyApiKey).not.toHaveBeenCalled();
else expect(verifyApiKey).toHaveBeenCalledExactlyOnceWith("xai", "fixture-xai-key");
expect(result.checks[0]).toMatchObject({ code: "acpx_remote_runtime_unverified", level: "warn" });
if (state === "verified") expect(() => assertHarnessTestPassed(result)).not.toThrow();
else expect(() => assertHarnessTestPassed(result)).toThrow("The harness test failed");
expect(probeGrokInstallation).not.toHaveBeenCalled();
});
it.each(["ai_connection_api_key_rejected", "ai_connection_verification_failed"])("fails setup on %s", async code => {
verifyApiKey.mockRejectedValue({ details: { code } });
const result = await requireServerAdapter("paperclip_runner").testEnvironment!({ ...context,
+11 -12
View File
@@ -150,29 +150,28 @@ import {
resolvePaperclipRunnerProviderProfile,
} from "../services/native-runtime/provider-profile.js";
/** Reuse the existing provider key check without requiring a global Claude CLI.
* Readiness and account access are separate checks; model access is proven by a task.
*/
async function verifyNativeClaudeApiKey(
async function verifyNativeAcpxApiKey(
config: Record<string, unknown>,
result: AdapterEnvironmentTestResult,
): Promise<AdapterEnvironmentTestResult> {
const env = config.env as Record<string, unknown> | undefined;
if (config.acpxAgent !== "claude" || !env || typeof env.ANTHROPIC_API_KEY !== "string"
|| !env.ANTHROPIC_API_KEY || config.managedAiRouting || config.managedAiConnection
|| ["ANTHROPIC_BASE_URL", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY"].some(key => env[key])) return result;
const account = config.acpxAgent === "claude" ? { provider: "anthropic" as const, name: "Anthropic", key: "ANTHROPIC_API_KEY",
routing: ["ANTHROPIC_BASE_URL", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY"] }
: config.acpxAgent === "grok" ? { provider: "xai" as const, name: "xAI", key: "XAI_API_KEY", routing: ["XAI_BASE_URL", "GROK_BASE_URL"] } : null;
if (!account || !env || typeof env[account.key] !== "string" || !env[account.key]
|| config.managedAiRouting || config.managedAiConnection || account.routing.some(key => env[key])) return result;
try {
await validateAiApiKey("anthropic", env.ANTHROPIC_API_KEY);
await validateAiApiKey(account.provider, env[account.key] as string);
return { ...result, checks: [...result.checks, {
code: "ai_connection_api_key_reverified", level: "info",
message: "Anthropic verified this API key. Model access is checked when the agent runs.",
message: `${account.name} verified this API key. Model access is checked when the agent runs.`,
}] };
} catch (error) {
const details = error && typeof error === "object" && "details" in error
? error.details as { code?: string } | undefined : undefined;
return { ...result, status: "fail", checks: [...result.checks, {
code: details?.code === "ai_connection_api_key_rejected" ? details.code : "ai_connection_verification_failed",
level: "error", message: "Could not verify the Anthropic API key. Check the connection and try again.",
level: "error", message: `Could not verify the ${account.name} API key. Check the connection and try again.`,
}] };
}
}
@@ -527,7 +526,7 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
if (!((os === "Linux" && (arch === "x86_64" || (profile.acpxAgent === "claude" && ["aarch64", "arm64"].includes(arch)))) || (os === "Darwin" && (arch === "arm64" || (!["grok"].includes(profile.acpxAgent) && arch === "x86_64"))))) {
throw new Error(`ACPX ${profile.acpxAgent} requires a qualified Linux ${profile.acpxAgent === "claude" ? "x64/ARM64" : "x64"} or macOS architecture.`);
}
return verifyNativeClaudeApiKey(context.config, {
return verifyNativeAcpxApiKey(context.config, {
adapterType: "paperclip_runner", status: "warn" as const, testedAt: new Date().toISOString(),
checks: [{ code: "acpx_remote_runtime_unverified", level: "warn" as const,
message: "The remote platform is supported. Runtime package integrity and readiness must still be verified by the remote runner before launch." }],
@@ -535,7 +534,7 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
}
const { probeAcpxClaudeInstallation, probeAcpxGrokInstallation, probeAcpxCursorInstallation, probeAcpxPiInstallation } = await import("../vendor/paperclip-runner/live/index.js");
await (profile.acpxAgent === "pi" ? probeAcpxPiInstallation : profile.acpxAgent === "grok" ? probeAcpxGrokInstallation : profile.acpxAgent === "cursor" ? probeAcpxCursorInstallation : probeAcpxClaudeInstallation)(profile.model);
return verifyNativeClaudeApiKey(context.config, {
return verifyNativeAcpxApiKey(context.config, {
adapterType: "paperclip_runner", status: "pass" as const, testedAt: new Date().toISOString(),
checks: [{ code: "acpx_runtime_ready", level: "info" as const, message: `ACPX ${profile.acpxAgent} runtime is installed and verified. Model access is checked when it runs.` }],
});
@@ -9,6 +9,8 @@ import { skillPackageMaintenance } from "../services/skill-package-maintenance.j
import { hasUnsettledLocalInstructionCopies } from "../services/agent-instruction-working-copies.js";
import { providerTraceStore } from "../services/provider-trace-store.js";
import { migrateWorkspaceTrees } from "../services/remote-workspace-migration-batch.js";
import { migrateNativeHarnessBackups } from "../services/native-runtime/native-harness-backup-migration.js";
import { migrateNativeControllers } from "../services/native-runtime/native-controller-migration.js";
/** Runs only in the maintenance process after Cloud has confirmed all legacy
* deployments stopped. Each owner retains its own migration/publication rules;
@@ -25,12 +27,22 @@ export async function migrateEphemeralServerData(db: Db, input: {
const before = await readEphemeralServerDataEligibility(db);
const unsupported = before.blockers.filter(blocker => ![
"agent_files_require_migration", "run_logs_require_migration", "provider_traces_require_migration", "skills_require_migration", "workspaces_require_migration",
"native_harness_backups_require_migration",
"native_controllers_require_migration",
].includes(blocker));
if (unsupported.length) throw new Error(unsupported[0]);
phase("workspaces");
const storage = getStorageService();
if (storage.provider !== "s3") throw new Error("shared_object_storage_required");
phase("native_controllers");
const controllers = await migrateNativeControllers(db, { writersStopped: true, signal, storage });
counts.nativeControllers = controllers.imported;
counts.nativeControllersUnchanged = controllers.unchanged;
counts.nativeControllerSourcesAlreadyAbsent = controllers.sourceAlreadyAbsent;
counts.nativeControllersArchivedFailed = controllers.archivedFailed;
phase("native_harness_backups");
counts.nativeHarnessBackups = await migrateNativeHarnessBackups(db, { storage, writersStopped: true, signal });
phase("workspaces");
counts.workspaces = await migrateWorkspaceTrees(db, { storage, signal, phase });
phase("agent_files");
+5 -9
View File
@@ -56,7 +56,7 @@ import {
import { environmentService } from "./environments.js";
import { instanceSettingsService } from "./instance-settings.js";
import { hasSandboxSnapshotQualification, qualifiedSandboxSnapshotId, usesManagedCloudSandboxImage, type SandboxTemplateIdentity } from "./sandbox-snapshot-qualification.js";
import { verifyNativeHarnessBackupStamp } from "./native-runtime/native-harness-backup-stamp.js";
import { verifyRecoverableNativeHarnessBackup } from "./native-runtime/native-harness-backup-storage.js";
import {
collectEnvironmentSecretRefs,
parseEnvironmentDriverConfig,
@@ -1847,10 +1847,8 @@ function createSandboxEnvironmentDriver(
if (!providerLease) {
if (
input.adapterType === "paperclip_runner" &&
!verifyNativeHarnessBackupStamp(
reusableLease.metadata?.nativeHarnessBackup,
reusableLease.providerLeaseId,
)
!await verifyRecoverableNativeHarnessBackup({ db, companyId: reusableLease.companyId,
stamp: reusableLease.metadata?.nativeHarnessBackup, providerLeaseId: reusableLease.providerLeaseId })
) {
throw new RunnerHarnessBackupUnavailableError(
reusableLease.providerLeaseId,
@@ -3986,10 +3984,8 @@ export function environmentRuntimeService(
"paperclip_runner" &&
leaseSnapshot.metadata?.sandboxLeaseAcquisition &&
(!leaseSnapshot.providerLeaseId ||
!verifyNativeHarnessBackupStamp(
leaseSnapshot.metadata.nativeHarnessBackup,
leaseSnapshot.providerLeaseId,
))
!await verifyRecoverableNativeHarnessBackup({ db, companyId: leaseSnapshot.companyId,
stamp: leaseSnapshot.metadata.nativeHarnessBackup, providerLeaseId: leaseSnapshot.providerLeaseId }))
) {
throw new RunnerHarnessBackupUnavailableError(
leaseSnapshot.providerLeaseId ?? leaseSnapshot.id,
@@ -4,6 +4,8 @@ import type { Config } from "../config.js";
import { isValidLocalEncryptedMasterKey } from "../secrets/local-encrypted-provider.js";
import { agentInstructionsBundleMode } from "./agent-instructions.js";
import { skillInstallationMaintenance } from "./skill-installation-maintenance.js";
import { hasLocalNativeHarnessObligations } from "./native-runtime/native-harness-backup-migration.js";
import { hasLocalNativeControllerObligations } from "./native-runtime/native-controller-migration.js";
export const ephemeralServerEnabled = () => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true";
@@ -64,6 +66,8 @@ export async function readEphemeralServerDataEligibility(db: Db, options: { purp
isNull(providerTraceRecords.deletedAt), sql`${providerTraceRecords.traceRef} not like 's3-segments-v1:%'`,
)).limit(1);
if (localTrace) blockers.push("provider_traces_require_migration");
if (await hasLocalNativeHarnessObligations(db)) blockers.push("native_harness_backups_require_migration");
if (await hasLocalNativeControllerObligations(db)) blockers.push("native_controllers_require_migration");
const skills = skillInstallationMaintenance(db);
let after: string | undefined;
for (;;) {
@@ -432,14 +432,14 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
const { input, run } = await fixture("process");
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
expect(input.nativeRuntimeResolution.kind).toBe("legacy");
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner");
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native Paperclip Runner");
expect((await read(run.id)).runtimeMode).toBe("legacy");
expect(await coordinators(run.id)).toHaveLength(0);
expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled();
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
});
it.each(["warm", "per_turn"] as const)("checks resolved %s policy for ephemeral admission", async mode => {
it.each(["warm", "per_turn"] as const)("preserves resolved %s policy on an ephemeral server", async mode => {
const { input, run } = await fixture();
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
// Deliberately disagree with the resolved provider policy in both cases.
@@ -447,14 +447,41 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true,
effectiveCapabilities: { reusableLeases: true } as never,
runnerLifecyclePolicy: mode === "warm" ? { mode, idleTimeoutMs: 300_000 } : { mode, idleTimeoutMs: null } };
if (mode === "per_turn") {
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner");
expect(await coordinators(run.id)).toHaveLength(0);
} else {
const selected = await selectHeartbeatRuntime(db, input);
expect(selected.selected && selected.nativeExecution?.session.lifecyclePolicy.mode).toBe("warm");
expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { session: { lifecyclePolicy: { mode: "warm" } } } });
}
const selected = await selectHeartbeatRuntime(db, input);
expect(selected.selected && selected.nativeExecution?.session.lifecyclePolicy.mode).toBe(mode);
expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { session: { lifecyclePolicy: { mode } } } });
expect(await coordinators(run.id)).toHaveLength(1);
});
it.each([
{ provider: "acpx", acpxAgent: "claude", model: "claude-opus-5-5", credentialProvider: "anthropic" as const, method: "subscription" as const },
{ provider: "acpx", acpxAgent: "grok", model: "grok-4.7", credentialProvider: "xai" as const, method: "subscription" as const },
{ provider: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731", credentialProvider: "openrouter" as const, method: "api_key" as const },
])("admits managed $provider/$acpxAgent on an ephemeral server without changing credential lifecycle", async profile => {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
const { input, run, issue } = await fixture();
const { credentialProvider, method, ...providerConfig } = profile;
const adapterConfig = { ...providerConfig, lifecycleMode: "warm" };
input.agent = { ...input.agent, adapterConfig };
await db.update(agents).set({ adapterConfig }).where(eq(agents.id, input.agent.id));
input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true,
effectiveCapabilities: { reusableLeases: true } as never, runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } };
input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig,
agent: input.agent, issue, target: input.workspace.executionTarget, workspaceId: null });
expect(input.nativeRuntimeResolution.kind).toBe("native");
const attribution = { connectionId: randomUUID(), grantId: randomUUID(), provider: credentialProvider, method,
mode: "responsible_user" as const, responsibleUserId: "task-owner" };
input.config.managedAiRuntime = { sessionIdentity: "account:epoch-1", identity: "account:generation-1",
config: { env: {}, managedAiConnection: { ...attribution, identity: "account:generation-1", sessionIdentity: "account:epoch-1" } },
attribution, accountName: "Saved account", accountOwnerUserId: "task-owner", home: instructionRoot, cleanup: async () => {} };
const selected = await selectHeartbeatRuntime(db, input);
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
expect(selected.nativeExecution.provider.kind).toBe(profile.provider);
expect(selected.nativeExecution.session.lifecyclePolicy).toEqual({ mode: "per_turn", idleTimeoutMs: null });
expect((await read(run.id)).runnerProfileJson).toMatchObject({
nativeExecutionInput: { session: { lifecyclePolicy: { mode: "per_turn" } } },
});
expect(await coordinators(run.id)).toHaveLength(1);
});
it.each([
@@ -1,7 +1,7 @@
import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js";
import { museRunnerBroker } from "../muse-runner-broker.js";
import { assertEphemeralNativeLifecycle, readRemoteWorkspaceAuthority } from "../remote-workspaces.js";
import { assertEphemeralNativeRuntime, readRemoteWorkspaceAuthority } from "../remote-workspaces.js";
import { measureRunTransaction } from "../execution-transaction-timing.js";
import { measureRunPhase } from "../execution-preparation-timing.js";
import {
@@ -269,7 +269,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
const runOptions = input.recovery;
const { providerTraceRequested, providerTraceCapture } = input.trace;
if (nativeRuntimeResolution.kind !== "native" && !isDotRun) assertEphemeralNativeLifecycle(undefined);
if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native");
let nativeExecution: NativeExecutionInput | null = null;
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
let nativeRunnerInstanceId: string | null = null;
@@ -522,9 +522,6 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
(["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession
? { mode: "per_turn" as const, idleTimeoutMs: null }
: environmentLifecyclePolicy ?? agentLifecyclePolicy);
if (!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend)) {
assertEphemeralNativeLifecycle(effectiveLifecyclePolicy.mode);
}
if (
effectiveLifecyclePolicy.mode === "warm" &&
executionTarget?.kind === "remote" &&
@@ -88,13 +88,14 @@ postgresTests("upgrade readiness persisted execution evidence", () => {
finally { await core.stop(); await rm(root, { recursive: true, force: true }); }
expect(await readUpgradeReadiness(db, input)).toMatchObject({ ready: true, recoverableRuns: 1 });
const cipher = createLocalEncryptedCipher("write");
// Real staging headers contain retained launch metadata around 113 KiB.
const representativeHeader = JSON.stringify({ ...JSON.parse(header), padding: "x".repeat(100_000) });
const parsedHeader = JSON.parse(header);
const representativeHeader = JSON.stringify({ ...parsedHeader, state: { ...parsedHeader.state,
runAttachTemplate: { provider: { instructions: "x".repeat(300_000) } } } });
await db.update(nativeRunnerCheckpoints).set({ material: cipher.encrypt(JSON.stringify({ ...binding, part: "header", contents: representativeHeader })) })
.where(eq(nativeRunnerCheckpoints.runId, runId));
expect((await readUpgradeReadiness(db, input)).ready).toBe(true);
for (const contents of ["{", "{}", JSON.stringify({ ...JSON.parse(header), schema: "paperclip.runner.controller-checkpoint.v1" }),
JSON.stringify({ ...JSON.parse(header), padding: "x".repeat(400_000) }),
JSON.stringify({ ...JSON.parse(header), padding: "x".repeat(1_100_000) }),
JSON.stringify({ ...JSON.parse(header), state: { ...JSON.parse(header).state, identity: { ...JSON.parse(header).state.identity, runId: randomUUID() } } })]) {
await db.update(nativeRunnerCheckpoints).set({ material: cipher.encrypt(JSON.stringify({ ...binding, part: "header", contents })) })
.where(eq(nativeRunnerCheckpoints.runId, runId));
@@ -1,7 +1,8 @@
import type { Db } from "@paperclipai/db";
import { sql } from "drizzle-orm";
import { remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
import { inspectNativeRunnerUpgradeHeader } from "../native-runtime/native-runner-checkpoint.js";
import { inspectNativeRunnerUpgradeHeader, NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES,
NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES } from "../native-runtime/native-runner-checkpoint.js";
import { readNativeWorkspaceSyncReference } from "../native-runtime/native-workspace-sync.js";
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../native-runtime/native-runner-recovery-contract.js";
import { currentNativeControllerIdentity } from "../native-runtime/native-restart-recovery.js";
@@ -30,7 +31,8 @@ export async function readUpgradeReadiness(db: Db, input: {
r.runner_profile_json->'nativeWorkspaceSync' as workspace,
r.runner_profile_json->'remoteWorkspace' as "remoteWorkspace",
r.runner_profile_json ? 'remoteWorkspace' as "hasRemoteWorkspace", p.id as "pluginLeaseId",
case when pg_column_size(c.material) <= 262144 and sum(pg_column_size(c.material)) over () <= 8388608 then c.material else null end as material, (
case when pg_column_size(c.material) <= ${NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES}
and sum(pg_column_size(c.material)) over () <= ${NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES} then c.material else null end as material, (
r.runtime_mode = 'native' and a.adapter_type = 'paperclip_runner'
and r.runner_instance_id is not null and r.native_session_id is not null
and r.process_pid is not null and r.process_started_at is not null
@@ -444,7 +444,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local folder retained"));
process.env.PAPERCLIP_EPHEMERAL_SERVER = String(ephemeral);
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(ephemeral
? "Volume-free servers require warm Paperclip Runner execution with remote workspace storage." : "local folder retained");
? "Volume-free servers require native Paperclip Runner execution with remote workspace storage." : "local folder retained");
if (ephemeral) expect(resolve).not.toHaveBeenCalled();
else expect(resolve).toHaveBeenCalledOnce();
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
@@ -0,0 +1,40 @@
import { createHash } from "node:crypto";
import { mkdtemp, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { Db } from "@paperclipai/db";
import type { StorageService } from "../../storage/types.js";
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
import { persistNativeHarnessBackup, verifyDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
type Source = { companyId: string; issueId: string; runId: string; controller: string; supervisor?: string };
const sha256 = (value: string) => createHash("sha256").update(value).digest("hex");
export function legacyNativeControllerArchive(source: Source) {
const manifest = JSON.stringify({ schema: "paperclip.legacy-controller-archive.v1", companyId: source.companyId,
issueId: source.issueId, runId: source.runId, reason: "original_runner_artifact_unproven",
files: { "control-plane-state.json": sha256(source.controller),
...(source.supervisor === undefined ? {} : { "supervisor.json": sha256(source.supervisor) }) } });
return { manifest, scopeDigest: sha256(manifest), manifestSha256: `sha256:${sha256(manifest)}` };
}
export async function hasArchivedLegacyNativeController(db: Db, input: Source & { storage: StorageService; signal?: AbortSignal }) {
return verifyDurableNativeHarnessBackup({ db, storage: input.storage, companyId: input.companyId,
...legacyNativeControllerArchive(input), signal: input.signal });
}
export async function archiveFailedLegacyNativeController(db: Db, input: Source & {
storage: StorageService; writersStopped: true; signal: AbortSignal;
}) {
if (await hasArchivedLegacyNativeController(db, input)) return;
const descriptor = legacyNativeControllerArchive(input);
const root = await mkdtemp(join(tmpdir(), "pc-legacy-controller-archive-"));
try {
await writeFile(join(root, "manifest.json"), descriptor.manifest, { mode: 0o600 });
await writeFile(join(root, "control-plane-state.json"), input.controller, { mode: 0o600 });
if (input.supervisor !== undefined) await writeFile(join(root, "supervisor.json"), input.supervisor, { mode: 0o600 });
await persistNativeHarnessBackup({ db, storage: input.storage, root, scopeDigest: descriptor.scopeDigest, signal: input.signal,
owner: { kind: "offline_migration", writersStopped: input.writersStopped, terminalFailureOnly: true,
companyId: input.companyId, issueId: input.issueId, runId: input.runId } });
} finally { await removeNativeHarnessBackup(root); }
}
@@ -0,0 +1,156 @@
import { createHash, randomUUID } from "node:crypto";
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { eq } from "drizzle-orm";
import { afterEach, expect, it, vi } from "vitest";
import { agentTaskSessions, heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations, nativeRunnerCheckpoints } from "@paperclipai/db";
import { DurablePrpControlPlane, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
import { nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
import { nativeControllerMigrationDescriptor } from "./native-session-executor.js";
import { hasLocalNativeControllerObligations, migrateNativeControllers } from "./native-controller-migration.js";
import { resolveNativeControllerRecovery } from "./native-controller-recovery.js";
import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js";
import { nativeSha256 } from "./canonical.js";
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
import { createStorageService } from "../../storage/service.js";
import { legacyNativeControllerArchive } from "./native-controller-archive.js";
import { restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
describeEmbeddedPostgres.each(["import", "archive", "success"])("offline native controller migration (%s)", scenario => {
const environment = nativeRestartFixture();
afterEach(() => vi.unstubAllEnvs());
async function fixture() {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "73".repeat(32));
const f = await environment.seedRun(randomUUID());
const { companyId, agentId, paperclipHome, runtimeRoot } = environment.current;
const binding = { companyId, agentId, issueId: f.issueId, runId: f.runId, executionWorkspaceId: f.native.environmentLeaseId };
const execution = parseNativeExecutionInput({
schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null }, binding,
task: { identifier: "MIG-1", title: "Retained work", description: null, prompt: "Continue the work.", workMode: "standard" },
workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: f.native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1,
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: randomUUID(), sha256: "fixture", schemaVersion: "paperclip.completion-contract.v1",
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Save work." }] } },
credentialBindings: [], interactionResponses: [],
});
const fingerprint = nativeToolContractFingerprintForTarget("remote", "codex");
const paths = nativeControllerMigrationDescriptor(execution, fingerprint);
const identity = { runId: f.runId, runnerInstanceId: f.native.runnerInstanceId, normalizedSessionId: f.native.normalizedSessionId,
environmentLeaseId: f.native.environmentLeaseId, turnId: f.native.turnId, itemId: f.native.itemId };
const core = new DurablePrpControlPlane({ stateDirectory: join(paths.root, "control-plane"), identity,
expectedRunnerVersion: "migration-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
core.issueBootstrapTicket(); core.queueCommand("session.snapshot", {}, "retained-command"); await core.stop();
const file = join(paths.root, "control-plane/control-plane-state.json");
const state = JSON.parse(await readFile(file, "utf8"));
state.tickets = {};
const source = JSON.stringify(state); await writeFile(file, source, { mode: 0o600 });
const packBinding = { companyId, agentId, workspaceId: binding.executionWorkspaceId,
sessionId: f.native.normalizedSessionId, runnerInstanceId: f.native.runnerInstanceId,
environmentLeaseId: f.native.environmentLeaseId, remoteCwd: "/remote/workspace", providerDigest: nativeSha256(execution.provider) };
const payload = {
runnerSourceRevision: "a".repeat(40), target: { platform: "linux", architecture: "x64" },
pins: { opencode: "1.18.34" }, artifacts: {
nodeCommand: { path: "node", sha256: `sha256:${"1".repeat(64)}` },
opencodeCommand: { path: "opencode", sha256: `sha256:${"2".repeat(64)}` },
},
};
const pack = resolveRemoteProviderPackAuthority({ stateDirectory: paths.root, reattach: false, binding: packBinding,
current: () => ({ schema: "paperclip-runner/remote-provider-pack/v1", payload,
digest: `sha256:${createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex")}` }),
runnerArtifact: () => ({ version: "migration-fixture", digest: `sha256:${"a".repeat(64)}` }) });
pack.verified();
const snapshot = { backendKind: "mock", sessionId: f.native.normalizedSessionId,
identity: { companyId, agentId, issueId: f.issueId, runId: f.runId, sessionId: f.native.normalizedSessionId } };
const profile = { ...f.preparedProfile, nativeExecutionInput: execution, nativeToolContractFingerprint: fingerprint, sessionCheckpoint: snapshot,
nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared", descriptorSha256: "a".repeat(64),
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: f.native.environmentLeaseId,
providerLeaseId: randomUUID(), remoteCwd: "/remote/workspace", resourceDisposition: "destroy" } };
await f.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
await f.db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null }).where(eq(nativeRunFinalizations.runId, f.runId));
const queuedId = randomUUID();
await f.db.insert(heartbeatRuns).values({ id: queuedId, companyId, agentId, issueId: f.issueId, status: "queued" });
await f.db.update(issues).set({ executionRunId: queuedId }).where(eq(issues.id, f.issueId));
const [session] = await f.db.insert(agentTaskSessions).values({ companyId, agentId, adapterType: "paperclip_runner",
taskKey: f.issueId, lastRunId: f.runId }).returning();
return { ...f, companyId, agentId, paperclipHome, runtimeRoot, paths, file, source, profile, session, queuedId, identity, snapshot };
}
if (scenario === "import") it("imports an exact settled authority once, preserves queued work, and resumes from an empty home", async () => {
const f = await fixture();
const { companyId, agentId, paperclipHome, runtimeRoot, profile, file, source, session, queuedId, identity, snapshot } = f;
expect(await hasLocalNativeControllerObligations(f.db)).toBe(true);
await f.db.update(heartbeatRuns).set({ runnerProfileJson: { ...profile,
nativeWorkspaceSync: { ...profile.nativeWorkspaceSync, leaseId: randomUUID() } } }).where(eq(heartbeatRuns.id, f.runId));
await expect(migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) }))
.rejects.toThrow("native_controller_migration_artifact_binding");
await f.db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
await f.db.update(nativeRunFinalizations).set({ leaseOwner: "unsettled-owner" }).where(eq(nativeRunFinalizations.runId, f.runId));
await expect(migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) })).rejects.toThrow("normal_ownership_recovery_required");
expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toHaveLength(0);
await f.db.update(nativeRunFinalizations).set({ leaseOwner: "expired-owner", leaseExpiresAt: new Date(0),
phase: "retryable_failure", recoveryState: "resuming_session" }).where(eq(nativeRunFinalizations.runId, f.runId));
const [preservedOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
expect(await migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) }))
.toMatchObject({ imported: 1, sourceAlreadyAbsent: 0 });
expect(await migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) })).toMatchObject({ unchanged: 1, imported: 0 });
expect(await hasLocalNativeControllerObligations(f.db)).toBe(false);
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(preservedOwner);
expect(await readFile(file, "utf8")).toBe(source);
expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, session.id)))[0].lastRunId).toBe(f.runId);
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queuedId)))[0].status).toBe("queued");
await rm(paperclipHome, { recursive: true, force: true });
await f.db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, nativeSessionId: f.native.normalizedSessionId,
status: "running", runtimeMode: "native" }).where(eq(heartbeatRuns.id, queuedId));
const recovered = await resolveNativeControllerRecovery(f.db, { companyId, agentId, issueId: f.issueId,
normalizedSessionId: f.native.normalizedSessionId, currentRunId: queuedId });
expect(recovered.kind).toBe("recoverable");
if (recovered.kind !== "recoverable") throw new Error(JSON.stringify(recovered));
const replacement = join(runtimeRoot, "replacement"); await mkdir(replacement, { mode: 0o700 });
recovered.checkpoint.restoreSessionDirectory(replacement);
expect(JSON.parse(await readFile(join(replacement, "control-plane/control-plane-state.json"), "utf8")).identity).toEqual(identity);
expect(recovered.snapshot).toEqual(snapshot);
});
if (scenario === "archive") it("preserves an already-failed legacy controller without inventing executable recovery authority", async () => {
const f = await fixture();
await rm(join(f.paths.root, "provider-pack-authority.json"));
await f.db.update(nativeRunFinalizations).set({ recoveryState: "blocked", failureCode: "native_provider_terminal_failed" })
.where(eq(nativeRunFinalizations.runId, f.runId));
const [originalRun] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const [originalOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
const storage = createStorageService(createLocalDiskStorageProvider(join(f.runtimeRoot, "durable-objects")));
const options = { storage, writersStopped: true as const, signal: AbortSignal.timeout(30_000) };
const put = vi.spyOn(storage, "putFile").mockRejectedValue(new Error("storage unavailable"));
await expect(migrateNativeControllers(f.db, options)).rejects.toThrow("storage unavailable");
put.mockRestore();
expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(true);
expect(await migrateNativeControllers(f.db, options)).toMatchObject({ archivedFailed: 1, imported: 0 });
expect(await migrateNativeControllers(f.db, options)).toMatchObject({ archivedFailed: 1, imported: 0 });
expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(false);
expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toHaveLength(0);
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(originalRun);
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(originalOwner);
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(f.queuedId);
expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, f.session.id)))[0].lastRunId).toBe(f.runId);
await rm(f.paperclipHome, { recursive: true, force: true });
const descriptor = legacyNativeControllerArchive({ companyId: f.companyId, issueId: f.issueId, runId: f.runId, controller: f.source });
const replacement = join(f.runtimeRoot, "archived-inspection");
expect(await restoreDurableNativeHarnessBackup({ db: f.db, storage, companyId: f.companyId,
scopeDigest: descriptor.scopeDigest, backupRoot: replacement })).toBe(true);
expect(await readFile(join(replacement, "current", "control-plane-state.json"), "utf8")).toBe(f.source);
const recovery = await resolveNativeControllerRecovery(f.db, { companyId: f.companyId, agentId: f.agentId,
issueId: f.issueId, normalizedSessionId: f.native.normalizedSessionId, currentRunId: f.queuedId });
expect(recovery.kind).not.toBe("recoverable");
});
if (scenario === "success") it("does not substitute a failure archive for a successful retained session", async () => {
const f = await fixture();
await rm(join(f.paths.root, "provider-pack-authority.json"));
await f.db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId));
const storage = createStorageService(createLocalDiskStorageProvider(join(f.runtimeRoot, "durable-objects")));
await expect(migrateNativeControllers(f.db, { storage, writersStopped: true, signal: AbortSignal.timeout(30_000) }))
.rejects.toThrow("native_runner_checkpoint_artifact_unproven");
expect(await f.db.select().from(nativeHarnessBackups)).toHaveLength(0);
expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(true);
});
});
@@ -0,0 +1,163 @@
import { constants } from "node:fs";
import { lstat, open, readdir } from "node:fs/promises";
import { join } from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { heartbeatRuns, type Db } from "@paperclipai/db";
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
import { findNativeSessionResumeRun } from "./native-session-resume.js";
import { nativeControllerMigrationDescriptor } from "./native-session-executor.js";
import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { importNativeControllerRecovery, readNativeControllerRecovery } from "./native-controller-recovery.js";
import { nativeHarnessStateBase } from "./native-harness-backup-stamp.js";
import { providerPackWorkspaceId, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
import { nativeSha256 } from "./canonical.js";
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
import type { StorageService } from "../../storage/types.js";
import { durableNativeHarnessStorage } from "./native-harness-backup-storage.js";
import { archiveFailedLegacyNativeController, hasArchivedLegacyNativeController } from "./native-controller-archive.js";
type Selected = { id: string; companyId: string; agentId: string; issueId: string; sessionId: string; createdAt: Date };
async function* retainedSources(db: Db, signal?: AbortSignal) {
let after: string | undefined;
for (;;) {
signal?.throwIfAborted();
const rows = await db.execute<Selected>(sql`select s.id, s.company_id as "companyId", s.agent_id as "agentId",
r.native_issue_id as "issueId", r.native_session_id as "sessionId", r.created_at as "createdAt"
from agent_task_sessions s join heartbeat_runs r on r.id = s.last_run_id
and r.company_id = s.company_id and r.agent_id = s.agent_id
where s.adapter_type = 'paperclip_runner' and r.native_issue_id is not null and r.native_session_id is not null
${after ? sql`and s.id > ${after}::uuid` : sql``} order by s.id limit 100`);
for (const selected of rows) {
signal?.throwIfAborted();
const source = await findNativeSessionResumeRun(db, { companyId: selected.companyId, agentId: selected.agentId,
issueId: selected.issueId, normalizedSessionId: selected.sessionId,
beforeCreatedAt: new Date(selected.createdAt), includeProgressBarrier: true });
if (source) yield { selected, source };
}
if (rows.length < 100) return;
after = rows.at(-1)!.id;
}
}
async function readPrivateSource(file: string, maxBytes: number): Promise<string | undefined> {
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return undefined; throw error;
});
if (!handle) return undefined;
try {
const stat = await handle.stat();
if (!stat.isFile() || stat.nlink !== 1 || stat.size > maxBytes) throw new Error("native_controller_migration_unsafe_source");
return await handle.readFile("utf8");
} finally { await handle.close(); }
}
async function localPaths(source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>) {
const execution = parseNativeExecutionInput(source.runnerProfileJson.nativeExecutionInput);
const descriptor = nativeControllerMigrationDescriptor(execution, source.runnerProfileJson.nativeToolContractFingerprint);
const root = await lstat(descriptor.root).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return undefined; throw error;
});
if (root && (!root.isDirectory() || root.isSymbolicLink())) throw new Error("native_controller_migration_unsafe_source");
const control = join(descriptor.root, "control-plane");
const directory = await lstat(control).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return undefined; throw error;
});
if (directory && (!directory.isDirectory() || directory.isSymbolicLink())) throw new Error("native_controller_migration_unsafe_source");
return { ...descriptor, execution, controlFile: join(control, "control-plane-state.json") };
}
export async function hasLocalNativeControllerObligations(db: Db, storage?: StorageService | null): Promise<boolean> {
const roots = await readdir(nativeHarnessStateBase()).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return [] as string[]; throw error;
});
if (!roots.length) return false;
for await (const { selected, source } of retainedSources(db)) {
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
if (readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint)
&& await hasNativeRunnerCheckpoint(db, binding)) continue;
const paths = await localPaths(source);
const present = await lstat(paths.controlFile).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return undefined; throw error;
});
if (present && source.status === "failed") {
const archiveStorage = storage === undefined ? durableNativeHarnessStorage() : storage;
const controller = await readPrivateSource(paths.controlFile, 256 * 1024 * 1024);
const supervisor = paths.execution.session.lifecyclePolicy.mode === "warm"
? await readPrivateSource(paths.supervisorPath, 4 * 1024 * 1024) : undefined;
if (controller !== undefined && archiveStorage && await hasArchivedLegacyNativeController(db, { ...binding, storage: archiveStorage, controller, supervisor })) continue;
}
if (present || await hasNativeRunnerCheckpoint(db, binding)) return true;
}
return false;
}
export async function migrateNativeControllers(db: Db, input: { writersStopped: true; signal: AbortSignal; storage?: StorageService }) {
const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
if (!input.writersStopped || active) throw new Error("native_controller_migration_requires_offline_writers");
const counts = { inspected: 0, imported: 0, unchanged: 0, sourceAlreadyAbsent: 0, archivedFailed: 0 };
for await (const { selected, source } of retainedSources(db, input.signal)) {
counts.inspected++;
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
const metadata = readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint);
const snapshot = source.runnerProfileJson.sessionCheckpoint as { identity?: Record<string, unknown> } | null;
if (checkpoint && metadata?.checkpoint?.runId === source.id
&& metadata.checkpoint.runnerInstanceId === source.runnerInstanceId
&& metadata.checkpoint.normalizedSessionId === selected.sessionId
&& snapshot?.identity?.runId === source.id && snapshot.identity.companyId === selected.companyId
&& snapshot.identity.agentId === selected.agentId && snapshot.identity.issueId === selected.issueId
&& snapshot.identity.sessionId === selected.sessionId
&& checkpoint.recoveryEvidence.runnerInstanceId === source.runnerInstanceId
&& checkpoint.recoveryEvidence.normalizedSessionId === selected.sessionId) { counts.unchanged++; continue; }
const paths = await localPaths(source);
const legacyControllerState = checkpoint ? undefined : await readPrivateSource(paths.controlFile, 256 * 1024 * 1024);
if (!checkpoint && legacyControllerState === undefined) { counts.sourceAlreadyAbsent++; continue; }
let legacyRunnerArtifact: { version: string; digest: string } | undefined;
if (legacyControllerState !== undefined && await readPrivateSource(join(paths.root, "provider-pack-authority.json"), 512 * 1024)) {
const identity = JSON.parse(legacyControllerState).identity;
const execution = paths.execution;
const [admission] = await db.select({ workspace: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'` })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.id), eq(heartbeatRuns.companyId, selected.companyId),
eq(heartbeatRuns.agentId, selected.agentId)));
const workspace = readNativeWorkspaceSyncReference(admission?.workspace);
const remoteCwd = workspace?.remoteCwd ?? execution.workspace.cwd;
if (typeof identity?.environmentLeaseId !== "string" || !remoteCwd
|| (workspace && workspace.leaseId !== identity.environmentLeaseId)) {
throw new Error("native_controller_migration_artifact_binding");
}
const unavailable = (): never => { throw new Error("native_controller_migration_artifact_unavailable"); };
legacyRunnerArtifact = resolveRemoteProviderPackAuthority({ stateDirectory: paths.root, reattach: true,
binding: { companyId: selected.companyId, agentId: selected.agentId,
workspaceId: providerPackWorkspaceId(execution.binding.executionWorkspaceId, source.id, identity.environmentLeaseId),
sessionId: selected.sessionId, runnerInstanceId: source.runnerInstanceId!,
environmentLeaseId: identity.environmentLeaseId, remoteCwd,
providerDigest: nativeSha256(execution.provider) },
current: unavailable, runnerArtifact: unavailable }).runnerArtifact;
}
const supervisor = paths.execution.session.lifecyclePolicy.mode === "warm"
? await readPrivateSource(paths.supervisorPath, 4 * 1024 * 1024) : undefined;
input.signal.throwIfAborted();
let result: Awaited<ReturnType<typeof importNativeControllerRecovery>>;
try {
result = await importNativeControllerRecovery(db, { ...binding, agentId: selected.agentId,
normalizedSessionId: selected.sessionId, expectedConfigDigest: paths.configDigest,
supervisor: supervisor === undefined ? undefined : JSON.parse(supervisor),
offline: { writersStopped: true, legacyControllerState, legacyRunnerArtifact } });
} catch (error) {
const storage = input.storage ?? durableNativeHarnessStorage();
if (!(error instanceof Error) || error.message !== "native_runner_checkpoint_artifact_unproven"
|| source.status !== "failed" || legacyControllerState === undefined || !storage) throw error;
await archiveFailedLegacyNativeController(db, { ...binding, storage, controller: legacyControllerState, supervisor,
writersStopped: true, signal: input.signal });
counts.archivedFailed++;
continue;
}
if (result.kind !== "imported" && result.kind !== "unchanged") {
throw new Error(`native_controller_migration_blocked:${source.id}:${"reason" in result ? result.reason : result.kind}`);
}
counts[result.kind]++;
}
return counts;
}
@@ -1,6 +1,7 @@
import type { Db } from "@paperclipai/db";
import { sql } from "drizzle-orm";
import { inspectNativeRunnerUpgradeHeader } from "./native-runner-checkpoint.js";
import { inspectNativeRunnerUpgradeHeader, NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES,
NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES } from "./native-runner-checkpoint.js";
/** An explicit, paginated diagnostic. Never called by an idle/background scan.
* This inventories selected retained task-session pointers, not all history.
@@ -23,7 +24,8 @@ export async function readNativeControllerRecoveryInventory(db: Db, input: { aft
r.native_session_id as "sessionId", r.runner_instance_id as "runnerId",
case when pg_column_size(r.runner_profile_json->'nativeControllerRecovery') <= 4096
then r.runner_profile_json->'nativeControllerRecovery' else null end as metadata,
case when pg_column_size(c.material) <= 262144 and sum(pg_column_size(c.material)) over () <= 8388608
case when pg_column_size(c.material) <= ${NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES}
and sum(pg_column_size(c.material)) over () <= ${NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES}
then c.material else null end as material,
(r.finished_at is null or r.status not in ('succeeded', 'failed', 'cancelled', 'interrupted', 'timed_out')
or f.run_id is null or f.lease_owner is not null or f.recovery_state is not null
@@ -3,7 +3,7 @@ import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agentTaskSessions, agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
import { DurablePrpControlPlane, parseNativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
import { startRecoveryPostgres, recoveryPostgresSupport } from "../../__tests__/helpers/recovery-postgres.js";
@@ -92,6 +92,18 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
expect(await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId })).toEqual({ kind: "waiting_for_owner", deadline });
});
it("uses the database clock when an import finds an expired owner", async () => {
const fixture = await seed();
const now = Date.now();
await db.update(nativeRunFinalizations).set({ leaseOwner: "expired-owner", leaseExpiresAt: new Date(now - 60_000) })
.where(eq(nativeRunFinalizations.runId, fixture.scope.runId));
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(now - 3_600_000);
try {
expect(await importNativeControllerRecovery(db, { ...fixture.scope, expectedConfigDigest: compatibility.configDigest,
supervisor: fixture.supervisor })).toEqual({ kind: "missing_evidence", reason: "normal_ownership_recovery_required" });
} finally { vi.useRealTimers(); }
});
it("holds mismatched snapshots and runner bindings instead of falling back to fresh execution", async () => {
const fixture = await seed(), currentRunId = randomUUID();
const metadata = createNativeControllerRecovery(fixture.snapshot, compatibility, { runId: fixture.scope.runId,
@@ -1,12 +1,13 @@
import { and, eq, ne, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import type { DurablePrpCheckpoint, PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
import { captureLegacyDurablePrpCheckpoint, parseNativeExecutionInput, type DurablePrpCheckpoint, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { readRunCancellation } from "../run-cancellation.js";
import { nativeSha256 } from "./canonical.js";
import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js";
import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024;
@@ -223,10 +224,11 @@ export async function resolveNativeControllerRecovery(db: Db, input: {
export async function importNativeControllerRecovery(db: Db, input: {
companyId: string; agentId: string; issueId: string; runId: string; normalizedSessionId: string;
expectedConfigDigest: string;
supervisor: unknown;
supervisor?: unknown;
offline?: { writersStopped: true; legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } };
}): Promise<{ kind: "imported" | "unchanged" } | Exclude<NativeControllerRecoveryOutcome, { kind: "recoverable" } | { kind: "no_prior_execution" }>> {
const supervisor = record(input.supervisor);
if (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest) {
if (input.supervisor !== undefined && (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest)) {
return { kind: "unsupported", reason: "supervisor_compatibility" };
}
return db.transaction(async tx => {
@@ -236,25 +238,48 @@ export async function importNativeControllerRecovery(db: Db, input: {
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
eq(nativeRunFinalizations.runId, input.runId))).for("update");
if (!task || !owner) return { kind: "missing_evidence" as const, reason: "owner_binding" };
if (owner.leaseOwner || task.executionRunId) {
return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date()
if (input.offline) {
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
if (!input.offline.writersStopped || active) return { kind: "missing_evidence" as const, reason: "offline_writers_required" };
}
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
const liveOwner = owner.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt > new Date(clock.now));
if (input.offline ? liveOwner : owner.leaseOwner || task.executionRunId) {
return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date(clock.now)
? { kind: "waiting_for_owner" as const, deadline: owner.leaseExpiresAt }
: { kind: "missing_evidence" as const, reason: "normal_ownership_recovery_required" };
}
if (owner.phase !== "committed" || !owner.resultId || owner.recoveryState) {
if (!input.offline && (!(owner.phase === "committed" && owner.resultId) || owner.recoveryState)) {
return { kind: "missing_evidence" as const, reason: "pending_finalization" };
}
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).for("update");
const profile = record(run?.runnerProfileJson), snapshot = record(profile.sessionCheckpoint), identity = record(snapshot.identity);
if (run?.runtimeMode !== "native" || run.status !== "succeeded" || !run.finishedAt
if (run?.runtimeMode !== "native" || !(run.status === "succeeded" || input.offline && ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) || !run.finishedAt
|| identity.runId !== input.runId || identity.companyId !== input.companyId || identity.issueId !== input.issueId
|| identity.agentId !== input.agentId || identity.sessionId !== input.normalizedSessionId
|| nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) {
|| input.supervisor !== undefined && nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) {
return { kind: "missing_evidence" as const, reason: "stale_supervisor_snapshot" };
}
const checkpoint = await loadNativeRunnerCheckpoint(tx, input);
if (input.supervisor === undefined) {
try {
const admitted = parseNativeExecutionInput(profile.nativeExecutionInput);
if (!input.offline || admitted.session.lifecyclePolicy.mode !== "per_turn"
|| admitted.binding.runId !== input.runId || admitted.binding.companyId !== input.companyId
|| admitted.binding.agentId !== input.agentId || admitted.binding.issueId !== input.issueId) {
return { kind: "missing_evidence" as const, reason: "admitted_per_turn_binding" };
}
} catch { return { kind: "unsupported" as const, reason: "admitted_input_format" }; }
}
let checkpoint = await loadNativeRunnerCheckpoint(tx, input);
const importing = !checkpoint && input.offline?.legacyControllerState !== undefined;
if (importing) {
if (!run.runnerInstanceId) return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
checkpoint = captureLegacyDurablePrpCheckpoint(input.offline!.legacyControllerState!, {
runId: input.runId, runnerInstanceId: run.runnerInstanceId, normalizedSessionId: input.normalizedSessionId,
}, input.offline!.legacyRunnerArtifact);
}
if (!checkpoint || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId
|| checkpoint.recoveryEvidence.runnerInstanceId !== run.runnerInstanceId) {
return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
@@ -265,10 +290,13 @@ export async function importNativeControllerRecovery(db: Db, input: {
}, { runId: input.runId, runnerInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, normalizedSessionId: input.normalizedSessionId });
if (!readNativeControllerRecovery(metadata, snapshot)) return { kind: "unsupported" as const, reason: "metadata_format" };
if (profile.nativeControllerRecovery) {
return nativeSha256(profile.nativeControllerRecovery) === nativeSha256(metadata)
? { kind: "unchanged" as const }
: { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" };
if (nativeSha256(profile.nativeControllerRecovery) !== nativeSha256(metadata)) {
return { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" };
}
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
return { kind: importing ? "imported" as const : "unchanged" as const };
}
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeControllerRecovery: metadata }, updatedAt: new Date() })
.where(eq(heartbeatRuns.id, input.runId));
return { kind: "imported" as const };
@@ -0,0 +1,80 @@
import { mkdtemp, mkdir, open, readFile, readlink, rm, stat, symlink, writeFile } from "node:fs/promises";
import { createWriteStream } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { pipeline } from "node:stream/promises";
import tar from "tar-stream";
import { afterEach, expect, it } from "vitest";
import { captureNativeHarnessArchive, restoreNativeHarnessArchive, MAX_NATIVE_HARNESS_BYTES } from "./native-harness-backup-archive.js";
const roots: string[] = [];
afterEach(async () => { for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true }); });
async function fixture() { const root = await mkdtemp(join(tmpdir(), "native-backup-archive-test-")); roots.push(root); return root; }
async function maliciousArchive(root: string, headers: Array<tar.Headers>) {
const archive = tar.pack(), file = join(root, "malicious.tar");
const written = pipeline(archive, createWriteStream(file));
for (const header of headers) archive.entry(header, Buffer.alloc(0));
archive.finalize(); await written; return file;
}
it.each([
{ name: "../escaped", type: "file" as const },
{ name: "/absolute", type: "file" as const },
{ name: "link", type: "symlink" as const, linkname: "../outside" },
{ name: "device", type: "character-device" as const },
{ name: "parent/child", type: "file" as const },
])("rejects unsafe archive entry $name", async header => {
const root = await fixture(), file = await maliciousArchive(root, [header]);
await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid");
await expect(stat(join(root, "escaped"))).rejects.toMatchObject({ code: "ENOENT" });
});
it("rejects duplicate paths instead of overwriting a restored file", async () => {
const root = await fixture(), file = await maliciousArchive(root, [{ name: "manifest.json", type: "file" }, { name: "manifest.json", type: "file" }]);
await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid");
});
it("refuses a source symlink without reading its target", async () => {
const root = await fixture(), outside = join(root, "outside");
await writeFile(outside, "private external file");
const { mkdir } = await import("node:fs/promises"); const source = join(root, "source"); await mkdir(source);
await symlink(outside, join(source, "linked"));
await expect(captureNativeHarnessArchive(source, join(root, "archive"))).rejects.toThrow("native_harness_archive_invalid");
expect(await readFile(outside, "utf8")).toBe("private external file");
});
it("honors cancellation before walking the checkpoint tree", async () => {
const root = await fixture();
await expect(captureNativeHarnessArchive(root, join(root, "archive"), AbortSignal.abort(new Error("test cancelled"))))
.rejects.toThrow();
});
it("preserves contained provider links", async () => {
const root = await fixture(), source = join(root, "source"), target = join(root, "restored");
await mkdir(join(source, "provider", ".bin"), { recursive: true });
await writeFile(join(source, "manifest.json"), "{}");
await writeFile(join(source, "provider", "tool.js"), "provider state");
await symlink("../tool.js", join(source, "provider", ".bin", "tool"));
const archive = join(root, "archive");
await captureNativeHarnessArchive(source, archive);
await restoreNativeHarnessArchive(archive, target);
expect(await readlink(join(target, "provider", ".bin", "tool"))).toBe("../tool.js");
expect(await readFile(join(target, "provider", ".bin", "tool"), "utf8")).toBe("provider state");
});
it("rejects writes beneath a symlink even when its target is contained", async () => {
const root = await fixture(), file = await maliciousArchive(root, [
{ name: "manifest.json", type: "file" }, { name: "directory", type: "directory" },
{ name: "link", type: "symlink", linkname: "directory" }, { name: "link/child", type: "file" },
]);
await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid");
});
it("rejects indirect symlink targets instead of allowing chained traversal", async () => {
const root = await fixture(), file = await maliciousArchive(root, [
{ name: "manifest.json", type: "file" }, { name: "first", type: "symlink", linkname: "second/child" },
{ name: "second", type: "symlink", linkname: "elsewhere" },
]);
await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid");
});
it("rejects oversized source files before streaming their payload", async () => {
const root = await fixture(), source = join(root, "source"); await mkdir(source);
const handle = await open(join(source, "oversized"), "wx");
await handle.truncate(MAX_NATIVE_HARNESS_BYTES + 1); await handle.close();
await expect(captureNativeHarnessArchive(source, join(root, "archive"))).rejects.toThrow("native_harness_archive_invalid");
expect((await stat(join(root, "archive"))).size).toBeLessThan(1024);
});
@@ -0,0 +1,114 @@
import { constants, createReadStream, createWriteStream } from "node:fs";
import { chmod, lstat, mkdir, open, readdir, readlink, symlink } from "node:fs/promises";
import { join, posix } from "node:path";
import { Transform } from "node:stream";
import { pipeline } from "node:stream/promises";
import tar from "tar-stream";
export const MAX_NATIVE_HARNESS_BYTES = 4 * 1024 * 1024 * 1024;
export const MAX_NATIVE_HARNESS_OBJECT_BYTES = MAX_NATIVE_HARNESS_BYTES + 768 * 1024 * 1024;
export const MAX_NATIVE_HARNESS_ENTRIES = 100_000;
const fail = () => new Error("native_harness_archive_invalid");
const safeName = (name: string) => name.length > 0 && Buffer.byteLength(name) <= 4096 && !name.includes("\\")
&& !/[\u0000-\u001f\u007f]/u.test(name) && !posix.isAbsolute(name)
&& name.split("/").every(part => part !== ".." && part !== "." && part !== "");
const safeLink = (name: string, target: string) => target.length > 0 && Buffer.byteLength(target) <= 4096
&& !target.includes("\\") && !/[\u0000-\u001f\u007f]/u.test(target) && !posix.isAbsolute(target)
&& !["..", "."].includes(posix.normalize(posix.join(posix.dirname(name), target)))
&& !posix.normalize(posix.join(posix.dirname(name), target)).startsWith("../");
const validateLinks = (links: Map<string, string>) => {
for (const [name, target] of links) {
const resolved = posix.normalize(posix.join(posix.dirname(name), target)).split("/");
for (let index = 1; index <= resolved.length; index++) {
if (links.has(resolved.slice(0, index).join("/"))) throw fail();
}
}
};
const boundedArchive = () => {
let bytes = 0;
return new Transform({ transform(chunk, _encoding, callback) {
bytes += chunk.length;
callback(bytes > MAX_NATIVE_HARNESS_OBJECT_BYTES ? fail() : null, chunk);
} });
};
export async function captureNativeHarnessArchive(root: string, target: string, signal?: AbortSignal) {
if (!(await lstat(root)).isDirectory()) throw fail();
const archive = tar.pack();
const output = pipeline(archive, boundedArchive(), createWriteStream(target, { flags: "wx", mode: 0o600 }), { signal });
void output.catch(() => undefined);
let count = 0, bytes = 0;
const links = new Map<string, string>();
const visit = async (relative: string): Promise<void> => {
for (const name of (await readdir(join(root, relative))).sort()) {
signal?.throwIfAborted();
const entry = relative ? `${relative}/${name}` : name;
if (!safeName(entry) || ++count > MAX_NATIVE_HARNESS_ENTRIES) throw fail();
const file = join(root, entry), stat = await lstat(file);
if (stat.isDirectory()) {
await new Promise<void>((resolve, reject) => archive.entry({ name: entry, type: "directory", mode: stat.mode & 0o777 },
error => error ? reject(error) : resolve()));
await visit(entry);
} else if (stat.isFile()) {
bytes += stat.size;
if (bytes > MAX_NATIVE_HARNESS_BYTES) throw fail();
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const before = await handle.stat();
if (!before.isFile() || before.ino !== stat.ino || before.dev !== stat.dev || before.size !== stat.size) throw fail();
const stream = archive.entry({ name: entry, size: stat.size, type: "file", mode: stat.mode & 0o777 });
await pipeline(handle.createReadStream({ autoClose: false }), stream, { signal });
const after = await handle.stat();
if (before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw fail();
} finally { await handle.close(); }
} else if (stat.isSymbolicLink()) {
const linkname = await readlink(file);
if (!safeLink(entry, linkname)) throw fail();
links.set(entry, linkname);
await new Promise<void>((resolve, reject) => archive.entry({ name: entry, type: "symlink", linkname },
error => error ? reject(error) : resolve()));
} else throw fail();
}
};
try { await visit(""); validateLinks(links); archive.finalize(); await output; }
catch (error) { archive.destroy(error as Error); await output.catch(() => undefined); throw error; }
}
export async function restoreNativeHarnessArchive(source: string, target: string, signal?: AbortSignal) {
await mkdir(target, { mode: 0o700 });
const archive = tar.extract(), names = new Set<string>(), links = new Map<string, string>();
const directories: Array<{ path: string; mode: number }> = [], directoryNames = new Set<string>();
let bytes = 0;
archive.on("entry", (header, body, next) => {
body.on("error", () => undefined);
void (async () => {
signal?.throwIfAborted();
const name = header.type === "directory" ? header.name.replace(/\/$/u, "") : header.name;
if (!safeName(name) || names.has(name) || names.size >= MAX_NATIVE_HARNESS_ENTRIES
|| !["directory", "file", "symlink"].includes(header.type ?? "") || !Number.isSafeInteger(header.size ?? 0)
|| (header.size ?? 0) < 0 || (bytes += header.size ?? 0) > MAX_NATIVE_HARNESS_BYTES) throw fail();
names.add(name);
const file = join(target, name), mode = (header.mode ?? 0o600) & 0o777;
const parent = posix.dirname(name);
if (parent !== "." && !directoryNames.has(parent)) throw fail();
if (header.type === "directory") {
if (header.size) throw fail();
await mkdir(file, { mode: 0o700 }); directories.push({ path: file, mode }); directoryNames.add(name);
for await (const _ of body) throw fail();
} else if (header.type === "symlink") {
if (header.size || !header.linkname || !safeLink(name, header.linkname)) throw fail();
links.set(name, header.linkname);
await symlink(header.linkname, file);
for await (const _ of body) throw fail();
} else {
await pipeline(body, createWriteStream(file, { flags: "wx", mode }), { signal });
await chmod(file, mode);
}
})().then(() => next(), error => archive.destroy(error as Error));
});
await pipeline(createReadStream(source), boundedArchive(), archive, { signal });
if (!names.has("manifest.json")) throw fail();
validateLinks(links);
for (const directory of directories.reverse()) await chmod(directory.path, directory.mode);
}
@@ -2,7 +2,7 @@ import { notifyExternalConversationSettlement } from "../external-conversation-s
import { randomUUID } from "node:crypto";
import { promises as fs } from "node:fs";
import path from "node:path";
import { and, asc, eq, gt, inArray, isNull, lt, or, sql } from "drizzle-orm";
import { and, asc, eq, gt, gte, inArray, isNull, lt, or, sql } from "drizzle-orm";
import {
agentTaskSessions,
environmentLeases,
@@ -10,6 +10,7 @@ import {
issues,
nativeRunFinalizations,
nativeRunnerCheckpoints,
nativeHarnessBackups,
type Db,
} from "@paperclipai/db";
import { logger } from "../../middleware/logger.js";
@@ -18,6 +19,9 @@ import { nativeSessionScopeDigestIsBusy } from "./native-session-executor.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import type { NativeHarnessBackupExpirationScope } from "../native-harness-backup-expiration-notifications.js";
import { deleteRetiredNativeHarnessBackup, durableNativeHarnessStorage } from "./native-harness-backup-storage.js";
import { retireObsoleteNativeHarnessBackup } from "./native-harness-backup-retention.js";
import type { StorageService } from "../../storage/types.js";
export const NATIVE_HARNESS_BACKUP_EXPIRATION_INTERVAL_MS = 60 * 60 * 1000;
export const NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS = 2 * 60 * 1000;
@@ -51,6 +55,7 @@ export type NativeHarnessBackupExpirationOptions = {
stateBase?: string;
shouldContinue?: () => boolean;
isSessionBusy?: (scopeDigest: string) => boolean;
storage?: StorageService;
};
export type NativeHarnessBackupExpirationResult = {
@@ -291,6 +296,73 @@ export function nativeHarnessBackupExpirationService(
let evidenceDueAt = 0;
let passDueAt = Number.POSITIVE_INFINITY;
let checkpointCursor: string | null = null;
let durableBackupCursor: string | null = null;
let checkpointsEnumerated = false;
async function expireDurableBackups(options: {
closedBefore: Date; dryRun: boolean; maxEntries: number; maxRemovals: number; shouldContinue: () => boolean;
}, result: NativeHarnessBackupExpirationResult) {
const storage = defaults.storage ?? durableNativeHarnessStorage();
if (!storage) return;
const candidates = await db.select({ id: nativeHarnessBackups.id, scopeDigest: nativeHarnessBackups.scopeDigest,
companyId: nativeHarnessBackups.companyId, runId: nativeHarnessBackups.runId, survivingRunId: heartbeatRuns.id,
sessionId: heartbeatRuns.nativeSessionId, status: nativeHarnessBackups.status })
.from(nativeHarnessBackups).leftJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeHarnessBackups.runId))
.where(and(durableBackupCursor ? gt(nativeHarnessBackups.id, durableBackupCursor) : undefined,
or(eq(nativeHarnessBackups.status, "retired"), lt(nativeHarnessBackups.createdAt, options.closedBefore))))
.orderBy(asc(nativeHarnessBackups.id)).limit(options.maxEntries + 1);
let processed = 0;
for (const candidate of candidates) {
if (!options.shouldContinue() || processed++ >= options.maxEntries || result.expired >= options.maxRemovals) {
result.truncated = true; return;
}
durableBackupCursor = candidate.id;
if (candidate.survivingRunId && !candidate.sessionId) continue;
try {
const retired = candidate.status === "retired" || await db.transaction(async tx => {
if (["pending", "superseded"].includes(candidate.status)) {
const obsolete = await retireObsoleteNativeHarnessBackup(tx, { ...candidate,
closedBefore: options.closedBefore, dryRun: options.dryRun });
if (obsolete) result.candidates++;
return obsolete && !options.dryRun;
}
if (!candidate.survivingRunId) {
result.candidates++;
if (options.dryRun) return false;
await tx.update(nativeHarnessBackups).set({ status: "retired", updatedAt: new Date() })
.where(eq(nativeHarnessBackups.id, candidate.id));
return true;
}
const blocker = await sessionRetentionBlocker(tx, { scopeDigest: candidate.scopeDigest, checkpoint: true,
closedBefore: options.closedBefore,
identity: { runId: candidate.runId, normalizedSessionId: candidate.sessionId!, environmentLeaseId: null } });
if ("reason" in blocker) return false;
const [recent] = await tx.select({ id: nativeHarnessBackups.id }).from(nativeHarnessBackups).where(and(
eq(nativeHarnessBackups.companyId, blocker.companyId), eq(nativeHarnessBackups.scopeDigest, candidate.scopeDigest),
gte(nativeHarnessBackups.createdAt, options.closedBefore))).limit(1);
if (recent) return false;
result.candidates++;
if (options.dryRun) return false;
await notifyExternalConversationSettlement(tx, { companyId: blocker.companyId });
await tx.update(nativeRunFinalizations).set({ controllerGeneration: sql`${nativeRunFinalizations.controllerGeneration} + 1` })
.where(inArray(nativeRunFinalizations.runId, blocker.runIds));
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${blocker.companyId}:${candidate.scopeDigest}`}, 0))`);
const cleared = await tx.delete(agentTaskSessions).where(and(eq(agentTaskSessions.companyId, blocker.companyId),
eq(agentTaskSessions.agentId, blocker.agentId), or(inArray(agentTaskSessions.lastRunId, blocker.runIds),
sql`${agentTaskSessions.sessionParamsJson}->>'sessionId' = ${candidate.sessionId}`))).returning({ id: agentTaskSessions.id });
result.clearedTaskSessions += cleared.length;
await tx.update(nativeHarnessBackups).set({ status: "retired", updatedAt: new Date() }).where(and(
eq(nativeHarnessBackups.companyId, blocker.companyId), eq(nativeHarnessBackups.scopeDigest, candidate.scopeDigest)));
return true;
});
if (retired && !options.dryRun && await deleteRetiredNativeHarnessBackup(db, storage, candidate.id, AbortSignal.timeout(60_000))) result.expired++;
} catch (err) {
result.failed++;
logger.warn({ err, backupId: candidate.id }, "durable native harness backup expiration skipped an object");
}
}
durableBackupCursor = null;
}
async function expireCheckpoints(options: {
closedBefore: Date; dryRun: boolean; maxEntries: number; maxRemovals: number; shouldContinue: () => boolean;
@@ -704,13 +776,20 @@ export function nativeHarnessBackupExpirationService(
const result = emptyResult();
const shouldContinue = options.shouldContinue ?? defaults.shouldContinue ?? (() => true);
if (!shouldContinue()) return { ...result, truncated: true };
await expireCheckpoints({
const pass = {
closedBefore: new Date((options.now ?? defaults.now ?? new Date()).getTime() - closedForDays * DAY_MS),
dryRun: options.dryRun ?? defaults.dryRun ?? false,
maxEntries: Math.max(1, options.maxEntries ?? defaults.maxEntries ?? 200),
maxRemovals: Math.max(1, options.maxRemovals ?? defaults.maxRemovals ?? 50),
shouldContinue,
}, result);
};
if (!checkpointsEnumerated) {
await expireCheckpoints(pass, result);
if (result.truncated) return result;
checkpointsEnumerated = true;
}
await expireDurableBackups(pass, result);
if (!result.truncated) checkpointsEnumerated = false;
return result;
}
@@ -0,0 +1,79 @@
import { createHash } from "node:crypto";
import { lstat, readFile, readdir } from "node:fs/promises";
import { join } from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, nativeHarnessBackups, type Db } from "@paperclipai/db";
import type { StorageService } from "../../storage/types.js";
import { nativeHarnessStateBase, readNativeHarnessBackupStamp, verifyNativeHarnessBackupStampAsync } from "./native-harness-backup-stamp.js";
import { persistNativeHarnessBackup, verifyDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
export async function hasLocalNativeHarnessObligations(db: Db): Promise<boolean> {
const rows = await db.execute(sql`select 1 from agent_task_sessions s
join heartbeat_runs r on r.id = s.last_run_id and r.company_id = s.company_id and r.agent_id = s.agent_id
join environment_leases e on e.company_id = r.company_id and e.issue_id = r.native_issue_id
and e.metadata->'nativeHarnessBackup'->>'normalizedSessionId' = r.native_session_id::text
where s.adapter_type = 'paperclip_runner'
and e.metadata->'nativeHarnessBackup'->>'schema' = 'paperclip.native-harness-backup-stamp.v2'
and not exists (select 1 from native_harness_backups b where b.company_id = r.company_id
and b.scope_digest = e.metadata->'nativeHarnessBackup'->>'sessionScopeSha256' and b.status = 'current')
limit 1`);
return rows.length > 0;
}
export async function migrateNativeHarnessBackups(db: Db, input: {
storage: StorageService; writersStopped: true; signal: AbortSignal;
}): Promise<number> {
if (!input.writersStopped) throw new Error("native_harness_migration_requires_offline_writers");
const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
if (active) throw new Error("native_harness_migration_requires_offline_writers");
const base = nativeHarnessStateBase();
const directories = await readdir(base).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return [] as string[]; throw error;
});
let count = 0;
for (const scopeDigest of directories.filter(name => /^[a-f0-9]{64}$/.test(name)).sort()) {
input.signal.throwIfAborted();
const root = join(base, scopeDigest, "failover-backups", "current");
const stat = await lstat(root).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null; throw error;
});
if (!stat) continue;
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("native_harness_migration_unsafe_source");
const manifestBytes = await readFile(join(root, "manifest.json"));
const manifest = JSON.parse(manifestBytes.toString("utf8"));
const state = JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8"));
if (state.lifecycle !== "suspended" || typeof state.runId !== "string"
|| manifest.schema !== "paperclip.native-harness-backup.v1") throw new Error("native_harness_migration_source_unsettled");
const [run] = await db.select({ companyId: heartbeatRuns.companyId, issueId: heartbeatRuns.nativeIssueId,
sessionId: heartbeatRuns.nativeSessionId, runnerId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, state.runId));
if (!run) continue;
if (!run.issueId || run.sessionId !== manifest.normalizedSessionId || run.runnerId !== manifest.runnerInstanceId) {
throw new Error("native_harness_migration_binding_mismatch");
}
const [current] = await db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.companyId, run.companyId),
eq(nativeHarnessBackups.scopeDigest, scopeDigest), eq(nativeHarnessBackups.status, "current")));
const manifestSha256 = `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`;
if (current) {
if (current.manifestSha256 !== manifestSha256) throw new Error("native_harness_migration_durable_source_conflict");
await verifyDurableNativeHarnessBackup({ db, storage: input.storage, companyId: run.companyId,
scopeDigest, manifestSha256: current.manifestSha256, signal: input.signal });
count++; continue;
}
const leases = await db.select({ stamp: sql<unknown>`${environmentLeases.metadata}->'nativeHarnessBackup'`,
providerLeaseId: environmentLeases.providerLeaseId }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.issueId, run.issueId),
sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'manifestSha256' = ${manifestSha256}`));
let valid = false;
for (const lease of leases) {
const stamp = readNativeHarnessBackupStamp(lease.stamp, lease.providerLeaseId ?? "");
if (stamp?.sessionScopeSha256 === scopeDigest && await verifyNativeHarnessBackupStampAsync(stamp, lease.providerLeaseId!, input.signal)) { valid = true; break; }
}
if (!valid) throw new Error("native_harness_migration_backup_unverified");
await persistNativeHarnessBackup({ db, storage: input.storage, root, scopeDigest, signal: input.signal,
owner: { kind: "offline_migration", writersStopped: true, companyId: run.companyId, issueId: run.issueId, runId: state.runId } });
count++;
}
return count;
}
@@ -53,6 +53,15 @@ describe("native harness backup publication", () => {
expect(await fs.readdir(backupRoot)).toEqual(["current"]);
expect(changes).toEqual([{ companyId: input.companyId, scopeDigest: input.scopeDigest }]);
});
it("restores the old current directory when the pending rename fails", async () => {
const input = await fixture(); const changes: unknown[] = [];
unsubscribe = onNativeHarnessBackupChange(input.db, change => changes.push(change));
await fs.rm(input.pendingRoot, { recursive: true });
await expect(publishNativeHarnessBackup(input)).rejects.toMatchObject({ code: "ENOENT" });
expect(await fs.readFile(path.join(input.current, "manifest.json"), "utf8")).toBe("saved");
expect(changes).toEqual([]);
});
it("serves an unrelated request while a retired backup is being removed", async () => {
const input = await fixture();
for (let index = 0; index < 40; index++) {
@@ -0,0 +1,34 @@
import { and, eq, inArray, isNotNull, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, nativeHarnessBackups, nativeRunFinalizations, type Db } from "@paperclipai/db";
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
export async function retireObsoleteNativeHarnessBackup(tx: Transaction, input: {
id: string; companyId: string; scopeDigest: string; closedBefore: Date; dryRun: boolean;
}): Promise<boolean> {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${input.companyId}:${input.scopeDigest}`}, 0))`);
const [candidate] = await tx.select().from(nativeHarnessBackups).where(and(
eq(nativeHarnessBackups.id, input.id), eq(nativeHarnessBackups.companyId, input.companyId))).for("update");
if (!candidate || !["pending", "superseded"].includes(candidate.status) || candidate.updatedAt >= input.closedBefore) return false;
const [current] = await tx.select({ id: nativeHarnessBackups.id }).from(nativeHarnessBackups).where(and(
eq(nativeHarnessBackups.companyId, input.companyId), eq(nativeHarnessBackups.scopeDigest, input.scopeDigest),
eq(nativeHarnessBackups.status, "current"))).limit(1);
if (!current && candidate.status === "superseded") return false;
const [lease] = await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId),
inArray(environmentLeases.status, ["active", "released", "retained", "pending_cleanup"]),
sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'sessionScopeSha256' = ${input.scopeDigest}`,
sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'manifestSha256' = ${candidate.manifestSha256}`)).limit(1);
if (lease) return false;
if (candidate.status === "pending") {
const [run] = await tx.select({ status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt })
.from(heartbeatRuns).where(eq(heartbeatRuns.id, candidate.runId));
if (run && (!run.finishedAt || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) return false;
const [owner] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, candidate.runId), isNotNull(nativeRunFinalizations.leaseOwner))).limit(1);
if (owner) return false;
}
if (!input.dryRun) await tx.update(nativeHarnessBackups).set({ status: "retired", updatedAt: new Date() })
.where(eq(nativeHarnessBackups.id, candidate.id));
return true;
}
@@ -1,5 +1,7 @@
import { createHash } from "node:crypto";
import { lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs";
import { constants, lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs";
import { lstat, open, readdir, readlink } from "node:fs/promises";
import { MAX_NATIVE_HARNESS_BYTES, MAX_NATIVE_HARNESS_ENTRIES } from "./native-harness-backup-archive.js";
import { dirname, resolve } from "node:path";
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
@@ -142,12 +144,12 @@ export function createNativeHarnessBackupStamp(input: {
};
}
export function verifyNativeHarnessBackupStamp(
export function readNativeHarnessBackupStamp(
value: unknown,
expectedProviderLeaseId: string,
): boolean {
if (!expectedProviderLeaseId) return false;
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
): NativeHarnessBackupStamp | null {
if (!expectedProviderLeaseId) return null;
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
const stamp = value as Record<string, unknown>;
if (
stamp.schema !== "paperclip.native-harness-backup-stamp.v2" ||
@@ -160,69 +162,96 @@ export function verifyNativeHarnessBackupStamp(
!stamp.sourceProviderLeaseId ||
stamp.authorizedProviderLeaseId !== expectedProviderLeaseId ||
typeof stamp.manifestSha256 !== "string" ||
!stamp.manifestSha256.startsWith("sha256:")
!/^sha256:[a-f0-9]{64}$/.test(stamp.manifestSha256) ||
!/^[a-f0-9]{64}$/.test(stamp.sessionScopeSha256) ||
typeof stamp.completedAt !== "string" || !Number.isFinite(Date.parse(stamp.completedAt))
)
return false;
return null;
return stamp as unknown as NativeHarnessBackupStamp;
}
function matchingBackupDirectories(candidate: string, stamp: NativeHarnessBackupStamp): Array<{ path: string; sha256: string; bytes: number }> | null {
const manifestPath = resolve(candidate, "manifest.json");
if (!isRealDirectory(candidate) || !isRealFile(manifestPath) || lstatSync(manifestPath).size > 4 * 1024 * 1024) return null;
const bytes = readFileSync(manifestPath);
if (`sha256:${createHash("sha256").update(bytes).digest("hex")}` !== stamp.manifestSha256) return null;
const manifest = JSON.parse(bytes.toString("utf8"));
if (manifest.schema !== "paperclip.native-harness-backup.v1" || manifest.normalizedSessionId !== stamp.normalizedSessionId
|| manifest.runnerInstanceId !== stamp.runnerInstanceId || manifest.sourceProviderLeaseId !== stamp.sourceProviderLeaseId
|| !Array.isArray(manifest.directories) || manifest.directories.length === 0) return null;
const directories: Array<{ path: string; sha256: string; bytes: number }> = [];
for (const entry of manifest.directories) {
if (!entry || typeof entry !== "object" || Array.isArray(entry) || typeof entry.name !== "string"
|| !/^[A-Za-z0-9._-]+$/.test(entry.name) || [".", ".."].includes(entry.name)
|| typeof entry.sha256 !== "string" || !Number.isSafeInteger(entry.bytes) || entry.bytes < 0) return null;
const path = resolve(candidate, entry.name);
if (!isRealDirectory(path)) return null;
directories.push({ path, sha256: entry.sha256, bytes: entry.bytes });
}
return directories;
}
function backupCandidates(stamp: NativeHarnessBackupStamp) {
const root = stateRootFromDigest(stamp.sessionScopeSha256);
if (!root || !isRealDirectory(root)) return false;
const backupRoot = resolve(root, "failover-backups");
for (const candidate of [
resolve(backupRoot, "current"),
resolve(backupRoot, "previous"),
]) {
const manifestPath = resolve(candidate, "manifest.json");
if (!isRealDirectory(candidate) || !isRealFile(manifestPath)) continue;
return !root || !isRealDirectory(root) ? [] : ["current", "previous"].map(name => resolve(root, "failover-backups", name));
}
export function verifyNativeHarnessBackupStamp(value: unknown, expectedProviderLeaseId: string): boolean {
const stamp = readNativeHarnessBackupStamp(value, expectedProviderLeaseId);
if (!stamp) return false;
for (const candidate of backupCandidates(stamp)) {
try {
const bytes = readFileSync(manifestPath);
const manifestSha256 = `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
if (manifestSha256 !== stamp.manifestSha256) continue;
const manifest = JSON.parse(bytes.toString("utf8")) as Record<
string,
unknown
>;
if (
manifest.schema !== "paperclip.native-harness-backup.v1" ||
manifest.normalizedSessionId !== stamp.normalizedSessionId ||
manifest.runnerInstanceId !== stamp.runnerInstanceId ||
manifest.sourceProviderLeaseId !== stamp.sourceProviderLeaseId ||
!Array.isArray(manifest.directories) ||
manifest.directories.length === 0
)
continue;
let valid = true;
for (const entry of manifest.directories) {
if (!entry || typeof entry !== "object" || Array.isArray(entry)) {
valid = false;
break;
}
const declared = entry as Record<string, unknown>;
if (
typeof declared.name !== "string" ||
!/^[A-Za-z0-9._-]+$/.test(declared.name) ||
typeof declared.sha256 !== "string" ||
typeof declared.bytes !== "number"
) {
valid = false;
break;
}
const directory = resolve(candidate, declared.name);
if (!isRealDirectory(directory)) {
valid = false;
break;
}
const actual = digestDirectory(directory);
if (
actual.sha256 !== declared.sha256 ||
actual.bytes !== declared.bytes
) {
valid = false;
break;
}
const directories = matchingBackupDirectories(candidate, stamp);
if (directories?.every(directory => {
const actual = digestDirectory(directory.path);
return actual.sha256 === directory.sha256 && actual.bytes === directory.bytes;
})) return true;
} catch {}
}
return false;
}
export async function verifyNativeHarnessBackupStampAsync(value: unknown, expectedProviderLeaseId: string, signal: AbortSignal): Promise<boolean> {
const stamp = readNativeHarnessBackupStamp(value, expectedProviderLeaseId);
if (!stamp) return false;
for (const candidate of backupCandidates(stamp)) {
signal.throwIfAborted();
try {
const directories = matchingBackupDirectories(candidate, stamp);
if (!directories || directories.reduce((total, entry) => total + entry.bytes, 0) > MAX_NATIVE_HARNESS_BYTES) continue;
let valid = true, count = 0;
for (const directory of directories) {
const hash = createHash("sha256"); let bytes = 0;
const visit = async (current: string, relative: string): Promise<void> => {
signal.throwIfAborted();
const entries = (await readdir(current, { withFileTypes: true })).sort((a, b) => a.name.localeCompare(b.name));
if (!entries.length) hash.update(`directory:${relative}\0`);
for (const entry of entries) {
if (++count > MAX_NATIVE_HARNESS_ENTRIES) throw new Error("native_harness_backup_entry_limit");
const path = resolve(current, entry.name), name = relative ? `${relative}/${entry.name}` : entry.name;
const stats = await lstat(path);
if (stats.isDirectory()) { hash.update(`directory:${name}:${stats.mode & 0o777}\0`); await visit(path, name); }
else if (stats.isSymbolicLink()) hash.update(`symlink:${name}:${await readlink(path)}\0`);
else if (stats.isFile()) {
hash.update(`file:${name}:${stats.mode & 0o777}:${stats.size}\0`);
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const before = await handle.stat();
if (!before.isFile() || stats.ino !== before.ino || stats.dev !== before.dev || stats.size !== before.size) throw new Error("native_harness_backup_source_changed");
for await (const chunk of handle.createReadStream({ signal, autoClose: false })) {
bytes += chunk.length; if (bytes > directory.bytes) throw new Error("native_harness_backup_size_mismatch"); hash.update(chunk);
}
const after = await handle.stat();
if (before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw new Error("native_harness_backup_source_changed");
} finally { await handle.close(); }
} else throw new Error("native_harness_backup_unsupported_entry");
}
};
await visit(directory.path, "");
if (`sha256:${hash.digest("hex")}` !== directory.sha256 || bytes !== directory.bytes) { valid = false; break; }
}
if (valid) return true;
} catch {
// Try the previous atomically-published backup.
}
} catch { signal.throwIfAborted(); }
}
return false;
}
@@ -0,0 +1,257 @@
import { createHash, randomUUID } from "node:crypto";
import { chmod, mkdir, readFile, rename, stat, symlink, readlink, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { and, eq } from "drizzle-orm";
import { agentTaskSessions, environmentLeases, heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations } from "@paperclipai/db";
import { afterEach, expect, it, vi } from "vitest";
import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
import { createStorageService } from "../../storage/service.js";
import { nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup,
verifyDurableNativeHarnessBackup, deleteRetiredNativeHarnessBackup } from "./native-harness-backup-storage.js";
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
import { buildNativeHarnessBackupManifest } from "./native-session-executor.js";
import { createNativeHarnessBackupStamp, verifyNativeHarnessBackupStamp, verifyNativeHarnessBackupStampAsync } from "./native-harness-backup-stamp.js";
import { hasLocalNativeHarnessObligations, migrateNativeHarnessBackups } from "./native-harness-backup-migration.js";
import { nativeHarnessBackupExpirationService } from "./native-harness-backup-expiration.js";
describeEmbeddedPostgres("durable native provider backups", () => {
const environment = nativeRestartFixture();
const ownedHomes: string[] = [];
afterEach(async () => {
for (const home of ownedHomes.splice(0)) await removeNativeHarnessBackup(home);
vi.unstubAllEnvs();
});
async function fixture(provider = "codex") {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "51".repeat(32));
const f = await environment.seedRun(randomUUID());
const { companyId, runtimeRoot, successor } = environment.current;
await f.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId, controllerGeneration: 1,
leaseOwner: "backup-writer", leaseExpiresAt: new Date(Date.now() + 120_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
const owner = await nativeHarnessBackupOwner(f.db, { companyId, issueId: f.issueId, runId: f.runId, bootId: successor.bootId });
const home = join(runtimeRoot, randomUUID()), root = join(home, "failover-backups", "current");
ownedHomes.push(home);
await mkdir(join(root, provider, "sessions"), { recursive: true });
await mkdir(join(root, "runner"));
const identity = { normalizedSessionId: randomUUID(), providerSessionId: randomUUID(), provider };
await writeFile(join(root, provider, "sessions", "history.json"), JSON.stringify(identity), { mode: 0o600 });
await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({ runId: f.runId, lifecycle: "suspended" }), { mode: 0o600 });
const manifest = JSON.stringify({ schema: "paperclip.native-harness-backup.v1", ...identity, directories: [provider, "runner"] });
await writeFile(join(root, "manifest.json"), manifest, { mode: 0o600 });
await chmod(join(root, provider, "sessions"), 0o500);
const sessionScopeId = randomUUID(), scopeDigest = createHash("sha256").update(sessionScopeId).digest("hex");
const storage = createStorageService(createLocalDiskStorageProvider(join(runtimeRoot, "durable-objects")));
return { ...f, home, root, companyId, owner, scopeDigest, sessionScopeId, storage, identity,
manifestSha256: `sha256:${createHash("sha256").update(manifest).digest("hex")}` };
}
it.each(["codex", "claude", "opencode", "grok", "new-provider"])("restores %s session bytes after deleting the entire controller home", async provider => {
const f = await fixture(provider);
await persistNativeHarnessBackup(f);
const [row] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
expect(row.status).toBe("current");
const object = await f.storage.getObject(f.companyId, row.reference.objectKey as string);
const parts: Buffer[] = []; for await (const part of object.stream) parts.push(part);
expect(Buffer.concat(parts).includes(Buffer.from(f.identity.providerSessionId))).toBe(false);
await removeNativeHarnessBackup(f.home);
const replacement = join(environment.current.runtimeRoot, randomUUID(), "failover-backups");
ownedHomes.push(replacement);
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
expect(JSON.parse(await readFile(join(replacement, "current", provider, "sessions", "history.json"), "utf8"))).toEqual(f.identity);
expect((await stat(join(replacement, "current", provider, "sessions"))).mode & 0o777).toBe(0o500);
expect(await verifyDurableNativeHarnessBackup(f)).toBe(true);
});
it("keeps the last published backup when object storage fails", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
const put = vi.spyOn(f.storage, "putFile").mockRejectedValue(new Error("storage unavailable"));
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("storage unavailable"); put.mockRestore();
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
expect(rows.filter(row => row.status === "current").map(row => row.id)).toEqual([before.id]);
expect(rows.filter(row => row.status === "pending")).toHaveLength(1);
expect(await readFile(join(f.root, "manifest.json"), "utf8")).toContain("native-harness-backup");
});
it("fences an old controller whose upload finishes after ownership changes", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
const put = f.storage.putFile.bind(f.storage);
f.storage.putFile = async input => {
const result = await put(input);
await f.db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(), controllerGeneration: 2 })
.where(eq(nativeRunFinalizations.runId, f.runId));
return result;
};
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_runner_checkpoint_owner_lost");
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
expect(rows.filter(row => row.status === "current").map(row => row.id)).toEqual([before.id]);
});
it("refuses publication after a pending upload was retired without demoting the current backup", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const [current] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
const put = f.storage.putFile.bind(f.storage);
f.storage.putFile = async input => {
const result = await put(input);
await f.db.update(nativeHarnessBackups).set({ status: "retired" }).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest), eq(nativeHarnessBackups.status, "pending")));
return result;
};
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_harness_storage_publication_retired");
expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, current.id)))[0].status).toBe("current");
const [retired] = await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest), eq(nativeHarnessBackups.status, "retired")));
expect(await deleteRetiredNativeHarnessBackup(f.db, f.storage, retired.id)).toBe(true);
});
it("rejects corrupt objects and cross-company lookups without publishing a local cache", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const replacement = join(environment.current.runtimeRoot, randomUUID(), "failover-backups");
expect(await restoreDurableNativeHarnessBackup({ ...f, companyId: randomUUID(), backupRoot: replacement })).toBe(false);
const [row] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
await f.storage.deleteObject(f.companyId, row.reference.objectKey as string);
await f.storage.putFile({ companyId: f.companyId, namespace: "native-harness-backups", objectKey: row.reference.objectKey as string,
originalFilename: null, contentType: "application/octet-stream", body: Buffer.from("corrupted") });
await expect(restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).rejects.toThrow("hash_mismatch");
await expect(stat(join(replacement, "current"))).rejects.toMatchObject({ code: "ENOENT" });
});
async function settle(f: { db: Awaited<ReturnType<typeof fixture>>["db"]; runId: string; issueId: string }, closedAt = new Date()) {
await f.db.update(heartbeatRuns).set({ status: "failed", finishedAt: closedAt }).where(eq(heartbeatRuns.id, f.runId));
await f.db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, f.runId));
await f.db.update(issues).set({ status: "done", executionRunId: null, completedAt: closedAt }).where(eq(issues.id, f.issueId));
}
it("migrates a lease-verified local backup once and keeps its selected session intact", async () => {
const f = await fixture("codex-home");
const base = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(base); await mkdir(base);
await rename(f.home, join(base, f.scopeDigest));
const root = join(base, f.scopeDigest, "failover-backups", "current");
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", base);
const runnerInstanceId = randomUUID(), providerLeaseId = randomUUID();
const execution = { provider: { kind: "codex" }, binding: { companyId: f.companyId, issueId: f.issueId,
runId: f.runId, agentId: environment.current.agentId, executionWorkspaceId: randomUUID() },
workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: f.identity.normalizedSessionId, driverKind: "codex_app_server",
protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } } as never;
await chmod(join(root, "codex-home", "sessions"), 0o700);
await symlink("history.json", join(root, "codex-home", "sessions", "history-link"));
await chmod(join(root, "codex-home", "sessions"), 0o500);
const manifest = buildNativeHarnessBackupManifest({ backupRoot: root, execution, runnerInstanceId,
providerSessionIdentity: { providerSessionId: f.identity.providerSessionId }, sourceProviderLeaseId: providerLeaseId });
await writeFile(join(root, "manifest.json"), JSON.stringify(manifest));
const stamp = createNativeHarnessBackupStamp({ manifestPath: join(root, "manifest.json"), sessionScopeId: f.sessionScopeId,
authorizedProviderLeaseId: providerLeaseId, runnerInstanceId, normalizedSessionId: f.identity.normalizedSessionId, completedAt: manifest.completedAt });
expect(verifyNativeHarnessBackupStamp(stamp, providerLeaseId)).toBe(true);
expect(await verifyNativeHarnessBackupStampAsync(stamp, providerLeaseId, AbortSignal.timeout(10_000))).toBe(true);
await expect(verifyNativeHarnessBackupStampAsync(stamp, providerLeaseId, AbortSignal.abort(new Error("cancelled")))).rejects.toThrow("cancelled");
await f.db.update(heartbeatRuns).set({ nativeSessionId: f.identity.normalizedSessionId, runnerInstanceId }).where(eq(heartbeatRuns.id, f.runId));
await f.db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: f.runId,
providerLeaseId, status: "expired", metadata: { nativeHarnessBackup: stamp } });
const [selected] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId,
adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId,
sessionParamsJson: { sessionId: f.identity.normalizedSessionId } }).returning();
expect(await hasLocalNativeHarnessObligations(f.db)).toBe(true);
await expect(migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) }))
.rejects.toThrow("native_harness_migration_requires_offline_writers");
const ownedRuns = await f.db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, f.companyId));
for (const run of ownedRuns) await settle({ ...f, runId: run.id, issueId: run.issueId! });
await f.db.update(nativeRunFinalizations).set({ recoveryState: "blocked" }).where(eq(nativeRunFinalizations.runId, f.runId));
const [preservedOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
const [queued] = await f.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: environment.current.agentId,
status: "queued", invocationSource: "on_demand", issueId: f.issueId }).returning();
await f.db.update(issues).set({ executionRunId: queued.id }).where(eq(issues.id, f.issueId));
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(1);
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(1);
expect(await hasLocalNativeHarnessObligations(f.db)).toBe(false);
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(1);
expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, selected.id)))[0]).toEqual(selected);
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(preservedOwner);
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id)))[0]).toEqual(queued);
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(queued.id);
expect(JSON.parse(await readFile(join(root, "manifest.json"), "utf8"))).toEqual(manifest);
const restored = join(base, "replacement");
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: restored })).toBe(true);
expect(await readlink(join(restored, "current", "codex-home", "sessions", "history-link"))).toBe("history.json");
const changedManifest = JSON.stringify({ ...manifest, completedAt: new Date(Date.now() + 1_000).toISOString() });
await writeFile(join(root, "manifest.json"), changedManifest);
await expect(migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) }))
.rejects.toThrow("native_harness_migration_durable_source_conflict");
expect(await readFile(join(root, "manifest.json"), "utf8")).toBe(changedManifest);
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(1);
});
it("preserves orphaned local backup files without blocking other enrollment", async () => {
const f = await fixture();
const base = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(base); await mkdir(base);
await rename(f.home, join(base, f.scopeDigest));
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", base);
const root = join(base, f.scopeDigest, "failover-backups", "current");
const orphan = JSON.stringify({ runId: randomUUID(), lifecycle: "suspended" });
await writeFile(join(root, "runner", "runner-state.json"), orphan);
const ownedRuns = await f.db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, f.companyId));
for (const run of ownedRuns) await settle({ ...f, runId: run.id, issueId: run.issueId! });
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(0);
expect(await readFile(join(root, "runner", "runner-state.json"), "utf8")).toBe(orphan);
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(0);
});
it("retains live work and retries an uncertain object deletion through the existing startup retention pass", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const old = new Date(Date.now() - 10 * 86_400_000);
await f.db.update(nativeHarnessBackups).set({ createdAt: old }).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
expect((await retention.tickCheckpoints()).expired).toBe(0);
await settle(f, old);
const drop = vi.spyOn(f.storage, "deleteObject").mockRejectedValueOnce(new Error("delete reply lost"));
expect((await retention.tickCheckpoints()).failed).toBe(1);
expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)))[0].status).toBe("retired");
expect((await retention.tickCheckpoints()).expired).toBe(1);
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(0);
expect(drop).toHaveBeenCalledTimes(2);
});
it("expires obsolete objects without clearing an open conversation or its exact lease backup", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const [obsolete] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
await writeFile(join(f.root, "manifest.json"), JSON.stringify({ revision: 2 }));
await persistNativeHarnessBackup(f);
const old = new Date(Date.now() - 10 * 86_400_000);
await f.db.update(nativeHarnessBackups).set({ createdAt: old, updatedAt: old }).where(eq(nativeHarnessBackups.id, obsolete.id));
const [selected] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId,
adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId }).returning();
const [lease] = await f.db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
heartbeatRunId: f.runId, status: "pending_cleanup", metadata: { nativeHarnessBackup: {
sessionScopeSha256: f.scopeDigest, manifestSha256: obsolete.manifestSha256 } } }).returning();
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
expect((await retention.tickCheckpoints()).expired).toBe(0);
await f.db.update(environmentLeases).set({ status: "expired" }).where(eq(environmentLeases.id, lease.id));
expect((await retention.tickCheckpoints({ dryRun: true })).expired).toBe(0);
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, obsolete.id))).toHaveLength(1);
expect((await retention.tickCheckpoints()).expired).toBe(1);
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
expect(rows).toHaveLength(1); expect(rows[0].status).toBe("current");
expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, selected.id)))[0]).toEqual(selected);
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running");
});
it("retains an in-flight pending upload and removes it once its owner settles", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const put = vi.spyOn(f.storage, "putFile").mockRejectedValueOnce(new Error("upload unavailable"));
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("upload unavailable"); put.mockRestore();
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
const pending = rows.find(row => row.status === "pending")!;
const old = new Date(Date.now() - 10 * 86_400_000);
await f.db.update(nativeHarnessBackups).set({ createdAt: old, updatedAt: old }).where(eq(nativeHarnessBackups.id, pending.id));
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
expect((await retention.tickCheckpoints()).expired).toBe(0);
await settle(f);
expect((await retention.tickCheckpoints()).expired).toBe(1);
const remaining = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
expect(remaining).toHaveLength(1); expect(remaining[0].status).toBe("current");
});
it("keeps deleted-source objects unchanged during a retention dry run", async () => {
const f = await fixture(); await persistNativeHarnessBackup(f);
const old = new Date(Date.now() - 10 * 86_400_000);
await f.db.update(nativeHarnessBackups).set({ createdAt: old, runId: randomUUID() })
.where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
expect((await retention.tickCheckpoints({ dryRun: true })).expired).toBe(0);
expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, before.id)))[0]).toEqual(before);
expect(await verifyDurableNativeHarnessBackup(f)).toBe(true);
expect((await retention.tickCheckpoints()).expired).toBe(1);
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, before.id))).toHaveLength(0);
});
});
@@ -0,0 +1,228 @@
import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js";
import { createCipheriv, createDecipheriv, createHash, randomBytes, randomUUID } from "node:crypto";
import { createReadStream, createWriteStream } from "node:fs";
import { mkdtemp, readFile } from "node:fs/promises";
import { join } from "node:path";
import os from "node:os";
import { pipeline } from "node:stream/promises";
import { Transform } from "node:stream";
import { and, eq, inArray, sql } from "drizzle-orm";
import { heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js";
import type { StorageService } from "../../storage/types.js";
import { assertNativeRunnerCheckpointOwner, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js";
import { loadConfig } from "../../config.js";
import { getStorageService } from "../../storage/index.js";
import { readNativeHarnessBackupStamp, verifyNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js";
import { captureNativeHarnessArchive, restoreNativeHarnessArchive, MAX_NATIVE_HARNESS_OBJECT_BYTES } from "./native-harness-backup-archive.js";
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
const SCHEMA = "paperclip.native-harness-object.v1";
type Binding = { companyId: string; scopeDigest: string; id: string };
type Reference = Binding & {
schema: typeof SCHEMA; objectKey: string; sha256: string; byteSize: number;
material: Record<string, unknown>; manifestSha256: string;
};
type Row = typeof nativeHarnessBackups.$inferSelect;
type OfflineOwner = { kind: "offline_migration"; companyId: string; issueId: string; runId: string;
writersStopped: true; terminalFailureOnly?: true };
const fail = (reason: string): never => { throw new Error(`native_harness_storage_${reason}`); };
const aad = (binding: Binding) => Buffer.from(JSON.stringify([SCHEMA, binding.companyId, binding.scopeDigest, binding.id]));
const scope = (binding: Pick<Binding, "companyId" | "scopeDigest">) => and(
eq(nativeHarnessBackups.companyId, binding.companyId), eq(nativeHarnessBackups.scopeDigest, binding.scopeDigest));
export function durableNativeHarnessStorage(): StorageService | null {
if (loadConfig().storageProvider === "s3") return getStorageService();
if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true") fail("durable_storage_required");
return null;
}
export async function verifyRecoverableNativeHarnessBackup(input: {
db: Db; companyId: string; stamp: unknown; providerLeaseId: string;
}): Promise<boolean> {
const stamp = readNativeHarnessBackupStamp(input.stamp, input.providerLeaseId);
if (!stamp) return false;
const storage = durableNativeHarnessStorage();
if (!storage) return verifyNativeHarnessBackupStamp(stamp, input.providerLeaseId);
const durable = await verifyDurableNativeHarnessBackup({ db: input.db, storage, companyId: input.companyId,
scopeDigest: stamp.sessionScopeSha256, manifestSha256: stamp.manifestSha256, signal: AbortSignal.timeout(300_000) });
return durable || process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" && verifyNativeHarnessBackupStamp(stamp, input.providerLeaseId);
}
function reference(row: Row): Reference {
const ref = row.reference as Reference;
if (ref.schema !== SCHEMA || ref.id !== row.id || ref.companyId !== row.companyId || ref.scopeDigest !== row.scopeDigest
|| !/^[a-f0-9]{64}$/.test(ref.scopeDigest) || !/^[a-f0-9]{64}$/.test(ref.sha256)
|| ref.manifestSha256 !== row.manifestSha256 || !/^sha256:[a-f0-9]{64}$/.test(ref.manifestSha256)
|| ref.objectKey !== `${row.companyId}/native-harness-backups/${row.scopeDigest}/${row.id}.pchn`
|| !Number.isSafeInteger(ref.byteSize) || ref.byteSize <= 0 || ref.byteSize > MAX_NATIVE_HARNESS_OBJECT_BYTES
|| !ref.material || typeof ref.material !== "object") fail("reference_invalid");
return ref;
}
async function digest(file: string, signal?: AbortSignal) {
const hash = createHash("sha256"); let byteSize = 0;
for await (const bytes of createReadStream(file, { signal })) { hash.update(bytes); byteSize += bytes.length; }
return { sha256: hash.digest("hex"), byteSize };
}
async function receive(storage: StorageService, ref: Reference, target?: string, signal?: AbortSignal) {
const object = await storage.getObject(ref.companyId, ref.objectKey, { signal });
const hash = createHash("sha256"); let byteSize = 0;
const observed = new Transform({ transform(bytes, _encoding, callback) {
hash.update(bytes); byteSize += bytes.length;
callback(byteSize > ref.byteSize ? new Error("native_harness_storage_size_mismatch") : null, bytes);
} });
try {
if (target) await pipeline(object.stream, observed, createWriteStream(target, { flags: "wx", mode: 0o600 }), { signal });
else for await (const bytes of object.stream) {
signal?.throwIfAborted(); hash.update(bytes); byteSize += bytes.length;
if (byteSize > ref.byteSize) fail("size_mismatch");
}
if (byteSize !== ref.byteSize || hash.digest("hex") !== ref.sha256) fail("hash_mismatch");
} finally { object.stream.destroy(); }
}
export async function nativeHarnessBackupOwner(db: Db, input: { companyId: string; issueId: string; runId: string; bootId: string }) {
const [row] = await db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
eq(nativeRunFinalizations.runId, input.runId), eq(nativeRunFinalizations.controllerBootId, input.bootId)));
if (!row?.leaseOwner) fail("owner_missing");
return { ...input, authorityRunId: input.runId, leaseOwner: row!.leaseOwner!, generation: row!.controllerGeneration } satisfies NativeRunnerCheckpointOwner;
}
async function fence(tx: Parameters<Parameters<Db["transaction"]>[0]>[0], owner: NativeRunnerCheckpointOwner | OfflineOwner) {
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
.where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId))).for("share");
const authorityRunId = "kind" in owner ? owner.runId : owner.authorityRunId;
if (!task || !("kind" in owner) && task.executionRunId && task.executionRunId !== authorityRunId) fail("owner_superseded");
if ("kind" in owner) {
if (!owner.writersStopped) fail("offline_writers_required");
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
if (active) fail("offline_writers_required");
const [run] = await tx.select({ finishedAt: heartbeatRuns.finishedAt, status: heartbeatRuns.status }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, owner.runId), eq(heartbeatRuns.companyId, owner.companyId),
eq(heartbeatRuns.nativeIssueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "native"))).for("share");
const [coordinator] = await tx.select({ leaseOwner: nativeRunFinalizations.leaseOwner,
leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, phase: nativeRunFinalizations.phase })
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, owner.runId)).for("share");
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
if (!run?.finishedAt || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status)
|| !coordinator || coordinator.leaseOwner && (!coordinator.leaseExpiresAt || coordinator.leaseExpiresAt > new Date(clock.now))
|| owner.terminalFailureOnly && (run.status !== "failed" || coordinator.phase !== "terminal_failure")) {
fail("offline_owner_unsettled");
}
return;
}
await assertNativeRunnerCheckpointOwner(tx, owner, false);
}
export async function persistNativeHarnessBackup(input: {
db: Db; storage: StorageService; owner: NativeRunnerCheckpointOwner | OfflineOwner;
scopeDigest: string; root: string; signal?: AbortSignal;
}): Promise<void> {
if (!/^[a-f0-9]{64}$/.test(input.scopeDigest)) fail("scope_invalid");
const manifestBytes = await readFile(join(input.root, "manifest.json"));
const manifestSha256 = `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`;
const binding = { companyId: input.owner.companyId, scopeDigest: input.scopeDigest, id: randomUUID() };
const scratch = await mkdtemp(join(os.tmpdir(), "pc-harness-upload-"));
const key = randomBytes(32);
try {
const archive = join(scratch, "archive"), encrypted = join(scratch, "encrypted");
await captureNativeHarnessArchive(input.root, archive, input.signal);
const plaintext = await digest(archive, input.signal), iv = randomBytes(12);
const cipher = createCipheriv("aes-256-gcm", key, iv); cipher.setAAD(aad(binding));
await pipeline(createReadStream(archive), cipher, createWriteStream(encrypted, { flags: "wx", mode: 0o600 }), { signal: input.signal });
const ref: Reference = { schema: SCHEMA, ...binding, manifestSha256,
objectKey: `${binding.companyId}/native-harness-backups/${binding.scopeDigest}/${binding.id}.pchn`,
...await digest(encrypted, input.signal),
material: createLocalEncryptedCipher("write").encrypt(JSON.stringify({ schema: SCHEMA, ...binding,
key: key.toString("base64"), iv: iv.toString("base64"), tag: cipher.getAuthTag().toString("base64"), plaintext })) };
if (ref.byteSize > MAX_NATIVE_HARNESS_OBJECT_BYTES) fail("size_limit_exceeded");
await input.db.transaction(async tx => {
await fence(tx, input.owner);
await tx.insert(nativeHarnessBackups).values({ ...binding, issueId: input.owner.issueId, runId: input.owner.runId,
manifestSha256, reference: ref });
});
await input.storage.putFile({ companyId: ref.companyId, namespace: "native-harness-backups", objectKey: ref.objectKey,
originalFilename: null, contentType: "application/octet-stream", body: createReadStream(encrypted),
byteSize: ref.byteSize, sha256: ref.sha256, signal: input.signal });
await receive(input.storage, ref, undefined, input.signal);
await input.db.transaction(async tx => {
await fence(tx, input.owner);
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${ref.companyId}:${ref.scopeDigest}`}, 0))`);
const [pending] = await tx.select({ status: nativeHarnessBackups.status }).from(nativeHarnessBackups)
.where(eq(nativeHarnessBackups.id, ref.id)).for("update");
if (pending?.status !== "pending") fail("publication_retired");
const [successor] = await tx.select({ id: nativeHarnessBackups.id }).from(nativeHarnessBackups)
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeHarnessBackups.runId)).where(and(scope(ref),
eq(nativeHarnessBackups.status, "current"), sql`${nativeHarnessBackups.runId} <> ${input.owner.runId}::uuid`,
sql`${heartbeatRuns.createdAt} >= (select created_at from heartbeat_runs where id = ${input.owner.runId}::uuid)`)).limit(1);
if (successor) fail("owner_superseded");
await fence(tx, input.owner);
await tx.update(nativeHarnessBackups).set({ status: "superseded", updatedAt: new Date() })
.where(and(scope(ref), eq(nativeHarnessBackups.status, "current")));
await tx.update(nativeHarnessBackups).set({ status: "current", updatedAt: new Date() })
.where(and(eq(nativeHarnessBackups.id, ref.id), eq(nativeHarnessBackups.status, "pending")));
});
} finally { key.fill(0); await removeNativeHarnessBackup(scratch); }
}
export async function restoreDurableNativeHarnessBackup(input: {
db: Db; storage: StorageService; companyId: string; scopeDigest: string;
backupRoot: string; signal?: AbortSignal;
}): Promise<boolean> {
const [row] = await input.db.select().from(nativeHarnessBackups).where(and(scope(input), eq(nativeHarnessBackups.status, "current")));
if (!row) return false;
const ref = reference(row);
const wrapped = JSON.parse(createLocalEncryptedCipher("read").decrypt(ref.material));
if (wrapped.schema !== SCHEMA || wrapped.id !== ref.id || wrapped.companyId !== ref.companyId
|| wrapped.scopeDigest !== ref.scopeDigest) fail("binding_mismatch");
const current = join(input.backupRoot, "current");
const localManifest = await readFile(join(current, "manifest.json")).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null; throw error;
});
if (localManifest && `sha256:${createHash("sha256").update(localManifest).digest("hex")}` === ref.manifestSha256) return true;
const scratch = await mkdtemp(join(os.tmpdir(), "pc-harness-restore-"));
const key = Buffer.from(wrapped.key, "base64");
try {
if (key.length !== 32) fail("key_invalid");
const encrypted = join(scratch, "encrypted"), archive = join(scratch, "archive");
await receive(input.storage, ref, encrypted, input.signal);
const decipher = createDecipheriv("aes-256-gcm", key, Buffer.from(wrapped.iv, "base64"));
decipher.setAAD(aad(ref)); decipher.setAuthTag(Buffer.from(wrapped.tag, "base64"));
await pipeline(createReadStream(encrypted), decipher, createWriteStream(archive, { flags: "wx", mode: 0o600 }), { signal: input.signal });
const actual = await digest(archive, input.signal);
if (actual.sha256 !== wrapped.plaintext?.sha256 || actual.byteSize !== wrapped.plaintext?.byteSize) fail("plaintext_mismatch");
const restored = join(scratch, "restored");
await restoreNativeHarnessArchive(archive, restored, input.signal);
const manifest = await readFile(join(restored, "manifest.json"));
if (`sha256:${createHash("sha256").update(manifest).digest("hex")}` !== ref.manifestSha256) fail("manifest_mismatch");
const { mkdir, cp } = await import("node:fs/promises");
await mkdir(input.backupRoot, { recursive: true, mode: 0o700 });
const pending = join(input.backupRoot, `.durable-${randomUUID()}`);
try {
await cp(restored, pending, { recursive: true, preserveTimestamps: true, verbatimSymlinks: true, errorOnExist: true, force: false });
await publishNativeHarnessBackup({ db: input.db, companyId: input.companyId, scopeDigest: input.scopeDigest,
backupRoot: input.backupRoot, pendingRoot: pending });
} finally { await removeNativeHarnessBackup(pending); }
return true;
} finally { key.fill(0); await removeNativeHarnessBackup(scratch); }
}
export async function verifyDurableNativeHarnessBackup(input: {
db: Db; storage: StorageService; companyId: string; scopeDigest: string; manifestSha256: string; signal?: AbortSignal;
}): Promise<boolean> {
const [row] = await input.db.select().from(nativeHarnessBackups).where(and(scope(input),
eq(nativeHarnessBackups.manifestSha256, input.manifestSha256), inArray(nativeHarnessBackups.status, ["current", "superseded"]))).limit(1);
if (!row) return false;
await receive(input.storage, reference(row), undefined, input.signal);
return true;
}
export async function deleteRetiredNativeHarnessBackup(db: Db, storage: StorageService, id: string, signal?: AbortSignal) {
const [row] = await db.select().from(nativeHarnessBackups).where(and(
eq(nativeHarnessBackups.id, id), eq(nativeHarnessBackups.status, "retired")));
if (!row) return false;
const ref = reference(row);
await storage.deleteObject(ref.companyId, ref.objectKey, { allVersions: true, signal });
await db.delete(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.id, id), eq(nativeHarnessBackups.status, "retired")));
return true;
}
@@ -6,6 +6,8 @@ import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provid
import { legacyControllerBootId } from "../legacy-controller-lease.js";
type Binding = { companyId: string; issueId: string; runId: string };
export const NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES = 1024 * 1024;
export const NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES = 8 * 1024 * 1024;
export type NativeRunnerCheckpointOwner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string };
function encryptParts(binding: Binding, records: ReadonlyMap<string, string>) {
if (!records.size) return [];
@@ -16,6 +18,11 @@ function encryptParts(binding: Binding, records: ReadonlyMap<string, string>) {
}));
}
export async function insertMigratedNativeRunnerCheckpoint(tx: CheckpointTransaction, binding: Binding, checkpoint: DurablePrpCheckpoint) {
if (checkpoint.runId !== binding.runId) throw new Error("native_runner_checkpoint_binding_mismatch");
await tx.insert(nativeRunnerCheckpoints).values(encryptParts(binding, checkpoint.records));
}
function decryptPart(cipher: ReturnType<typeof createLocalEncryptedCipher>, binding: Binding,
part: string, material: Record<string, unknown>): string {
const envelope = JSON.parse(cipher.decrypt(material));
@@ -1632,6 +1632,18 @@ describe("native provider usage normalization", () => {
it("does not treat a protocol-default price as free usage", () => {
expect(nativeUsageCostUsd({ runDelta: { inputTokens: 20, outputTokens: 10, providerCostUsd: 0 } })).toBeUndefined();
});
it("attributes Grok API usage while preserving managed billing identity", () => {
const provider = { kind: "acpx", agent: "grok", model: "grok-4.7" } as NativeExecutionInput["provider"];
expect(nativeUsageBiller(provider)).toBe("xai");
expect(resolveNativeBilling(provider)).toEqual({ provider: "xai", biller: "xai", billingType: "unknown" });
expect(resolveNativeBilling(provider, { XAI_API_KEY: "fixture" }))
.toEqual({ provider: "xai", biller: "xai", billingType: "metered_api" });
const subscription = { provider: "xai", biller: "xai", billingType: "subscription_included" as const };
expect(resolveNativeBilling(provider, { XAI_API_KEY: "fixture" }, subscription)).toEqual(subscription);
expect(nativeUsageCostUsd({ runDelta: { inputTokens: 30787, outputTokens: 5763, cacheReadTokens: 475008,
cacheWriteTokens: 0, providerCostUsd: 0.336842 } }, provider)).toBe(0.336842);
});
it("reports the actual runtime billing identity", () => {
expect(resolveNativeBilling({ kind: "opencode", model: "openrouter/deepseek/test", permissionMode: "allow" }))
.toEqual({ provider: "deepseek", biller: "openrouter", billingType: "unknown" });
@@ -4,6 +4,7 @@ import { providerFailureFromError } from "../../vendor/paperclip-runner/index.js
import { nativeCancellationHasUnsettledOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js";
import { readClaimedNativeStopRecovery } from "./native-cancellation-request.js";
import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js";
import { durableNativeHarnessStorage, nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js";
import { rehydratePinnedSkillAssets } from "./runtime-context.js";
import { createNativeUsageDelivery } from "./native-usage-delivery.js";
import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js";
@@ -5964,6 +5965,7 @@ function nativeSessionConfigDigest(
execution: NativeExecutionInput,
executionTargetKind: "local" | "remote" = "local",
legacyProjectlessRunId?: string,
recordedToolContractFingerprint?: string,
): string {
const executionLocation = {
executionKind: "local_process",
@@ -5999,13 +6001,25 @@ function nativeSessionConfigDigest(
// changes so an older thread is rotated instead of falsely resuming
// without newly required tools.
nativeToolContractFingerprint:
nativeToolContractFingerprintForTarget(executionTargetKind, execution.provider.kind),
recordedToolContractFingerprint ?? nativeToolContractFingerprintForTarget(executionTargetKind, execution.provider.kind),
runtimeContract: nativeRuntimeContractForProvider(execution.provider),
}),
)
.digest("hex")}`;
}
export function nativeControllerMigrationDescriptor(execution: NativeExecutionInput, recordedToolContractFingerprint: unknown) {
if (typeof recordedToolContractFingerprint !== "string" || !/^sha256:[a-f0-9]{64}$/.test(recordedToolContractFingerprint)) {
throw new Error("native_controller_migration_tool_contract_missing");
}
return {
root: scopedRunnerdStateRoot(execution),
supervisorPath: resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner", "sessions",
`${createHash("sha256").update(nativeSessionScopeKey(execution)).digest("hex")}.json`),
configDigest: nativeSessionConfigDigest(execution, "remote", undefined, recordedToolContractFingerprint),
};
}
function idleWarmNativeSessionOwnerRunId(input: {
execution: NativeExecutionInput;
runnerExecutionTarget?: AdapterExecutionTarget | null;
@@ -6027,6 +6041,7 @@ async function restorePerTurnRunnerdController(input: {
if (input.restartRecovery || input.runnerExecutionTarget?.kind !== "remote" ||
input.execution.session.lifecyclePolicy.mode !== "per_turn") return;
const root = scopedRunnerdStateRoot(input.execution);
await restoreDurableHarnessForExecution(input.db, input.execution, root);
const identity = readRunnerdDurableIdentity(root);
const reference = identity ?? readRunnerdDurableIdentity(root, true);
if (!reference) {
@@ -6091,6 +6106,7 @@ async function restoreWarmRunnerdController(input: {
if (input.restartRecovery || input.runnerExecutionTarget?.kind !== "remote" ||
input.execution.session.lifecyclePolicy.mode !== "warm") return;
const root = scopedRunnerdStateRoot(input.execution);
await restoreDurableHarnessForExecution(input.db, input.execution, root);
const identity = readRunnerdDurableIdentity(root);
const reference = identity ?? readRunnerdDurableIdentity(root, true);
if (!idleWarmNativeSessionOwnerRunId(input) && (!reference || !existsSync(nativeSessionCheckpointPath(input.execution)))) {
@@ -6599,6 +6615,14 @@ function harnessBackupRoot(root: string): string {
return resolve(root, "failover-backups");
}
async function restoreDurableHarnessForExecution(db: Db, execution: NativeExecutionInput, root: string) {
const storage = durableNativeHarnessStorage();
if (!storage) return;
await restoreDurableNativeHarnessBackup({ db, storage, companyId: execution.binding.companyId,
scopeDigest: createHash("sha256").update(nativeSessionScopeKey(execution)).digest("hex"),
backupRoot: harnessBackupRoot(root), signal: AbortSignal.timeout(300_000) });
}
function harnessBackupCandidates(root: string): string[] {
const backupRoot = harnessBackupRoot(root);
return [resolve(backupRoot, "current"), resolve(backupRoot, "previous")];
@@ -10987,6 +11011,9 @@ export function resolveNativeBilling(provider: NativeExecutionInput["provider"],
const biller = nativeUsageBiller(provider);
return { provider: biller, biller, billingType: "unknown" };
}
if (provider.kind === "acpx" && provider.agent === "grok") {
return { provider: "xai", biller: "xai", billingType: env.XAI_API_KEY ? "metered_api" : "unknown" };
}
if (provider.kind === "acpx" && provider.agent === "claude") {
if (env.CLAUDE_CODE_USE_BEDROCK === "1") return { provider: "anthropic", biller: "aws_bedrock", billingType: "metered_api" };
return { provider: "anthropic", biller: "anthropic", billingType: env.ANTHROPIC_API_KEY ? "metered_api" : env.CLAUDE_CODE_OAUTH_TOKEN ? "subscription_included" : "unknown" };
@@ -14069,6 +14096,11 @@ async function createRunnerdBackendWithinSessionClaim(
!remoteRunnerFilesystemRoot
)
return;
const durableStorage = durableNativeHarnessStorage();
if (durableStorage && !input.controllerPersistence) throw new Error("native_harness_storage_owner_missing");
const backupOwner = durableStorage ? await nativeHarnessBackupOwner(input.db, { ...input.execution.binding,
runId: sessionToolAuthorityEpochs.get(sessionScopeId)?.runId ?? input.execution.binding.runId,
bootId: input.controllerPersistence!.bootId }) : null;
// Transport release also runs when provider bootstrap failed. In that case
// runnerd has no durable provider identity (and may not have created the
// provider persistence directory at all), so attempting a failover backup
@@ -14174,6 +14206,11 @@ async function createRunnerdBackendWithinSessionClaim(
});
renameSync(temporaryManifest, manifestPath);
if (durableStorage && backupOwner) {
await persistNativeHarnessBackup({ db: input.db, storage: durableStorage, owner: backupOwner,
scopeDigest: createHash("sha256").update(sessionScopeId).digest("hex"), root: pendingRoot,
signal: AbortSignal.timeout(300_000) });
}
await publishNativeHarnessBackup({
db: input.db, companyId: input.execution.binding.companyId,
scopeDigest: createHash("sha256").update(sessionScopeId).digest("hex"), backupRoot, pendingRoot,
@@ -225,7 +225,7 @@ type NativeSessionResumeSelection = {
agentId: string;
issueId: string;
normalizedSessionId: string;
currentRunId: string;
currentRunId?: string;
beforeCreatedAt: Date;
/** Recovery discovery must retain an ineligible newest progress barrier. */
includeProgressBarrier?: boolean;
@@ -306,7 +306,7 @@ async function selectNativeSessionResumeBarrier(db: Db, input: NativeSessionResu
.from(heartbeatRuns)
.where(
and(
ne(heartbeatRuns.id, input.currentRunId),
input.currentRunId ? ne(heartbeatRuns.id, input.currentRunId) : undefined,
eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId),
eq(heartbeatRuns.nativeIssueId, input.issueId),
@@ -51,6 +51,7 @@ export function nativeUsageBiller(provider: NativeExecutionInput["provider"]): s
if (provider.agent === "cursor") return "cursor";
if (provider.agent === "copilot") return "github";
if (provider.agent === "pi") return "openrouter";
if (provider.agent === "grok") return "xai";
}
return "openai";
}
@@ -387,7 +387,8 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
}
}, 60_000);
it("prepares, saves and restores task files without a controller checkout or original allocation", async () => {
it.each(["same root", "different root", "destroyed allocation", "occupied root"] as const)(
"preserves the published workspace path and files on a cold restore (%s)", async mode => {
const reserved = await reserve({ kind: "empty" }), first = await target("sandbox-A");
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first });
expect(readRemoteWorkspaceAuthority(prepared.workspace.metadata)?.snapshotId).toBeTruthy();
@@ -399,8 +400,16 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
lease: prepared.lease, target: first, assertOwned: async () => undefined });
const current = await workspace(reserved.id);
expect(readRemoteWorkspaceAuthority(current.metadata)).toMatchObject({ dirty: false, writerRunId: null, snapshotId: saved.snapshotId });
await fs.rm(path.dirname(prepared.remoteRoot), { recursive: true, force: true });
const second = await target("sandbox-B");
if (mode === "destroyed allocation") {
const destroyed: EnvironmentLease = { ...prepared.lease, status: "released", releasedAt: new Date(), cleanupStatus: "success",
metadata: { ...prepared.lease.metadata, remoteExecutionTermination: remoteTerminationReceipt(prepared.lease,
{ providerLeaseId: prepared.lease.providerLeaseId, state: "destroyed" }) } };
await db.transaction(tx => clearRemoteWorkspaceLocationForReleasedLease({ db: tx, lease: destroyed }));
expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata)?.location).toBeNull();
}
await fs.rm(path.join(root, "sandbox-A"), { recursive: true, force: true });
const second = await target(mode === "different root" ? "sandbox-B" : "sandbox-A");
const expectedRemoteRoot = path.join(second.remoteCwd, ".paperclip-workspaces", path.basename(prepared.remoteRoot));
const nextRun = randomUUID(), nextLease = randomUUID();
await db.insert(heartbeatRuns).values({ id: nextRun, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" });
await db.insert(environmentLeases).values({ id: nextLease, companyId: ids.company, environmentId: ids.environment,
@@ -408,7 +417,19 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
const reservedAgain = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, runId: nextRun,
agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null, existingWorkspaceId: reserved.id, source: { kind: "empty" } });
const newLease = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, nextLease)))[0]! as EnvironmentLease;
if (mode === "occupied root") {
await fs.mkdir(expectedRemoteRoot, { recursive: true });
await fs.writeFile(path.join(expectedRemoteRoot, "unpublished.txt"), "preserve unknown preparation\n");
await expect(prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second }))
.rejects.toThrow("remote_workspace_operation_failed");
expect(await fs.readFile(path.join(expectedRemoteRoot, "unpublished.txt"), "utf8")).toBe("preserve unknown preparation\n");
expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata))
.toEqual(readRemoteWorkspaceAuthority(reservedAgain.metadata));
await fs.rename(expectedRemoteRoot, path.join(root, "retained-unknown-preparation"));
}
const restored = await prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second });
expect(restored.remoteRoot).toBe(expectedRemoteRoot);
if (mode !== "different root") expect(restored.remoteRoot).toBe(prepared.remoteRoot);
expect(await fs.readFile(path.join(restored.remoteRoot, "report.md"), "utf8")).toBe("useful saved result\n");
expect(await fs.readFile(path.join(restored.remoteRoot, "binary.bin"))).toEqual(Buffer.from([0, 1, 255]));
expect((await fs.stat(path.join(restored.remoteRoot, "empty-folder"))).isDirectory()).toBe(true);
@@ -619,8 +640,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
existingWorkspaceId: reserved.id, source: { kind: "empty" } });
const nextTarget = await target("restore-cleanup-destination");
const sandbox = nextTarget as Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
const expectedRemoteRoot = path.join(sandbox.remoteCwd, ".paperclip-workspaces",
`${reserved.id}-${readRemoteWorkspaceAuthority(nextWorkspace.metadata)!.epoch}-${nextLease}`);
const expectedRemoteRoot = path.join(sandbox.remoteCwd, ".paperclip-workspaces", path.basename(prepared.remoteRoot));
const execute = sandbox.runner!.execute.bind(sandbox.runner!);
let failCleanup = true, cleanupAttempts = 0;
sandbox.runner!.execute = async input => {
+20 -12
View File
@@ -73,28 +73,22 @@ export function shouldUseRemoteWorkspaces(input: {
return configuredRemoteWorkspacePersistence() !== null;
}
/** Deployment profile admission belongs beside workspace authority. A warm
* remote controller may replace its local projection; host-owned execution
* cannot make that promise. External conversation providers own no task disk. */
export function assertEphemeralWorkspaceExecution(input: {
externalConversation: boolean; adapterType: string;
pluginTaskWorkspace: boolean; remoteAuthority: boolean;
}) {
if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" || input.externalConversation) return;
// Runtime selection has not promoted a fresh run yet. Validate storage here;
// the runtime selector separately admits only its resolved warm native policy.
if (input.adapterType !== "paperclip_runner"
|| (!input.pluginTaskWorkspace && !input.remoteAuthority)) {
throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.",
throw unprocessable("Volume-free servers require native Paperclip Runner execution with remote workspace storage.",
{ code: "ephemeral_execution_configuration_required" });
}
}
/** The runtime selector calls this after environment, credentials and saved
* execution policy have resolved. Raw agent settings are not lifecycle authority. */
export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn" | undefined) {
if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && mode !== "warm") {
throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.",
export function assertEphemeralNativeRuntime(native: boolean) {
if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && !native) {
throw unprocessable("Volume-free servers require native Paperclip Runner execution with remote workspace storage.",
{ code: "ephemeral_execution_configuration_required" });
}
}
@@ -327,8 +321,22 @@ function isSavedWorkspaceRepository(workspace: Workspace, authority: RemoteWorks
&& (saved.readOnly ?? false) === (repository.readOnly ?? false));
}
function publishedWorkspaceDirectory(workspace: Workspace & Pick<typeof executionWorkspaces.$inferSelect, "providerRef">,
authority: RemoteWorkspaceAuthority) {
const publishedRoot = authority.snapshotId && (authority.location?.remoteCwd ?? workspace.providerRef);
if (!publishedRoot) return null;
const directory = path.posix.basename(publishedRoot);
const suffix = directory.slice(workspace.id.length);
if (!path.posix.isAbsolute(publishedRoot) || path.posix.basename(path.posix.dirname(publishedRoot)) !== ".paperclip-workspaces"
|| !directory.startsWith(workspace.id) || (suffix !== "" && !/^-[1-9][0-9]*-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(suffix))) {
throw new Error("remote_workspace_published_path_invalid");
}
return directory;
}
export async function prepareRemoteWorkspace(input: {
db: Db; workspace: Workspace; runId: string; lease: EnvironmentLease; target: AdapterExecutionTarget | null;
db: Db; workspace: Workspace & Pick<typeof executionWorkspaces.$inferSelect, "providerRef">;
runId: string; lease: EnvironmentLease; target: AdapterExecutionTarget | null;
source?: RemoteWorkspaceSource; assertOwned?: () => Promise<void>;
provisionCommand?: string | null; runtimeProvisionCommand?: string | null; storage?: StorageService;
}) {
@@ -345,7 +353,7 @@ export async function prepareRemoteWorkspace(input: {
&& initial.location?.environmentId === input.lease.environmentId;
if (initial.location && !sameAllocation && initial.dirty) throw conflict("The prior workspace has unsaved work; recover it before replacing its allocation.");
const remoteRoot = sameAllocation ? initial.location!.remoteCwd : path.posix.join(target.remoteCwd, ".paperclip-workspaces",
`${input.workspace.id}-${initial.epoch}-${input.lease.id}`);
publishedWorkspaceDirectory(input.workspace, initial) ?? `${input.workspace.id}-${initial.epoch}-${input.lease.id}`);
const preparationIdentity = createHash("sha256").update(JSON.stringify([input.workspace.companyId, input.workspace.id,
input.runId, initial.epoch, input.lease.id, input.lease.providerLeaseId, remoteRoot, initial.source])).digest("hex");
let restoredExport: Awaited<ReturnType<typeof restoreRemoteWorkspaceDirectory>> | null = null;
+1
View File
@@ -94,6 +94,7 @@ export const RUNNER_TOOL_SCOPE_TOKEN_HEADER = runner.RUNNER_TOOL_SCOPE_TOKEN_HEA
export const DurablePrpControlPlane = runner.DurablePrpControlPlane;
export const providerFailureFromError = runner.providerFailureFromError;
export const decodeDurablePrpCheckpoint = runner.decodeDurablePrpCheckpoint;
export const captureLegacyDurablePrpCheckpoint = runner.captureLegacyDurablePrpCheckpoint;
export const inspectDurablePrpCheckpointHeader = runner.inspectDurablePrpCheckpointHeader;
export const RUNNER_PRP_PROTOCOL_MIN = runner.RUNNER_PRP_PROTOCOL_MIN;
export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX;