diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index c7dc8f46975..3e23f56507a 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -4174,6 +4174,23 @@ capabilities, stable keys, operating limits, automatic history migration and qua Do not remove a web volume until its complete retained evidence inventory and the independent disposable-server requirements are satisfied. +## Volume-free execution admission + +With `PAPERCLIP_EPHEMERAL_SERVER=true`, workspace preparation requires Paperclip +Runner and remote workspace storage. A fresh run still has its initial `legacy` +label at this phase; that label is not the selected execution mode. The runtime +selector checks the resolved native warm lifecycle before admission and rejects +legacy or per-turn execution. External conversation providers keep their existing +exemption because they own no task filesystem. Source resolution checks this +contract again before a configured host folder can fall back to local storage. + +A replacement controller reconstructs an active remote agent-directory cache +from the registered sandbox, including unsaved files. The remote directory, +lease and save baseline remain unchanged; current canonical files are never +uploaded over the retained execution. Remote collection and cleanup serialize +through Postgres across controller homes. A missing remote collector retains a +pending save instead of treating an empty local cache as the agent's files. + ## Cloud startup configuration Cloud stores nonsecret startup choices in a private PostgreSQL singleton. diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index afbbe4d8282..a85a79356f5 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -1,7 +1,68 @@ # Zero-downtime web deployments -Date: 2026-10-31. Status: runtime and Cloud overlap policy merged; qualification -and existing-volume enrollment remain in progress. No hosted acceptance claim. +Date: 2026-10-31. Status (2026-11-01 14:05 UTC): existing-volume staging enrollment +is complete and its read/save/file-integrity checks pass. Active-agent recovery, +natural database idling and the ten-deploy staging campaign remain unqualified. +No production canary yet. + +Latest critical path: Core #18543 includes runtime admission, immutable-cache +retirement, remote agent-file projection, and adoption without rematerializing +launch assets. The live test at `67714af9ff` transferred the same runner/session to +controller generation two after deleting A's entire home, then failed in the full +backend constructor: composing system instructions still read A's retired bundle. +All three owned Daytona sandboxes were independently confirmed absent; the child +credential was revoked and the test object bucket deleted. This remains failed +evidence, not a successful deployment qualification. + +The backend now accepts admitted instructions from a validated same-run Runner +checkpoint. Runner owns checkpoint/template inspection; Core binds it to the +current run, normalized session and context digest. Staged attachments select the +validated target template. Fresh runs and ordinary later continuations still +compose current instructions. The deleted-bundle reproduction failed before the +fix, then passed with real backend session recovery and no new thread/turn start. +All 59 backend factory tests and the focused checkpoint validation tests pass. +At `cd690de4d3`, the real Codex/Daytona run survives removal of A's entire home, +reattaches the same runner/session under generation two, and succeeds. Its next +revision also succeeds. The handoff records 372 reads, 38 saves and 38 verified +saves with zero failures. Full qualification still fails: the supervisor created +an unrelated open probe task in the business company, so the unchanged all-tasks- +done oracle could never advance to the second replacement. The attempt was +interrupted and retained as incomplete. A regression reproduces this harness +scope defect; probes now use their own disposable company. The oracle is unchanged. +Exact-head review of the recovery fix is 5/5 with no unresolved threads; new +fixture checks and another bounded live attempt are next. + +Review also identified save quotas incorrectly applied during cache recovery. +Recovery uses the existing inbound remote projection and preserves path/link +validation without enforcing save quotas on admission. Ordinary saves still reject +oversized changes. Related transport recovery tests passed 71 cases; directory +cleanup/cache-loss tests passed 20 cases after three timing failures in a broad +run. A separate CI attention-retention timeout reproduced on clean master: its +fixed clock fell behind database-assigned updatedAt. Advancing across the saved +deadline fixes it; all 53 tests in that file pass with original timeouts. + +Cloud #1102 merged as `461c4aea28` and is serving on the staging controller. +The original enrollment completed using its original qualified image, a retained +detached volume, and the migrated decision key. The post-enrollment probe passed +20 reads and three saves with no HTTP/WebSocket failures and verified all 34 files. +The original browser tab reconnected and persisted a task title. Server logs later +showed old Agents chunks returned 404 and navigation recovered via a page reload: +this is migration continuity evidence, not uninterrupted old-tab asset loading. + +Two eight-minute natural database-idle observations failed. A separate bounded SQL +attribution window saw no other app-database sessions; it is diagnostic stimulus, +not idle proof, and does not establish the source of provider activity. Automatic +idle sleep was correctly refused because this development fixture is not eligible +for the free/demo policy. Manual sleep with force disabled passed: the registry +reports sleeping, Railway has no active web deployment or attached volume, and +Neon is idle. The normal organization entry started wake, and Railway became healthy on the +same volume-free service. Post-wake proof passed 21 reads, three saves, a connected +WebSocket and all 34 file hashes. The browser displayed the retained task and saved +a new title, independently read back through the API. Automatic return from the +refreshing entry page was not verified; the browser navigated to the tenant after +readiness. Wake took several minutes while existing shared UI assets were checked. +Private receipts remain under the task's live-test, staging enrollment and hosted +probe directories; no credential or live target identity belongs here. ## Outcome and endpoint @@ -63,10 +124,11 @@ Measure one-time setup separately from repeated attempts. Keep failed evidence. | Item | State | Next action | | --- | --- | --- | -| Prior prerequisites | #18403 now merged; current source fetched | Characterize remaining local authority paths | -| Core implementation | `/private/tmp/paperclip-zero-downtime-20261031`, durable files, readiness/drain and migration/cleanup implemented in draft PR #18518 | Cross-process commands, browser/assets, active Runner qualification | -| Cloud implementation | `/private/tmp/paperclip-cloud-zero-downtime-20261031`, persisted ephemeral shape and image/variable gates implemented in draft PR #1096 | Complete lifecycle guards, test and deploy qualification | -| Staging / production | Not selected or deployed | Record exact target, owner, artifact and campaign before action | +| Prior prerequisites | #18403 and all five storage workstreams merged | Retain the ephemeral filesystem contract | +| Core implementation | #18518, #18526, #18529, #18530, #18531 and #18533 merged; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Complete live remote execution, accounting and later continuation qualification | +| Cloud implementation | #1096, #1098, #1099, #1100, #1101 and #1102 merged; #1102 serving on staging | Verify repeated Railway overlap behavior after Core recovery qualification | +| Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Diagnose idle behavior, verify sleep/wake and repair, then qualify ten app-only replacements | +| Production | No canary mutation performed | Proceed only after the staging and active-agent acceptance gates pass | Keep PRs, tested revisions, artifact identities, evidence paths, deployment targets and unresolved gates here. A green CI/build/deploy status alone is not acceptance. @@ -528,3 +590,196 @@ and after logs are `/private/tmp/paperclip-zdt-buffer-test-{before,after}-pg.log A first local attempt failed before tests because embedded PostgreSQL could not start; the existing external test database enabled the comparison. The failed post-merge shard was retried once while the assertion fix proceeds through review. + +### 2026-11-01 11:07 UTC package compatibility prerequisite + +The post-merge source retry passed. The final Cloud composition for Core +`1d348bbef0` is `cloud-41298718800-1`, digest +`sha256:800cb8578368c63e33bdd048786ec66b3e7a9902f0f221d7fe41ea3c35134020`. +It passed the real-volume migration and empty-home rehearsal with 121 reads, +25 saves and zero failures; receipt: +`/private/tmp/paperclip-zdt-container-drdrQS/evidence.json`. + +The forward/rollback manifest comparison exposed a real packaging blocker. +The Agent Computer bundle differed only in its embedded Runner source revision +and source map, despite unchanged Runner inputs, SDKs and dependency lock. +The exact-package gate correctly requires maintenance for those two images. +Do not use them as proof of an app-only overlap pair. + +The Cloud source-selection owner now selects the last first-parent Runner input +change when the older reviewed ROM pin differs, then proves complete input +equality. The plugin and ROM builder share that selection and keep temporary +history alive through archive verification. Actual clean checkouts of both +released Core commits select `66e8fab9447aa21350f87ddec79fcd027f3e23ad`. +A real shallow-checkout probe selected the same source in 25.6 seconds without +modifying the caller. The focused suite passed 32 cases; full Cloud tests passed +4,830 cases with 129 gated skips and no failures. Rebuilt package equality and +hosted overlap remain unverified. Private receipts are +`/private/tmp/paperclip-zdt-canonical-source-{proof,shallow-proof}.json`. + +Core's assertion fix is in #18533, reviewed at 5/5. One CI shard received a +runner shutdown signal, so that cancelled shard and its downstream aggregate +gate were retried. The original cancellation log remains at +`/private/tmp/paperclip-zdt-buffer-pr-cancelled-job.log`. +No hosted enrollment has occurred. After the controller became quiet, the +qualified maintenance image was added to the staging allowlist without an +automatic deployment. An explicit controller configuration restart was then +submitted with a mutation-confirmed deployment identity. The prior busy-campaign +and pending-default checks produced no mutations. Private receipt: +`/private/tmp/paperclip-zdt-controller-allowlist-RzRa7k/receipt.json`. +The paid-agent credential blocker and all remaining acceptance criteria remain. + + +### 2026-11-01 11:15 UTC hosted enrollment response correction + +Core #18533 merged as `247552fe0719dcc92f7b42c50b658be6f624986d`; Cloud +#1099 merged as `1c5c5133c2d3740fdb716b26c31f038add273b55`. Both latest +heads passed all checks and review at 5/5 with no unresolved findings. Real +ROM-input generation also verified the complete source archive and Linux +dependency lock in 53.1 seconds from a clean checkout. + +The staging controller configuration restart succeeded with unchanged source +`316163b28`. The first real enrollment request at 11:08 UTC failed before +provider mutation with `Enrollment requires the owned Railway service domain`. +The registry correctly retains maintenance routing. The original web deployment +and attached volume were not stopped or changed. Railway's scoped domain query +returns the matching service/environment but a null `projectId`. Reproducing +that response in the existing lifecycle test fails identically. The correction +accepts the nullable project field from the project-scoped query while rejecting +explicit project, service or environment mismatches before provider mutation. +The original attempt identity is retained for retry after reviewed deployment. +Private failure receipt: `/private/tmp/paperclip-zdt-enrollment-controller-logs.json`. +No hosted overlap deployment or production canary has passed yet. + + +### 2026-11-01 11:25 UTC exact release pair qualified + +Cloud #1100 merged as `abcb017123013f2baa215fd665b4092495de00a1` after +4,831 local tests, 129 explicit skips, typecheck, golden-path smoke and all +latest-head CI checks passed, with 5/5 review and no unresolved findings. Its +automatic staging controller deployment is starting; enrollment retry waits for +that serving revision. The original held deployment and attached volume were +independently confirmed unchanged after the first failure. + +The rebuilt pair now has byte-identical Cloud bundles, SDKs and dependency lock: +`cloud-41300941768-1` (Core `1d348bbef0`) and `cloud-41301234904-1` (Core +`d941ed3879`). Both select the same canonical Runner source. The production +`ephemeralPackageOverlap` function accepts this pair. There is no database diff +between the two Core commits. Exact manifest receipt: +`/private/tmp/paperclip-zdt-canonical-pair-proof.json`. + +Each exact image passed its own packaged migration, complete predecessor-volume +deletion and two empty-home replacements against real PostgreSQL and S3: 118 +reads/24 saves and 116 reads/24 saves respectively, no failures, all owned +resources cleaned. Receipts: `/private/tmp/paperclip-zdt-container-woB7ci/evidence.json` +and `/private/tmp/paperclip-zdt-container-tMUSnD/evidence.json`. Both images were +added to the staging allowlist with deployment skipped, under the normal quiet +guards, before the reviewed controller rollout. The repeated hosted experiment +will invoke the existing fleet workflow driver with exact release references and +a single pinned fixture; it retains campaign/release/serving verification while +avoiding repeated GitHub scheduler waits. No hosted success is inferred from +these local receipts. Active-agent credentials remain blocked as previously noted. + + +### 2026-11-01 11:34 UTC maintenance entrypoint qualification + +The corrected controller is serving `abcb017123013f2baa215fd665b4092495de00a1`. +The first prepare reached the durable `preparing` phase and stopped the old +deployment, but Railway's remove call timed out. A read-only provider check +confirmed removal and retention of the attached volume, then the same attempt +resumed successfully to `migrating`. Its healthy maintenance process reported +`blocked` during configuration; no detach was attempted. + +Railway's start-command override replaces ENTRYPOINT, whereas the original local +Docker rehearsal replaced only CMD. The maintenance process therefore ran as root +and refused the original UID-1000-owned signing key. An explicit local entrypoint +override reproduced the same blocked status and captured root process identity +and key ownership, without reading key contents into evidence. Failed receipt: +`/private/tmp/paperclip-zdt-container-5nGSUs/evidence.json`. + +Cloud #1101 preserves Core's init and entrypoint wrapper in the maintenance +command. With actual entrypoint replacement, the corrected packaged rehearsal +passed original-key preservation, volume deletion and later empty-home recovery: +116 reads, 24 saves, no failures, cleanup confirmed in +`/private/tmp/paperclip-zdt-container-POFr0C/evidence.json`. Twelve lifecycle tests, +root typecheck, the full 4,831-pass suite and ordinary golden-path smoke passed. +The PR is awaiting CI/review; hosted preparation will resume with the same +attempt after deployment. The original live volume remains attached and retained. +No hosted overlap or active-agent acceptance has passed. + +### 2026-11-01 11:48 UTC reviewed entrypoint fix + +Cloud #1101 merged as `76f2dedf076ed139af3a7136e988c6748589d50b` with +all latest-head checks passing and no unresolved review findings. The exact image +currently held in staging also passed the corrected Railway-entrypoint rehearsal: +118 reads, 24 saves, zero failures, original signing authority retained, predecessor +volume deleted and all owned test resources cleaned. Private receipt: +`/private/tmp/paperclip-zdt-container-vyIrsD/evidence.json`. + +Automatic staging controller workflow `41302710585` passed preflight and is +waiting for the ordinary controller coordination gate. The concurrent app campaign +has completed 16 of 17 targets with one still running; the pinned qualification +fixture is excluded. No cancellation or coordination bypass is permitted. The +serving controller remains `abcb017123013f2baa215fd665b4092495de00a1`. +Enrollment must resume with its original attempt identity after the new controller +is serving. The active-model credential remains unavailable; no paid-agent result +or hosted overlap acceptance is inferred from the local container proofs. + + +### 2026-11-01 12:34 UTC live enrollment and runtime admission findings + +Cloud #1101 reached staging at 12:05 UTC. The owning enrollment attempt completed +migration and authenticated eligibility: 34 agent objects, 12 skill versions, +zero blockers. The original signing key was stored before the volume detached. +The original volume remains retained. Final app boot failed because Railway left +the maintenance start command unchanged when the provider sent null. + +The same attempt exposed delayed detach reads, automatic maintenance deployment +after detach, and successful deployment stops whose API responses timed out. +Cloud #1102 handles those provider transitions with bounded ownership-checked +reconciliation, an empty-string start-command reset, and app reconfiguration on +same-attempt boot retry. Root typecheck, 4,835 tests (129 gated skips), ordinary +routing/sleep smoke and 352 focused provider tests passed. Review/deployment is +pending. Tenant routing remains in maintenance; no ten-roll acceptance is claimed. + +The user supplied a working credential location; the OpenAI authentication gate +is resolved without exposing the key. Two live attempts exposed premature runtime +admission: fresh run rows still say legacy while workspace preparation runs, +before the native runtime selector commits its decision. Workspace preparation +must validate storage authority; runtime selection must validate the resolved +native warm lifecycle. Two focused regressions failed before this source fix; +all 156 workspace/runtime/ephemeral-contract tests passed after it. Full typecheck +and build passed; the full test suite and rebuilt live cell are running. + +A third live attempt mistakenly used the old compiled server and reproduced the +old guard. That retained failure is not evidence against the source fix. The +launcher now records the compiled server hash and rejects that stale guard. +All three completed attempts revoked their owned Daytona credentials and deleted +their disposable object buckets. No paid-agent acceptance has passed yet. + + +### 2026-11-01 12:43 UTC recovery fix merged and live cleanup boundary + +Cloud #1102 merged as `461c4aea28a1ffc31b1175db42158338dd9de1ae` after +4,853 passing tests (129 gated skips), typecheck, routing/sleep smoke, latest-head +required CI and 5/5 review with no unresolved threads. The optional security +scanner was inconclusive because its sandbox could not start; a rerun request +returned 404. Automatic staging deployment is workflow `41307729362`. + +The rebuilt live cell completed two native Codex/Daytona runs. Its first web +replacement then failed while deleting an immutable skill-cache directory in the +stopped predecessor home. The attempt remains failed, with partial estimated LLM +cost $0.1072965 and incomplete usage coverage. The shared harness cleanup now +makes stopped owned directories removable without following symlinks; seven +focused regressions and the E2E typecheck pass. No receipt is accepted without +whole-home deletion. Its leaked remote fixture is being reconciled through exact +company/environment labels and recorded lease IDs; the first delete was refused +while the sandbox was stopping. Original attempt evidence is retained. + +Required broad checks exposed separate failures: `pnpm test:run` stopped at a +15-second chat integration timeout (338 passed); that exact test passed in +isolation without changing its limit. Product E2E unit checks reported 5,898 +passes and 13 failures in untouched onboarding expectations, operating receipts, +Pi catalog assertions, workflow security assertions and embedded PostgreSQL +startup. These are not counted as passes; baseline attribution is pending. +Workspace typecheck/build and all 156 affected Core tests remain passing. diff --git a/packages/paperclip-runner/src/backends/codex-native-backend.ts b/packages/paperclip-runner/src/backends/codex-native-backend.ts index d5e94f5c62c..69bcbf411ee 100644 --- a/packages/paperclip-runner/src/backends/codex-native-backend.ts +++ b/packages/paperclip-runner/src/backends/codex-native-backend.ts @@ -21,6 +21,8 @@ import { } from "./runtime-context.js"; export interface CodexNativeSessionBackendOptions { + /** Admitted instructions from a validated same-run checkpoint, supplied by the recovery owner. */ + retainedSystemInstructions?: string; /** Effective provider environment, including the assigned workspace boundary. */ environment?: NodeJS.ProcessEnv; observability?: import("../drivers/agent-observability.js").ProviderObservabilityLaunch; @@ -149,7 +151,7 @@ function createTransportBackedNativeSessionBackend( ...nativeTaskConstraints(input), ]; - const baseInstructions = nativeSystemInstructions(input); + const baseInstructions = options.retainedSystemInstructions ?? nativeSystemInstructions(input); const transportFactory = options.transportFactory; return new HarnessDriverBackend( new CodexAppServerDriver({ diff --git a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts index 23e2b6fd89d..91e3a46b958 100644 --- a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts +++ b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts @@ -360,6 +360,36 @@ describe("native backend factory", () => { await session.close({ reason: "test complete" }); }); + it("recovers with admitted instructions after the old bundle is removed", async () => { + const input = preparedExecution(execution()); + if (!("runtimeContext" in input)) throw new Error("fixture requires runtime context"); + const retainedSystemInstructions = composeNativeSystemInstructions(input.runtimeContext, "Keep assigned instructions."); + const original = await createNativeSessionBackend(input, { + codexTransportFactory: () => new FakeCodexTransport(), + environment: { ...process.env, PAPERCLIP_WORKSPACE_CWD: WORKSPACE }, + }).openSession({ + identity: { runId: "run", sessionId: "session", companyId: "company", agentId: "agent", issueId: "issue" }, + workingDirectory: WORKSPACE, + }); + const snapshot = await original.snapshot(); + await original.close({ reason: "controller replacement" }); + rmSync(input.runtimeContext.instructions.bundle.rootPath, { recursive: true }); + const transport = new FakeCodexTransport(); + transport.readResponse = { thread: { id: transport.threadId, sessionId: transport.providerSessionId, cwd: WORKSPACE, turns: [] } }; + const factory = vi.fn(() => transport); + expect(() => createNativeSessionBackend(input, { codexTransportFactory: factory })).toThrow(); + expect(factory).not.toHaveBeenCalled(); + const backend = createNativeSessionBackend(input, { + retainedSystemInstructions, codexTransportFactory: factory, + workingDirectoryAuthority: "remote_runner", environment: { PAPERCLIP_WORKSPACE_CWD: WORKSPACE }, + }); + const recovered = await backend.recoverSession!({ ...snapshot, activeTurnId: null }, { signal: new AbortController().signal }); + expect(recovered).toMatchObject({ recovered: true }); + expect(factory).toHaveBeenCalledWith(expect.objectContaining({ baseInstructions: retainedSystemInstructions })); + expect(transport.calls.some(call => call.method === "thread/start" || call.method === "turn/start")).toBe(false); + if (recovered.recovered) await recovered.session.close({ reason: "test complete" }); + }); + describe("runner-prepared launch working directory", () => { // The transport reports the prepared directory; it exists only on the VM. class PreparedLaunchTransport extends FakeCodexTransport { diff --git a/packages/paperclip-runner/src/backends/native-backend-factory.ts b/packages/paperclip-runner/src/backends/native-backend-factory.ts index f50bd0c9a4f..f603fb0a8c8 100644 --- a/packages/paperclip-runner/src/backends/native-backend-factory.ts +++ b/packages/paperclip-runner/src/backends/native-backend-factory.ts @@ -68,6 +68,7 @@ export function createNativeSessionBackend( } if (options.codexTransportFactory) { return createRunnerdNativeSessionBackend(input, { + retainedSystemInstructions: options.retainedSystemInstructions, completionFeedback: options.completionFeedback, runnerInstanceId: options.runnerInstanceId, onSpawn: options.onSpawn, diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index c4f29bdbb6a..b55f34797cf 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -33,6 +33,7 @@ import { decodeDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, inspectStagedRunAttachment, + inspectRetainedRunInstructions, inspectStagedRunAttachmentHeader, type DurablePrpCheckpoint, durableRecoveryInternals, @@ -55,6 +56,33 @@ const identity: DurableRecoveryIdentity = { const expectedRunnerVersion = "0.3.0"; const expectedRunnerDigest = `sha256:${"a".repeat(64)}`; +it("recovers only the matching run's admitted instructions from a valid checkpoint", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-retained-instructions-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + const digest = "b".repeat(64); + const expected = { runId: identity.runId, normalizedSessionId: identity.normalizedSessionId, runtimeContextDigest: digest }; + const template = { provider: { instructions: "Exact admitted instructions.\n", runtimeContext: { aggregateDigest: digest } } }; + try { + core.persistRunAttachTemplate(template); + expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions); + for (const field of ["runId", "normalizedSessionId", "runtimeContextDigest"] as const) { + expect(inspectRetainedRunInstructions(core.store.state, { ...expected, [field]: "foreign" })).toBeNull(); + } + expect(inspectRetainedRunInstructions({ ...core.store.state, commands: null }, expected)).toBeNull(); + expect(inspectRetainedRunInstructions({ ...core.store.state, runAttachTemplate: { provider: { ...template.provider, instructions: "" } } }, expected)).toBeNull(); + const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" }; + const targetTemplate = { provider: { instructions: "Target instructions", runtimeContext: { aggregateDigest: "c".repeat(64) } } }; + core.stageRunAttachment(target, targetTemplate); + const targetExpected = { ...expected, runId: target.runId, runtimeContextDigest: "c".repeat(64) }; + expect(inspectRetainedRunInstructions(core.store.state, targetExpected)).toBe("Target instructions"); + expect(inspectRetainedRunInstructions(core.store.state, { ...targetExpected, runtimeContextDigest: digest })).toBeNull(); + expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions); + const corrupted = structuredClone(core.store.state); + corrupted.stagedRunAttachment!.templateDigest = `sha256:${"f".repeat(64)}`; + expect(inspectRetainedRunInstructions(corrupted, targetExpected)).toBeNull(); + } finally { await core.stop(); rmSync(root, { recursive: true, force: true }); } +}); + it("persists an undispatched source-to-target attachment before bootstrap and preserves it after cache loss", async () => { const root = mkdtempSync(resolve(tmpdir(), "runner-staged-attachment-")); const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" }; diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index ea82cd674e0..fad19bfada3 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -923,6 +923,26 @@ export function inspectStagedRunAttachment(state: unknown, expectedTarget?: Dura } catch { return null; } } +/** Recover admitted instructions without opening the retired controller's assets. + * Inspection is evidence only; the caller still authorizes the runner and lease. */ +export function inspectRetainedRunInstructions(state: unknown, expected: { + runId: string; normalizedSessionId: string; runtimeContextDigest: string; +}): string | null { + try { + if (!isRecord(state) || !exactIdentity(state.identity) || !isStoredCoreState(state, state.identity)) return null; + const matches = (identity: DurableRecoveryIdentity) => identity.runId === expected.runId + && identity.normalizedSessionId === expected.normalizedSessionId; + const staged = state.stagedRunAttachment; + const template = staged && matches(staged.targetIdentity) + ? stagedAttachmentTemplate(state) + : matches(state.identity) ? state.runAttachTemplate : null; + if (!isRecord(template) || !isRecord(template.provider)) return null; + const { instructions, runtimeContext } = template.provider; + return isRecord(runtimeContext) && runtimeContext.aggregateDigest === expected.runtimeContextDigest + && typeof instructions === "string" && instructions.trim().length > 0 ? instructions : null; + } catch { return null; } +} + function isStoredCoreState( value: unknown, identity: DurableRecoveryIdentity, diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 9f628c39f0a..72e5aa5eb1e 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -39,6 +39,7 @@ export { RUNNER_PRP_PROTOCOL_MAX, inspectWarmRunTransition, inspectStagedRunAttachment, + inspectRetainedRunInstructions, inspectStagedRunAttachmentHeader, type StagedRunAttachment, type DurablePrpControlPlaneOptions, diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index b1749f4653a..8219cc6f600 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -20,7 +20,7 @@ import { } from "node:fs/promises"; import { createHash, randomUUID } from "node:crypto"; import { execFileSync, spawn } from "node:child_process"; -import { readFileSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import { bindAcpxAgentFiles } from "../drivers/acpx/agent-files-binding.js"; import { resolveRunnerProviderAssetsRoot } from "../drivers/acpx/provider-assets-root.js"; import { createServer } from "node:http"; @@ -7646,9 +7646,14 @@ it("rotates PRP authority in place for a warm cross-run attachment", async () => } }, 30_000); -it("recovers an active second warm business run with its opaque attachment identity", async () => { +it("recovers an active second warm business run after deleting its controller context assets", async () => { const root = await mkdtemp(join(tmpdir(), "runnerd-warm-second-active-")); const firstRoot = join(root, "first-controller"), nextRoot = join(root, "next-controller"); + const assetRoot = join(root, "retired-controller-assets"); + const skillRoot = join(assetRoot, "skill"), instructionRoot = join(assetRoot, "instructions"); + await mkdir(skillRoot, { recursive: true }); await mkdir(instructionRoot, { recursive: true }); + await writeFile(join(skillRoot, "SKILL.md"), "# Pinned skill\n"); + await writeFile(join(instructionRoot, "AGENTS.md"), "Pinned instructions\n"); const server = createServer(); let authority: DurablePrpControlPlane | null = null, generation = 0; let checkpoint: durableControlPlane.DurablePrpCheckpoint | undefined; @@ -7670,6 +7675,7 @@ it("recovers an active second warm business run with its opaque attachment ident runId: "run-warm-first-healthy", normalizedSessionId: "session-warm-second-active", turnId: "turn-first", itemId: "item-first" }; const options = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, codexArgs: fakeCodexArgs(root, "--emit-tool-call", "--durable-tool-ids"), stateDirectory: firstRoot, + runtimeContext: assignedRuntimeContext(skillRoot, instructionRoot), prpIdentity: initialIdentity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 }, controlPlaneRegistration: registration }; const first = createCapabilityRunnerdCodexTransport(options); const within = async (label: string, promise: Promise) => { @@ -7707,6 +7713,8 @@ it("recovers an active second warm business run with its opaque attachment ident await activeCore.flushCheckpoint(); await within("controller detach", first.detachControllerForRestart()); expect(() => process.kill(runnerPid!, 0)).not.toThrow(); + await rm(assetRoot, { recursive: true, force: true }); + expect(existsSync(assetRoot)).toBe(false); checkpoint!.restoreSessionDirectory(nextRoot); const noSpawn = vi.fn(() => { throw new Error("Duplicate Runner launch"); }); const openedThread = opened.thread as Record; @@ -7758,6 +7766,7 @@ it("recovers an active second warm business run with its opaque attachment ident if (runnerPid) { try { process.kill(-runnerPid, "SIGKILL"); } catch {} } server.closeAllConnections(); if (server.listening) await new Promise(done => server.close(() => done())); + for (const controller of [firstRoot, nextRoot]) await releaseMaterializedNativeRuntimeSkills(join(controller, "codex-home", "skills")); await rm(root, { recursive: true, force: true }); } }, 30_000); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 811133f5b0a..fbf1a9f6c6f 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -6046,16 +6046,16 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#assertStartupOpen(); // Route and immutable endpoint admission precedes every launch-material // write. A refused transition leaves its retained home/context untouched. - if (runtimeContext !== null && !this.options.prepareLaunch && this.#sourceRecoveryIdentity === null) { + if (runtimeContext !== null && preparesProviderLaunch && this.#sourceRecoveryIdentity === null) { writeFileSync( localRuntimeContextPath, `${JSON.stringify(runtimeContext)}\n`, { mode: 0o600 }, ); } - if (provider === "aws_agentcore" && !this.options.prepareLaunch) + if (provider === "aws_agentcore" && preparesProviderLaunch) mkdirSync(codexHome, { recursive: true, mode: 0o700 }); - if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured" && this.#sourceRecoveryIdentity === null) { + if (preparesProviderLaunch && provider === "codex" && this.options.homeMode !== "preconfigured" && this.#sourceRecoveryIdentity === null) { await releaseMaterializedNativeRuntimeSkills( resolve(localCodexHome, "skills"), ); diff --git a/server/src/__tests__/agent-directory-working-copies.test.ts b/server/src/__tests__/agent-directory-working-copies.test.ts index 183769a38d6..fcf3c5ebd6a 100644 --- a/server/src/__tests__/agent-directory-working-copies.test.ts +++ b/server/src/__tests__/agent-directory-working-copies.test.ts @@ -14,8 +14,8 @@ import os from "node:os"; import path from "node:path"; import { createHash, randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { eq } from "drizzle-orm"; -import { agentFileStore, fileHash, inspectAgentFile, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js"; +import { eq, sql } from "drizzle-orm"; +import { agentFileStore, fileHash, inspectAgentFile, inspectAgentDirectory, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js"; import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js"; @@ -902,6 +902,87 @@ describe("persistent agent directories", () => { expect((await fs.stat(active.localRoot)).isDirectory()).toBe(true); }); + it.each(["new-home", "same-path-empty", "foreign-lease", "other-controller", "missing-collector", "over-quota"] as const)("recovers remote agent files after controller cache loss: %s", async scenario => { + const copy = await run({ warm: true }); + const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/retained-task"; + const executionRoot = path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId); + await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" }); + await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: copy.runId, + provider: "daytona", providerLeaseId: "original-sandbox", status: "active", metadata: { remoteCwd } }); + await db.update(agentInstructionWorkingCopies).set({ location: `remote:${environmentId}`, executionRoot, + receipt: { ...copy.receipt, cleanup: { leaseId, remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + const saved = (await copies.get(companyId, copy.runId))!; + const replacementHome = path.join(home, `replacement-${randomUUID()}`); + const remoteFiles = await fs.mkdtemp(path.join(home, "remote-checkpoint-")); + await fs.mkdir(path.join(remoteFiles, "files", "policy"), { recursive: true }); + await fs.writeFile(path.join(remoteFiles, "files", entryFile), "Remote instructions changed during the turn"); + await fs.writeFile(path.join(remoteFiles, "files", "unsaved.txt"), "Only the running sandbox has this"); + await fs.rm(path.dirname(copy.localRoot), { recursive: true, force: true }); + if (scenario === "over-quota") { + const handle = await fs.open(path.join(remoteFiles, "files", "too-large.bin"), "w"); + try { await handle.truncate(MAX_AGENT_FILE_BYTES + 1); } finally { await handle.close(); } + } + const transport = vi.spyOn(executionTargetTools, "prepareAdapterExecutionTargetRuntime").mockImplementation(async options => ({ + target: options.target!, workspaceRemoteDir: options.workspaceRemoteDir!, runtimeRootDir: options.runtimeRootDir!, + assetDirs: {}, additionalSourceDirs: {}, additionalSourceFailures: [], workspaceSyncSnapshot: null, + restoreWorkspace: async () => { await fs.cp(path.join(remoteFiles, "files"), options.workspaceLocalDir, { recursive: true }); }, + })); + const ephemeralBefore = process.env.PAPERCLIP_EPHEMERAL_SERVER; + process.env.PAPERCLIP_EPHEMERAL_SERVER = "true"; + if (scenario === "new-home") process.env.PAPERCLIP_HOME = replacementHome; + const remote = { kind: "remote" as const, transport: "sandbox" as const, environmentId, + leaseId: scenario === "foreign-lease" ? randomUUID() : leaseId, remoteCwd, runner: { execute: vi.fn() } }; + try { + if (scenario === "missing-collector") { + const retained = await copies.collectStopped({ companyId, runId: copy.runId }); + expect(retained).toMatchObject({ state: "pending_collection", errorCode: "INSTRUCTION_REMOTE_COLLECTOR_UNAVAILABLE", baseHash: saved.baseHash }); + expect(retained?.receipt?.baseline).toEqual(saved.receipt?.baseline); + expect(retained?.nextAttemptAt).toBeInstanceOf(Date); + expect(transport).not.toHaveBeenCalled(); + return; + } + if (scenario === "other-controller") { + await db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`agent-directory:${companyId}:${agentId}`}, 0))`); + await expect(copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true })) + .rejects.toThrow("Agent directory operation is already active"); + expect(transport).not.toHaveBeenCalled(); + }); + return; + } + const preparing = copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true }); + if (scenario === "foreign-lease") { + await expect(preparing).rejects.toThrow("Agent directory recovery lease changed"); + expect(transport).not.toHaveBeenCalled(); + return; + } + const recovered = (await preparing)!; + expect(recovered.executionRoot).toBe(executionRoot); + expect(recovered.baseHash).toBe(saved.baseHash); + expect(recovered.receipt).toEqual(saved.receipt); + if (scenario === "new-home") { + expect(recovered.localRoot.startsWith(replacementHome)).toBe(true); + await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" }); + } else expect(recovered.localRoot).toBe(copy.localRoot); + expect(await fs.readFile(path.join(recovered.localRoot, "unsaved.txt"), "utf8")).toBe("Only the running sandbox has this"); + expect(await fs.readFile(path.join(recovered.localRoot, entryFile), "utf8")).toBe("Remote instructions changed during the turn"); + expect(transport).toHaveBeenNthCalledWith(1, expect.objectContaining({ adapterKey: "agent-files-recovery", workspaceRemoteDir: executionRoot, + workspaceBaseline: { exclude: [".paperclip-runtime"], entries: new Map() }, workspaceInboundMode: "adopt_remote" })); + if (scenario === "over-quota") { + expect((await fs.stat(path.join(recovered.localRoot, "too-large.bin"))).size).toBe(MAX_AGENT_FILE_BYTES + 1); + await expect(inspectAgentDirectory(recovered.localRoot)).rejects.toThrow(); + } + expect(transport).toHaveBeenCalledWith(expect.objectContaining({ workspaceLocalDir: recovered.localRoot, workspaceInboundMode: "adopt_remote" })); + await copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true }); + expect(transport).toHaveBeenCalledTimes(2); + } finally { + process.env.PAPERCLIP_HOME = home; + if (ephemeralBefore === undefined) delete process.env.PAPERCLIP_EPHEMERAL_SERVER; + else process.env.PAPERCLIP_EPHEMERAL_SERVER = ephemeralBefore; + transport.mockRestore(); + } + }); + it("retains remote cleanup across restart and failed retries until the original lease is cleaned", async () => { const copy = await run(); const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task"; diff --git a/server/src/__tests__/attention-retention-work.test.ts b/server/src/__tests__/attention-retention-work.test.ts index b722c75f395..63baf94217e 100644 --- a/server/src/__tests__/attention-retention-work.test.ts +++ b/server/src/__tests__/attention-retention-work.test.ts @@ -161,7 +161,10 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(await db.select().from(issueRelations).where(and(eq(issueRelations.issueId, child.id), eq(issueRelations.relatedIssueId, parent.id)))).toHaveLength(1); await settle(); expect(coordinator!.nextWakeAt()).not.toBeNull(); - await vi.advanceTimersByTimeAsync(delay + 2000); await archived("review", child.id); await quiet(); + // SQL assigns updatedAt using the database clock, independently of the fake + // scheduler clock. Cross the saved deadline rather than a fixed duration. + await vi.advanceTimersByTimeAsync(Math.max(0, child.updatedAt.getTime() + delay - Date.now()) + 2000); + await archived("review", child.id); await quiet(); expect(await db.select().from(decisionArchiveNotificationOutbox).where(eq(decisionArchiveNotificationOutbox.sourceId, child.id))).toHaveLength(1); }); diff --git a/server/src/__tests__/ephemeral-server-contract.test.ts b/server/src/__tests__/ephemeral-server-contract.test.ts index 72c319fac84..bfa5e4acae8 100644 --- a/server/src/__tests__/ephemeral-server-contract.test.ts +++ b/server/src/__tests__/ephemeral-server-contract.test.ts @@ -26,14 +26,15 @@ it("never connects the listener with migration privileges, another tenant, or th }); it("rejects disk-dependent execution and instructions while retaining remote warm execution", () => { vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); - const remote = { externalConversation: false, native: true, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true }; + const remote = { externalConversation: false, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true }; expect(() => assertEphemeralWorkspaceExecution(remote)).not.toThrow(); expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false, pluginTaskWorkspace: true })).not.toThrow(); - for (const patch of [{ native: false }, { remoteAuthority: false }, { adapterType: "process" }]) { + for (const patch of [{ remoteAuthority: false }, { adapterType: "process" }]) { expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("warm Paperclip Runner"); } expect(() => assertEphemeralNativeLifecycle("warm")).not.toThrow(); expect(() => assertEphemeralNativeLifecycle("per_turn")).toThrow("warm Paperclip Runner"); + expect(() => assertEphemeralNativeLifecycle(undefined)).toThrow("warm Paperclip Runner"); const agent = { id: "agent", companyId: "company", name: "Agent", adapterConfig: {} }; expect(() => assertEphemeralAgentInstructions(agent)).not.toThrow(); expect(() => assertEphemeralAgentInstructions({ ...agent, adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/old-volume" } })).toThrow("managed agent instructions"); diff --git a/server/src/services/agent-directory-working-copies.ts b/server/src/services/agent-directory-working-copies.ts index 435d295a442..c1fe658aab6 100644 --- a/server/src/services/agent-directory-working-copies.ts +++ b/server/src/services/agent-directory-working-copies.ts @@ -3,7 +3,7 @@ import fs from "node:fs/promises"; import path from "node:path"; import { isDeepStrictEqual } from "node:util"; import { and, eq, inArray, isNull, sql } from "drizzle-orm"; -import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db"; +import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { syncDirectoryToSsh, restoreWorkspaceFromSshExecution } from "@paperclipai/adapter-utils/ssh"; import { prepareAdapterExecutionTargetRuntime, runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget, type PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target"; import { withDirectoryMergeLock, directorySnapshotSha256, parseDirectorySnapshot, serializeDirectorySnapshot, type DirectoryMergeLockOperation } from "@paperclipai/adapter-utils/workspace-restore-merge"; @@ -18,6 +18,7 @@ import type { Environment, EnvironmentLease } from "@paperclipai/shared"; import { agentDirectoryBaselineDigest, agentDirectoryProbeProgram, observeLocalAgentDirectory } from "./agent-directory-probe.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import { cachedAgentFileManifest, captureAgentFileCheckpoint, checkpointBaseline, checkpointSnapshot, type AgentFileManifest } from "./agent-file-checkpoints.js"; +import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; import { logger } from "../middleware/logger.js"; type Copy = typeof copies.$inferSelect; @@ -54,7 +55,16 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string }); } const ownerFile = (row: Copy) => path.join(path.dirname(row.localRoot), "owner.json"); + const disposableRemoteCopy = (row: Copy) => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && row.location.startsWith("remote:"); + function projectionRoot(row: Copy): string { + const origin = String(row.receipt?.materializationRunId ?? row.receipt?.directoryRunId ?? row.runId); + if (!/^[a-zA-Z0-9_-]{1,128}$/.test(origin)) throw conflict("Invalid agent directory owner"); + return path.join(path.dirname(resolveManagedInstructionsRoot({ id: row.agentId, companyId: row.companyId, name: "", adapterConfig: {} })), "file-sync", "runs", origin, "live"); + } async function owns(row: Copy) { + // Remote ownership is durable. A marker on a discarded controller cannot + // establish ownership of the retained sandbox on its replacement. + if (disposableRemoteCopy(row)) return row.state !== "superseded" && !row.receipt?.retainedByRunId; if (row.receipt?.warm !== true) return true; return await fs.readFile(ownerFile(row), "utf8").then(s => JSON.parse(s).runId === row.runId).catch(error => error.code === "ENOENT" && row.processStoppedAt !== null); } @@ -159,7 +169,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string ? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "agent-files", input.agentId, origin) : localRoot; const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local"; - if (row && (row.localRoot !== localRoot || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment"); + if (row && ((!disposableRemoteCopy(row) && row.localRoot !== localRoot) || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment"); if (row && !serialHeld) { // Restart preparation can replace a completed copy under the same run ID. // Join cleanup's lock before resetting its receipt or creating new bytes. @@ -168,7 +178,10 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string return serial(row, () => prepareCopy(input, true), "agent_directory_prepare"); } if (row && !completed.has(row.state) && row.state !== "preparing") { - if (input.target?.kind === "remote" && !transports.has(key(row))) transports.set(key(row), await transport(row, input.target, true)); + if (input.target?.kind === "remote") { + if (disposableRemoteCopy(row)) await restoreProjection(row, input.target); + if (!transports.has(key(row))) transports.set(key(row), await transport(row, input.target, true)); + } return row; } // Register ownership before creating any bytes, so a crash during the copy @@ -241,6 +254,39 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string } } + async function restoreProjection(row: Copy, target: AdapterExecutionTarget) { + const exists = await fs.lstat(row.localRoot).then(stat => { + if (!stat.isDirectory() || stat.isSymbolicLink()) throw conflict("Agent directory projection is not a directory"); + return true; + }, error => { + if (error.code === "ENOENT") return false; + throw error; + }); + if (exists) return; + const lease = await ownedRemoteLease(row); + const cleanup = row.receipt?.cleanup as { leaseId?: string; remoteCwd?: string } | undefined; + if (target.kind !== "remote" || !lease || target.leaseId !== lease.id || lease.status !== "active" + || lease.releasedAt || hasRemoteTerminationReceipt(lease) || cleanup?.remoteCwd !== target.remoteCwd + || row.location !== `remote:${target.environmentId}`) throw conflict("Agent directory recovery lease changed"); + // Reconstruct a private mirror without applying save quotas to admission. + // An empty merge baseline downloads all remote bytes without uploading any; + // the original save baseline remains unchanged in Postgres. + await fs.mkdir(row.localRoot, { recursive: true, mode: 0o700 }); + let projection: PreparedAdapterExecutionTargetRuntime | undefined; + try { + projection = await prepareAdapterExecutionTargetRuntime({ target, runId: row.runId, adapterKey: "agent-files-recovery", + workspaceLocalDir: row.localRoot, workspaceRemoteDir: row.executionRoot, + runtimeRootDir: sandboxTransferRoot(row, target.remoteCwd), syncWorkspace: true, + workspaceInboundMode: "adopt_remote", workspaceBaseline: { exclude: [".paperclip-runtime"], entries: new Map() }, + workspaceGitSnapshot: null, workspaceRepositories: [], workspaceFileMode: "all", workspaceExclude: [".paperclip-runtime", ".paperclip-runtime/**"] }); + await projection.restoreWorkspace(); + await inspectAgentDirectory(row.localRoot, false); + if (row.receipt?.warm === true) await fs.writeFile(ownerFile(row), JSON.stringify({ runId: row.runId })); + } catch (error) { + await fs.rm(row.localRoot, { recursive: true, force: true }); + throw error; + } finally { await projection?.cleanupWorkspaceSnapshot?.(); } + } async function retrieve(row: Copy, target?: AdapterExecutionTarget | null) { if (row.location !== "local") { let runtime = transports.get(key(row)); @@ -356,6 +402,14 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string } async function checkpoint(row: Copy, target?: AdapterExecutionTarget | null, stopped = false): Promise { if (!await owns(row) || row.state === "superseded") return row; + target ??= transports.get(key(row))?.target; + if (row.location !== "local" && target?.kind !== "remote") { + // A missing controller cache is not the remote directory. Retain the + // pending save until its exact remote collector or destruction proof exists. + return patch(row, { ...(stopped ? { state: "pending_collection", processStoppedAt: row.processStoppedAt ?? new Date() } : {}), + errorCode: "INSTRUCTION_REMOTE_COLLECTOR_UNAVAILABLE", errorMessage: "The remote agent directory is retained; its collector is unavailable. No save or discard is claimed.", + nextAttemptAt: stopped ? new Date(Date.now() + 30_000) : null }); + } let failure: unknown; for (let attempt = 0; attempt < 3; attempt++) { let captured: Awaited> | undefined; @@ -575,13 +629,29 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string return result.exitCode === 0 && !result.timedOut; } async function serial(row: Copy, fn: (current: Copy) => Promise, operation: DirectoryMergeLockOperation): Promise { - // Duplicate stop callbacks and restart recovery must not race while moving - // the stopped working copy. This lock is outside the writable tree. - return withDirectoryMergeLock(path.resolve(row.localRoot, "../../.."), async () => { - const current = await get(row.companyId, row.runId); - if (!current) throw notFound("Agent directory copy not found"); - return fn(current); - }, process.env, operation); + const run = async () => { + const lockRoot = path.resolve(disposableRemoteCopy(row) ? projectionRoot(row) : row.localRoot, "../../.."); + if (disposableRemoteCopy(row)) await fs.mkdir(lockRoot, { recursive: true, mode: 0o700 }); + return withDirectoryMergeLock(lockRoot, async () => { + let current = await get(row.companyId, row.runId); + if (!current) throw notFound("Agent directory copy not found"); + if (disposableRemoteCopy(current) && current.localRoot !== projectionRoot(current)) { + current = await patch(current, { localRoot: projectionRoot(current) }); + if (current.localRoot !== projectionRoot(current)) throw conflict("Agent directory projection changed"); + transports.delete(key(current)); + } + return fn(current); + }, process.env, operation); + }; + if (!disposableRemoteCopy(row)) return run(); + // Different controllers have different cache roots. Serialize their active + // collection/cleanup operations in Postgres, without holding row locks over I/O. + return withDedicatedDbConnection(db, dedicated => dedicated.transaction(async tx => { + const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`agent-directory:${row.companyId}:${row.agentId}`}, 0)) as acquired, + (select setting::bigint from pg_settings where name = 'idle_in_transaction_session_timeout') as idle_timeout_ms`); + if (lock?.acquired !== true) throw conflict("Agent directory operation is already active"); + return withWorkspaceLockKeepalive(tx as unknown as Db, Number(lock.idle_timeout_ms), run); + })); } return { prepare: (input: Parameters[0]) => prepareCopy(input), hasChanges, adopt, canReuse, recoverUnavailable, recoverStoppedRemote: (row: Copy) => serial(row, current => current.receipt?.retainedByRunId ? Promise.resolve(current) : recoverStoppedRemote(current), "agent_directory_collect"), diff --git a/server/src/services/agent-instruction-working-copies.ts b/server/src/services/agent-instruction-working-copies.ts index 3e7a6363ecf..0d990331d4a 100644 --- a/server/src/services/agent-instruction-working-copies.ts +++ b/server/src/services/agent-instruction-working-copies.ts @@ -121,7 +121,7 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen if (signalsCleanup) await notifyDirectoryCleanup(tx, { companyId: row.companyId, runId: row.runId }); if (signalsCollection) await notifyInstructionCollection(tx, { companyId: row.companyId, runId: row.runId }); const [updated] = await tx.update(copies).set({ ...values, updatedAt: new Date() }).where(and( - scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash), + scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash), eq(copies.localRoot, row.localRoot), sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`, requireDue ? or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())) : undefined, condition, )).returning(); diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index 7b777887f85..f5d4603d050 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -328,6 +328,17 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = expect(await coordinators(run.id)).toHaveLength(0); }); + it("rejects resolved legacy execution on an ephemeral server before runtime admission", async () => { + const { input, run } = await fixture("process"); + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + expect(input.nativeRuntimeResolution.kind).toBe("legacy"); + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner"); + expect((await read(run.id)).runtimeMode).toBe("legacy"); + expect(await coordinators(run.id)).toHaveLength(0); + expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled(); + expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled(); + }); + it.each(["warm", "per_turn"] as const)("checks resolved %s policy for ephemeral admission", async mode => { const { input, run } = await fixture(); vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 4f10a467302..49d96cf762b 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -267,6 +267,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config; const runOptions = input.recovery; const { providerTraceRequested, providerTraceCapture } = input.trace; + if (nativeRuntimeResolution.kind !== "native" && !isDotRun) assertEphemeralNativeLifecycle(undefined); let nativeExecution: NativeExecutionInput | null = null; let getNativeFreshSessionHandoff: (() => Promise) | undefined; let nativeRunnerInstanceId: string | null = null; diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index 5aa66c1d9cd..beb43a34b0f 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -52,7 +52,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace let database: Awaited>; let db: Db; let home: string; - const persistenceKeys = ["DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"]; + const persistenceKeys = ["PAPERCLIP_EPHEMERAL_SERVER", "DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"]; let persistenceBefore = new Map(); const configureRemotePersistence = () => Object.assign(process.env, { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" }); @@ -170,14 +170,15 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace }); it.each([ - { kind: "empty", repoUrl: null, branchMode: null, fresh: false }, + { kind: "empty", repoUrl: null, branchMode: null, fresh: false, ephemeral: true }, { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false }, { kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false }, { kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false }, { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false }, { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false }, { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true }, - ] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async ({ kind, repoUrl, branchMode, fresh }) => { + ] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async (scenario) => { + const { kind, repoUrl, branchMode, fresh } = scenario; const { input, source, issue, envOrchestrator } = await fixture(); const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`, driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning(); @@ -227,10 +228,13 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace throw new Error("reached remote acquisition"); }); const configured = { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "s3", - PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" }; + PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture", + PAPERCLIP_EPHEMERAL_SERVER: "ephemeral" in scenario ? "true" : "false" }; const previous = new Map(Object.keys(configured).map(key => [key, process.env[key]])); Object.assign(process.env, configured); try { + // Fresh runs are still marked legacy until the runtime selector owns promotion. + expect(input.run.runtimeMode).toBe("legacy"); await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition"); expect(resolve).not.toHaveBeenCalled(); expect(repositoryPreparation).not.toHaveBeenCalled(); @@ -268,7 +272,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace expect(await db.select().from(executionWorkspaces)).toHaveLength(0); }); - it("auto-enrollment leaves a configured local folder on the host authority", async () => { + it.each([false, true])("configured local folders retain host authority only on durable servers (ephemeral=%s)", async ephemeral => { const { input, envOrchestrator } = await fixture(); input.agent.adapterType = "paperclip_runner"; input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment, @@ -277,8 +281,11 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace const acquire = vi.spyOn(envOrchestrator, "acquireForRun"); configureRemotePersistence(); const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local folder retained")); - await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("local folder retained"); - expect(resolve).toHaveBeenCalledOnce(); + process.env.PAPERCLIP_EPHEMERAL_SERVER = String(ephemeral); + await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(ephemeral + ? "Volume-free servers require warm Paperclip Runner execution with remote workspace storage." : "local folder retained"); + if (ephemeral) expect(resolve).not.toHaveBeenCalled(); + else expect(resolve).toHaveBeenCalledOnce(); expect(await db.select().from(executionWorkspaces)).toHaveLength(0); expect(acquire).not.toHaveBeenCalled(); }); diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index 15c91735f71..e4c76adb48b 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -241,7 +241,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace && !projectExecutionWorkspacePolicy?.storageProviderPluginId, existingWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace, }); - assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, native: run.runtimeMode === "native", + assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, adapterType: agent.adapterType, pluginTaskWorkspace, remoteAuthority }); const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({ @@ -322,7 +322,13 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace issue: issueRef, agent, projectId: issueRef!.projectId, repoRef: null, })) : null), }); - if (sourceState.authority === "host") remoteAuthority = false; + if (sourceState.authority === "host") { + remoteAuthority = false; + // Source resolution can retain a configured host folder. Check the + // resolved authority before materializing it or acquiring a sandbox. + assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, + adapterType: agent.adapterType, pluginTaskWorkspace, remoteAuthority }); + } else { remoteSourceState.value = sourceState; if (!enrolledRemoteAuthority) await assertRemotePreparationOwned(); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index b2799be84cb..452a290b23d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -145,6 +145,7 @@ import { executeNativeSession, applyNativeSessionGoalControl, inspectStagedRunAttachment, + inspectRetainedRunInstructions, inspectWarmRunTransition, parseNativeExecutionInput, parsePaperclipQuestionSet, @@ -12545,6 +12546,19 @@ async function createRunnerdBackendWithinSessionClaim( : null; const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner" ? await verifyStagedAttachmentRestart(input) : null; + const retainedSystemInstructions = input.restartRecovery?.kind === "reattach_remote_runner" + && "runtimeContext" in input.execution + && input.execution.schema !== "paperclip.native-execution-input.v6" + && input.execution.schema !== "paperclip.native-execution-input.v8" + ? inspectRetainedRunInstructions(JSON.parse(readBoundedNativeFile( + resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, + "native_runner_recovery_instructions_unavailable").toString("utf8")), { + runId: input.execution.binding.runId, + normalizedSessionId: nativeSessionKey(input.execution), + runtimeContextDigest: input.execution.runtimeContext.aggregateDigest, + }) + : undefined; + if (retainedSystemInstructions === null) throw new Error("native_runner_recovery_instructions_unavailable"); let stagedRecoveryPending = stagedRecovery !== null; const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity; const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity @@ -12647,6 +12661,7 @@ async function createRunnerdBackendWithinSessionClaim( return current.execute(call); }; const backend = createNativeSessionBackend(input.execution, { + retainedSystemInstructions, // The plugin resolves the VM directory while preparing the launch. Core keeps // its host environment out of the plugin and records the prepared directory. runnerInstanceId: prior?.runnerInstanceId ?? effectiveRunnerInstanceId, environment: {}, @@ -13981,6 +13996,7 @@ async function createRunnerdBackendWithinSessionClaim( runId: input.execution.binding.runId, prepareRemote: prepareRemoteRunner }); } const backend = createNativeSessionBackend(runnerExecution, { + retainedSystemInstructions, runnerInstanceId: retainedTransition?.runnerInstanceId ?? input.runnerInstanceId, environment: effectiveRunnerEnvironment, diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 61776878ae0..37c41fd330e 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -76,11 +76,13 @@ export function shouldUseRemoteWorkspaces(input: { * remote controller may replace its local projection; host-owned execution * cannot make that promise. External conversation providers own no task disk. */ export function assertEphemeralWorkspaceExecution(input: { - externalConversation: boolean; native: boolean; adapterType: string; + externalConversation: boolean; adapterType: string; pluginTaskWorkspace: boolean; remoteAuthority: boolean; }) { if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" || input.externalConversation) return; - if (!input.native || input.adapterType !== "paperclip_runner" + // Runtime selection has not promoted a fresh run yet. Validate storage here; + // the runtime selector separately admits only its resolved warm native policy. + if (input.adapterType !== "paperclip_runner" || (!input.pluginTaskWorkspace && !input.remoteAuthority)) { throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.", { code: "ephemeral_execution_configuration_required" }); @@ -89,7 +91,7 @@ export function assertEphemeralWorkspaceExecution(input: { /** The runtime selector calls this after environment, credentials and saved * execution policy have resolved. Raw agent settings are not lifecycle authority. */ -export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn") { +export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn" | undefined) { if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && mode !== "warm") { throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.", { code: "ephemeral_execution_configuration_required" }); diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 63a918160ea..c2122cb1c6e 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -95,6 +95,7 @@ export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX; export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit; export const inspectWarmRunTransition = runner.inspectWarmRunTransition; export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment; +export const inspectRetainedRunInstructions = runner.inspectRetainedRunInstructions; export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader; export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment; export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding; diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 8d6610f5358..77d8deacc21 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -100,9 +100,12 @@ The separate explicit-only `rolling-web-deployment` suite uses the same syntheti projectless workflow, four-admission budget, thirty-minute limit, and independent run/file oracles. Its supervisor starts an independent empty-home server, waits for `/api/ready`, moves ingress, sends SIGTERM to the predecessor, and deletes -that predecessor's whole home and build. Continuous ordinary task reads and +that predecessor's whole home and build. Cleanup makes read-only cache directories +removable only after shutdown, without following dependency symlinks. Continuous ordinary task reads and comment saves must have zero failures, and every accepted save must be read back -exactly once. Forced process cleanup fails graceful qualification even if the +exactly once. Probe tasks live in a separate disposable company, so they cannot +block the business workflow's all-tasks-done oracle or consume its agent budget. +Forced process cleanup fails graceful qualification even if the successor recovers the execution. The read-only observer shares the controllers' disposable encryption key. It repeats replacement after completion before the retained-session follow-up. The five-minute outage suites are unchanged. diff --git a/tests/runner-e2e/directory-cleanup.test.ts b/tests/runner-e2e/directory-cleanup.test.ts new file mode 100644 index 00000000000..1852250a4bf --- /dev/null +++ b/tests/runner-e2e/directory-cleanup.test.ts @@ -0,0 +1,29 @@ +import { chmod, lstat, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { expect, it } from "vitest"; +import { makeDirectoryTreeRemovable } from "./directory-cleanup.js"; + +it("removes a retired home with immutable bundles without changing linked dependencies", async () => { + const root = await mkdtemp(path.join(tmpdir(), "paperclip-retired-home-")); + const home = path.join(root, "controller"), external = path.join(root, "external"); + const bundle = path.join(home, "runtime-context-assets/bundles/digest"); + try { + await mkdir(bundle, { recursive: true }); await mkdir(external); + await writeFile(path.join(bundle, "SKILL.md"), "immutable skill"); + await writeFile(path.join(external, "keep"), "dependency"); + await chmod(bundle, 0o555); await chmod(external, 0o555); + await symlink(external, path.join(home, "node_modules")); + await makeDirectoryTreeRemovable(home); + await rm(home, { recursive: true }); + await expect(lstat(home)).rejects.toMatchObject({ code: "ENOENT" }); + expect((await lstat(external)).mode & 0o777).toBe(0o555); + expect(await readFile(path.join(external, "keep"), "utf8")).toBe("dependency"); + const link = path.join(root, "not-an-owned-home"); + await symlink(external, link); + await expect(makeDirectoryTreeRemovable(link)).rejects.toThrow("not a symlink"); + expect((await lstat(external)).mode & 0o777).toBe(0o555); + } finally { + await makeDirectoryTreeRemovable(root); await rm(root, { recursive: true }); + } +}); diff --git a/tests/runner-e2e/directory-cleanup.ts b/tests/runner-e2e/directory-cleanup.ts new file mode 100644 index 00000000000..ffe0d6ca5c9 --- /dev/null +++ b/tests/runner-e2e/directory-cleanup.ts @@ -0,0 +1,13 @@ +import { chmod, lstat, readdir } from "node:fs/promises"; +import path from "node:path"; + +/** Cleanup of a stopped, test-owned tree. Never follow links to dependencies + * or external workspaces when making immutable bundle directories removable. */ +export async function makeDirectoryTreeRemovable(directory: string): Promise { + const stat = await lstat(directory); + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Cleanup requires an owned directory, not a symlink"); + await chmod(directory, 0o700); + const entries = await readdir(directory, { withFileTypes: true }); + await Promise.all(entries.filter(entry => entry.isDirectory() && !entry.isSymbolicLink()) + .map(entry => makeDirectoryTreeRemovable(path.join(directory, entry.name)))); +} diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index 3a278d79dbe..471982ed396 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -1,3 +1,4 @@ +import { makeDirectoryTreeRemovable } from "./directory-cleanup.js"; import { readUpgradeArtifacts, verifyUpgradeArtifact, upgradeArtifactKey } from "./upgrade-artifacts.js"; import { recoverWorkerTiming } from "./worker-timing.js"; import { openEvalOperations, type EvalOperations } from "./operations.js"; @@ -118,17 +119,6 @@ function secret(bytes = 32) { return randomBytes(bytes).toString("base64url"); } -async function makeDirectoryTreeRemovable(directory: string): Promise { - await chmod(directory, 0o700); - const entries = await readdir(directory, { withFileTypes: true }); - await Promise.all( - entries - .filter((entry) => entry.isDirectory() && !entry.isSymbolicLink()) - .map((entry) => - makeDirectoryTreeRemovable(path.join(directory, entry.name)), - ), - ); -} function stopProcessGroup(pid: number, signal: NodeJS.Signals = "SIGTERM") { try { diff --git a/tests/runner-e2e/rolling-deployment-smoke.ts b/tests/runner-e2e/rolling-deployment-smoke.ts index 6154d39b9d2..ee57d473ca6 100644 --- a/tests/runner-e2e/rolling-deployment-smoke.ts +++ b/tests/runner-e2e/rolling-deployment-smoke.ts @@ -1,3 +1,4 @@ +import { makeDirectoryTreeRemovable } from "./directory-cleanup.js"; /** Credential-free full-server overlap characterization. No model or Runner * continuity is claimed. Requires an explicitly selected disposable S3 service. * node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/rolling-deployment-smoke.ts */ @@ -208,7 +209,7 @@ try { proxy.routeTo(b.origin); await delay(500); await a.stop({ requireGraceful: true }); - await rm(a.home, { recursive: true, force: true }); + await makeDirectoryTreeRemovable(a.home); await rm(a.home, { recursive: true }); await delay(1_000); report.handoffElapsedMs = Math.round(performance.now() - switchedAt); if (browser) await check("Existing browser reconnects and opens an old lazy chunk after A removal", async () => { @@ -270,7 +271,7 @@ try { if (fixture.seeded) assert.equal((await api(b.origin, `${route}?path=AGENTS.md`)).content, "Durable original"); } }); - await b.stop({ requireGraceful: true }); await rm(b.home, { recursive: true, force: true }); + await b.stop({ requireGraceful: true }); await makeDirectoryTreeRemovable(b.home); await rm(b.home, { recursive: true }); const c = await start("C", false); proxy.routeTo(c.origin); await check("Fresh C restores files after both homes are deleted, even with adoption opt-in disabled", async () => { assert.equal((await api(c.origin, `${fileRoute}?path=reference.txt`)).content, content); diff --git a/tests/runner-e2e/rolling-deployment-supervisor.test.ts b/tests/runner-e2e/rolling-deployment-supervisor.test.ts index 13b7f9320a0..4ce8087c722 100644 --- a/tests/runner-e2e/rolling-deployment-supervisor.test.ts +++ b/tests/runner-e2e/rolling-deployment-supervisor.test.ts @@ -4,17 +4,17 @@ import { createLocalEncryptedCipher } from "../../server/src/secrets/local-encry const mocks = vi.hoisted(() => ({ start: vi.fn(), observer: vi.fn(), stop: vi.fn(), close: vi.fn(), writeFile: vi.fn(), })); -vi.mock("node:fs/promises", () => ({ mkdir: async () => {}, writeFile: mocks.writeFile, +vi.mock("node:fs/promises", () => ({ mkdir: async () => {}, rename: async () => {}, writeFile: mocks.writeFile, readFile: async () => JSON.stringify({ database: { mode: "postgres", connectionString: "postgres://fixture" }, secrets: { localEncrypted: { keyFilePath: "/fixture/secrets/master.key" } } }) })); vi.mock("../../packages/shared/src/config-schema.js", () => ({ paperclipConfigSchema: { parse: (value: unknown) => value } })); vi.mock("./rolling-deployment-process.js", () => ({ startRollingDeploymentProcess: mocks.start })); vi.mock("./rolling-deployment-proxy.js", () => ({ startRollingDeploymentProxy: async () => ({ origin: "http://127.0.0.1:1", routeTo() {}, close: mocks.close }) })); vi.mock("./controller-recovery-observer.js", () => ({ createControllerRecoveryObserver: mocks.observer })); -vi.mock("./server-restart-request.js", () => ({ readRestartRequest: async () => ({ requestId: "key-check", rollingDeployment: true, recoveryBinding: {} }) })); +vi.mock("./server-restart-request.js", () => ({ readRestartRequest: async () => ({ requestId: "key-check", rollingDeployment: true, recoveryBinding: { companyId: "business-company" } }) })); import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js"; -afterEach(() => { vi.unstubAllEnvs(); vi.clearAllMocks(); }); +afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); vi.clearAllMocks(); }); it.each([undefined, "prior-disposable-key"])("shares the server key with checkpoint observation and restores %s on failure", async prior => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", prior); @@ -40,3 +40,24 @@ it.each([undefined, "prior-disposable-key"])("shares the server key with checkpo expect(mocks.close).toHaveBeenCalledOnce(); expect(process.env.PAPERCLIP_SECRETS_MASTER_KEY).toBe(prior); }); + + +it("keeps open deployment probe tasks out of the agent workflow's company", async () => { + mocks.start.mockResolvedValue({ child: { exitCode: null, signalCode: null }, configPath: "/fixture/config.json", + origin: "http://127.0.0.1:2", waitForReady: async () => {}, stop: mocks.stop }); + mocks.observer.mockResolvedValue(undefined); + const routes: string[] = []; + vi.stubGlobal("fetch", vi.fn(async (url: string) => { + const route = new URL(url).pathname; + routes.push(route); + if (route === "/api/companies") return new Response(JSON.stringify({ id: "probe-company" })); + if (route === "/api/companies/probe-company/issues") throw new Error("probe fixture boundary reached"); + throw new Error(`Probe modified business workflow: ${route}`); + })); + await expect(superviseRollingDeployment({ temporaryRoot: "/fixture", repositoryRoot: "/fixture", + configPath: "/fixture/config.json", port: 1, executionIds: ["rolling"], cancelled: () => false, + environment: { PAPERCLIP_STORAGE_PROVIDER: "s3" }, + })).rejects.toThrow("probe fixture boundary reached"); + expect(routes).toEqual(["/api/companies", "/api/companies/probe-company/issues"]); + expect(mocks.stop).toHaveBeenCalledOnce(); +}); diff --git a/tests/runner-e2e/rolling-deployment-supervisor.ts b/tests/runner-e2e/rolling-deployment-supervisor.ts index 0f3a9814f2c..0db28f93792 100644 --- a/tests/runner-e2e/rolling-deployment-supervisor.ts +++ b/tests/runner-e2e/rolling-deployment-supervisor.ts @@ -1,3 +1,4 @@ +import { makeDirectoryTreeRemovable } from "./directory-cleanup.js"; import { randomBytes, randomUUID } from "node:crypto"; import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises"; import path from "node:path"; @@ -76,7 +77,9 @@ export async function superviseRollingDeployment(input: { const writes: Array<{ id: string; body: string }> = []; try { if (!probeIssueId) { - const companyId = request.recoveryBinding?.companyId ?? (await api<{ id: string }>("/companies", "POST", { name: "Synthetic deployment probes" })).id; + // The business oracle observes every task in its company. Keep the + // unassigned read/save probe outside that workload and its agent budget. + const companyId = (await api<{ id: string }>("/companies", "POST", { name: "Synthetic deployment probes" })).id; const issue = await api<{ id: string }>(`/companies/${companyId}/issues`, "POST", { title: "Synthetic deployment read/save probe", description: "Unassigned fixture for ordinary user saves during deployment.", status: "todo", }); @@ -103,7 +106,7 @@ export async function superviseRollingDeployment(input: { waitForReady: async replacement => { await replacement.server.waitForReady(); }, switchTraffic: replacement => { current = replacement.server; proxy.routeTo(current.origin); }, stopPrevious: () => previous.stop({ requireGraceful: true }), - deletePreviousHome: () => rm(previous.home, { recursive: true }), + deletePreviousHome: async () => { await makeDirectoryTreeRemovable(previous.home); await rm(previous.home, { recursive: true }); }, beforeHandoff: observer?.beforeHandoff, afterPreviousStopped: observer?.stoppedAfterHandoff, afterHandoff: observer ? (receipt, record) => observer.recovered(receipt, record, input.cancelled) : undefined,