From 53645a010402913193cfab709cf656f9a1fc9db7 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 09:54:13 -0600 Subject: [PATCH 01/13] perf(workspaces): record per-phase timings for workspace saves (#19472) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a sandbox save the workspace to object storage. A run saves before it starts work and again at the end. > - A save of a large repository takes 150 to 250 seconds when it runs alone. When several large saves overlap, each save takes 510 to 900 seconds. > - A slow save before the run starts delays the agent. It also gives a deploy a wider window to interrupt the run. > - The save operation records only a start time and an end time. It does not show which phase uses the time: in-sandbox capture, provider download, re-hash, encrypted upload, or publication. > - Without that data, a speed fix is a guess. Each possible bottleneck needs a different fix. > - This pull request records a bounded duration for each phase in the save operation's local metadata. > - The benefit is that one busy window on a deployment identifies the bottleneck. Then a targeted fix can be measured. ## Linked Issues or Issue Description **Is your feature request related to a problem?** Saves of large repository workspaces are slow, and a save does not record where the time goes. On one deployment, saves of a 460MB workspace archive (a full repository with history) took 150 to 250 seconds when they ran alone. During a window where eight large saves overlapped, each save took 510 to 895 seconds. Saves of small repositories in the same window stayed at 2 to 6 seconds. The operation row has only `startedAt` and `finishedAt`, so the contended resource is unknown. It can be the sandbox CPU, the provider download, or server disk and network. **Describe the solution you'd like** Each save invocation records per-phase durations and the archive size in the operation's metadata, for both outcomes. **Describe alternatives you've considered** Changes to the speed of a single phase without phase data. A local profile of one capture showed that `git fsck --full` on the archived history uses about half of the capture time. But full object integrity is a tested requirement, and the cheaper `--connectivity-only` check does not meet it. A larger delta-base cache gave no gain. A threaded check helps only on sandboxes with more than one core. Also, the overlap slowdown can come from the server, not the sandbox. A fix chosen without phase data could miss the real bottleneck. Related: #19444 (skip redundant saves on follow-up runs), #19458 (overlap file hashing in capture), #19462 (bounded bootstrap retry after a closed runtime listener). ## What Changed - `captureRemoteWorkspaceArchive` accepts an optional `timing` record. It writes `captureMs` (helper install and in-sandbox capture), `transferMs` (provider download), `verifyMs` (server re-hash), `byteSize`, and `entries` as each phase ends. - `saveRemoteWorkspace` adds `uploadMs` (encrypt, hash, and store) and `publishMs` (manifest read and authority commit). It also sets `reusedUpload` when it reuses an earlier upload. - `settleSaveInvocation` writes a `saveTiming` record (`paperclip.remote-workspace-save-timing.v1`) with `outcome`, `totalMs`, and the finished phases. It writes the record on success and on failure. A failed save keeps the phases that it finished. - The record holds only numbers, a fixed outcome, and a timestamp. It holds no paths or provider text. It stays in the instance database (`workspace_operations.metadata`), so it is not Telemetry. ## Verification - Extended `retains the writer and previous snapshot when object storage fails` in `server/src/services/remote-workspaces.integration.test.ts`. After the storage failure, the `final` operation has a `failed` timing with capture, transfer, and verify phases and no upload phase. After the retry, it has a `succeeded` timing with every phase. - Red/green: with the source change reverted, the test fails (`expected [ undefined ] to deeply equal [ { …(9) } ]`). With the change, it passes. - `npx vitest run src/services/remote-workspaces.integration.test.ts -t "object storage fails"` passes. - `npx tsc --noEmit -p server` passes. - A reviewer can read the record through `GET /api/heartbeat-runs/:runId/workspace-operations` (`metadata.saveTiming`). ## Risks - Low risk. The change adds metadata only. It does not change save order, fencing, retries, or archive bytes. - The latest settled invocation replaces `saveTiming`. Concurrent invocations of the same boundary keep only the last record. That is acceptable for diagnosis. - `performance.now()` calls add negligible cost. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, browser for read-only staging GETs, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../services/remote-workspace-transport.ts | 28 +++++++++++++- .../remote-workspaces.integration.test.ts | 24 +++++++++++- server/src/services/remote-workspaces.ts | 37 +++++++++++++++++-- 3 files changed, 83 insertions(+), 6 deletions(-) diff --git a/server/src/services/remote-workspace-transport.ts b/server/src/services/remote-workspace-transport.ts index 3e821827f68..59df98adf52 100644 --- a/server/src/services/remote-workspace-transport.ts +++ b/server/src/services/remote-workspace-transport.ts @@ -398,9 +398,29 @@ export async function cleanupRemoteWorkspaceArchive(input: { await command(input.target, "cleanup", ["-e", script, input.remoteCwd, input.operationId], undefined, 10_000); } +/** + * Wall-clock milliseconds for each save phase, filled in as each phase ends so a + * failed save still reports the phases it finished. It holds only durations and + * archive counts, never paths or provider text. + */ +export type RemoteWorkspaceSavePhaseTiming = { + /** Archive helper install plus in-sandbox capture. */ + captureMs?: number; + /** Provider download of the captured archive. */ + transferMs?: number; + /** Server-side re-hash of the downloaded archive. */ + verifyMs?: number; + byteSize?: number; + entries?: number; +}; + +const elapsedSince = (started: number) => Math.max(0, Math.round(performance.now() - started)); + export async function captureRemoteWorkspaceArchive(input: { - target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string; + target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string; timing?: RemoteWorkspaceSavePhaseTiming; }) { + const timing = input.timing ?? {}; + const captureStarted = performance.now(); const staged = await installArchiveHelper(input.target, input.remoteCwd, input.operationId); // Publish the archive and receipt as one immutable directory. Concurrent // invocations own separate attempts; no invocation unlinks another's archive. @@ -444,14 +464,20 @@ export async function captureRemoteWorkspaceArchive(input: { }; if (!["legacy", "captured"].includes(layout) || !/^[a-f0-9]{64}$/.test(receipt.manifestDigest) || !/^[a-f0-9]{64}$/.test(receipt.sha256) || !Number.isSafeInteger(receipt.byteSize) || receipt.byteSize <= 0) throw new Error("remote_workspace_capture_receipt_invalid"); + timing.captureMs = elapsedSince(captureStarted); timing.byteSize = receipt.byteSize; + if (Number.isSafeInteger(receipt.entries) && receipt.entries >= 0) timing.entries = receipt.entries; const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-upload-")); const archivePath = path.join(root, "workspace.pcws"); try { + const transferStarted = performance.now(); await transferArchive(input.target, "syncOut", [{ operationId: input.operationId, files: [{ kind: "file", sourcePath: layout === "captured" ? path.posix.join(staged.directory, "captured", "workspace.pcws") : staged.archive, targetPath: archivePath }], }], receipt.byteSize); + timing.transferMs = elapsedSince(transferStarted); + const verifyStarted = performance.now(); const hash = createHash("sha256"); let bytes = 0; for await (const chunk of createReadStream(archivePath)) { hash.update(chunk); bytes += chunk.length; } + timing.verifyMs = elapsedSince(verifyStarted); if (hash.digest("hex") !== receipt.sha256 || bytes !== receipt.byteSize) throw new Error("remote_workspace_capture_transfer_mismatch"); return { ...receipt, archivePath, cleanup: () => rm(root, { recursive: true, force: true }) }; } catch (error) { diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 7b8e13bf206..8c46f3edb43 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -1115,9 +1115,26 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata)).toMatchObject({ dirty: true, writerRunId: ids.run, snapshotId: readRemoteWorkspaceAuthority(prepared.workspace.metadata)!.snapshotId }); expect((await fs.readdir(path.join(prepared.remoteRoot, ".paperclip-runtime", "workspace-snapshots"))).length).toBe(1); - const saved = await saveRemoteWorkspace({ db, storage, workspace: await workspace(reserved.id), runId: ids.run, + // Each settled save records where its time went: the failed upload keeps the + // phases it finished, and the retry of the same boundary records every phase. + const finalTiming = async () => (await db.select().from(workspaceOperations).where(and( + eq(workspaceOperations.executionWorkspaceId, reserved.id), eq(workspaceOperations.phase, "remote_workspace_snapshot"), + sql`${workspaceOperations.metadata}->>'boundary' = 'final'`))).map(save => save.metadata?.saveTiming); + const duration = expect.any(Number), archive = { byteSize: expect.any(Number), entries: expect.any(Number) }; + expect(await finalTiming()).toEqual([{ schema: "paperclip.remote-workspace-save-timing.v1", outcome: "failed", totalMs: duration, + captureMs: duration, transferMs: duration, verifyMs: duration, ...archive, at: expect.any(String) }]); + const beforeSuccess = await workspace(reserved.id); + const saved = await saveRemoteWorkspace({ db, storage, workspace: beforeSuccess, runId: ids.run, lease: prepared.lease, target: first, assertOwned: async () => undefined }); expect(saved.generation).toBe(2); + const [succeededTiming] = await finalTiming(); + // A lost-reply replay of the committed save keeps the measured record. + await expect(saveRemoteWorkspace({ db, storage, workspace: beforeSuccess, runId: ids.run, + lease: prepared.lease, target: first, assertOwned: async () => undefined })).resolves.toMatchObject({ snapshotId: saved.snapshotId }); + expect(await finalTiming()).toEqual([succeededTiming]); + expect(await finalTiming()).toEqual([{ schema: "paperclip.remote-workspace-save-timing.v1", outcome: "succeeded", totalMs: duration, + captureMs: duration, transferMs: duration, verifyMs: duration, uploadMs: duration, publishMs: duration, ...archive, + at: expect.any(String) }]); expect(await fs.readdir(path.join(prepared.remoteRoot, ".paperclip-runtime", "workspace-snapshots"))).toEqual([]); }, 60_000); @@ -1224,9 +1241,12 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem .rejects.toThrow("remote_workspace_snapshot_operation_binding_changed"); expect(upload).not.toHaveBeenCalled(); const [held] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, poisoned!.id)); - const { saveWork, ...originalMetadata } = held!.metadata!; + const { saveWork, saveTiming, ...originalMetadata } = held!.metadata!; expect(originalMetadata).toEqual(poisoned!.metadata); expect(saveWork).toMatchObject({ activeIds: [], unknown: true, quiescedAt: expect.any(String) }); + // The refused invocation records only its own failed duration; it never captured or uploaded. + expect(saveTiming).toEqual({ schema: "paperclip.remote-workspace-save-timing.v1", outcome: "failed", + totalMs: expect.any(Number), at: expect.any(String) }); expect(Number.isFinite(Date.parse((saveWork as { quiescedAt: string }).quiescedAt))).toBe(true); expect(await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.sourceOperationId, poisoned!.id))) .toHaveLength(0); diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 8af46c45c4e..63aa0bfe1e1 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -28,6 +28,7 @@ import { captureRemoteWorkspaceArchive, cleanupRemoteWorkspaceArchive, prepareRemoteWorkspaceDirectory, prepareRemoteRepository, remoteWorkspaceRepositoryPath, RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError, remoteWorkspaceFailureCause, requireRemoteWorkspaceTarget, restoreRemoteWorkspaceDirectory, RemoteWorkspaceRestoreError, type RemoteWorkspaceRestoreStep, + type RemoteWorkspaceSavePhaseTiming, } from "./remote-workspace-transport.js"; import { beginWorkspaceSnapshot, getWorkspaceSnapshot, loadWorkspaceSnapshotArchive, uploadWorkspaceSnapshot, @@ -46,6 +47,16 @@ type Workspace = Pick[0]>[0]; const object = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; const SAVE_WORK_SCHEMA = "paperclip.remote-workspace-save-work.v1"; +const SAVE_TIMING_SCHEMA = "paperclip.remote-workspace-save-timing.v1"; + +/** Per-phase durations of the latest settled save invocation, kept in the operation's local metadata. */ +type RemoteWorkspaceSaveTiming = RemoteWorkspaceSavePhaseTiming & { + /** Encrypt, hash, and store the archive (zero work when a prior upload is reused). */ + uploadMs?: number; + /** Read the archive manifest and commit the saved authority. */ + publishMs?: number; + reusedUpload?: boolean; +}; const RETIRED_SAVE_SCHEMA = "paperclip.remote-workspace-retired-save.v1"; type SaveWork = { schema: typeof SAVE_WORK_SCHEMA; activeIds: string[]; unknown: boolean; quiescedAt: string | null }; function readSaveWork(metadata: Record | null | undefined): SaveWork | null { @@ -610,9 +621,15 @@ export function remoteWorkspaceSaveFailure(error: unknown): { code: string; caus return cause ? { code, cause } : { code }; } +function saveTimingRecord(timing: RemoteWorkspaceSaveTiming & { startedAt: number }, outcome: "succeeded" | "failed", at: Date) { + const { startedAt, ...phases } = timing; + return { schema: SAVE_TIMING_SCHEMA, outcome, totalMs: Math.max(0, Math.round(performance.now() - startedAt)), + ...phases, at: at.toISOString() }; +} + async function settleSaveInvocation(input: { db: Db; workspace: Workspace; runId: string; epoch: number; location: NonNullable; operationId: string; invocationId: string; - failure?: { code: string; cause?: string } }) { + failure?: { code: string; cause?: string }; timing?: RemoteWorkspaceSaveTiming & { startedAt: number } }) { return input.db.transaction(async tx => { const workspace = await currentWorkspace(tx, input.workspace.companyId, input.workspace.id); const [operation] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.id, input.operationId), @@ -629,6 +646,7 @@ async function settleSaveInvocation(input: { db: Db; workspace: Workspace; runId const failed = Boolean(failure) && operation.status === "running" && !activeIds.length && !work.unknown; await tx.update(workspaceOperations).set({ metadata: { ...operation.metadata, saveWork: { ...work, activeIds, quiescedAt: activeIds.length ? null : now.toISOString() }, + ...(input.timing ? { saveTiming: saveTimingRecord(input.timing, input.failure ? "failed" : "succeeded", now) } : {}), ...(failed && failure ? { lastFailure: { ...failure, at: now.toISOString() } } : {}) }, ...(failed && failure ? { status: "failed", exitCode: 1, finishedAt: now, stderrExcerpt: `${[failure.code, failure.cause].filter(Boolean).join("\n")}\n` } : {}), @@ -644,6 +662,7 @@ export async function saveRemoteWorkspace(input: { assertOwned: () => Promise; storage?: StorageService; boundary?: string; repositories?: NonNullable; }) { + const saveStarted = performance.now(); const target = requireRemoteWorkspaceTarget(input.target), boundary = input.boundary ?? "final"; const expected = readRemoteWorkspaceAuthority(input.workspace.metadata); if (!expected?.location || expected.location.providerLeaseId !== input.lease.providerLeaseId || expected.location.leaseId !== input.lease.id) throw new Error("remote_workspace_save_binding_mismatch"); @@ -696,21 +715,31 @@ export async function saveRemoteWorkspace(input: { let captured: Awaited> | null = null; let publicationScratch: string | undefined; let failure: { code: string; cause?: string } | undefined; + const timing: RemoteWorkspaceSaveTiming & { startedAt: number } = { startedAt: saveStarted }; + // A replay of an already-committed save measures no phase; it keeps the + // original save's timing instead of overwriting it with an empty record. + let replayedCommittedSave = false; try { const snapshot = await beginWorkspaceSnapshot(input.db, { companyId: input.workspace.companyId, workspaceId: input.workspace.id, operationId: operation.id, runId: input.runId, parentSnapshotId: operation.metadata?.parentSnapshotId as string | null, generation: operation.metadata?.generation as number, epoch: expected.epoch }); if (operation.status === "succeeded" && ["committed", "superseded"].includes(snapshot.status)) { + replayedCommittedSave = true; return { snapshotId: snapshot.id, generation: snapshot.generation, sha256: snapshot.sha256!, savedAt: snapshot.updatedAt.toISOString() }; } const existingUpload = snapshot.status === "uploaded" ? await getWorkspaceSnapshot(input.db, { companyId: input.workspace.companyId, workspaceId: input.workspace.id, snapshotId: snapshot.id }) : null; - captured = existingUpload ? null : await captureRemoteWorkspaceArchive({ target, remoteCwd: expected.location.remoteCwd, operationId: operation.id }); + if (existingUpload) timing.reusedUpload = true; + captured = existingUpload ? null : await captureRemoteWorkspaceArchive({ target, remoteCwd: expected.location.remoteCwd, + operationId: operation.id, timing }); await input.assertOwned(); + const uploadStarted = performance.now(); const saved = await uploadWorkspaceSnapshot({ db: input.db, storage: input.storage ?? getStorageService(), snapshotId: snapshot.id, companyId: input.workspace.companyId, workspaceId: input.workspace.id, archivePath: captured?.archivePath ?? "", manifestDigest: existingUpload?.manifestDigest ?? captured!.manifestDigest }); + timing.uploadMs = Math.max(0, Math.round(performance.now() - uploadStarted)); await input.assertOwned(); + const publishStarted = performance.now(); let publicationArchive = captured?.archivePath; if (!publicationArchive) { publicationScratch = await mkdtemp(path.join(os.tmpdir(), "pc-workspace-publication-")); @@ -756,6 +785,7 @@ export async function saveRemoteWorkspace(input: { deleteAfter: new Date(at.getTime() + 7 * 86400_000), updatedAt: at }).where(and(eq(executionWorkspaceSnapshots.id, snapshot.parentSnapshotId), eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id))); }); + timing.publishMs = Math.max(0, Math.round(performance.now() - publishStarted)); return { snapshotId: saved.id, generation: saved.generation, sha256: saved.sha256, savedAt: at.toISOString() }; } catch (error) { failure = remoteWorkspaceSaveFailure(error); @@ -768,7 +798,8 @@ export async function saveRemoteWorkspace(input: { catch { logger.warn({ runId: input.runId, operationId: operation.id }, "Remote workspace local archive cleanup failed"); } try { if (await settleSaveInvocation({ db: input.db, workspace: input.workspace, runId: input.runId, epoch: expected.epoch, - location: expected.location, operationId: operation.id, invocationId, failure })) { + location: expected.location, operationId: operation.id, invocationId, failure, + timing: replayedCommittedSave ? undefined : timing })) { await cleanupSavedExports({ db: input.db, companyId: input.workspace.companyId, workspaceId: input.workspace.id, location: expected.location, target, assertOwned: input.assertOwned }); } From e2f0fc9e7dca780c4356478522540f5203d11479 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 10:08:20 -0600 Subject: [PATCH 02/13] fix(runner): end the detached Codex controller's event stream so graceful shutdown exits promptly (#19483) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane runs native Codex turns through the Paperclip runner, and a restart must hand those turns to the next controller > - On a graceful shutdown (SIGTERM), the server detaches each native Codex session and leaves the runner alive for restart recovery > - After the detach, the shutdown waited the full 30 s "bounded heartbeat execution finalizer drain" every time > - A supervisor with a 30 s graceful window (for example the runner-e2e supervisor) then sent SIGKILL, so a graceful restart became a hard kill > - This pull request ends the detached controller's event stream, so the old run execution unwinds immediately > - The benefit is that graceful restarts and deploys exit in about one second and stay graceful ## Linked Issues or Issue Description **What happened?** Every graceful shutdown with an active or restart-suspended native Codex run waited 30 s before exit. The server log showed `graceful heartbeat run drain complete` with the run in `restartSuspendedRunIds`, then `bounded heartbeat execution finalizer drain timed out` 30 s later. Under a supervisor with a 30 s grace period, the server got SIGKILL. **Steps to reproduce** 1. Start a native Codex run (runner-e2e cell `everyday-workflows.runner-codex.local.recover-controller`). 2. Stop the server with SIGTERM while the turn is active (`PAPERCLIP_RUNNER_E2E_CONTROLLER_STOP_SIGNAL=SIGTERM`). 3. See that the server does not exit until the 30 s drain timeout, or until SIGKILL. **Expected behavior** The server exits promptly. The detached run is left for restart recovery. **Root cause** - `drainRunningRunsForShutdown` (`server/src/services/heartbeat/recovery.ts`) calls `detachNativeSessionsForRestart` for the native run. This calls `CodexHarnessSession.detachControllerForRestart`. - `CodexHarnessSession.detachControllerForRestart` (`packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts`) detached the transport but did not close the session `eventQueue`. - The transport closed its notification queue. `pumpNotifications` then returned normally and did not close the session queue. - `executeNativeSession` waited on `eventIterator.next()` for an event that could never arrive. The heartbeat execution promise (registered at `server/src/services/heartbeat/queue.ts` in `activeRunExecutionPromises`) never settled. - `drainActiveRunExecutions` awaited that promise until the 30 s bound in `drainRunExecutionFinalizersForShutdown`. Temporary diagnostics confirmed this: during the hang, the only pending entry was the `queue.ts` run execution, in the `execution.runtime` phase, with no pending wakes. Other drivers (Cursor Cloud, external runnerd) already end their event stream on detach. ## What Changed - `CodexHarnessSession.detachControllerForRestart` now clears and closes the session event queue after the transport detaches. The detached consumer gets end-of-stream. The executor then throws `NativeControllerDetachedForRestartError`, and heartbeat leaves the run for restart recovery. - Buffered events are not delivered to the detached consumer. The next controller replays them from the runner journal. - Pending runtime requests stay open for the adopting controller (no `cancelPendingRequests`). - If the transport detach fails, the queue stays open. The existing uncertain-ownership path is unchanged. - Added a regression test: after detach, the session event iterator ends, and a late provider notification does not reach the detached consumer. ## Verification - Red, then green: `npx vitest run src/drivers/codex/codex-app-server-driver.lifecycle.test.ts` in `packages/paperclip-runner`. Before the fix, the new test failed with `hung`. After the fix, all 22 tests pass. - `npx vitest run src/drivers/codex src/backends`: 27 files, 457 tests pass. - `tsc --noEmit` for `@paperclipai/paperclip-runner` passes. - Live: `PAPERCLIP_RUNNER_E2E_CONTROLLER_STOP_SIGNAL=SIGTERM pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-codex.local.recover-controller` - Before (unchanged code, reproduced locally): `Paperclip did not stop within 30000ms; sending SIGKILL`, `server exited code=null signal=SIGKILL`. - After, run 1: PASS. Drain complete with the run in `restartSuspendedRunIds`, then `server exited code=0 signal=null` within the same second. The next server resumed the run and the task completed. - After, run 2: PASS. Same clean exit, no SIGKILL. - `pnpm verify` returned exit 2 (broader verification required). Required CI supplies that coverage. - Note: 21 tests in `src/live/runnerd-restart-continuation.test.ts`, `runnerd-controller-outage.test.ts` and `runnerd-codex-transport-warm-attachment.test.ts` fail on this machine with and without this change. They use the transport directly, not `CodexHarnessSession`. ## Risks - Low risk. The change only affects a session after its controller detaches for restart. That controller has no authority to settle the run. - The 30 s finalizer bound stays in place. It still applies when a run execution is past the provider stream, for example while it persists an accepted result or settles a workspace. Those durable writes must finish (or reach their recovery records) before exit, so waiting for them is correct. - Dropped buffered events: the old consumer could not persist them anyway after the transport closed. The next controller replays from the runner journal. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../codex-app-server-driver.lifecycle.test.ts | 55 +++++++++++++++++++ .../drivers/codex/codex-harness-session.ts | 6 ++ 2 files changed, 61 insertions(+) diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts index a0baee374a6..62e97dc75e4 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts @@ -426,6 +426,61 @@ describe("Codex app-server Codex driver", () => { expect(close).not.toHaveBeenCalled(); }); + it("ends the detached controller's event stream so its consumer can unwind", async () => { + const transport = new FakeCodexTransport(); + Object.assign(transport, { detachControllerForRestart: vi.fn(async () => undefined) }); + const driver = makeDriver([transport]); + const session = await driver.openSession({ + runId: "run-detach-consumer", + normalizedSessionId: "normalized-detach-consumer", + workingDirectory: WORKSPACE, + }); + await session.startTurn({ message: { role: "user", text: "Keep working" } }); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "turn-1", status: "inProgress", items: [] } }); + const iterator = session.events()[Symbol.asyncIterator](); + // Drain what the provider published before the restart. + for (;;) { + const pending = await Promise.race([ + iterator.next(), + new Promise<"waiting">(resolve => setTimeout(() => resolve("waiting"), 20)), + ]); + if (pending === "waiting") break; + expect(pending.done).toBe(false); + } + const next = iterator.next(); + + await session.detachControllerForRestart?.(); + // The provider keeps running for the next controller; a late notification + // must not reach the detached consumer. + transport.push("item/agentMessage/delta", { threadId: "thread-1", turnId: "turn-1", itemId: "m", delta: "late" }); + + await expect(Promise.race([ + next, + new Promise<"hung">(resolve => setTimeout(() => resolve("hung"), 200)), + ])).resolves.toEqual({ value: undefined, done: true }); + }); + + it("drops events the detached controller had not read yet", async () => { + const transport = new FakeCodexTransport(); + Object.assign(transport, { detachControllerForRestart: vi.fn(async () => undefined) }); + const driver = makeDriver([transport]); + const session = await driver.openSession({ + runId: "run-detach-buffered", + normalizedSessionId: "normalized-detach-buffered", + workingDirectory: WORKSPACE, + }); + await session.startTurn({ message: { role: "user", text: "Keep working" } }); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "turn-1", status: "inProgress", items: [] } }); + // Let the notification pump buffer these events without a reader. + await new Promise(resolve => setTimeout(resolve, 20)); + + await session.detachControllerForRestart?.(); + + // The next controller replays them from the runner journal instead. + await expect(session.events()[Symbol.asyncIterator]().next()) + .resolves.toEqual({ value: undefined, done: true }); + }); + it("sends direct chat as plain text and permits a follow-up turn", async () => { const transport = new FakeCodexTransport(); const driver = makeDriver([transport], { conversationMode: "direct" }); diff --git a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts index c9c2ef14354..fdde2d14d90 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts @@ -895,5 +895,11 @@ export class CodexHarnessSession async detachControllerForRestart(): Promise { if (this.transport.detachControllerForRestart === undefined) return; await this.transport.detachControllerForRestart(); + // The next controller replays the runner journal. End this controller's + // stream without its buffered suffix so the detached consumer unwinds + // instead of waiting for provider events that will never arrive here. + // Pending runtime requests stay open for the adopting controller. + this.eventQueue.clear(); + this.eventQueue.close(); } } From 5e62d8ee821cec11448474524fa674612698dfd5 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 08:18:03 -0800 Subject: [PATCH 03/13] Report the native private host-control protocol version (#19488) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner executes agent work and reports its binary contract through build metadata. > - Unix Runner already supports a private host-control socket for park, claim, and retire requests. > - A host cannot identify that protocol from the current metadata. > - This PR adds an explicit version field for that existing private protocol. > - Hosts can verify the selected binary before opening the private channel. ## Linked Issues or Issue Description Refs #19140 and #19129, which added the existing native lifecycle and private channel. **What existing behavior does this improve?** The `paperclip-runnerd --build-metadata` response. **Current behavior** The native binary implements the private fd 4 protocol on Unix. Its metadata has no field that identifies this protocol. Recovery versions describe different contracts. **Proposed behavior** Report `privateHostControlVersion: 1` on Unix builds. Omit the field on other builds. Keep ordinary artifact checks and runtime behavior unchanged. **Reason and benefit** A host can verify protocol support on the selected binary before using the channel. This field does not mean a provider is ready to park or that a host implements public release/resume or qualified sleep/wake. **Breaking changes** The JSON response gains one optional field on Unix. Current artifact parsers and launch checks accept extra fields. Binary contract version 2 stays unchanged. ## What Changed - Add the Unix-only private host-control metadata version. - Check the exact version and non-Unix absence in metadata unit and real CLI tests. Keep hosted parking and public release/resume fields absent. - Add a regression test that proves ordinary artifact parsing accepts and ignores the new field. - Document the field, exact-version checks, and separate provider and host qualification. ## Verification - On macOS arm64 with Rust 1.97.1, passed narrow offline, locked Cargo tests for the metadata unit, six existing private-protocol cases, wrong-fd CLI rejection, and real CLI metadata output. - Passed `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`. - Passed `vitest run src/evals/runnerd-artifact.test.ts` from the Runner package: 5 tests. Initial collection could not resolve Ajv in the fresh worktree. Connecting existing local dependencies resolved that setup failure; no source change was needed. - `git diff --check` passed. Reviewed the diff for secrets, personal data, and private references. - GitHub CI passed on commit `0790af690afaff10f3b69fd9130b5be1c5bab38c`: [main PR workflow](https://github.com/paperclipai/paperclip/actions/runs/42248006277), [Docker Runner check](https://github.com/paperclipai/paperclip/actions/runs/42248006207), and [Windows Runner](https://github.com/paperclipai/paperclip/actions/runs/42248006518). Windows tests cover omission of the field on non-Unix builds. All 61 current-head check runs settled without failure; the latest status context passed. - [Standard Greptile review](https://github.com/paperclipai/paperclip/pull/19488#issuecomment-6689260747) scored that same commit 5/5 with no actionable findings or unresolved threads. - No live provider or host sleep/wake test was run for this metadata-only change. ## Risks The main risk is treating protocol support as permission to park a provider or as proof that a host supports release/resume. The documentation keeps those checks separate. No caller enables the channel from this field in this PR. Current parsers check named fields or project known fields, so the optional field does not change ordinary artifact checks. No migrations, new dependencies, runtime flags, checkpoints, or provider pins. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), High reasoning, through Codex with tool use and code execution. Independent agent review checked the metadata boundary and compatibility. The context window size is not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../paperclip-runner/runner/DURABLE_TRANSPORT.md | 12 ++++++++++++ .../runner-core/src/bin/paperclip-runnerd.rs | 14 ++++++++++++-- .../crates/runner-core/tests/local_runner.rs | 7 +++++++ .../src/evals/runnerd-artifact.test.ts | 5 +++++ 4 files changed, 36 insertions(+), 2 deletions(-) diff --git a/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md b/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md index ca4697d8c46..ab44e35d6e1 100644 --- a/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md +++ b/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md @@ -179,6 +179,18 @@ validate the exact Runner identity and acknowledged sequence cut. ### Hosted park and reclaim loop +On Unix builds, `paperclip-runnerd --build-metadata` reports +`privateHostControlVersion: 1` for the existing private fd 4 protocol below. +Other builds omit this field. A consumer must verify the selected executable's +identity and require the exact supported version before using this channel; +an absent or unknown version does not establish support. Recovery and warm-task +versions do not establish it either. Ordinary artifact admission can ignore this +optional field. + +This field describes protocol support only. It does not mean a provider is ready +to park or that a host implements the public `releaseResumeVersion` contract. +Runner readiness checks and host sleep/wake qualification are still required. + A trusted Launcher may pass a connected Unix stream at fd 4 and start durable listener mode with `--host-control-fd 4`. The descriptor is close-on-exec before provider startup. The private channel accepts the `park`, `claim`, and `retire` diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs index 4d3c6da791c..895de31d866 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs @@ -115,7 +115,7 @@ fn install_crypto_provider() { } fn build_metadata() -> serde_json::Value { - json!({ + let mut metadata = json!({ "schema": RUNNERD_BUILD_METADATA_SCHEMA, "binaryName": "paperclip-runnerd", "packageName": "@paperclipai/paperclip-runner", @@ -133,7 +133,11 @@ fn build_metadata() -> serde_json::Value { "maximumVersion": 2 }, "prpTransportModes": ["dial_ws_loopback", "dial_wss", "listen_ws"] - }) + }); + if cfg!(unix) { + metadata["privateHostControlVersion"] = json!(1); + } + metadata } // The publisher seals image installations. Check only path metadata at startup; @@ -726,6 +730,12 @@ mod tests { assert_eq!(metadata["acpxRecoveryIntegrityVersion"], 1); assert!(metadata.get("acpxSourceRecoveryVersion").is_none()); assert!(metadata.get("hostedParkingVersion").is_none()); + assert!(metadata.get("releaseResumeVersion").is_none()); + if cfg!(unix) { + assert_eq!(metadata["privateHostControlVersion"], 1); + } else { + assert!(metadata.get("privateHostControlVersion").is_none()); + } assert_eq!(metadata["schema"], RUNNERD_BUILD_METADATA_SCHEMA); assert_eq!(metadata["binaryContractVersion"], 2); assert_eq!( diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/local_runner.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/local_runner.rs index e71ef5778a5..eb2eacbf6bd 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/local_runner.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/local_runner.rs @@ -79,6 +79,13 @@ fn runnerd_startup_reports_build_metadata_without_panicking() { let metadata: Value = serde_json::from_slice(&output.stdout).expect("build metadata should be valid JSON"); assert_eq!(metadata["binaryName"], "paperclip-runnerd"); + if cfg!(unix) { + assert_eq!(metadata["privateHostControlVersion"], 1); + } else { + assert!(metadata.get("privateHostControlVersion").is_none()); + } + assert!(metadata.get("hostedParkingVersion").is_none()); + assert!(metadata.get("releaseResumeVersion").is_none()); } #[test] diff --git a/packages/paperclip-runner/src/evals/runnerd-artifact.test.ts b/packages/paperclip-runner/src/evals/runnerd-artifact.test.ts index 4463571f3cf..8f208d99e02 100644 --- a/packages/paperclip-runner/src/evals/runnerd-artifact.test.ts +++ b/packages/paperclip-runner/src/evals/runnerd-artifact.test.ts @@ -28,6 +28,11 @@ describe("runnerd artifact metadata", () => { expect(parsePaperclipRunnerdBuildMetadata(valid)).toEqual(valid); }); + it("accepts private host-control metadata without changing ordinary artifact admission", () => { + expect(parsePaperclipRunnerdBuildMetadata({ ...valid, privateHostControlVersion: 1 })) + .toEqual(valid); + }); + it("rejects an unknown binary metadata schema", () => { expect(() => parsePaperclipRunnerdBuildMetadata({ ...valid, schema: "runnerd/v2" })) .toThrow(PaperclipRunnerdArtifactError); From 7840d894faf1f8ed7354c8b050a6462873f5d472 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 6 Nov 2026 08:23:36 -0800 Subject: [PATCH 04/13] fix: retain files after accepted remote copy-back exhausts retries (#19491) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces keep unsaved files bound to the run that wrote them. > - An accepted result can exhaust its copy-back retries before those files are saved. > - Failed-workspace preservation rejects every accepted result, so a stopped writer can keep the workspace held after a newer setup attempt fails. > - This pull request lets the existing owner retain files from that exact stopped remote writer. > - File recovery can then release the hold without applying the old result or repeating its provider work. ## Linked Issues or Issue Description **What happened?** A failed native remote writer had an accepted result and exhausted copy-back retries. Its sandbox was stopped with a valid retained-source receipt. A newer attempt failed on the writer's unsaved workspace. Task Retry could not register the missing file-recovery obligation because preservation returned as soon as it found the old accepted result. **Expected behavior** Task Retry must let the existing recovery owner save the exact retained files. The accepted result, failed attempts and current task must stay intact. The old provider work must not run again. **Steps to reproduce** 1. Finish native work in a remote task workspace and accept its completion result. 2. Exhaust copy-back retries, stop and retain that exact sandbox, and leave the workspace dirty. 3. Let a newer setup attempt fail on the old writer's hold. 4. Use ordinary task Retry. Before this fix, the missing file obligation is never recorded. **Paperclip version or commit** Reproduced against master at `51a65af9b81f3c60f8210047e9a3eeaada7bc155` with isolated PostgreSQL and real files. Related: #19385 preserves accepted copy-back interrupted by Stop. #19422 admits saved input after verified recovery. #19464 and #19466 repair preparation and Retry continuation. This change covers an older remote writer whose accepted copy-back has exhausted its retries. ## What Changed - Admit only terminal, exhausted accepted copy-back on a failed remote writer with a finished run and no live controller or finalizer. - Require the exact stop-and-retain receipt, accepted completion contract and current dirty workspace binding. Keep native-sync sources on their existing export path. - Use the existing file-preservation recorder and worker. Add no new recovery worker, schema, polling or replay path. - Cover bound and orphaned entry points, reusable leases, changed proof and the ordinary HTTP Retry journey. Document the recovery behavior. ## Verification - Causal baseline: the corrected bound-writer and orphaned-lease tests both fail without the source change and pass with it. - Assembled acceptance: authenticated HTTP Retry records the old writer's missing obligation; the real recovery service saves its files; concurrent continuation calls queue exactly one successor for the newer failed setup. PostgreSQL, filesystem, tar, Git and the API/service path are real. Provider transport is controlled, and no model turn is dispatched. - `pnpm check:module-boundaries`, `pnpm -r typecheck`, `pnpm build` and `git diff --check` pass. - Four full affected recovery suites pass: **659 tests**. Command: `pnpm --dir server exec vitest run --config vitest.config.ts src/services/workspace-recovery.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-cancelled-workspace-preservation.test.ts src/services/native-runtime/native-workspace-export-retry.test.ts`. This includes the final 16 new regression, refusal and assembled journey cases. - `pnpm verify --explain` completed. It selected broad transitive coverage; this is planning evidence only. Hosted CI will supply the remaining broad coverage. - Dependency cost: extra indexed source/result reads run only for this exact terminal retained-result path, inside the existing bounded transaction. Existing notifications activate the existing recovery owner. No idle scan is added. - Fixture failures are accounted for: the first fixture used an active-only stop helper on a released lease. The corrected fixture produced the causal baseline above. One negative fixture tried to remove a contract while an accepted result referenced it and failed the database foreign key. That impossible fixture was removed; the positive journey passed in the same run. Neither fixture failure is reported as a product regression or a passing full run. - Hosted CI is fully green at the reviewed commit: 54 successful checks and 3 intentional skips, with no incomplete or failed checks. Greptile is 5/5 with no unresolved threads. - Hosted rollout and a live task continuation remain pending. Local API admission does not prove a deployed model run. ## Risks This expands file preservation for a narrow terminal failure. Wrong source identity could retain unrelated files; unchanged writer, epoch, location, provider lease, stop receipt and accepted contract checks fail closed. Active ownership, unsafe archives, rejected results and native-sync sources stay on their existing paths. Existing recovery owns physical source locking, copy verification and bounded retries. It preserves unknown action outcomes and leaves the accepted result uncommitted. There is no migration or configuration change. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning effort. Context window size is unavailable. Used repository tools, isolated tests and code execution. No delegated agents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/execution-semantics.md | 10 +- .../native-failed-workspace-recovery.ts | 23 +++- .../src/services/workspace-recovery.test.ts | 122 +++++++++++++++++- 3 files changed, 149 insertions(+), 6 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index c405b8be27f..14dec9fe6ca 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -548,8 +548,14 @@ an active or ephemeral one. It records the save obligation for the exact writer run while the workspace is still bound to that writer. A Stop during an accepted result's copy-back retries ends that copy-back, and no finalization exports those files afterwards. Preservation admits that writer as well and -leaves the accepted result uncommitted. Other accepted results keep their own -export repair path. Paperclip's workspace recovery then saves those files, which +leaves the accepted result uncommitted. A failed remote writer whose accepted +result exhausted its copy-back retries can also retain its files through this +path. It needs the exact stopped sandbox receipt, accepted completion contract, +and unchanged dirty workspace binding. This works even when a newer setup +attempt has failed on the old writer's hold. File recovery preserves both +attempts and the accepted result; it does not apply that result or repeat the +provider work. Other accepted results keep their own export repair path. +Paperclip's workspace recovery then saves those files, which releases the hold. It never replays the stopped run. While that save is in progress, its last progress counts as the ended writer's activity for the holder cutoff. A waiter keeps waiting instead of failing setup. The save's retry budget bounds that wait. A save that exhausts diff --git a/server/src/services/native-runtime/native-failed-workspace-recovery.ts b/server/src/services/native-runtime/native-failed-workspace-recovery.ts index 9ec7ac330fe..19b87205a18 100644 --- a/server/src/services/native-runtime/native-failed-workspace-recovery.ts +++ b/server/src/services/native-runtime/native-failed-workspace-recovery.ts @@ -4,7 +4,7 @@ import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm"; -import { environmentLeases, executionWorkspaces, heartbeatRuns, workspaceRestoreFailureCodesSql, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { environmentLeases, executionWorkspaces, heartbeatRuns, workspaceRestoreFailureCodesSql, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js"; import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js"; import { logActivity } from "../activity-log.js"; @@ -102,7 +102,10 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: Pick Date.now() + || run.runnerProfileJson?.nativeWorkspaceSync != null || !remote.success || remote.data.state !== "prepared" + || !hasConfirmedSandboxStopAndRetain(source) + || !await preparedRemoteWorkspaceSource(tx as unknown as Db, source)) return; + const [accepted] = await tx.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.companyId, run.companyId), eq(nativeRunResults.issueId, issueId), + eq(nativeRunResults.runId, run.id), eq(nativeRunResults.id, coordinator.resultId!), + eq(nativeRunResults.completionContractId, run.completionContractId), eq(nativeRunResults.schemaStatus, "accepted"), + )).limit(1); + if (!accepted) return; + } // A stopped reusable sandbox is retained only with its exact stop receipt // while the workspace is still bound to this writer. Otherwise record nothing. - if (source.status !== "active" && reusableLeasePolicies.includes(source.leasePolicy) && (!hasConfirmedSandboxStopAndRetain(source) + if (!exhaustedAcceptedCopyback && source.status !== "active" && reusableLeasePolicies.includes(source.leasePolicy) && (!hasConfirmedSandboxStopAndRetain(source) || !await preparedRemoteWorkspaceSource(tx as unknown as Db, source))) return; // Ownership settlement still applies to workspace-less external sessions. // Their Runner descriptor is not evidence that user files need exporting. diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index 20f7b40be8e..1a8b87ca3a9 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -33,7 +33,7 @@ import { workspaceOperationService } from "./workspace-operations.js"; import { instanceSettingsService } from "./instance-settings.js"; import { projectTaskInterruption, taskInterruptionPresentation } from "./task-interruption-presentation.js"; import * as nativeWorkspaceSync from "./native-runtime/native-workspace-sync.js"; -import { registerCancelledWorkspacePreservationWork } from "./native-runtime/native-failed-workspace-recovery.js"; +import { registerCancelledWorkspacePreservationWork, preserveTaskRemoteWorkspaceWriter, preserveOrphanedNativeWorkspace } from "./native-runtime/native-failed-workspace-recovery.js"; import { GitShallowMetadataError } from "@paperclipai/adapter-utils/git-workspace-sync"; import { readSystemRecovery } from "./system-recovery-state.js"; import { adoptSystemRecovery, updateSystemRecovery } from "./system-recovery.js"; @@ -974,6 +974,126 @@ describe("automatic retained workspace recovery", () => { return { ...f, ...(sourceLayout === "published" ? { remote: remoteCwd } : {}), source, workspaceId, resultId, git, commit }; } + async function exhaustedAcceptedRemoteWriter(leasePolicy: "ephemeral" | "reuse_by_environment" = "ephemeral") { + const f = await acceptedRemoteExport(); + await db.update(issueRecoveryActions).set({ status: "cancelled", cause: "native_workspace_sync_out_failed", + evidence: { runId: f.runId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } }, + }).where(eq(issueRecoveryActions.id, f.actionId)); + await db.update(heartbeatRuns).set({ errorCode: "native_workspace_sync_out_retry_exhausted" }) + .where(eq(heartbeatRuns.id, f.runId)); + const [lease] = await db.update(environmentLeases).set({ leasePolicy }) + .where(eq(environmentLeases.id, f.lease.id)).returning(); + const stopping = (await prepareStoppedSandboxForRecovery(db, lease))!; + const stopped = (await settleStopOnlyCleanup(db, stopping, { + attemptId: String(stopping.metadata!.pendingCleanupAttemptId), + receipt: { providerLeaseId: f.providerLeaseId, state: "stopped" }, + }))!; + const source = await run(f.runId); + const [preparation] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, + issueId: f.issueId, runtimeMode: "legacy", status: "failed", executionStage: "preparing", + startedAt: new Date(), finishedAt: new Date(), errorCode: "setup_failed", + error: "The workspace still has an unfinished writer or save operation.", + contextSnapshot: { issueId: f.issueId, executionWorkspaceId: f.workspaceId }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).returning(); + return { ...f, stopped, preparation }; + } + + it.each(["bound writer", "orphaned lease", "reusable writer"] as const)("retains an exhausted accepted remote copy-back through its %s", async entry => { + const f = await exhaustedAcceptedRemoteWriter(entry === "reusable writer" ? "reuse_by_environment" : "ephemeral"); + const [result] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)); + const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + const historical = await action(f.actionId); + if (entry === "orphaned lease") expect(await preserveOrphanedNativeWorkspace(db, f.stopped)).toBe(true); + else await preserveTaskRemoteWorkspaceWriter(db, f.companyId, f.issueId); + expect((await run(f.runId)).resultJson).toMatchObject({ workspaceRestoreFailure: "restore_failed", + workspaceRestoreRecovery: { leaseIds: [f.lease.id] } }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]) + .toMatchObject({ phase: "terminal_failure", resultId: result.id, leaseOwner: null }); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(result); + expect(await action(f.actionId)).toEqual(historical); + expect(await run(f.preparation.id)).toEqual(f.preparation); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toEqual(task); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(2); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); + expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n"); + }, 30_000); + + it.each(["active-finalizer", "retryable", "other-failure", "live-controller", "unfinished-run", "missing-stop", + "rejected-result", "new-writer", "saved-workspace", "wrong-epoch", "native-sync", "source-binding"] as const)( + "keeps accepted remote copy-back held when its %s proof changes", async gate => { + const f = await exhaustedAcceptedRemoteWriter(); + const source = await run(f.runId); + if (gate === "active-finalizer") await db.update(nativeRunFinalizations).set({ leaseOwner: "live-finalizer" }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + if (gate === "retryable") await db.update(nativeRunFinalizations).set({ phase: "retryable_failure" }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + if (gate === "other-failure") await db.update(nativeRunFinalizations).set({ failureCode: "native_workspace_sync_out_unsafe_archive" }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + if (gate === "live-controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }) + .where(eq(heartbeatRuns.id, f.runId)); + if (gate === "unfinished-run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, f.runId)); + if (gate === "missing-stop") await db.update(environmentLeases).set({ metadata: { ...f.stopped.metadata, + remoteExecutionTermination: null } }).where(eq(environmentLeases.id, f.lease.id)); + if (gate === "rejected-result") await db.update(nativeRunResults).set({ schemaStatus: "rejected" }) + .where(eq(nativeRunResults.id, f.resultId)); + if (gate === "new-writer" || gate === "saved-workspace") { + const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)); + const authority = readRemoteWorkspaceAuthority(workspace.metadata)!; + await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: { ...authority, + writerRunId: gate === "new-writer" ? randomUUID() : null, dirty: gate === "new-writer" } } }) + .where(eq(executionWorkspaces.id, f.workspaceId)); + } + if (gate === "wrong-epoch") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...source.runnerProfileJson, + remoteWorkspace: { ...(source.runnerProfileJson!.remoteWorkspace as object), epoch: 99 } } }).where(eq(heartbeatRuns.id, f.runId)); + if (gate === "native-sync") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...source.runnerProfileJson, + nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3" } } }).where(eq(heartbeatRuns.id, f.runId)); + if (gate === "source-binding") await db.update(environmentLeases).set({ metadata: { ...f.stopped.metadata, remoteCwd: "/other" } }) + .where(eq(environmentLeases.id, f.lease.id)); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)); + const [result] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)); + await preserveTaskRemoteWorkspaceWriter(db, f.companyId, f.issueId); + expect((await run(f.runId)).resultJson).not.toHaveProperty("workspaceRestoreRecovery"); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0]).toEqual(lease); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(result); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId))).toHaveLength(1); + expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n"); + }, 30_000); + + it("repairs an older accepted remote writer before one HTTP Retry continues a newer failed preparation", async () => { + const f = await exhaustedAcceptedRemoteWriter(); + const [result] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.preparation.agentId, status: "running" }); + const receipt = (await request(receiptOnlyRecoveryApp()).post(`/api/issues/${f.issueId}/retry`) + .send({ observedRunId: f.preparation.id, idempotencyKey: randomUUID() }).expect(202)).body; + expect((await run(f.runId)).resultJson).toHaveProperty("workspaceRestoreRecovery"); + const records = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId)); + const preservation = records.find(record => record.id !== f.actionId && record.evidence.runId === f.runId + && record.evidence.workspaceRestoreRecovery)!; + expect(preservation).toBeDefined(); + const input = { ...f.input, actionId: preservation.id }; + await adoptSystemRecovery(db, input); + await recoverRetainedWorkspace(input); + expect(readWorkspaceRecovery((await action(preservation.id)).evidence)?.status).toBe("recovered"); + const heartbeat = heartbeatService(db); + await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId), + heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]); + const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`)); + expect(successors).toHaveLength(1); + expect(successors[0]).toMatchObject({ status: "queued", contextSnapshot: { previousRunId: f.preparation.id } }); + const recovered = await run(f.runId), reference = recovered.runnerProfileJson!.remoteWorkspace as { snapshotId: string }; + const saved = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId, + snapshotId: reference.snapshotId }); + try { expect(await readFile(path.join(saved.root, "saved.txt"), "utf8")).toBe("finished work\n"); } + finally { await saved.cleanup(); } + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(result); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]) + .toMatchObject({ phase: "terminal_failure", resultId: result.id }); + expect(await run(f.preparation.id)).toEqual(f.preparation); + }, 30_000); + async function pressuredToolCache(pressure: "transfer" | "sandbox-write" = "transfer") { const f = await acceptedRemoteExport(); const runner = path.join(f.remote, ".paperclip-runtime", "paperclip-runner"); From e1332b997f457b8c032f6a4b64fa75e4a66c2bc7 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 08:25:43 -0800 Subject: [PATCH 05/13] fix(plugins): align tool run context and interaction input types (#19493) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins add tools and issue-thread interactions through `@paperclipai/plugin-sdk` > - The SDK types are the public contract between the host and plugin authors > - Two parts of that contract do not match what the host does at runtime > - First, `ToolRunContext.projectId` says it is always a project UUID. The tool gateway sends an empty string when a run has no project. The direct tool route rejects such runs with 400 > - Second, three interaction helpers ask for the parsed output type, so plugin authors must pass fields that the host fills in with defaults > - This pull request makes the types and both tool paths agree with the runtime > - The benefit is that agents with no project can run plugin tools, and plugin authors can omit the fields that have defaults ## Linked Issues or Issue Description Fixes #3273 Related: #3274 removes the `projectId` requirement from the route. That change alone is not enough today: the route now also looks up the project to check its company, and that lookup needs a value. This pull request also skips that lookup when the run has no project. It gives plugins one value for "no project" on both tool paths. The new test typecheck in #18739 and #18839 also listed both problems as possible product problems. ## What Changed - `server/src/routes/plugins.ts`: `POST /api/plugins/tools/execute` accepts a `runContext.projectId` that is missing, `null`, or an empty string. It passes an empty string to the plugin, the same value that the tool gateway sends. It checks the project's company only when a project ID is present. A `projectId` that is not a string returns 400. - `server/src/routes/openapi.ts`: the request schema marks `runContext.projectId` as optional and nullable. - `packages/plugins/sdk/src/types.ts`: - The `ToolRunContext.projectId` comment says the value is an empty string when the run has no project. - `suggestTasks`, `requestConfirmation` and `requestCheckboxConfirmation` take `CreateIssueThreadInteractionInput`, as `askUserQuestions` already does. The host parses this input with the schema, so the defaults apply. - `packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts`: the example passes `undefined` instead of an empty project ID to `issues.create`. - `doc/plugins/PLUGIN_SPEC.md` (section 13.10): documents the empty project ID. - Tests: - `server/src/__tests__/plugin-routes-authz.test.ts`: tests a run with an empty, missing, or `null` project ID, which skips the project lookup. Also tests that a project ID that is not a string returns 400. - `packages/plugins/sdk/tests/testing-actions.test.ts`: removes the cast that the output type required. ## Verification - `pnpm exec vitest run server/src/__tests__/plugin-routes-authz.test.ts server/src/__tests__/openapi-routes.test.ts packages/plugins/sdk/tests/testing-actions.test.ts`: 3 files, 81 tests pass. - `cd server && pnpm typecheck`: passes. This includes the source layers and the three test-tree shards. - `tsc --noEmit` passes for `packages/plugins/sdk` (`tsconfig.json` and `tsconfig.test.json`) and for the kitchen-sink example. - Not run locally: the full `pnpm test:run` and the root `pnpm build`. Required CI covers them. ## Risks - Low risk. The direct tool route now accepts calls that it rejected before. It still checks that the agent and the run belong to the company. With no project ID, there is no project to check. - A plugin that reads `runCtx.projectId` can get an empty string on both tool paths. The tool gateway already sent this value before this change. Only the documentation is new. - The SDK parameter types are wider. Existing plugin code still compiles. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/plugins/PLUGIN_SPEC.md | 1 + .../plugin-kitchen-sink-example/src/worker.ts | 3 +- packages/plugins/sdk/src/types.ts | 12 +-- .../plugins/sdk/tests/testing-actions.test.ts | 6 +- .../src/__tests__/plugin-routes-authz.test.ts | 88 +++++++++++++++++++ server/src/routes/openapi.ts | 2 +- server/src/routes/plugins.ts | 37 +++++--- 7 files changed, 126 insertions(+), 23 deletions(-) diff --git a/doc/plugins/PLUGIN_SPEC.md b/doc/plugins/PLUGIN_SPEC.md index af3b060021b..d028c2867b1 100644 --- a/doc/plugins/PLUGIN_SPEC.md +++ b/doc/plugins/PLUGIN_SPEC.md @@ -682,6 +682,7 @@ The host provides: - tool name (without plugin namespace prefix) - parsed parameters matching the tool's declared schema - run context: agent ID, run ID, company ID, project ID + (the project ID is an empty string when the run has no project) The worker executes the tool and returns a typed result (string content, structured data, or error). diff --git a/packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts b/packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts index cff71c53767..91b3c0cbaa6 100644 --- a/packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts +++ b/packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts @@ -897,7 +897,8 @@ async function registerToolHandlers(ctx: PluginContext): Promise { } const issue = await ctx.issues.create({ companyId: runCtx.companyId, - projectId: runCtx.projectId, + // The run context has an empty project ID when the run has no project. + projectId: runCtx.projectId || undefined, title: payload.title, description: payload.description, }); diff --git a/packages/plugins/sdk/src/types.ts b/packages/plugins/sdk/src/types.ts index 7bd956f0fe0..e6be5116e5d 100644 --- a/packages/plugins/sdk/src/types.ts +++ b/packages/plugins/sdk/src/types.ts @@ -36,7 +36,6 @@ import type { AskUserQuestionsInteraction, RequestConfirmationInteraction, RequestCheckboxConfirmationInteraction, - CreateIssueThreadInteraction, CreateIssueThreadInteractionInput, PluginIssueOriginKind, IssueSurfaceVisibility, @@ -270,7 +269,10 @@ export interface ToolRunContext { runId: string; /** UUID of the company the run belongs to. */ companyId: string; - /** UUID of the project the run belongs to. */ + /** + * UUID of the project the run belongs to, or an empty string when the run + * has no project. Check the value before you use it as a project ID. + */ projectId: string; } @@ -1600,7 +1602,7 @@ export interface PluginIssuesClient { ): Promise; suggestTasks( issueId: string, - interaction: Omit, "kind">, + interaction: Omit, "kind">, companyId: string, options?: { authorAgentId?: string }, ): Promise; @@ -1612,13 +1614,13 @@ export interface PluginIssuesClient { ): Promise; requestConfirmation( issueId: string, - interaction: Omit, "kind">, + interaction: Omit, "kind">, companyId: string, options?: { authorAgentId?: string }, ): Promise; requestCheckboxConfirmation( issueId: string, - interaction: Omit, "kind">, + interaction: Omit, "kind">, companyId: string, options?: { authorAgentId?: string }, ): Promise; diff --git a/packages/plugins/sdk/tests/testing-actions.test.ts b/packages/plugins/sdk/tests/testing-actions.test.ts index 203a9851d9d..987bd37e5f2 100644 --- a/packages/plugins/sdk/tests/testing-actions.test.ts +++ b/packages/plugins/sdk/tests/testing-actions.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { createTestHarness } from "../src/testing.js"; -import type { PaperclipPluginManifestV1, PluginIssuesClient } from "../src/types.js"; +import type { PaperclipPluginManifestV1 } from "../src/types.js"; const manifest = { id: "paperclip.test-actions", @@ -136,8 +136,8 @@ describe("createTestHarness issue interactions", () => { minSelected: 1, maxSelected: 2, }, - // Omits the fields the schema defaults, so the assertions below cover those defaults. - } as Parameters[1], + // Omits the fields the schema defaults. The input type allows that. + }, "company-1", { authorAgentId: "agent-1" }, ); diff --git a/server/src/__tests__/plugin-routes-authz.test.ts b/server/src/__tests__/plugin-routes-authz.test.ts index 4cd0828c79e..49b08a9efea 100644 --- a/server/src/__tests__/plugin-routes-authz.test.ts +++ b/server/src/__tests__/plugin-routes-authz.test.ts @@ -673,6 +673,94 @@ describe("plugin tool and bridge authz", () => { ); }); + it("allows tool execution for a run with no project and skips the project lookup", async () => { + const executeTool = vi.fn().mockResolvedValue({ content: "ok" }); + const { app } = await createApp(boardActor(), {}, { + // Only the agent and run lookups. A project lookup would get no row and fail. + db: createSelectQueueDb([ + [{ companyId: companyA }], + [{ companyId: companyA, agentId: agentA }], + ]), + toolDeps: { + toolDispatcher: { + listToolsForAgent: vi.fn(), + getTool: vi.fn(() => ({ name: "paperclip.example:search" })), + executeTool, + }, + }, + }); + + const res = await request(app) + .post("/api/plugins/tools/execute") + .send({ + tool: "paperclip.example:search", + parameters: {}, + runContext: { agentId: agentA, runId: runA, companyId: companyA, projectId: "" }, + }); + + expect(res.status).toBe(200); + expect(executeTool).toHaveBeenCalledWith( + "paperclip.example:search", + {}, + { agentId: agentA, runId: runA, companyId: companyA, projectId: "" }, + ); + }); + + it.each([ + ["missing", {}], + ["null", { projectId: null }], + ])("passes an empty projectId to the plugin when runContext.projectId is %s", async (_label, project) => { + const executeTool = vi.fn().mockResolvedValue({ content: "ok" }); + const { app } = await createApp(boardActor(), {}, { + db: createSelectQueueDb([ + [{ companyId: companyA }], + [{ companyId: companyA, agentId: agentA }], + ]), + toolDeps: { + toolDispatcher: { + listToolsForAgent: vi.fn(), + getTool: vi.fn(() => ({ name: "paperclip.example:search" })), + executeTool, + }, + }, + }); + + const res = await request(app) + .post("/api/plugins/tools/execute") + .send({ + tool: "paperclip.example:search", + parameters: {}, + runContext: { agentId: agentA, runId: runA, companyId: companyA, ...project }, + }); + + expect(res.status).toBe(200); + expect(executeTool).toHaveBeenCalledWith( + "paperclip.example:search", + {}, + { agentId: agentA, runId: runA, companyId: companyA, projectId: "" }, + ); + }); + + it("rejects tool execution when runContext.projectId is not a string", async () => { + const executeTool = vi.fn(); + const { app } = await createApp(boardActor(), {}, { + toolDeps: { + toolDispatcher: { listToolsForAgent: vi.fn(), getTool: vi.fn(), executeTool }, + }, + }); + + const res = await request(app) + .post("/api/plugins/tools/execute") + .send({ + tool: "paperclip.example:search", + parameters: {}, + runContext: { agentId: agentA, runId: runA, companyId: companyA, projectId: 42 }, + }); + + expect(res.status).toBe(400); + expect(executeTool).not.toHaveBeenCalled(); + }); + it.each([ ["legacy data", "post", `/api/plugins/${pluginId}/bridge/data`, { key: "health" }], ["legacy action", "post", `/api/plugins/${pluginId}/bridge/action`, { key: "sync" }], diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 09f23d3eb92..98d52aeff80 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -9586,7 +9586,7 @@ registry.registerPath({ agentId: z.string(), runId: z.string(), companyId: z.string(), - projectId: z.string(), + projectId: z.string().nullable().optional(), }), }), ), diff --git a/server/src/routes/plugins.ts b/server/src/routes/plugins.ts index 9c1a82df448..0479f3466c6 100644 --- a/server/src/routes/plugins.ts +++ b/server/src/routes/plugins.ts @@ -482,8 +482,8 @@ interface PluginToolExecuteRequest { tool: string; /** Parameters matching the tool's declared JSON Schema. */ parameters?: unknown; - /** Agent run context. */ - runContext: ToolRunContext; + /** Agent run context. projectId may be omitted or null when the run has no project. */ + runContext: Omit & { projectId?: string | null }; } /** @@ -862,13 +862,15 @@ export function pluginRoutes( return '"runContext.runId" does not belong to "runContext.agentId"'; } - const [project] = await db - .select({ companyId: projects.companyId }) - .from(projects) - .where(eq(projects.id, runContext.projectId)) - .limit(1); - if (!project || project.companyId !== runContext.companyId) { - return '"runContext.projectId" does not belong to "runContext.companyId"'; + if (runContext.projectId) { + const [project] = await db + .select({ companyId: projects.companyId }) + .from(projects) + .where(eq(projects.id, runContext.projectId)) + .limit(1); + if (!project || project.companyId !== runContext.companyId) { + return '"runContext.projectId" does not belong to "runContext.companyId"'; + } } return null; @@ -1060,7 +1062,7 @@ export function pluginRoutes( return; } - const { tool, parameters, runContext } = body; + const { tool, parameters, runContext: requestedRunContext } = body; // Validate required fields if (!tool || typeof tool !== "string") { @@ -1068,18 +1070,27 @@ export function pluginRoutes( return; } - if (!runContext || typeof runContext !== "object") { + if (!requestedRunContext || typeof requestedRunContext !== "object") { res.status(400).json({ error: '"runContext" is required and must be an object' }); return; } - if (!runContext.agentId || !runContext.runId || !runContext.companyId || !runContext.projectId) { + if (!requestedRunContext.agentId || !requestedRunContext.runId || !requestedRunContext.companyId) { res.status(400).json({ - error: '"runContext" must include agentId, runId, companyId, and projectId', + error: '"runContext" must include agentId, runId, and companyId', }); return; } + const requestedProjectId: unknown = requestedRunContext.projectId; + if (requestedProjectId != null && typeof requestedProjectId !== "string") { + res.status(400).json({ error: '"runContext.projectId" must be a string when present' }); + return; + } + // A run can have no project. Plugins then get an empty projectId, the same + // value the tool gateway sends, so both paths give plugins one contract. + const runContext: ToolRunContext = { ...requestedRunContext, projectId: requestedProjectId ?? "" }; + assertCompanyAccess(req, runContext.companyId); const scopeError = await validateToolRunContextScope(runContext); if (scopeError) { From fcf5dc2395ffc9eaa53b25807cd7877d50b37e59 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 08:28:25 -0800 Subject: [PATCH 06/13] fix(ai-connections): reject unknown binding modes before grant checks (#19494) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run on AI accounts that people connect as AI connections > - Before a run, `aiConnectionService.select` resolves the agent's binding to one account grant and checks who may use it > - The grant-kind checks cover only the `responsible_user`, `shared` and `delegated` modes > - Every caller parses the binding first, so no other mode reaches `select` today. But if one did, it would skip the grant-kind checks > - This pull request rejects any other mode before `select` reads any data > - The benefit is that an authorization check cannot be skipped through a bad binding in the future ## Linked Issues or Issue Description No issue exists. The new test typecheck in #18739 showed that a test passed `mode: "connection"`, which is not a valid mode. #18739 fixed the test and listed the production question. **What existing behavior does this improve?** `select` in `server/src/services/ai-connections.ts` reads `binding.grantId` for every mode except `responsible_user`. It checks the grant kind only when the mode is `shared` or `delegated`. **Current behavior** All runtime callers parse the binding with `aiRuntimeConnectionBindingSchema` or `aiConnectionBindingSchema` first (`heartbeat.ts`, `agent-harness-verification.ts`, `routes/agents.ts`). So no live path sends an unknown mode. If a future caller skipped parsing, an unknown mode would pass with no grant-kind check. **Proposed behavior** `select` rejects an unknown mode with the existing `ai_connection_incompatible` failure (422), before it reads any data. **Reason and benefit** A guard in the owner of the check is cheaper than trusting every future caller. **Breaking changes** None. Valid bindings behave the same. ## What Changed - `server/src/services/ai-connections.ts`: `select` throws `ai_connection_incompatible` for any mode other than `responsible_user`, `shared` or `delegated`. - `server/src/__tests__/ai-connection-binding-mode.test.ts`: new test. It sends `mode: "connection"` and expects a 422 with no database read. ## Verification - `pnpm exec vitest run server/src/__tests__/ai-connection-binding-mode.test.ts`: passes. Without the service change, the same test fails because `select` reads the database. - `pnpm exec vitest run server/src/__tests__/ai-connections.test.ts server/src/__tests__/agent-hire-ai-connections.test.ts server/src/__tests__/ai-connection-binding-mode.test.ts`: 3 files, 243 tests pass. - `cd server && pnpm typecheck`: passes. This includes the source layers and the three test-tree shards. - Not run locally: the full `pnpm test:run` and the root `pnpm build`. Required CI covers them. ## Risks - Low risk. The new check only rejects bindings that the schemas already reject. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../ai-connection-binding-mode.test.ts | 34 +++++++++++++++++++ server/src/services/ai-connections.ts | 4 +++ 2 files changed, 38 insertions(+) create mode 100644 server/src/__tests__/ai-connection-binding-mode.test.ts diff --git a/server/src/__tests__/ai-connection-binding-mode.test.ts b/server/src/__tests__/ai-connection-binding-mode.test.ts new file mode 100644 index 00000000000..e3b4a6d0ea6 --- /dev/null +++ b/server/src/__tests__/ai-connection-binding-mode.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import type { AiConnectionBinding } from "@paperclipai/shared"; +import { readAiConnectionConfigurationFailure } from "../services/ai-connection-configuration-failure.js"; +import { aiConnectionService } from "../services/ai-connections.js"; + +describe("AI connection binding mode guard", () => { + it("rejects an unknown binding mode before it reads any connection", async () => { + // Any database read fails the test: the guard must run first. + const select = vi.fn(() => { + throw new Error("unexpected database read"); + }); + const service = aiConnectionService({ select } as unknown as Db); + const binding = { + provider: "openai", + method: "api_key", + mode: "connection", + connectionId: "11111111-1111-4111-8111-111111111111", + grantId: "22222222-2222-4222-8222-222222222222", + } as unknown as AiConnectionBinding; + + const error = await service.select({ + companyId: "33333333-3333-4333-8333-333333333333", + userId: "user-1", + agentId: "44444444-4444-4444-8444-444444444444", + adapterType: "codex_local", + binding, + }).catch((caught: unknown) => caught); + + expect(error).toMatchObject({ status: 422 }); + expect(readAiConnectionConfigurationFailure(error)).toBe("ai_connection_incompatible"); + expect(select).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/ai-connections.ts b/server/src/services/ai-connections.ts index 3622fc2c1c8..a4081a4b4d7 100644 --- a/server/src/services/ai-connections.ts +++ b/server/src/services/ai-connections.ts @@ -281,6 +281,10 @@ export function aiConnectionService(db: Db) { binding: AiConnectionBinding; }) { const { companyId, userId, agentId, binding } = input; + // The grant-kind checks below cover only these three modes. Callers parse + // the binding first, but an unknown mode must never skip those checks. + if (binding.mode !== "responsible_user" && binding.mode !== "shared" && binding.mode !== "delegated") + throw aiConnectionConfigurationFailure("ai_connection_incompatible", "Select a supported AI connection mode"); if ( !isAiConnectionCompatible( binding, From 819420a60afa1bfeadcddd8d1c11c5edc306fe3a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 10:33:51 -0600 Subject: [PATCH 07/13] fix(runner): give a cold OpenCode restore the 60 s admission budget (#19487) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode agents run through a local runner. After a control-plane restart, a replacement runner restores the saved provider session. > - The replacement runner does run preparation, `opencode serve` startup and session resume inside one `run.attach` command. > - The controller waits only 30 s for that command. A fresh start gets separate bounds for `run.prepare` and `session.open` for the same work. > - On a loaded host, a cold OpenCode restore took 10-38 s. When it took more than 30 s, the continuation failed with `runner_local_connect_failed: PRP command run.attach timed out`. > - This pull request gives a cold OpenCode restore the same 60 s cold-admission budget that Pi already uses. > - The benefit is that a task that waited on a question across a restart continues when the user answers, without a failed run. ## Linked Issues or Issue Description **What happened?** A native OpenCode agent asked a structured question. The control plane restarted while the task waited. The user answered. The continuation run failed with `native_session_recovery_failed: Error: runner_local_connect_failed: PRP command run.attach timed out`, about 35 s after the runner transport connected. This happened in 1 of 2 runs of the `local-session-integrity.runner-opencode.local.structured-question-restart-resume` runner E2E cell. **Expected behavior** The continuation restores the saved OpenCode session and finishes the task. A restart must not cause a failed run. **Steps to reproduce** 1. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id local-session-integrity.runner-opencode.local.structured-question-restart-resume` on a host with other load. 2. The case asks a question, restarts Paperclip and answers the question. 3. When the cold restore takes more than 30 s, the continuation run fails at `run.attach`. **Paperclip version or commit** `master` at 0252421d5c. Evidence from the runner spans (`runner.session.startup` of the continuation run): 31.0 s (passed, close to the bound), 38.0 s (failed), 14.0 s and 10.1 s (passed). The first (fresh) runs took 4.9-13.7 s. ## What Changed - `coldAdmissionTimeoutMs` returns 60 s for a cold OpenCode runner process. Live adoption (a warm runner) keeps 30 s. Other providers keep 30 s, except Pi, which already had 60 s. - The unit test for the cold admission budget covers the OpenCode cold and live cases. ## Verification - Red before the fix: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "cold process admission budget"` failed on `coldAdmissionTimeoutMs("opencode", undefined, true)` (30000, expected 60000). - Green after the fix: the same command passes. - Live runner E2E on macOS arm64 with OpenRouter (`openrouter/deepseek/deepseek-v4-flash-0731`), local runner, at the fixed revision. Command: `pnpm test:e2e:runner -- --max-automatic-retries 0 --id local-session-integrity.runner-opencode.local.structured-question-restart-resume`. - Before the fix: 1 of 2 campaign runs failed at `run.attach` (restore 38.0 s). The passing run restored in 31.0 s. Two more local runs passed (restore 14.0 s and 10.1 s). - After the fix: 4 of 4 runs that reached the restart passed (restore 6.2 s, 11.3 s, 5.1 s and 11.8 s). One more run failed before the restart because the model sent malformed `request_human_input` arguments. That failure is not related to this change. - Limitation: host load was lower during the after-fix runs, so no restore took more than 30 s. The unit test proves the new bound. - Other failures in `runnerd-codex-transport.test.ts` are unchanged on `master` in this local environment (the fake Rust fixture binary is not built locally). CI builds it. ## Risks - Low risk. The change only makes a cold OpenCode admission wait longer before it reports a failure. A runner that is really stuck now fails after 60 s instead of 30 s. - A fresh OpenCode start now shares one absolute 60 s deadline for `session.open` and provider identity, the same as Pi. Fresh OpenCode starts took 5-14 s. - Inner bounds stay in place: the runner still bounds each provider request (30 s) and OpenCode health checks (10 s). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Reasoning mode was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/live/runnerd-codex-transport.test.ts | 6 +++++- .../paperclip-runner/src/live/runnerd-codex-transport.ts | 3 ++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index 87253a57f48..e16292d7641 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -5890,12 +5890,16 @@ describe("cold process admission budget", () => { it("bounds Pi cold startup and recovery at 60 seconds", () => { expect(coldAdmissionTimeoutMs("acpx", "pi", true)).toBe(60_000); }); + it("bounds a cold OpenCode restore at 60 seconds", () => { + expect(coldAdmissionTimeoutMs("opencode", undefined, true)).toBe(60_000); + }); it("preserves live adoption and every other provider at 30 seconds", () => { expect(coldAdmissionTimeoutMs("acpx", "pi", false)).toBe(30_000); + expect(coldAdmissionTimeoutMs("opencode", undefined, false)).toBe(30_000); for (const agent of ["codex", "claude", "cursor", "copilot", "grok", undefined]) { expect(coldAdmissionTimeoutMs("acpx", agent, true)).toBe(30_000); } - for (const provider of ["codex", "opencode", undefined]) { + for (const provider of ["codex", "cursor_cloud", undefined]) { expect(coldAdmissionTimeoutMs(provider, "pi", true)).toBe(30_000); } }); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 551fef7991e..2f19d3aa461 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -8288,7 +8288,8 @@ export function resolveRunnerdCodexSkillInputs( /** Controller admission budget; ordinary command and turn deadlines are independent. */ export function coldAdmissionTimeoutMs(provider: string | undefined, agent: string | undefined, coldProcess: boolean): number { - return coldProcess && provider === "acpx" && agent === "pi" ? 60_000 : 30_000; + if (!coldProcess) return 30_000; + return (provider === "acpx" && agent === "pi") || provider === "opencode" ? 60_000 : 30_000; } export interface RunnerCommandFailure extends Error { From 871a9b31805d07dd4eacf4bedef1c76c3074ba4d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 10:45:52 -0600 Subject: [PATCH 08/13] test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane owns task execution, recovery and Retry for every agent run > - Server restarts were the largest remaining source of stalled tasks in the staging soak campaign > - The soak could only observe restarts that staging happened to perform, so a failure could not be repeated > - This pull request adds a script harness that causes the restarts on purpose and grades each task on fixed rules > - The benefit is a repeatable restart failure with a replay command and an evidence bundle for each fixer ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart test coverage. Today the runner-e2e restart cells test one fixed restart point per cell. They cannot run many tasks at once, cannot choose a random fault time, and cannot replay a failure. **Subsystem affected** Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process code in `tests/runner-e2e/`. Product code does not change. **Current behavior** Restart robustness is checked through a small set of fixed cells. A random restart during a real task has no defined pass rule. **Proposed behavior** A harness starts an isolated control plane and seeds a "Chaos Lab" company. It launches seeded workloads, injects K1, K2 or K4 faults at seeded times, and grades each task on rules R1 to R6 plus a deterministic task oracle. `--replay /` repeats one task. **Reason and benefit** Fixers get a replay command and an evidence bundle for each failure signature. The first smoke wave already found three product signatures (dead Retry after a restart, a false over-budget stop after restarts, and an OpenCode turn killed mid-turn that ends as a provider failure). **Breaking changes** None. The change adds new test files, one entry in the test-project typecheck list and a gitignore entry. ## What Changed - Add `tests/restart-chaos/`: - `supervisor.ts` owns one isolated server. It reuses `replaceController`, the process-tree stopper, the environment isolation helpers, the config seed, the port reservation, the wrapper-owned Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM, restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the controller runtime cache stay across a fault. - `seed.ts` creates the company, encrypted provider secrets, native Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project. The project is a git repository with a Python CLI, one failing test and a four-minute `bench.sh`. - `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire time, hold and nonce for each task. It saves the plan so that `--replay` can run one task again. - `observer.ts` polls each issue and its runs. When a task stops and offers Retry, it sends `POST /api/issues/:id/retry` one time, with the same body as `useTaskRetry.ts`. - `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines workloads A1, B1, B5 and C1 and their oracles. - `evidence.ts` and `ledger.ts` write an evidence bundle for each failure, a JSONL ledger and `SUMMARY.md`. All results go to the gitignored `tests/restart-chaos/results/`. Known provider key values are redacted. - Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing) and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines). These tests need no credentials. - Add `tests/restart-chaos` to the test-project typecheck list (`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json` like `tests/runner-e2e`. The harness runs through direct script invocations, not `package.json` scripts. - After the first Greptile review: - A failed fault or an unexpected server exit now gets one bounded recovery. If the recovery fails, the wave fails and does not hang. - Incomplete cleanup is now a harness error, and the harness keeps the instance. - After the C1 follow-up comment, the observer needs a fresh rest window before it grades the task. - `--replay` now fires the recorded faults that crossed the task, at the same offsets, including the faults that other tasks requested. - A failed workspace-operation read is now a sweep error. It cannot give a clean rest. - R2 now checks that workspace files from before the first fault still exist at rest. - B5 now needs the completion record that `bench.sh` writes for each task. - After the second round of reviews: - An idle `todo` or `in_progress` task with no live run and no named wait is no longer treated as resting. The observer keeps watching it until the task resumes or passes the R5 stall limit. - A failed injected fault now fails the wave with exit 2. - T-oracle commands run agent-editable code. They now run under a macOS Seatbelt sandbox with a scratch `HOME` and no network, and they cannot read the operator's home directory. On other platforms these commands are refused unless the operator explicitly allows them. - R4 now counts every active run. A wake comment may be served again only after a run that did not succeed. - Evidence redaction now covers every environment variable name that ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`. - If the pre-fault snapshot fails, only R2 for that task is left unchecked. The wave continues. - B5 also needs bench progress that the harness observed while it polled. - Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its campaign ledger. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 tests pass. The tests include observer tests with a fake API, a real `bench.sh` run, and a sandbox test that cannot read a canary secret. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including "covers every project under tests/". - A supervisor check with no credentials passed. K1 was ready again in about 6 seconds. K2 with a 3-second hold was ready again in about 9 seconds. Company data stayed readable. - One paid smoke wave ran: `--env local --adapters codex,claude,opencode --workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`. The harness worked from start to end, and all three tasks crossed a restart. The 0/3 result comes from product failures, which the campaign ledger records. A scan found no provider key in the results. - To run a wave: ```sh pnpm --filter @paperclipai/paperclip-runner build:runner-binaries set -a; source ~/.secrets; set +a node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay / ``` ## Risks - Low product risk. The change adds test tooling only and does not change product code. - A wave spends provider money. The company budget hard cap (`--budget-cents`, default 3000) is the only spend limit. The plan's per-wave and per-night refusal is not implemented yet. - Known gaps: - `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The Daytona sandbox sweep for R6 is not implemented yet. - The project requests isolated git-worktree workspaces. A B1 task still ran in the shared fixture repository. Concurrent B1 and B5 tasks can share files. The oracles use marker files for each task. - Only K1, K2 and K4 at seeded random times are available. Boundary triggers (B1 to B16), other faults and the UI probe come in Phase 2. - The paid smoke did not exercise B5 or K4. - On macOS, B1 oracle commands need `python3` outside the operator's home directory, for example `/usr/bin` or Homebrew. On Linux they are refused until a sandbox is added. - The B5 proof is best-effort against a deliberately adversarial agent, because that agent can write to the workspace. - Startup phase timings come from polling `/api/ready`. The server records its own timings only as OpenTelemetry spans. - `pnpm verify` and the CI Vitest jobs do not select these unit tests. The root Vitest projects cover workspace packages only. CI typechecks the files, but the tests run only through the vitest command above. - After the smoke, rest detection and the review fixes changed. A recovery action that the board owns now counts as a rest, and R6 reports it. A paid wave did not run again after these changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code as an agent with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .gitignore | 1 + ...1-06-control-plane-restart-chaos-ledger.md | 55 +++ .../2026-11-06-control-plane-restart-chaos.md | 282 +++++++++++++++ scripts/typecheck-test-projects.mjs | 1 + tests/restart-chaos/README.md | 71 ++++ tests/restart-chaos/api.ts | 31 ++ tests/restart-chaos/config.ts | 108 ++++++ tests/restart-chaos/evidence.ts | 104 ++++++ tests/restart-chaos/grader.test.ts | 198 +++++++++++ tests/restart-chaos/grader.ts | 185 ++++++++++ tests/restart-chaos/ledger.ts | 91 +++++ tests/restart-chaos/main.ts | 188 ++++++++++ tests/restart-chaos/observer.test.ts | 118 +++++++ tests/restart-chaos/observer.ts | 334 ++++++++++++++++++ tests/restart-chaos/oracle-sandbox.test.ts | 49 +++ tests/restart-chaos/oracle-sandbox.ts | 65 ++++ tests/restart-chaos/scheduler.test.ts | 116 ++++++ tests/restart-chaos/scheduler.ts | 100 ++++++ tests/restart-chaos/seed.ts | 119 +++++++ tests/restart-chaos/supervisor.ts | 299 ++++++++++++++++ tests/restart-chaos/timeline.ts | 18 + tests/restart-chaos/tsconfig.json | 13 + tests/restart-chaos/tsconfig.layers.json | 8 + tests/restart-chaos/types.ts | 182 ++++++++++ tests/restart-chaos/vitest.config.ts | 12 + tests/restart-chaos/workloads.ts | 241 +++++++++++++ 26 files changed, 2989 insertions(+) create mode 100644 doc/plans/2026-11-06-control-plane-restart-chaos-ledger.md create mode 100644 doc/plans/2026-11-06-control-plane-restart-chaos.md create mode 100644 tests/restart-chaos/README.md create mode 100644 tests/restart-chaos/api.ts create mode 100644 tests/restart-chaos/config.ts create mode 100644 tests/restart-chaos/evidence.ts create mode 100644 tests/restart-chaos/grader.test.ts create mode 100644 tests/restart-chaos/grader.ts create mode 100644 tests/restart-chaos/ledger.ts create mode 100644 tests/restart-chaos/main.ts create mode 100644 tests/restart-chaos/observer.test.ts create mode 100644 tests/restart-chaos/observer.ts create mode 100644 tests/restart-chaos/oracle-sandbox.test.ts create mode 100644 tests/restart-chaos/oracle-sandbox.ts create mode 100644 tests/restart-chaos/scheduler.test.ts create mode 100644 tests/restart-chaos/scheduler.ts create mode 100644 tests/restart-chaos/seed.ts create mode 100644 tests/restart-chaos/supervisor.ts create mode 100644 tests/restart-chaos/timeline.ts create mode 100644 tests/restart-chaos/tsconfig.json create mode 100644 tests/restart-chaos/tsconfig.layers.json create mode 100644 tests/restart-chaos/types.ts create mode 100644 tests/restart-chaos/vitest.config.ts create mode 100644 tests/restart-chaos/workloads.ts diff --git a/.gitignore b/.gitignore index afa5f1c5db6..d0b463a8b6e 100644 --- a/.gitignore +++ b/.gitignore @@ -70,6 +70,7 @@ diagnostics/ tests/e2e/test-results/ tests/e2e/playwright-report/ tests/runner-e2e/results/ +tests/restart-chaos/results/ .env.runner-e2e.local tests/release-smoke/test-results/ tests/release-smoke/playwright-report/ diff --git a/doc/plans/2026-11-06-control-plane-restart-chaos-ledger.md b/doc/plans/2026-11-06-control-plane-restart-chaos-ledger.md new file mode 100644 index 00000000000..43ebbc312f4 --- /dev/null +++ b/doc/plans/2026-11-06-control-plane-restart-chaos-ledger.md @@ -0,0 +1,55 @@ +# Restart chaos ledger + +Companion to `2026-11-06-control-plane-restart-chaos.md`. One section per wave. +Classes: P product · Q agent quality · T test/harness · E unrelated environment. + +## Phase 0 — 2026-11-06 + +Master: `6cafcaf89b`. Host: macOS, local runner (Daytona image pending a CI build). + +| Check | Result | +| --- | --- | +| `controller-replacement-smoke.ts` (no agent) | PASS, 39 s, Postgres survived replacement | +| Local restart cells, first run | 0/7 — **T**: worktree `node_modules` older than lockfile (UI missing `mdast-util-from-markdown`). Not counted. | +| Local restart cells, rerun | **2/7** (below) | +| Daytona runner image | Blocked. Local build failed: stale Dockerfile lock pin, then the Docker VM ran out of disk. Moved to CI (`runner-full-stack-e2e.yml`, manual dispatch) after PR #19467. | + +### Local baseline rerun (`local-2026-11-06T14-27-42-229Z`) + +| Cell | Result | First error | Fixer | +| --- | --- | --- | --- | +| stock-harness · Codex · continuity-restart | PASS | — | — | +| stock-harness · ACPX Claude · continuity-restart | PASS | — | — | +| everyday-workflows · Codex · recover-controller | FAIL (cleanup_failure) | `kill EPERM` during owned-tree cleanup at the restart | subagent: EPERM | +| everyday-workflows · ACPX Claude · recover-controller | FAIL | Timed out waiting for an active run with saved source | subagent: Claude recover | +| stock-harness · OpenCode · continuity-restart | FAIL | Timed out waiting for the chat turn to settle to waiting | subagent: OpenCode | +| local-session-integrity · OpenCode · question-restart-resume | FAIL | Timed out waiting for the pending user question | subagent: OpenCode | +| completion-updates · Codex · handoff-completion-restart | FAIL | Timed out waiting for the worker to wait while the originating chat is idle | subagent: handoff | + +#### Signatures found + +| ID | Class | Signature | Status | +| --- | --- | --- | --- | +| RC-01 | T | Harness looked for saved source only at the workspace root; ACPX Claude saved `slugify-tool/slugify.py`, so the restart never fired | PR #19471 (open) | +| RC-02 | **P** | ACPX turn active at restart: runner closes it with `acpx_active_turn_recovery_closed` and no error, the control plane reports a generic "Provider turn failed", no continuation, queued follow-up unconsumed (fails R1). Codex has had this handled since #15297; ACPX is excluded by `provider.kind === "codex"` and `driverKind === "codex_app_server"` gates | fixer running | + +| RC-03 | T | macOS `killpg` returns EPERM (not ESRCH) on a zombie-only group; the supervisor rethrew it and aborted the restart. Fixed cell passed 2/2, and native Codex recovery itself works | PR #19470 (open) | +| RC-04 | P (latency) | Graceful shutdown always waits the full 30 s heartbeat finalizer drain when a native run is restart-suspended (`server/src/index.ts` ~L2032, `drainActiveRunExecutions`). Every graceful restart (K2) or deploy takes ≥30 s and, under the harness, escalates to SIGKILL | queued for a fixer | + +| RC-05 | **P** | Dead Retry after restart (fails R3): Retry accepted but no run for >15 min, stuck at `retryWait.reason=execution_recovery`. After the next restart the Retry vanished (`waiting_for_user`, `canRetry=false`). OpenCode t01, Claude t02 | fixer running (with RC-06) | +| RC-06 | **P** | False over-budget stop after restarts: a new issue was blocked "original recovery target is over budget" at 22¢ of 500¢, then a company `hard_stop` with `pendingRunCount=1`, `reservationCents=0`, `unpricedUsagePolicy=block` (`recovery/service.ts` ~L4826). Codex t03 never ran | fixer running (with RC-05) | +| RC-02b | **P** | OpenCode mid-turn kill also ends as a provider failure (t01). Sent to the RC-02 fixer | with RC-02 | + +### Harness smoke `w-20261106T144559-s7` (seed 7, K1 ×3, local, concurrency 2) + +0/3 pass. The harness worked: every failure is product-side (RC-02b, RC-05, RC-06). Branch `worktree-agent-a826af5179b88689b` @ `e864139cec`, not yet a PR. + +Setup note: fresh worktrees need `pnpm install --frozen-lockfile` and a built `paperclip-runnerd` (`pnpm --filter @paperclipai/paperclip-runner build`) before native cells can run. The chaos harness must check this before starting a wave. + +Classification of the remaining cells is pending each fixer's diagnosis. A timeout before the restart point counts against setup or the agent, not restart robustness. + +### Side findings + +- Nightly Runner Full-Stack E2E never builds a Daytona image. Its operating-overhead allowance (`RUNNER_E2E_OPERATING_OVERHEAD_JSON`) expired 2026-11-03, and the nightly sets `needs_daytona=false` anyway. PR #19467 (merged, 7e6d882558) names the error and removes the stale Dockerfile lock default. The nightly stays red until the allowance is renewed (Dotta's call). +- Image build moved to CI: manual dispatch run 42240434011 (`core-compatibility.runner-codex.daytona.message-marker`), 2026-11-06 15:09Z. +- `tests/runner-e2e/workflow-security.test.ts` has 5 local failures in `pr-trusted.yml` and `chat-catalog-testops-case.yml`. Not investigated. diff --git a/doc/plans/2026-11-06-control-plane-restart-chaos.md b/doc/plans/2026-11-06-control-plane-restart-chaos.md new file mode 100644 index 00000000000..42cbc655427 --- /dev/null +++ b/doc/plans/2026-11-06-control-plane-restart-chaos.md @@ -0,0 +1,282 @@ +# Control-plane restart chaos loop: make tasks survive server restarts + +Date: 2026-11-06 +Target: a local, isolated Paperclip control plane with agent runs in Daytona sandboxes +Fix workflow: `fix-failed-paperclip-run` (one session per failure signature) +Predecessor: `2026-11-04-staging-soak-loop.md` (staging soak campaign, PR worktree `paperclip-staging-test-plan-24ca37`) + +## 1. Why this plan exists + +The staging soak campaign (2026-11-04 to 2026-11-06) ran about 430 graded tasks +in waves of about 10. It used a fixed task library (sections A to H), mechanical +oracles (U1 to U9), and a dedupe-by-signature ledger. Each new failure signature +went to its own `fix-failed-paperclip-run` session. The campaign merged 42 fixes, +and the hard-failure rate on test tasks fell from 43% to about 0%. + +The largest remaining source of failures was restarts. Staging restarted up to +about five times an hour. The campaign summary says "frequent restarts stall +in-flight work". Three fixes in this area merged late in the campaign: + +- #19399: lost runner after a restart continues automatically, and Retry is offered. +- #19426: a declined Retry no longer swallows later clicks (the "dead Retry" bug). +- #19431: queued runs start promptly after a restart, and the wait shows a reason. + +The campaign could only observe restarts that staging happened to perform. We +could not choose when the restart came, what kind it was, or what each run was +doing at that moment. This plan reverses that: **we cause the restarts on +purpose**, against a local control plane whose runs execute in Daytona. We run +many tasks at once and repeat until nothing breaks. + +## 2. Success rule (the oracle) + +A run is **restart-crossing** if any part of it overlaps a control-plane fault: +its queued wake, preparation, provider turn, finalization, or a follow-up. Every +restart-crossing task must meet all of the rules below. + +| ID | Rule | Failure means | +| --- | --- | --- | +| **R1** | **Automatic continuation.** If continuing is safe, the task resumes after the server is back, with no user action, and reaches its expected resting state within the task's time budget plus the outage time. | Not robust to restart | +| **R2** | **Durable workspace.** No workspace, durability, or recovery error stays visible at rest. This includes restore, snapshot, finalize, writer-pending, lease mismatch, and stranded lock errors. If such an error appears during recovery, it clears by itself. Files the agent wrote before the fault are present afterwards, and nothing is silently rolled back. | Durability bug | +| **R3** | **Working Retry.** If the task cannot safely continue on its own, it shows exactly one Retry or continue affordance. One click (`POST /api/issues/:id/retry`, and the button in the UI) starts a run that completes the task. "Send a message to continue", a Retry that does nothing, a missing Retry, or a Retry that needs several clicks all fail this rule. | Dead or missing Retry | +| R4 | No duplicate execution. There is at most one active run per issue, no side effect is repeated, and no user message is consumed twice. This is soak oracle U4 and U5. | Replay or duplicate run | +| R5 | No silent stall. A task is never `in_progress` or `queued` with no live run, no queued wake, and no named wait reason for more than 3 minutes after `/api/liveness` reports ready. | Stall | +| R6 | Clean rest. At the end of the wave there is no pending workspace operation, recovery action, orphaned lease, or orphaned Daytona sandbox. | Leak | + +The paid "kill at an arbitrary point" probes (`recover-runner*`, see +`tests/runner-recovery/README.md`) were retired because they demanded +**automatic completion** after any kill. For a turn whose outcome is unknown, +automatic completion is not always safe. R1 and R3 resolve this: when automatic +continuation is not provably safe, the correct result is a Retry that works on +the first click. A random kill therefore always has a defined pass condition. It +is never graded "environment, ignore". + +## 3. Topology + +``` + chaos harness (Node script, no LLM) + ├─ supervisor: owns the server process (start, SIGKILL, SIGTERM, hot restart, outage hold) + ├─ launcher: creates tasks through the board API + ├─ observer: polls issues, runs and run events; grades R1–R6; captures evidence + └─ ledger: one JSONL line per task execution, plus the fault schedule seed + + local control plane (isolated PAPERCLIP_HOME, embedded Postgres, one fixed port) + └─ dials out to runnerd in each Daytona sandbox (runner ingress, port 43127) + + Daytona: one sandbox per run or lease, holding the real runner and provider (Claude, Codex) +``` + +- **Use an isolated instance**, not `~/.paperclip/instances/default`. Chaos kills + and the leftover Postgres state must never touch Dotta's working data. +- **Own the server process.** Reuse the existing supervisor in + `tests/runner-e2e/server.ts` and `controller-outage.ts` + (`replaceController`: stop → wait for unhealthy → hold → start → ready, with a + receipt). It already handles process groups, SIGKILL escalation, and + embedded-Postgres ownership. Do not use `pnpm dev` (watch mode): when its child + dies, the dev runner exits too. +- **Readiness.** Use `/api/liveness` and the startup phases + (`starting → recovering → ready`) to time the outage and the recovery window. + `startup-recovery-timing.ts` logs each phase. +- **Keep across restarts:** the Postgres data and the instance secret key. A + restart that loses either one is a different fault (K11) and is graded on its + own. +- **Concurrency.** Start with 6 concurrent tasks and raise to 12, then 20. The + limits are the Daytona sandbox quota and provider spend. Settle the quota in + Phase 0. + +## 4. Fault menu + +### 4a. What the fault does (K) + +| ID | Fault | How | +| --- | --- | --- | +| K1 | Hard kill, fast return | SIGKILL the server, restart immediately. Hot-restart reconciliation reports `not_requested`. | +| K2 | Graceful stop | SIGTERM (shutdown drain and finalizers run), then restart. | +| K3 | Hot restart | `scripts/request-hot-restart.ts`, then SIGTERM. Runs should be adopted. | +| K4 | Medium outage | Hard kill and hold for 2–5 minutes. This exceeds the 60-second controller lease but stays within the 7-minute runner reconnect grace. | +| K5 | Long outage | Hard kill and hold for 8–20 minutes. This exceeds the reconnect grace, and past 15 minutes Daytona auto-stops idle sandboxes. The runner gives up, so recovery must go through saved work. | +| K6 | Crash loop | Kill again during `recovering`, then a third time just after `ready`. | +| K7 | Kill during the automatic retry | Kill again while the automatic retry or continuation started by R1 is running. | +| K8 | Kill during a user's Retry | Kill between the Retry click (receipt committed) and the start of the new run. | +| K9 | Postgres dies too | SIGKILL embedded Postgres along with the server. | +| K10 | Network partition, no restart | Block the server's outbound traffic to Daytona for 30 seconds to 5 minutes. | +| K11 | Lost local state | Restart with the controller cache and scoped state directory deleted, as on a new container (`remote_reattach_root_missing`). The database is kept. | +| K12 | Rolling deploy | Start a second server on the same database before stopping the first, as in `rolling-deployment-supervisor.ts`. | +| K13 | Restart storm | Restart every 2–4 minutes for an hour with 20 tasks running. | + +### 4b. When the fault hits (B, run boundary) + +Some faults fire at a **random time**: a seeded random generator chooses the +time, so a failure can be replayed. Others are **boundary-triggered**: the +observer watches run events and status and fires the fault at a chosen +transition. Each boundary below lists the run event or state it fires on. + +| ID | Boundary | Fires on | +| --- | --- | --- | +| B1 | Queued, not yet dispatched | Run `queued` | +| B2 | Sandbox acquire or provision | Lease acquire started, sandbox not ready | +| B3 | Workspace restore or prepare | Restore or prepare operation in progress | +| B4 | Runner bootstrap or session open | Native bootstrap events | +| B5 | Mid model turn, streaming | First assistant output event | +| B6 | Mid long tool call | Tool-start event without a tool-end, for example `bench.sh` running about 4 minutes | +| B7 | Waiting on a question or approval | Pending interaction | +| B8 | User message or steering queued during the run | Queued comment exists | +| B9 | Turn done, finalization or sync-out in progress | `workspace_finalize` or sync operation running | +| B10 | Snapshot or archive upload | Snapshot upload begun | +| B11 | Done, before publication | Durable done, reply not yet published (as in the chat-restart cell) | +| B12 | Scheduled-retry backoff | Run `scheduled_retry` | +| B13 | Stop or interrupt settling | Cancel requested, not settled | +| B14 | Parent waiting on delegated children | Parent blocked on children, children running | +| B15 | Routine fire or timer wake | Routine run being created | +| B16 | Reassignment in flight | Assignee changed, old run not ended | + +The `finalization-crash-boundary.ts` and `controller-recovery-observer.ts` +modules in `tests/runner-e2e/` already bind kills to some of these boundaries. +Reuse them; do not write them again. + +### 4c. Workload (W) + +Use a smaller subset of the soak library, chosen so the boundaries above occur +naturally: + +- **A1:** brief written to an issue document. A short run with no repository. +- **A3:** CSV analysis whose answer can be checked exactly. +- **B1:** Python CLI with tests. Workspace files must survive the fault. The + harness runs the tests itself. +- **B2:** fix a failing test, plus a revision follow-up. This exercises workspace + reuse across the fault. +- **B5:** a long `bench.sh` run, which reaches boundary B6. +- **C1:** a comment sent while the run is active (B8). +- **C2:** Stop, then a new direction (B13). +- **A5:** the agent asks a clarifying question (B7). +- **D1:** CEO delegates to children (B14). +- **E1** or **H10:** a routine (B15). +- **C6:** reassignment (B16). + +Each task has a deterministic T-oracle, so a restart cannot hide wrong output. + +### 4d. Adapter (A) + +- **Primary:** native Claude and native Codex on Daytona. +- **Control:** the same tasks with a local runner. This tells apart failures + that are Daytona-specific from failures in control-plane recovery. +- **Later:** other adapters, once the primary pair is green. + +The full cross product is about 13 × 16 × 11 × 2. That is too large to run +exhaustively. Waves therefore sample **pairwise coverage**: every fault paired +with every boundary, and every boundary on every adapter, at least N times. The +remaining slots go to open signatures and random K1 and K13 storms. + +## 5. Harness (`tests/restart-chaos/`) + +The harness is a script, not an LLM, for the same reasons as the soak harness: +deterministic grading and cheap polling. It reuses the runner-e2e process code. +It does not copy that code. + +1. **`config`:** isolated instance path, port, `DAYTONA_API_KEY`, company, + agents, budget, and concurrency. +2. **`seed`:** creates the "Chaos Lab" company, Claude and Codex agents on + Daytona, and the `chaos-sandbox` project with a fixture repository. This + includes `bench.sh` and one failing test. It sets hard budgets. +3. **`supervisor`:** wraps `server.ts` and `controller-outage.ts`, exposing + `kill(kind)`, `hold(ms)`, `start()`, `waitReady()`, `partition(ms)` (for + example a pf or iptables rule, or a proxy in front of runner ingress), and + `wipeLocalState()`. Every action is written to the timeline. +4. **`scheduler`:** a seeded plan of faults and boundaries for each wave. The + seed and the chosen cells are saved, so `--replay /` repeats the + same fault at the same boundary. +5. **`observer`:** polls every 2–5 seconds and evaluates R1–R6 and the + T-oracles. When a task ends in the R3 state, the observer acts as a user + would: it calls Retry once through the API, the same call the UI hook makes + in `useTaskRetry.ts`, and grades the result. +6. **`ui-probe`:** every few waves, a Playwright pass opens one task that is in + the R3 state, checks that the Retry button is visible, clicks it once, and + confirms the task continues. The dead-Retry bug (#19426) was in the UI path, + so checking the API alone is not enough. +7. **`evidence bundle`** (captured at failure time): run IDs and statuses, the + last 200 run events for each run, the server log around the fault, the + startup-phase timings, the recovery claim dispositions + (`claimNativeRestartRecoveries`), workspace operations, leases and their + Daytona sandbox state, the fault timeline, the seed, and the SHA. +8. **`ledger` and summary:** the soak ledger format, plus the fault, boundary + and adapter fields. A small dashboard artifact shows pass rate per cell and + the open signatures. +9. **`sweep`:** at the end of each wave, reconcile Daytona sandboxes against + leases (R6). Then destroy whatever the harness created. + +## 6. Triage and the fix loop + +``` +wave (N tasks × sampled K/B/A cells) + → grade R1–R6 and the T-oracles + → classify failures; signature = component + first causal error + boundary + → new P signature → fixer session running /fix-failed-paperclip-run with the bundle and a replay command + → known signature → add the bundle to the existing record + → fix PR merged → rebase the harness worktree on master, replay that signature 5× plus its neighboring cells +``` + +- **Classes:** the same as the soak campaign. **P** is a product bug, **Q** is + agent quality, **T** is a test or harness defect. **E** (environment) never + applies to the fault we injected. Only an unrelated provider or Daytona outage + counts as E. +- **Fixers start from the replay.** Each fixer gets `--replay`, so the first + step of `fix-failed-paperclip-run` (establish the incident) is already done. + The fixer then writes the smallest red test at the right layer. Usually that + is a server integration test on embedded Postgres, like the six in #19399, or + a `native-runner-restart-*.integration.test.ts`. The fix makes it green. +- **Merge authority:** the same as the soak campaign. Fixer sessions merge their + own PR at Greptile 5/5 with all required checks green, and run `check-pr`. +- **Concurrency:** at most 4 fixer sessions at once. +- **No deploy lag.** Fixes are retested locally as soon as they merge, which is + the main speed advantage over the staging loop. Staging stays the final + confirmation (Phase 4). + +## 7. Phases + +| Phase | Output | Exit | +| --- | --- | --- | +| **0. Baseline** | Isolated instance, Daytona key and quota confirmed, Chaos Lab seeded. Run the existing `everyday-workflows.runner-codex.daytona.recover-controller` cell and `controller-replacement-smoke.ts` as they are today. Also run 10 tasks by hand with a K1 kill each, to measure the starting failure rate. | The baseline failure rate is recorded and the first signatures are filed. | +| **1. Harness MVP** | Supervisor, launcher, observer with R1–R6, ledger and replay. Faults K1, K2 and K4 at random times. Workloads A1, B1, B5 and C1 on Claude and Codex. 6 concurrent tasks. | Two waves run without supervision, and the evidence bundles are good enough for a fixer to start from. | +| **2. Boundary matrix** | Boundary triggers B1–B16 and the full K list except K12 and K13. Pairwise sampling. UI probe. | Every K×B pair has been run at least 3 times. | +| **3. Stress** | Concurrency 12, then 20. K13 storms (an hour of restarts every 2–4 minutes). K12 rolling deploys. Overnight runs. | Finish line (§8). | +| **4. Lock in** | Each fixed signature has a red/green regression test in its PR. Stable cells are promoted to explicit runner-e2e cells (`recover-controller` style) for release qualification. Then one confirmation wave on staging. | Recurring regressions are caught in CI, not by chaos. | + +## 8. Finish line + +At one pinned master SHA: + +1. **500 consecutive restart-crossing task executions** across the sampled + matrix with **zero R1–R6 failures**. At least 20 of them cross K5 (the long + outage) and at least 20 cross K6 or K7 (repeat kills). +2. Every K×B cell has at least 5 passes on both Claude and Codex. +3. Two K13 storm hours at concurrency 20 in which every task reaches its + resting state, and the R3 Retry works on the first click every time. +4. No open P signature remains in the ledger. +5. One staging wave after deploy confirms the result. There, U9 sees no restart + failure across all restarts in a 24-hour window. + +## 9. Cost and safety + +- Hard budget per wave (start at $30) and per night (start at $150). The harness + refuses to start a wave above 80% of the cap. +- The harness signals only processes it started, using the PID and start-time + checks in `process-tree-owner.ts`. It never touches the default instance or + any other dev server. +- The Daytona cleanup sweep runs after every wave and on harness exit. Orphaned + sandboxes are counted as R6 failures before they are deleted. +- No real credentials are placed in fixtures. The canary secret check (soak U7) + stays on, because recovery paths replay logs. + +## 10. Decisions needed from Dotta + +1. **Daytona quota and concurrency ceiling:** is 20 concurrent sandboxes OK, or + is the cap lower? +2. **Budget:** are $30 per wave and $150 per night acceptable caps? +3. **K5 bar (outage longer than the 7-minute reconnect grace):** is a working + Retry an acceptable outcome, or must the task continue automatically from + saved work? The plan assumes automatic continuation is required when the + last provider turn settled, and Retry is acceptable when the turn was in + flight. This matches the #19399 rule. +4. **Adapters for the finish line:** Claude and Codex native only, or also the + ACPX adapters? +5. **K12 rolling deploys:** include them now, or wait until zero-downtime web + deploys (`2026-10-31-zero-downtime-web-deploys.md`) is the production path? diff --git a/scripts/typecheck-test-projects.mjs b/scripts/typecheck-test-projects.mjs index 0c6d537b216..3ef20da7961 100644 --- a/scripts/typecheck-test-projects.mjs +++ b/scripts/typecheck-test-projects.mjs @@ -24,6 +24,7 @@ export const PROJECTS = [ "server/src/__tests__/helpers/maintenance", "tests/e2e", "tests/lifecycle-baseline", + "tests/restart-chaos", "tests/runner-e2e", "tests/speko-native", ]; diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md new file mode 100644 index 00000000000..393d5fe9848 --- /dev/null +++ b/tests/restart-chaos/README.md @@ -0,0 +1,71 @@ +# Restart chaos harness (Phase 1 MVP) + +Plan: [`doc/plans/2026-11-06-control-plane-restart-chaos.md`](../../doc/plans/2026-11-06-control-plane-restart-chaos.md). + +A script, not an LLM. It starts an isolated Paperclip control plane (temporary +`PAPERCLIP_HOME`, wrapper-owned embedded Postgres, its own port), seeds a +"Chaos Lab" company, launches real agent tasks, kills or restarts the control +plane during those tasks, and grades every task on R1–R6 plus a deterministic +T-oracle. + +## Run + +```sh +pnpm install --frozen-lockfile +pnpm --filter @paperclipai/paperclip-runner build:runner-binaries # native runner (debug) +set -a; source ~/.secrets; set +a # provider keys, process env only +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ + --adapters codex,claude,opencode --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay w-20261106T101112-s42/t03 +pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts # credential-free unit tests +node scripts/typecheck-test-projects.mjs tests/restart-chaos # typecheck +``` + +Options: `--workloads A1,B1,B5,C1`, `--budget-cents` (company hard cap, +default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance). +`--env daytona` is reserved; it fails until the Daytona image and environment +are wired (see the TODO in `seed.ts`). + +## Pieces + +| File | Role | +| --- | --- | +| `supervisor.ts` | Owns the server. Reuses `tests/runner-e2e` process code (config, ports, env isolation, wrapper-owned Postgres, process-tree stopper, `replaceController`). Faults: K1 SIGKILL + immediate restart, K2 SIGTERM + restart, K4 SIGKILL + 2–5 min hold. The controller runtime cache is kept (discarding it is K11). | +| `seed.ts` | Chaos Lab company, encrypted provider secrets, native Codex / ACPX Claude / OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project (git repo: python CLI, one failing test, `bench.sh` ≈ 4 min) with isolated git-worktree workspaces. | +| `scheduler.ts` | Seeded plan: workload × adapter × fault cells and a random fire time per task. Saved to `results//plan.json`. `--replay` reruns one task alone with the same choices and fires the recorded sequence of faults that crossed it (other tasks' faults included) at the same offsets from launch. | +| `observer.ts` | Polls each issue and its runs. When a task rests needing Retry it calls `POST /api/issues/:id/retry` once, with the body `useTaskRetry.ts` sends. Collects T-oracle artifacts and the R6 sweep. | +| `grader.ts` | Pure R1–R6 grading over the samples. | +| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, with their T-oracles. | +| `evidence.ts`, `ledger.ts` | Evidence bundle for each failure and the JSONL ledger plus `SUMMARY.md`. | + +## Output (gitignored) + +`tests/restart-chaos/results/` (override with `RESTART_CHAOS_RESULTS_DIR`): + +- `ledger.jsonl`, `SUMMARY.md` — every graded task, per-cell pass rate, open signatures. +- `/plan.json`, `timeline.jsonl`, `server.log`, `faults.json`, `.grade.json`. +- `/evidence//` on failure — run ids and statuses, last 200 events per + run, workspace operations, server log window around each fault, startup phase + timings, fault timeline, seed, git SHA and the replay command. Known provider + key values are redacted. + +## Grading notes + +- A task is restart-crossing when a run overlaps a fault window or the task was + not yet at rest when the fault started. Non-crossing tasks are still graded. +- R2 also checks that workspace files present before the first fault still exist at rest. +- B5 passes only when `bench.sh` wrote its own completion record showing it + ran, and the harness itself saw at least three `.bench/progress` values + while polling. This is best-effort against a deliberately adversarial agent. +- R4 allows at most one active (queued, running or scheduled-retry) run per + sample. A wake comment may be served again only after a run that did not + succeed. +- T-oracle commands (B1 tests and CLI) run agent-editable code. They run under + a macOS Seatbelt sandbox with a scratch `HOME`, no network and no read access + to the operator's home directory. On other platforms they are refused unless + `RESTART_CHAOS_ALLOW_UNSANDBOXED_ORACLE=1` is set inside a disposable VM. +- R1 allows the workload budget plus outage time. When a Retry was needed R1 is + `n/a` and R3 decides; a task that rests unfinished with no Retry fails both. +- Faults are serialized; each one hits every task running at that moment. +- A `todo` or `in_progress` task with no live run and no named wait is not a rest. The observer keeps watching it until it resumes or passes the R5 stall limit. +- A fault that fails to complete fails the wave as a harness error (exit 2), even when recovery brings the server back. diff --git a/tests/restart-chaos/api.ts b/tests/restart-chaos/api.ts new file mode 100644 index 00000000000..2a37b89d71a --- /dev/null +++ b/tests/restart-chaos/api.ts @@ -0,0 +1,31 @@ +/** Minimal board API client for an isolated local_trusted instance. */ +export class ChaosApiError extends Error { + constructor(readonly status: number | null, message: string) { super(message); this.name = "ChaosApiError"; } +} + +export class ChaosApi { + constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000) {} + + private async request(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs): Promise { + let response: Response; + try { + response = await fetch(`${this.baseUrl}${route}`, { + method, + headers: { origin: this.baseUrl, ...(body === undefined ? {} : { "content-type": "application/json" }) }, + body: body === undefined ? undefined : JSON.stringify(body), + signal: AbortSignal.timeout(timeoutMs), + cache: "no-store", + }); + } catch (error) { + throw new ChaosApiError(null, `${method} ${route} unreachable: ${error instanceof Error ? error.message : String(error)}`); + } + const text = await response.text(); + if (!response.ok) throw new ChaosApiError(response.status, `${method} ${route} returned ${response.status}: ${text.slice(0, 500)}`); + return (text ? JSON.parse(text) : null) as T; + } + + get(route: string, timeoutMs?: number) { return this.request("GET", route, undefined, timeoutMs); } + post(route: string, body?: unknown, timeoutMs?: number) { return this.request("POST", route, body ?? {}, timeoutMs); } + put(route: string, body?: unknown) { return this.request("PUT", route, body ?? {}); } + patch(route: string, body?: unknown) { return this.request("PATCH", route, body ?? {}); } +} diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts new file mode 100644 index 00000000000..05479e63345 --- /dev/null +++ b/tests/restart-chaos/config.ts @@ -0,0 +1,108 @@ +import { parseArgs } from "node:util"; +import { + ADAPTER_IDS, ENVIRONMENT_KINDS, FAULT_KINDS, WORKLOAD_IDS, + type AdapterId, type EnvironmentKind, type FaultKind, type WaveOptions, type WorkloadId, +} from "./types.js"; + +export const CREDENTIAL_FOR_ADAPTER: Record = { + codex: "OPENAI_API_KEY", + claude: "ANTHROPIC_API_KEY", + opencode: "OPENROUTER_API_KEY", +}; + +/** Catalog profile reused from tests/runner-e2e/catalog.ts for each adapter. */ +export const PROFILE_FOR_ADAPTER: Record = { + codex: "runner-codex", + claude: "runner-acpx-claude", + opencode: "runner-opencode", +}; + +export type ParsedCommand = + | { mode: "wave"; options: WaveOptions; keep: boolean } + | { mode: "replay"; ref: ReplayRef; keep: boolean; overrides: Partial> }; + +export interface ReplayRef { waveId: string; taskId: string } + +const WAVE_ID = /^w-[0-9]{8}T[0-9]{6}-s[0-9]+(?:-replay-t[0-9]{2,3}-[0-9]+)?$/; +const TASK_ID = /^t[0-9]{2,3}$/; + +export function parseReplayRef(value: string): ReplayRef { + const trimmed = value.trim().replace(/\/+$/, ""); + const parts = trimmed.split("/"); + if (parts.length !== 2) throw new Error(`--replay expects /, got "${value}"`); + const [waveId, taskId] = parts as [string, string]; + if (!WAVE_ID.test(waveId)) throw new Error(`Invalid wave id "${waveId}"`); + if (!TASK_ID.test(taskId)) throw new Error(`Invalid task id "${taskId}"`); + return { waveId, taskId }; +} + +function list(raw: string | undefined, allowed: readonly T[], name: string, fallback: readonly T[]): T[] { + if (raw === undefined) return [...fallback]; + const values = raw.split(",").map(value => value.trim()).filter(Boolean); + if (!values.length) throw new Error(`--${name} must not be empty`); + for (const value of values) if (!allowed.includes(value as T)) throw new Error(`Unknown ${name} "${value}"; allowed: ${allowed.join(", ")}`); + return [...new Set(values)] as T[]; +} + +function integer(raw: string | undefined, name: string, fallback: number, min: number, max: number): number { + if (raw === undefined) return fallback; + if (!/^[0-9]+$/.test(raw)) throw new Error(`--${name} must be a non-negative integer`); + const value = Number(raw); + if (!Number.isSafeInteger(value) || value < min || value > max) throw new Error(`--${name} must be between ${min} and ${max}`); + return value; +} + +export function parseCommand(argv: readonly string[], now = Date.now()): ParsedCommand { + // Tolerate a forwarded literal `--` (for example from a package runner). + const args = argv[0] === "--" ? argv.slice(1) : [...argv]; + const { values } = parseArgs({ + args, + strict: true, + options: { + env: { type: "string" }, + adapters: { type: "string" }, + workloads: { type: "string" }, + faults: { type: "string" }, + concurrency: { type: "string" }, + tasks: { type: "string" }, + seed: { type: "string" }, + "budget-cents": { type: "string" }, + "poll-ms": { type: "string" }, + replay: { type: "string" }, + keep: { type: "boolean" }, + }, + }); + const keep = values.keep === true; + const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); + const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); + if (values.replay !== undefined) { + for (const key of ["env", "adapters", "workloads", "faults", "concurrency", "tasks", "seed"] as const) { + if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); + } + return { mode: "replay", ref: parseReplayRef(values.replay), keep, + overrides: { ...(values["poll-ms"] ? { pollMs } : {}), ...(values["budget-cents"] ? { budgetCents } : {}) } }; + } + const env = (values.env ?? "local") as EnvironmentKind; + if (!ENVIRONMENT_KINDS.includes(env)) throw new Error(`Unknown env "${env}"; allowed: ${ENVIRONMENT_KINDS.join(", ")}`); + const options: WaveOptions = { + env, + adapters: list(values.adapters, ADAPTER_IDS, "adapters", ["codex", "claude"]), + workloads: list(values.workloads, WORKLOAD_IDS, "workloads", WORKLOAD_IDS), + faults: list(values.faults, FAULT_KINDS, "faults", ["K1", "K2", "K4"]), + concurrency: integer(values.concurrency, "concurrency", 4, 1, 20), + tasks: integer(values.tasks, "tasks", 8, 1, 200), + seed: integer(values.seed, "seed", now % 2_147_483_647, 0, 2_147_483_647), + budgetCents, + pollMs, + }; + return { mode: "wave", options, keep }; +} + +export function waveIdFor(seed: number, at = new Date()): string { + const stamp = at.toISOString().replace(/[-:]/g, "").replace(/\.\d{3}Z$/, ""); + return `w-${stamp}-s${seed}`; +} + +export function replayCommand(waveId: string, taskId: string): string { + return `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay ${waveId}/${taskId}`; +} diff --git a/tests/restart-chaos/evidence.ts b/tests/restart-chaos/evidence.ts new file mode 100644 index 00000000000..67721b1cc95 --- /dev/null +++ b/tests/restart-chaos/evidence.ts @@ -0,0 +1,104 @@ +import { mkdir, open, writeFile } from "node:fs/promises"; +import path from "node:path"; +import type { ChaosApi } from "./api.js"; +import { replayCommand } from "./config.js"; +import type { TimelineEntry } from "./timeline.js"; +import type { FaultReceipt, OracleArtifacts, TaskGrade, TaskRecord, WavePlan } from "./types.js"; + +const LOG_BEFORE_BYTES = 128 * 1024; +const LOG_AFTER_BYTES = 512 * 1024; +const EVENTS_PER_RUN = 200; +const OBSERVER_POLL_LINE = /INFO\S*: \S*(?:GET|HEAD) \/(?:issues\/[0-9a-f-]+(?:\/runs)?|ready|liveness|health) 200/; + +type Json = Record; + +/** Canary (soak U7): never let a provider key reach an evidence file. */ +export function redactKnownSecrets(text: string, secrets: readonly string[]): string { + let out = text; + for (const value of secrets) if (value.length >= 8) out = out.split(value).join("[REDACTED]"); + return out; +} + +export function knownSecretValues(env: NodeJS.ProcessEnv = process.env): string[] { + // Any *KEY, *TOKEN, *SECRET or *PASSWORD name, including PAPERCLIP_AGENT_PRIVATE_KEY and AWS_SECRET_ACCESS_KEY. + return Object.entries(env).filter(([key, value]) => value && /(?:KEY|TOKEN|SECRET|PASSWORD)$/.test(key)).map(([, value]) => value!); +} + +async function lastEvents(api: ChaosApi, runId: string): Promise { + const kept: Json[] = []; + let afterSeq = 0; + for (let page = 0; page < 200; page += 1) { + const rows = await api.get(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=500`); + if (!rows?.length) break; + kept.push(...rows); + if (kept.length > EVENTS_PER_RUN) kept.splice(0, kept.length - EVENTS_PER_RUN); + const next = Number(rows[rows.length - 1]?.seq); + if (!Number.isFinite(next) || next <= afterSeq || rows.length < 500) break; + afterSeq = next; + } + return kept; +} + +async function logWindow(serverLogPath: string, faults: readonly FaultReceipt[]): Promise { + if (!faults.length) return ""; + const handle = await open(serverLogPath, "r").catch(() => null); + if (!handle) return ""; + try { + const size = (await handle.stat()).size; + const parts: string[] = []; + for (const fault of faults) { + const start = Math.max(0, fault.serverLogOffsetAtStart - LOG_BEFORE_BYTES); + const end = Math.min(size, (fault.serverLogOffsetAtReady ?? fault.serverLogOffsetAtStart) + LOG_AFTER_BYTES); + const buffer = Buffer.alloc(Math.max(0, end - start)); + await handle.read(buffer, 0, buffer.length, start); + // Drop the observer's own polling request lines; they bury the recovery log. + const text = buffer.toString("utf8").split("\n").filter(line => !OBSERVER_POLL_LINE.test(line)).join("\n"); + parts.push(`===== ${fault.faultId} ${fault.kind} bytes ${start}-${end} (observer GET lines removed) =====\n${text}`); + } + return parts.join("\n"); + } finally { await handle.close(); } +} + +/** Bundle everything a fix-failed-paperclip-run session needs to start (plan §5.7). */ +export async function writeEvidenceBundle(input: { + directory: string; api: ChaosApi; plan: WavePlan; record: TaskRecord; grade: TaskGrade; artifacts: OracleArtifacts | null; + faults: readonly FaultReceipt[]; timeline: readonly TimelineEntry[]; serverLogPath: string; gitSha: string; gitDirty: boolean; + issue: Json | null; secrets: readonly string[]; +}): Promise { + const { directory, api, record } = input; + await mkdir(directory, { recursive: true, mode: 0o700 }); + const write = (name: string, value: unknown) => writeFile(path.join(directory, name), + redactKnownSecrets(typeof value === "string" ? value : `${JSON.stringify(value, null, 2)}\n`, input.secrets), { mode: 0o600 }); + const errors: string[] = []; + const attempt = async (label: string, work: () => Promise) => { + try { await work(); } catch (error) { errors.push(`${label}: ${error instanceof Error ? error.message : String(error)}`); } + }; + const relevant = input.faults.filter(fault => input.grade.crossingFaults.includes(fault.faultId) || record.faultIds.includes(fault.faultId)); + await write("meta.json", { + schema: "paperclip.restart-chaos.evidence/v1", waveId: input.plan.waveId, seed: input.plan.seed, taskId: record.plan.taskId, + gitSha: input.gitSha, gitDirty: input.gitDirty, signature: input.grade.signature, replay: replayCommand(input.plan.waveId, record.plan.taskId), + issueId: record.issueId, identifier: record.identifier, task: record.plan, options: input.plan.options, + }); + await write("grade.json", input.grade); + await write("task-record.json", record); + await write("faults.json", relevant.length ? relevant : input.faults); + await write("startup-phases.json", (relevant.length ? relevant : input.faults).map(fault => ({ faultId: fault.faultId, kind: fault.kind, + startedAt: fault.startedAt, stoppedAt: fault.stoppedAt, restartedAt: fault.restartedAt, healthyAt: fault.healthyAt, readyAt: fault.readyAt, phases: fault.phases }))); + await write("timeline.json", input.timeline); + if (input.artifacts) await write("oracle-artifacts.json", input.artifacts); + await attempt("server-log", async () => write("server-log-window.txt", await logWindow(input.serverLogPath, relevant.length ? relevant : input.faults))); + if (record.issueId) { + await attempt("issue", async () => write("issue.json", input.issue ?? await api.get(`/api/issues/${record.issueId}`))); + let runs: Json[] = []; + await attempt("runs", async () => { runs = await api.get(`/api/issues/${record.issueId}/runs`); await write("runs.json", runs); }); + for (const run of runs) { + const runId = String(run.runId ?? run.id); + await attempt(`events ${runId}`, async () => write(`run-${runId}-events.json`, await lastEvents(api, runId))); + await attempt(`workspace-operations ${runId}`, async () => write(`run-${runId}-workspace-operations.json`, + await api.get(`/api/heartbeat-runs/${runId}/workspace-operations`))); + } + await attempt("comments", async () => write("comments.json", await api.get(`/api/issues/${record.issueId}/comments`))); + } + if (errors.length) await write("collection-errors.json", errors); + return directory; +} diff --git a/tests/restart-chaos/grader.test.ts b/tests/restart-chaos/grader.test.ts new file mode 100644 index 00000000000..97870f866d1 --- /dev/null +++ b/tests/restart-chaos/grader.test.ts @@ -0,0 +1,198 @@ +import { describe, expect, it } from "vitest"; +import { crossingFaults, gradeR3, gradeR4, gradeR5, gradeTask, STALL_LIMIT_MS } from "./grader.js"; +import { isResting } from "./observer.js"; +import type { FaultReceipt, OracleArtifacts, RunSample, TaskRecord, TaskSample } from "./types.js"; +import { BENCH_EXPECTED, parseBenchRecord, workloadDefinitions } from "./workloads.js"; + +const T0 = Date.parse("2026-11-06T12:00:00Z"); +const iso = (offsetMs: number) => new Date(T0 + offsetMs).toISOString(); + +function run(id: string, status: string, created: number, finished: number | null = null, extra: Partial = {}): RunSample { + return { id, status, createdAt: iso(created), startedAt: iso(created + 500), finishedAt: finished === null ? null : iso(finished), + errorCode: null, retryOfRunId: null, wakeCommentIds: [], livenessState: null, ...extra }; +} + +function sample(at: number, issueStatus: string, runs: RunSample[], extra: Partial = {}): TaskSample { + return { at: T0 + at, reachable: true, serverReady: true, issueStatus, interruption: null, activeRecoveryAction: false, + executionBlocker: false, scheduledRetry: false, runs, ...extra }; +} + +function record(samples: TaskSample[], extra: Partial = {}): TaskRecord { + return { + plan: { taskId: "t01", index: 0, workload: "A1", adapter: "codex", fault: "K1", fireAfterMs: 20_000, holdMs: 0, followUpAfterMs: null, nonce: "abcd1234" }, + issueId: "issue-1", identifier: "CHA-1", createdAt: T0, launchError: null, samples, retry: null, restAt: T0 + 120_000, + finalStatus: "done", deadlineAt: T0 + 600_000, timedOut: false, followUp: null, faultIds: ["f01"], + sweep: { activeRuns: [], runningWorkspaceOperations: [], activeRecoveryAction: false, error: null }, + preFaultFiles: null, lostFiles: null, ...extra, + }; +} + +const fault: FaultReceipt = { faultId: "f01", kind: "K1", requestedBy: "t01", signal: "SIGKILL", holdMs: 0, requestedAt: T0 + 20_000, + startedAt: T0 + 20_000, stoppedAt: T0 + 21_000, restartedAt: T0 + 21_000, healthyAt: T0 + 40_000, readyAt: T0 + 45_000, + phases: [{ phase: "unreachable", at: T0 + 21_000 }, { phase: "recovering", at: T0 + 38_000 }, { phase: "ready", at: T0 + 45_000 }], + serverLogOffsetAtStart: 0, serverLogOffsetAtReady: 100, error: null }; + +const passingBrief: OracleArtifacts = { issueStatus: "done", documents: { brief: "CHAOS-abcd1234\n- one\n- two\n- three\n" }, + agentComments: [], files: {}, commands: {}, workspacePath: null }; +const oracle = (artifacts = passingBrief) => workloadDefinitions.A1.grade("abcd1234", artifacts); + +describe("R1–R6 grading on fixture timelines", () => { + it("passes a task that auto-continued across a K1 kill", () => { + const samples = [ + sample(3_000, "in_progress", [run("r1", "running", 1_000)]), + { at: T0 + 25_000, reachable: false, serverReady: false }, + sample(50_000, "in_progress", [run("r1", "failed", 1_000, 30_000), run("r2", "running", 48_000)]), + sample(110_000, "done", [run("r1", "failed", 1_000, 30_000), run("r2", "succeeded", 48_000, 100_000)]), + ]; + const grade = gradeTask(record(samples), [fault], workloadDefinitions.A1.budgetMs, oracle()); + expect(grade.crossed).toBe(true); + expect(grade.crossingFaults).toEqual(["f01"]); + expect(grade.rules.R1.verdict).toBe("pass"); + expect(grade.rules.R3.verdict).toBe("n/a"); + expect(grade.pass).toBe(true); + expect(grade.signature).toBeNull(); + }); + + it("does not count a task that rested before the fault as restart-crossing", () => { + const samples = [sample(3_000, "done", [run("r1", "succeeded", 1_000, 10_000)])]; + expect(crossingFaults(record(samples, { restAt: T0 + 12_000 }), [fault])).toEqual([]); + }); + + it("fails R1 and R3 when a task rests unfinished with no Retry offered", () => { + const samples = [sample(60_000, "in_progress", [run("r1", "failed", 1_000, 30_000)])]; + const grade = gradeTask(record(samples, { finalStatus: "in_progress" }), [fault], workloadDefinitions.A1.budgetMs, + oracle({ ...passingBrief, issueStatus: "in_progress", documents: { brief: null } })); + expect(grade.rules.R1).toMatchObject({ verdict: "fail" }); + expect(grade.rules.R1.detail).toMatch(/rested_not_done/); + expect(grade.rules.R3.detail).toMatch(/missing_retry/); + expect(grade.signature).toBe("R1:rested_not_done:A1:K1"); + }); + + it("passes R3 when one Retry starts a run that completes the task", () => { + const samples = [ + sample(60_000, "blocked", [run("r1", "failed", 1_000, 30_000)], { interruption: { state: "failed", canRetry: true, retryWait: false, workspaceFailure: null, observedRunId: "r1", userAction: null } }), + sample(80_000, "in_progress", [run("r1", "failed", 1_000, 30_000), run("r2", "running", 75_000, null, { retryOfRunId: "r1" })]), + sample(140_000, "done", [run("r1", "failed", 1_000, 30_000), run("r2", "succeeded", 75_000, 130_000)]), + ]; + const retried = record(samples, { retry: { at: T0 + 74_000, observedRunId: "r1", httpStatus: 200, accepted: true, error: null }, restAt: T0 + 140_000 }); + expect(gradeR3(retried)).toMatchObject({ verdict: "pass" }); + const grade = gradeTask(retried, [fault], workloadDefinitions.A1.budgetMs, oracle()); + expect(grade.rules.R1.verdict).toBe("n/a"); + expect(grade.pass).toBe(true); + }); + + it("fails R3 for a dead Retry (accepted but no new run)", () => { + const samples = [sample(60_000, "blocked", [run("r1", "failed", 1_000, 30_000)])]; + const dead = record(samples, { finalStatus: "blocked", retry: { at: T0 + 74_000, observedRunId: "r1", httpStatus: 200, accepted: true, error: null } }); + expect(gradeR3(dead).detail).toBe("retry_no_new_run"); + const rejected = record(samples, { finalStatus: "blocked", retry: { at: T0 + 74_000, observedRunId: "r1", httpStatus: 409, accepted: null, error: "conflict" } }); + expect(gradeR3(rejected).detail).toMatch(/^retry_rejected/); + }); + + it("fails R4 for two concurrent running runs and for a comment consumed twice", () => { + expect(gradeR4(record([sample(10_000, "in_progress", [run("r1", "running", 1_000), run("r2", "running", 5_000)])])).detail).toMatch(/concurrent_runs/); + const twice = [sample(90_000, "done", [ + run("r1", "succeeded", 1_000, 40_000, { wakeCommentIds: ["c1"] }), + run("r2", "succeeded", 50_000, 80_000, { wakeCommentIds: ["c1"] }), + ])]; + expect(gradeR4(record(twice)).detail).toMatch(/comment_consumed_twice: c1/); + // Two active runs that are not both running also fail. + expect(gradeR4(record([sample(10_000, "in_progress", [run("r1", "queued", 1_000), run("r2", "scheduled_retry", 5_000)])])).detail).toMatch(/concurrent_runs/); + // A failed run's comment may be served again by the retry; a successful one may not. + const retried = [sample(90_000, "done", [ + run("r1", "failed", 1_000, 40_000, { wakeCommentIds: ["c1"] }), + run("r2", "succeeded", 50_000, 80_000, { wakeCommentIds: ["c1"], retryOfRunId: "r1" }), + ])]; + expect(gradeR4(record(retried)).verdict).toBe("pass"); + const afterSuccess = [sample(90_000, "done", [ + run("r1", "succeeded", 1_000, 40_000, { wakeCommentIds: ["c1"] }), + run("r2", "failed", 50_000, 80_000, { wakeCommentIds: ["c1"] }), + ])]; + expect(gradeR4(record(afterSuccess)).detail).toMatch(/comment_consumed_twice: c1 by r1,r2/); + }); + + it("fails R5 on a silent stall longer than three minutes after ready, but not during outages or named waits", () => { + const idle = (at: number, extra: Partial = {}) => sample(at, "in_progress", [run("r1", "failed", 1_000, 30_000)], extra); + const stalled = [idle(60_000), idle(60_000 + STALL_LIMIT_MS / 2), idle(60_000 + STALL_LIMIT_MS + 5_000)]; + expect(gradeR5(record(stalled))).toMatchObject({ verdict: "fail" }); + const outage = [idle(60_000), { at: T0 + 120_000, reachable: false, serverReady: false }, idle(200_000), idle(300_000)]; + expect(gradeR5(record(outage))).toMatchObject({ verdict: "pass" }); + const named = stalled.map(entry => ({ ...entry, scheduledRetry: true })); + expect(gradeR5(record(named))).toMatchObject({ verdict: "pass" }); + }); + + it("fails R2 when a workspace failure stays visible and R6 when work is left pending", () => { + const failure = { state: "failed", canRetry: false, retryWait: false, workspaceFailure: "snapshot restore failed", observedRunId: "r1", userAction: null }; + const grade = gradeTask(record([sample(90_000, "done", [run("r1", "succeeded", 1_000, 80_000)], { interruption: failure })]), [fault], 600_000, oracle()); + expect(grade.rules.R2.detail).toMatch(/workspace_failure_visible_at_rest/); + const leaky = record([sample(90_000, "done", [run("r1", "succeeded", 1_000, 80_000)])], + { sweep: { activeRuns: [], runningWorkspaceOperations: ["op1"], activeRecoveryAction: true, error: null } }); + expect(gradeTask(leaky, [fault], 600_000, oracle()).rules.R6.detail).toMatch(/running_workspace_operations=op1; active_recovery_action/); + }); + + it("fails R2 when files written before the fault are gone at rest", () => { + const done = [sample(90_000, "done", [run("r1", "succeeded", 1_000, 80_000)])]; + const lost = record(done, { preFaultFiles: ["NOTES.md", "chaoscli/cli.py"], lostFiles: ["NOTES.md"] }); + expect(gradeTask(lost, [fault], 600_000, oracle()).rules.R2.detail).toBe("pre_fault_files_lost: NOTES.md"); + expect(gradeTask(record(done, { preFaultFiles: ["NOTES.md"], lostFiles: [] }), [fault], 600_000, oracle()).rules.R2.verdict).toBe("pass"); + }); + + it("fails R2 visibly when the pre-fault snapshot could not run", () => { + const done = [sample(90_000, "done", [run("r1", "succeeded", 1_000, 80_000)])]; + const unchecked = record(done, { preFaultSnapshotError: "EACCES" }); + expect(gradeTask(unchecked, [fault], 600_000, oracle()).rules.R2.detail).toMatch(/^pre_fault_snapshot_failed/); + }); + + it("allows outage time on top of the budget", () => { + const k4: FaultReceipt = { ...fault, kind: "K4", holdMs: 240_000, readyAt: T0 + 290_000 }; + const late = record([sample(700_000, "done", [run("r2", "succeeded", 300_000, 690_000)])], { restAt: T0 + 700_000 }); + expect(gradeTask(late, [k4], 600_000, oracle()).rules.R1.verdict).toBe("pass"); + expect(gradeTask(late, [], 600_000, oracle()).rules.R1.detail).toMatch(/done_too_late/); + }); +}); + +describe("T-oracles", () => { + it("grades A1 and C1 markers", () => { + expect(workloadDefinitions.A1.grade("abcd1234", passingBrief).verdict).toBe("pass"); + expect(workloadDefinitions.C1.grade("abcd1234", passingBrief).checks.find(check => check.name.includes("followup"))?.ok).toBe(false); + expect(workloadDefinitions.C1.grade("abcd1234", { ...passingBrief, documents: { brief: `${passingBrief.documents.brief}FOLLOWUP-abcd1234\n` } }).verdict).toBe("pass"); + }); + + it("grades B1 by running the tests and the CLI", () => { + const base: OracleArtifacts = { issueStatus: "done", documents: {}, agentComments: [], workspacePath: "/tmp/ws", + files: { "NOTES-abcd1234.md": "B1-abcd1234\n" }, + commands: { unittest: { exitCode: 0, stdout: "", stderr: "OK" }, slugify: { exitCode: 0, stdout: "hello-chaos-world\n", stderr: "" } } }; + expect(workloadDefinitions.B1.grade("abcd1234", base).verdict).toBe("pass"); + expect(workloadDefinitions.B1.grade("abcd1234", { ...base, commands: { ...base.commands, unittest: { exitCode: 1, stdout: "", stderr: "FAILED" } } }).verdict).toBe("fail"); + }); + + it("grades B5 against the deterministic bench result", () => { + const base: OracleArtifacts = { issueStatus: "done", documents: {}, agentComments: [], workspacePath: "/tmp/ws", commands: {}, + benchProgressObserved: 6, + files: { "bench-result-abcd1234.txt": `${BENCH_EXPECTED}\n`, ".bench/run-abcd1234.txt": `started=1000 finished=1241 result=${BENCH_EXPECTED}\n` } }; + expect(BENCH_EXPECTED).toBe("BENCH_RESULT=4636840"); + expect(parseBenchRecord(base.files[".bench/run-abcd1234.txt"])).toEqual({ seconds: 241, result: BENCH_EXPECTED }); + expect(workloadDefinitions.B5.grade("abcd1234", base).verdict).toBe("pass"); + // A forged record without harness-observed progress does not pass. + expect(workloadDefinitions.B5.grade("abcd1234", { ...base, benchProgressObserved: 1 }).checks.find(c => c.name === "bench_progress_observed")?.ok).toBe(false); + expect(workloadDefinitions.B5.grade("abcd1234", { ...base, files: { ...base.files, "bench-result-abcd1234.txt": "BENCH_RESULT=1" } }).verdict).toBe("fail"); + // A computed answer without a bench.sh run, or a too-short run, does not pass. + expect(workloadDefinitions.B5.grade("abcd1234", { ...base, files: { "bench-result-abcd1234.txt": BENCH_EXPECTED } }).checks.find(c => c.name === "bench_actually_ran")?.ok).toBe(false); + expect(workloadDefinitions.B5.grade("abcd1234", { ...base, files: { ...base.files, ".bench/run-abcd1234.txt": `started=1000 finished=1010 result=${BENCH_EXPECTED}` } }).verdict).toBe("fail"); + }); +}); + +describe("rest detection", () => { + it("is not resting while a run, scheduled retry or recovery is live", () => { + expect(isResting(sample(0, "in_progress", [run("r1", "queued", 0)]))).toBe(false); + expect(isResting(sample(0, "in_progress", [], { scheduledRetry: true }))).toBe(false); + expect(isResting(sample(0, "done", [run("r1", "succeeded", 0, 10)]))).toBe(true); + // Unexplained idle work is a possible stall, not a rest; a visible Retry is a rest. + expect(isResting(sample(0, "in_progress", [run("r1", "failed", 0, 10)]))).toBe(false); + expect(isResting(sample(0, "in_progress", [run("r1", "failed", 0, 10)], { interruption: { state: "failed", canRetry: true, retryWait: false, workspaceFailure: null, observedRunId: "r1", userAction: null } }))).toBe(true); + expect(isResting({ at: T0, reachable: false, serverReady: false })).toBe(false); + expect(isResting(sample(0, "blocked", [], { activeRecoveryAction: true, recoveryOwner: "agent" }))).toBe(false); + // A board-owned recovery waits on a person, so the task rests (and R6 reports it). + expect(isResting(sample(0, "blocked", [], { activeRecoveryAction: true, recoveryOwner: "board" }))).toBe(true); + }); +}); diff --git a/tests/restart-chaos/grader.ts b/tests/restart-chaos/grader.ts new file mode 100644 index 00000000000..75eea1f037d --- /dev/null +++ b/tests/restart-chaos/grader.ts @@ -0,0 +1,185 @@ +import type { FaultReceipt, OracleResult, RuleId, RuleResult, TaskGrade, TaskRecord, TaskSample } from "./types.js"; + +/** Pure R1–R6 grading over observer samples (plan §2). No I/O. */ + +export const ACTIVE_RUN_STATUSES = new Set(["queued", "running", "scheduled_retry"]); +export const STALL_LIMIT_MS = 3 * 60_000; +const REST_STATUSES_WITH_WORK = new Set(["todo", "in_progress"]); + +const ms = (iso: string | null | undefined) => (iso ? Date.parse(iso) : NaN); + +function faultWindow(fault: FaultReceipt): [number, number] { + return [fault.startedAt, fault.readyAt ?? fault.healthyAt ?? Number.POSITIVE_INFINITY]; +} + +/** + * Restart-crossing: some run of the task (queued wake through finalization) + * overlaps the fault window, or the task was not yet at rest when it started. + */ +export function crossingFaults(record: TaskRecord, faults: readonly FaultReceipt[]): string[] { + if (record.createdAt === null) return []; + const latest = new Map[number]; at: number }>(); + for (const sample of record.samples) for (const run of sample.runs ?? []) latest.set(run.id, { run, at: sample.at }); + const runs = new Map(); + for (const [id, { run, at }] of latest) { + const start = ms(run.createdAt) || ms(run.startedAt) || at; + const end = ACTIVE_RUN_STATUSES.has(run.status) ? Number.POSITIVE_INFINITY : (ms(run.finishedAt) || at); + runs.set(id, { start, end }); + } + const restAt = record.restAt ?? Number.POSITIVE_INFINITY; + return faults.filter(fault => { + const [start, end] = faultWindow(fault); + if (record.createdAt! <= start && restAt > start) return true; + return [...runs.values()].some(run => run.start <= end && run.end >= start); + }).map(fault => fault.faultId); +} + +function reachable(samples: readonly TaskSample[]) { return samples.filter(sample => sample.reachable); } + +function lastReachable(record: TaskRecord): TaskSample | undefined { + const values = reachable(record.samples); + return values[values.length - 1]; +} + +export function outageOverlapMs(record: TaskRecord, faults: readonly FaultReceipt[]): number { + if (record.createdAt === null) return 0; + const end = record.restAt ?? Number.POSITIVE_INFINITY; + return faults.reduce((total, fault) => { + const [start, stop] = faultWindow(fault); + const overlap = Math.min(stop, end) - Math.max(start, record.createdAt!); + return total + (Number.isFinite(overlap) && overlap > 0 ? overlap : 0); + }, 0); +} + +export function gradeR1(record: TaskRecord, budgetMs: number, faults: readonly FaultReceipt[]): RuleResult { + if (record.launchError) return { verdict: "fail", detail: `launch_failed: ${record.launchError}` }; + if (record.retry) return { verdict: "n/a", detail: "task rested needing Retry; graded under R3" }; + if (record.finalStatus !== "done") { + return { verdict: "fail", detail: record.timedOut ? `no_rest_within_budget (last status ${record.finalStatus ?? "unknown"})` : `rested_not_done (status ${record.finalStatus ?? "unknown"}, no Retry offered)` }; + } + const allowed = budgetMs + outageOverlapMs(record, faults); + const took = (record.restAt ?? Number.POSITIVE_INFINITY) - record.createdAt!; + return took <= allowed + ? { verdict: "pass", detail: `done after ${Math.round(took / 1000)}s (allowed ${Math.round(allowed / 1000)}s)` } + : { verdict: "fail", detail: `done_too_late ${Math.round(took / 1000)}s > ${Math.round(allowed / 1000)}s` }; +} + +export function gradeR2(record: TaskRecord, oracle: OracleResult): RuleResult { + const last = lastReachable(record); + if (!last) return { verdict: "fail", detail: "no_reachable_sample" }; + if (last.interruption?.workspaceFailure) return { verdict: "fail", detail: `workspace_failure_visible_at_rest: ${last.interruption.workspaceFailure}` }; + if (record.preFaultSnapshotError) { + return { verdict: "fail", detail: `pre_fault_snapshot_failed: durability not checked (${record.preFaultSnapshotError})` }; + } + if (record.lostFiles?.length) { + return { verdict: "fail", detail: `pre_fault_files_lost: ${record.lostFiles.slice(0, 10).join(",")}` }; + } + const fileChecks = oracle.checks.filter(check => /file|workspace_located/.test(check.name) && !check.ok); + if (fileChecks.length && record.finalStatus === "done") { + return { verdict: "fail", detail: `workspace_files_missing_after_done: ${fileChecks.map(check => check.name).join(",")}` }; + } + const transient = record.samples.some(sample => sample.interruption?.workspaceFailure); + return { verdict: "pass", detail: transient ? "workspace failure appeared during recovery and cleared" : "no workspace failure at rest" }; +} + +export function gradeR3(record: TaskRecord): RuleResult { + const last = lastReachable(record); + if (!record.retry) { + if (record.finalStatus !== "done" && !record.timedOut && !record.launchError) { + return { verdict: "fail", detail: `missing_retry: rested ${record.finalStatus ?? "unknown"} with canRetry=${String(last?.interruption?.canRetry ?? false)}` }; + } + return { verdict: "n/a", detail: "no Retry needed" }; + } + if (record.retry.error || record.retry.accepted !== true) { + return { verdict: "fail", detail: `retry_rejected: http=${record.retry.httpStatus ?? "none"} accepted=${String(record.retry.accepted)} ${record.retry.error ?? ""}`.trim() }; + } + const retryAt = record.retry.at; + const newRun = record.samples.some(sample => (sample.runs ?? []).some(run => (ms(run.createdAt) || 0) >= retryAt - 1_000)); + if (!newRun) return { verdict: "fail", detail: "retry_no_new_run" }; + if (record.finalStatus !== "done") return { verdict: "fail", detail: `retry_did_not_complete (status ${record.finalStatus ?? "unknown"}${record.timedOut ? ", timed out" : ""})` }; + return { verdict: "pass", detail: "one Retry started a run that completed the task" }; +} + +/** + * R4: at most one active run (queued, running or scheduled_retry) per issue in + * any sample, and no comment consumed twice. A wake comment may be reused only + * by a run that follows a run which did not succeed (an explicit retry or + * continuation after a failure); reuse after a successful run is a replay. + */ +export function gradeR4(record: TaskRecord): RuleResult { + for (const sample of reachable(record.samples)) { + const active = (sample.runs ?? []).filter(run => ACTIVE_RUN_STATUSES.has(run.status)); + if (active.length > 1) return { verdict: "fail", detail: `concurrent_runs: ${active.map(run => `${run.id}(${run.status})`).join(",")}` }; + } + const finalRuns = new Map[number]>(); + for (const sample of reachable(record.samples)) for (const run of sample.runs ?? []) finalRuns.set(run.id, run); + const begun = [...finalRuns.values()].filter(run => run.startedAt) + .sort((a, b) => (ms(a.createdAt) || 0) - (ms(b.createdAt) || 0)); + const consumers = new Map(); + for (const run of begun) for (const commentId of run.wakeCommentIds) consumers.set(commentId, [...(consumers.get(commentId) ?? []), run]); + for (const [commentId, runs] of consumers) { + const replayed = runs.findIndex((_run, index) => index > 0 && runs.slice(0, index).some(prior => prior.status === "succeeded")); + if (replayed > 0) return { verdict: "fail", detail: `comment_consumed_twice: ${commentId} by ${runs.slice(0, replayed + 1).map(run => run.id).join(",")}` }; + } + return { verdict: "pass", detail: "at most one active run per sample; no comment consumed after a successful run" }; +} + +function namedWait(sample: TaskSample): boolean { + const interruption = sample.interruption; + return Boolean(sample.executionBlocker || sample.scheduledRetry || sample.activeRecoveryAction || interruption?.retryWait + || interruption?.state === "retry_scheduled" || interruption?.canRetry + || (interruption && ["failed", "waiting_for_user"].includes(interruption.state))); +} + +export function isStalledSample(sample: TaskSample): boolean { + if (!sample.reachable || !sample.serverReady) return false; + if (!REST_STATUSES_WITH_WORK.has(sample.issueStatus ?? "")) return false; + if ((sample.runs ?? []).some(run => ACTIVE_RUN_STATUSES.has(run.status))) return false; + return !namedWait(sample); +} + +export function gradeR5(record: TaskRecord): RuleResult { + let stallStart: number | null = null; + let worst = 0; + for (const sample of record.samples) { + if (!sample.reachable || !sample.serverReady) { stallStart = null; continue; } + if (isStalledSample(sample)) { + stallStart ??= sample.at; + worst = Math.max(worst, sample.at - stallStart); + if (sample.at - stallStart > STALL_LIMIT_MS) { + return { verdict: "fail", detail: `silent_stall ${Math.round((sample.at - stallStart) / 1000)}s in ${sample.issueStatus} with no live run, wake or wait reason` }; + } + } else stallStart = null; + } + return { verdict: "pass", detail: worst ? `longest unexplained idle ${Math.round(worst / 1000)}s` : "no unexplained idle" }; +} + +export function gradeR6(record: TaskRecord): RuleResult { + const sweep = record.sweep; + if (!sweep) return { verdict: "fail", detail: "rest_sweep_missing" }; + if (sweep.error) return { verdict: "fail", detail: `rest_sweep_error: ${sweep.error}` }; + const leaks = [ + ...(sweep.activeRuns.length ? [`active_runs=${sweep.activeRuns.join(",")}`] : []), + ...(sweep.runningWorkspaceOperations.length ? [`running_workspace_operations=${sweep.runningWorkspaceOperations.join(",")}`] : []), + ...(sweep.activeRecoveryAction ? ["active_recovery_action"] : []), + ]; + return leaks.length ? { verdict: "fail", detail: `leak: ${leaks.join("; ")}` } : { verdict: "pass", detail: "clean rest" }; +} + +export function gradeTask(record: TaskRecord, faults: readonly FaultReceipt[], budgetMs: number, oracle: OracleResult): TaskGrade { + const crossing = crossingFaults(record, faults); + const rules: Record = { + R1: gradeR1(record, budgetMs, faults), + R2: gradeR2(record, oracle), + R3: gradeR3(record), + R4: gradeR4(record), + R5: gradeR5(record), + R6: gradeR6(record), + }; + const failed = (Object.entries(rules) as Array<[RuleId, RuleResult]>).find(([, rule]) => rule.verdict === "fail"); + const pass = !failed && oracle.verdict === "pass"; + let signature: string | null = null; + if (failed) signature = `${failed[0]}:${failed[1].detail.split(/[\s:(]/)[0]}:${record.plan.workload}:${record.plan.fault}`; + else if (oracle.verdict === "fail") signature = `T:${oracle.checks.find(check => !check.ok)?.name ?? "unknown"}:${record.plan.workload}:${record.plan.fault}`; + return { crossed: crossing.length > 0, crossingFaults: crossing, rules, T: oracle, pass, signature }; +} diff --git a/tests/restart-chaos/ledger.ts b/tests/restart-chaos/ledger.ts new file mode 100644 index 00000000000..4569a116c8b --- /dev/null +++ b/tests/restart-chaos/ledger.ts @@ -0,0 +1,91 @@ +import { appendFile, mkdir, readFile, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { replayCommand } from "./config.js"; +import type { RuleId, TaskGrade, TaskRecord, WavePlan } from "./types.js"; + +export interface LedgerLine { + schema: "paperclip.restart-chaos.ledger/v1"; + waveId: string; + taskId: string; + seed: number; + gitSha: string; + env: string; + adapter: string; + workload: string; + fault: string; + holdMs: number; + fireAfterMs: number; + issue: string | null; + crossed: boolean; + crossingFaults: string[]; + rules: Record; + ruleDetails: Record; + T: string; + pass: boolean; + signature: string | null; + retried: boolean; + evidence: string | null; + replay: string; + recordedAt: string; +} + +export function ledgerLine(plan: WavePlan, record: TaskRecord, grade: TaskGrade, gitSha: string, evidence: string | null): LedgerLine { + const rules = Object.fromEntries(Object.entries(grade.rules).map(([id, rule]) => [id, rule.verdict])) as Record; + const ruleDetails = Object.fromEntries(Object.entries(grade.rules).map(([id, rule]) => [id, rule.detail])) as Record; + return { + schema: "paperclip.restart-chaos.ledger/v1", waveId: plan.waveId, taskId: record.plan.taskId, seed: plan.seed, gitSha, + env: plan.options.env, adapter: record.plan.adapter, workload: record.plan.workload, fault: record.plan.fault, + holdMs: record.plan.holdMs, fireAfterMs: record.plan.fireAfterMs, issue: record.identifier ?? record.issueId, + crossed: grade.crossed, crossingFaults: grade.crossingFaults, rules, ruleDetails, T: grade.T.verdict, pass: grade.pass, + signature: grade.signature, retried: Boolean(record.retry), evidence, replay: replayCommand(plan.waveId, record.plan.taskId), + recordedAt: new Date().toISOString(), + }; +} + +export async function appendLedger(resultsRoot: string, lines: readonly LedgerLine[]) { + await mkdir(resultsRoot, { recursive: true }); + await appendFile(path.join(resultsRoot, "ledger.jsonl"), lines.map(line => `${JSON.stringify(line)}\n`).join(""), { mode: 0o600 }); +} + +export async function readLedger(resultsRoot: string): Promise { + const text = await readFile(path.join(resultsRoot, "ledger.jsonl"), "utf8").catch(() => ""); + return text.split("\n").filter(Boolean).map(line => JSON.parse(line) as LedgerLine); +} + +/** Short summary: per-cell pass rate and open signatures across the ledger. */ +export function renderSummary(lines: readonly LedgerLine[], latestWave?: string): string { + const cells = new Map(); + const signatures = new Map(); + for (const line of lines) { + const key = `${line.env} · ${line.adapter} · ${line.workload} · ${line.fault}`; + const cell = cells.get(key) ?? { pass: 0, total: 0, crossed: 0 }; + cell.total += 1; if (line.pass) cell.pass += 1; if (line.crossed) cell.crossed += 1; + cells.set(key, cell); + if (line.signature) { + const entry = signatures.get(line.signature) ?? { count: 0, latest: line }; + entry.count += 1; entry.latest = line; + signatures.set(line.signature, entry); + } + } + const out = ["# Restart chaos summary", "", `Tasks graded: ${lines.length}; passed: ${lines.filter(line => line.pass).length}; restart-crossing: ${lines.filter(line => line.crossed).length}.`, ""]; + if (latestWave) { + out.push(`## Latest wave ${latestWave}`, "", "| Task | Cell | Crossed | R1 | R2 | R3 | R4 | R5 | R6 | T | Result |", "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |"); + for (const line of lines.filter(candidate => candidate.waveId === latestWave)) { + out.push(`| ${line.taskId} ${line.issue ?? ""} | ${line.adapter} ${line.workload} ${line.fault} | ${line.crossed ? "yes" : "no"} | ${line.rules.R1} | ${line.rules.R2} | ${line.rules.R3} | ${line.rules.R4} | ${line.rules.R5} | ${line.rules.R6} | ${line.T} | ${line.pass ? "PASS" : `FAIL ${line.signature}`} |`); + } + out.push(""); + } + out.push("## Cells", "", "| Cell | Pass | Total | Crossed |", "| --- | --- | --- | --- |"); + for (const [key, cell] of [...cells].sort()) out.push(`| ${key} | ${cell.pass} | ${cell.total} | ${cell.crossed} |`); + out.push("", "## Open signatures", ""); + if (!signatures.size) out.push("None."); + for (const [signature, entry] of signatures) out.push(`- \`${signature}\` × ${entry.count} — latest ${entry.latest.waveId}/${entry.latest.taskId}; evidence ${entry.latest.evidence ?? "n/a"}; replay \`${entry.latest.replay}\``); + out.push(""); + return out.join("\n"); +} + +export async function writeSummary(resultsRoot: string, latestWave: string) { + const summary = renderSummary(await readLedger(resultsRoot), latestWave); + await writeFile(path.join(resultsRoot, "SUMMARY.md"), summary, { mode: 0o600 }); + return summary; +} diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts new file mode 100644 index 00000000000..7fd6981e028 --- /dev/null +++ b/tests/restart-chaos/main.ts @@ -0,0 +1,188 @@ +/** + * Control-plane restart chaos harness, Phase 1 MVP. + * See doc/plans/2026-11-06-control-plane-restart-chaos.md and ./README.md. + * + * node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode \ + * --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 + * node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay / + */ +import { execFileSync, spawnSync } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { ChaosApi } from "./api.js"; +import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "./config.js"; +import { knownSecretValues, writeEvidenceBundle } from "./evidence.js"; +import { gradeTask } from "./grader.js"; +import { appendLedger, ledgerLine, writeSummary, type LedgerLine } from "./ledger.js"; +import { TaskObserver } from "./observer.js"; +import { observedFaultSchedule, planWave, replayWave } from "./scheduler.js"; +import { seedChaosLab } from "./seed.js"; +import { ChaosSupervisor } from "./supervisor.js"; +import { Timeline } from "./timeline.js"; +import type { FaultReceipt, OracleArtifacts, WavePlan } from "./types.js"; + +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); +const resultsRoot = path.resolve(process.env.RESTART_CHAOS_RESULTS_DIR ?? path.join(import.meta.dirname, "results")); +const QUIET_BEFORE_SWEEP_MS = 15_000; + +function git(args: string[]) { + try { return execFileSync("git", args, { cwd: repositoryRoot, encoding: "utf8" }).trim(); } catch { return ""; } +} + +async function loadPlan(): Promise<{ plan: WavePlan; keep: boolean }> { + const command = parseCommand(process.argv.slice(2)); + if (command.mode === "replay") { + const sourceDir = path.join(resultsRoot, command.ref.waveId); + const source = JSON.parse(await readFile(path.join(sourceDir, "plan.json"), "utf8")) as WavePlan; + // Replay the faults that actually crossed the task, including other tasks' faults. + const graded = await readFile(path.join(sourceDir, `${command.ref.taskId}.grade.json`), "utf8") + .then(text => JSON.parse(text) as { record: { createdAt: number | null; faultIds: string[] } }).catch(() => null); + const faults = await readFile(path.join(sourceDir, "faults.json"), "utf8").then(text => JSON.parse(text) as FaultReceipt[]).catch(() => []); + const observed = graded ? observedFaultSchedule(graded.record.createdAt, graded.record.faultIds, faults) : []; + if (!observed.length) console.warn("Replay: no recorded fault sequence for this task; using its planned fault only."); + return { plan: replayWave(source, command.ref, Date.now(), command.overrides, observed), keep: command.keep }; + } + return { plan: planWave(waveIdFor(command.options.seed), command.options), keep: command.keep }; +} + +function preflight(plan: WavePlan) { + const missing = [...new Set(plan.tasks.map(task => CREDENTIAL_FOR_ADAPTER[task.adapter]))].filter(key => !process.env[key]); + if (missing.length) throw new Error(`Missing credentials ${missing.join(", ")}. Run: set -a; source ~/.secrets; set +a`); + const runner = process.env.PAPERCLIP_RUNNER_BINARY + ?? path.join(repositoryRoot, "packages/paperclip-runner/runner/target/debug/paperclip-runnerd"); + if (!existsSync(runner)) throw new Error(`Runner binary missing at ${runner}. Run: pnpm --filter @paperclipai/paperclip-runner build:runner-binaries`); + // Same provider bundle preparation as tests/runner-e2e/verified-provider-bundles.setup.mjs. + const build = spawnSync(process.execPath, [path.join(repositoryRoot, "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs")], + { cwd: repositoryRoot, stdio: "inherit" }); + if (build.status !== 0) throw new Error("Verified provider entrypoint build failed"); +} + +async function main(): Promise { + const { plan, keep } = await loadPlan(); + const waveDir = path.join(resultsRoot, plan.waveId); + await mkdir(waveDir, { recursive: true, mode: 0o700 }); + await writeFile(path.join(waveDir, "plan.json"), `${JSON.stringify(plan, null, 2)}\n`, { mode: 0o600 }); + const timeline = new Timeline(path.join(waveDir, "timeline.jsonl")); + const gitSha = git(["rev-parse", "HEAD"]); + const gitDirty = git(["status", "--porcelain"]).length > 0; + timeline.add("wave_planned", { waveId: plan.waveId, seed: plan.seed, gitSha, gitDirty, replayOf: plan.replayOf ?? null, + tasks: plan.tasks.map(task => `${task.taskId}:${task.adapter}/${task.workload}/${task.fault}@${task.fireAfterMs}ms${task.holdMs ? ` hold ${task.holdMs}ms` : ""}`) }); + preflight(plan); + + const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline }); + let interrupted = false; + const onSignal = (signal: NodeJS.Signals) => { if (!interrupted) { interrupted = true; timeline.add("harness_interrupted", { signal }); } }; + process.on("SIGINT", onSignal); process.on("SIGTERM", onSignal); + const faults: FaultReceipt[] = []; + const faultWork: Promise[] = []; + const lines: LedgerLine[] = []; + let anyFailure = false; + let harnessError: unknown; + let faultFailures = 0; + try { + await supervisor.start(); + const api = new ChaosApi(supervisor.baseUrl); + const lab = await seedChaosLab({ api, env: plan.options.env, adapters: [...new Set(plan.tasks.map(task => task.adapter))], + concurrency: plan.options.concurrency, budgetCents: plan.options.budgetCents, + fixtureRoot: path.join(supervisor.paths.temporaryRoot, "fixtures"), waveId: plan.waveId, timeline }); + // The instance root holds the secrets master key and the database: oracle code may not read it. + const protectedRoots = [os.homedir(), supervisor.paths.temporaryRoot, resultsRoot]; + const observers = plan.tasks.map(task => new TaskObserver(task, api, lab, timeline, supervisor, protectedRoots)); + const pending = [...observers]; + const active: TaskObserver[] = []; + const lastPoll = new Map(); + let lastDoneAt = Date.now(); + + while (!interrupted && (pending.length || active.length)) { + const now = Date.now(); + // A failed fault or an unexpected exit leaves the server not ready. + // Recover within a bound, or fail the wave instead of waiting forever. + if (!supervisor.serverReady && !supervisor.faultInProgress) await supervisor.ensureServing(); + while (active.length < plan.options.concurrency && pending.length && supervisor.serverReady && !supervisor.faultInProgress) { + const next = pending.shift()!; + await next.launch(); + if (!next.done) active.push(next); + } + await Promise.all(active.filter(observer => now - (lastPoll.get(observer) ?? 0) >= plan.options.pollMs).map(async observer => { + lastPoll.set(observer, now); + await observer.poll(now); + })); + for (const observer of active) { + const due = observer.faultDue(Date.now()); + if (!due) continue; + if (observer.record.restAt !== null) { timeline.add("fault_skipped_task_resting", { taskId: observer.plan.taskId, fault: due.kind }); continue; } + const exposed = active.filter(candidate => !candidate.done); + await Promise.all(exposed.map(candidate => candidate.snapshotBeforeFault())); + faultWork.push(supervisor.fault(due.kind, due.holdMs, observer.plan.taskId).then(receipt => { + faults.push(receipt); + for (const candidate of exposed) { candidate.noteFault(receipt.faultId); candidate.extendDeadline((receipt.readyAt ?? Date.now()) - receipt.startedAt); } + }, (error: Error & { receipt?: FaultReceipt }) => { + if (error.receipt) faults.push(error.receipt); + faultFailures += 1; + timeline.add("fault_error", { taskId: observer.plan.taskId, error: error.message }); + })); + } + for (let index = active.length - 1; index >= 0; index -= 1) { + if (active[index]!.done) { active.splice(index, 1); lastDoneAt = Date.now(); } + } + await delay(1_000); + } + await Promise.all(faultWork); + // An injected fault that did not complete invalidates the wave, even when + // bounded recovery brought the server back and the tasks passed. + if (faultFailures) harnessError ??= new Error(`${faultFailures} injected fault(s) failed; see fault_error in the timeline`); + if (interrupted) throw new Error("Harness interrupted before the wave finished"); + + // R6: let finalizers settle, then sweep for anything still pending. + const quietFor = Date.now() - lastDoneAt; + if (quietFor < QUIET_BEFORE_SWEEP_MS) await delay(QUIET_BEFORE_SWEEP_MS - quietFor); + timeline.add("wave_sweep", {}); + const secrets = knownSecretValues(); + for (const observer of observers) { + observer.record.sweep = await observer.sweep(); + let artifacts: OracleArtifacts | null = null; + if (observer.record.issueId) artifacts = await observer.collectArtifacts().catch(() => null); + const oracle = artifacts ? observer.definition.grade(observer.plan.nonce, artifacts) + : { verdict: "fail" as const, checks: [{ name: "artifacts_collected", ok: false, detail: observer.record.launchError ?? "collection failed" }] }; + const grade = gradeTask(observer.record, faults, observer.definition.budgetMs, oracle); + let evidence: string | null = null; + if (!grade.pass) { + anyFailure = true; + evidence = await writeEvidenceBundle({ directory: path.join(waveDir, "evidence", observer.plan.taskId), api, plan, + record: observer.record, grade, artifacts, faults, timeline: timeline.entries, serverLogPath: supervisor.paths.serverLogPath, + gitSha, gitDirty, issue: observer.issue, secrets }).catch(error => { timeline.add("evidence_failed", { taskId: observer.plan.taskId, error: String(error) }); return null; }); + } + await writeFile(path.join(waveDir, `${observer.plan.taskId}.grade.json`), `${JSON.stringify({ record: observer.record, grade }, null, 2)}\n`, { mode: 0o600 }); + lines.push(ledgerLine(plan, observer.record, grade, gitSha, evidence)); + timeline.add("task_graded", { taskId: observer.plan.taskId, pass: grade.pass, crossed: grade.crossed, signature: grade.signature, + rules: Object.fromEntries(Object.entries(grade.rules).map(([id, rule]) => [id, rule.verdict])), T: grade.T.verdict }); + } + await writeFile(path.join(waveDir, "faults.json"), `${JSON.stringify(faults, null, 2)}\n`, { mode: 0o600 }); + } catch (error) { + harnessError = error; + timeline.add("harness_error", { error: error instanceof Error ? error.stack ?? error.message : String(error) }); + } finally { + try { await supervisor.stop(); } catch (error) { + // Incomplete cleanup is a harness failure; keep the instance for inspection. + harnessError ??= error; + timeline.add("supervisor_stop_failed", { error: String(error) }); + } + if (lines.length) { + await appendLedger(resultsRoot, lines); + const summary = await writeSummary(resultsRoot, plan.waveId); + await writeFile(path.join(waveDir, "SUMMARY.md"), summary, { mode: 0o600 }); + console.info(`\n${summary}`); + } + if (keep || anyFailure || harnessError) timeline.add("instance_kept", { temporaryRoot: supervisor.paths.temporaryRoot }); + else await rm(supervisor.paths.temporaryRoot, { recursive: true, force: true }).catch(() => undefined); + } + console.info(`Wave ${plan.waveId}: results ${waveDir}`); + for (const line of lines.filter(candidate => !candidate.pass)) console.info(` FAIL ${line.taskId} ${line.signature} → ${line.evidence ?? "no evidence"}\n replay: ${replayCommand(plan.waveId, line.taskId)}`); + if (harnessError) { console.error(harnessError); return 2; } + return anyFailure ? 1 : 0; +} + +process.exit(await main().catch(error => { console.error(error); return 2; })); diff --git a/tests/restart-chaos/observer.test.ts b/tests/restart-chaos/observer.test.ts new file mode 100644 index 00000000000..9af1752ed34 --- /dev/null +++ b/tests/restart-chaos/observer.test.ts @@ -0,0 +1,118 @@ +import { execFileSync } from "node:child_process"; +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import type { ChaosApi } from "./api.js"; +import { REST_CONFIRM_MS, STALL_GIVE_UP_MS, TaskObserver, listWorkspaceFiles } from "./observer.js"; +import { createFixtureRepository, type ChaosLab } from "./seed.js"; +import { Timeline } from "./timeline.js"; +import type { TaskPlan } from "./types.js"; +import { BENCH_EXPECTED, parseBenchRecord } from "./workloads.js"; + +process.env.RESTART_CHAOS_QUIET = "1"; +const lab: ChaosLab = { companyId: "c1", environmentId: "e1", projectId: "p1", fixtureRepositoryPath: "/tmp/none", + agents: { codex: { id: "a1", name: "Chaos codex" }, claude: { id: "a2", name: "Chaos claude" }, opencode: { id: "a3", name: "Chaos opencode" } } }; +const plan: TaskPlan = { taskId: "t01", index: 0, workload: "C1", adapter: "codex", fault: "K1", fireAfterMs: 999_999, + holdMs: 0, followUpAfterMs: 1_000, nonce: "abcd1234" }; +const ready = { serverReady: true, faultInProgress: false }; +const directories: string[] = []; +afterEach(async () => { await Promise.all(directories.splice(0).map(directory => rm(directory, { recursive: true, force: true }))); }); + +/** Fake board API: a C1 issue whose first run already finished. */ +function fakeApi(overrides: Partial unknown>> = {}, status = "done") { + const posted: string[] = []; + const api = { + async get(route: string) { + for (const [prefix, handler] of Object.entries(overrides)) if (route.includes(prefix)) return handler!(route); + if (route.endsWith("/runs")) return [{ runId: "r1", status: "succeeded", createdAt: new Date(0).toISOString(), finishedAt: new Date(1).toISOString() }]; + return { id: "i1", status, interruption: null }; + }, + async post(route: string) { posted.push(route); return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "comment-1" }; }, + }; + return { api: api as unknown as ChaosApi, posted }; +} + +describe("TaskObserver", () => { + it("does not finish a C1 task on the poll that posts the follow-up", async () => { + const { api, posted } = fakeApi(); + const observer = new TaskObserver(plan, api, lab, new Timeline(null), ready); + await observer.launch(); + const start = observer.record.createdAt!; + // The task rests before the follow-up is due, for longer than the rest window. + await observer.poll(start + 100); + await observer.poll(start + 100 + REST_CONFIRM_MS + 1); + expect(observer.done).toBe(false); + expect(observer.record.followUp).toMatchObject({ commentId: "comment-1" }); + expect(posted.some(route => route.endsWith("/comments"))).toBe(true); + // A fresh rest window must elapse after the comment before the task counts as resting. + await observer.poll(start + 200 + REST_CONFIRM_MS + 2); + expect(observer.done).toBe(false); + await observer.poll(start + 300 + 2 * REST_CONFIRM_MS + 3); + expect(observer.done).toBe(true); + }); + + it("keeps watching unexplained idle work until the stall limit, then records the stall", async () => { + const { api } = fakeApi({}, "in_progress"); + const observer = new TaskObserver({ ...plan, workload: "A1", followUpAfterMs: null }, api, lab, new Timeline(null), ready); + await observer.launch(); + const start = observer.record.createdAt!; + await observer.poll(start + 1_000); + await observer.poll(start + 1_000 + REST_CONFIRM_MS + 1); + expect(observer.done).toBe(false); + await observer.poll(start + 1_000 + STALL_GIVE_UP_MS + 1); + expect(observer.done).toBe(true); + expect(observer.record.finalStatus).toBe("in_progress"); + expect(observer.record.restAt).toBe(start + 1_000); + }); + + it("reports a failed workspace-operation read as a sweep error, not a clean rest", async () => { + const { api } = fakeApi({ "/workspace-operations": () => { throw new Error("timeout"); } }); + const observer = new TaskObserver(plan, api, lab, new Timeline(null), ready); + await observer.launch(); + const sweep = await observer.sweep(); + expect(sweep.error).toMatch(/timeout/); + }); + + it("leaves R2 unchecked instead of failing the wave when the pre-fault snapshot cannot read the workspace", async () => { + const { api } = fakeApi({ "/api/issues/i1": route => route.endsWith("/runs") ? [] + : { id: "i1", status: "in_progress", currentExecutionWorkspace: { cwd: "/nonexistent/restart-chaos-workspace" } } }); + const observer = new TaskObserver({ ...plan, workload: "B1", followUpAfterMs: null }, api, lab, new Timeline(null), ready); + await observer.launch(); + await observer.poll(observer.record.createdAt! + 1_000); + await expect(observer.snapshotBeforeFault()).resolves.toBeUndefined(); + expect(observer.record.preFaultFiles).toBeNull(); + expect(observer.record.preFaultSnapshotError).toMatch(/ENOENT/); + }); + + it("fires the planned fault once, after its offset", async () => { + const { api } = fakeApi(); + const observer = new TaskObserver({ ...plan, fireAfterMs: 5_000 }, api, lab, new Timeline(null), ready); + await observer.launch(); + const start = observer.record.createdAt!; + expect(observer.faultDue(start + 4_999)).toBeNull(); + expect(observer.faultDue(start + 5_000)).toEqual({ kind: "K1", holdMs: 0, atMs: 5_000 }); + expect(observer.faultDue(start + 6_000)).toBeNull(); + }); +}); + +describe("fixture repository", () => { + it("lists workspace files without VCS or cache directories", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-fixture-")); + directories.push(root); + await createFixtureRepository(root); + const files = await listWorkspaceFiles(root); + expect(files).toContain("chaoscli/cli.py"); + expect(files.some(file => file.startsWith(".git/"))).toBe(false); + }); + + it("bench.sh writes a tagged completion record with the deterministic result", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-bench-")); + directories.push(root); + await createFixtureRepository(root, 1); + const output = execFileSync("./bench.sh", ["abcd1234"], { cwd: root, encoding: "utf8", timeout: 60_000 }); + expect(output.trim().split("\n").at(-1)).toBe(BENCH_EXPECTED); + const record = parseBenchRecord(await readFile(path.join(root, ".bench", "run-abcd1234.txt"), "utf8")); + expect(record?.result).toBe(BENCH_EXPECTED); + }, 60_000); +}); diff --git a/tests/restart-chaos/observer.ts b/tests/restart-chaos/observer.ts new file mode 100644 index 00000000000..1f0eb328177 --- /dev/null +++ b/tests/restart-chaos/observer.ts @@ -0,0 +1,334 @@ +import { randomUUID } from "node:crypto"; +import { readdir, readFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { ChaosApiError, type ChaosApi } from "./api.js"; +import { ACTIVE_RUN_STATUSES, isStalledSample, STALL_LIMIT_MS } from "./grader.js"; +import type { ChaosLab } from "./seed.js"; +import type { Timeline } from "./timeline.js"; +import type { + InterruptionSample, OracleArtifacts, RestSweep, RetryAttempt, RunSample, ScheduledFault, TaskPlan, TaskRecord, TaskSample, +} from "./types.js"; +import { runOracleCommand } from "./oracle-sandbox.js"; +import { taskFaultSchedule } from "./scheduler.js"; +import { workloadDefinitions } from "./workloads.js"; + +const IGNORED_DIRECTORIES = new Set([".git", ".bench", "__pycache__", "node_modules"]); +const MAX_SNAPSHOT_FILES = 2_000; + +/** Relative paths of workspace files, excluding VCS and cache directories. */ +export async function listWorkspaceFiles(root: string): Promise { + const out: string[] = []; + const walk = async (relative: string) => { + if (out.length >= MAX_SNAPSHOT_FILES) return; + const entries = await readdir(path.join(root, relative), { withFileTypes: true }); + for (const entry of entries) { + const child = relative ? `${relative}/${entry.name}` : entry.name; + if (entry.isDirectory()) { if (!IGNORED_DIRECTORIES.has(entry.name)) await walk(child); } + else if (entry.isFile()) out.push(child); + if (out.length >= MAX_SNAPSHOT_FILES) return; + } + }; + await walk(""); + return out.sort(); +} + +/** Rest must hold for this long before the observer acts on it. */ +export const REST_CONFIRM_MS = 12_000; +/** Extra time granted after a Retry click before R3 is graded as timed out. */ +export const RETRY_BUDGET_MS = 15 * 60_000; + +type Json = Record; + +export function toRunSample(run: Json): RunSample { + const list = (value: unknown) => (Array.isArray(value) ? value.filter((item): item is string => typeof item === "string") : []); + const wake = new Set([...list(run.wakeCommentIds), ...(typeof run.wakeCommentId === "string" ? [run.wakeCommentId] : [])]); + return { + id: String(run.runId ?? run.id), status: String(run.status), createdAt: run.createdAt ?? null, startedAt: run.startedAt ?? null, + finishedAt: run.finishedAt ?? null, errorCode: run.errorCode ?? null, retryOfRunId: run.retryOfRunId ?? null, + wakeCommentIds: [...wake], livenessState: run.livenessState ?? null, + }; +} + +export function toInterruption(value: Json | null | undefined): InterruptionSample | null { + if (!value) return null; + return { state: String(value.state), canRetry: value.canRetry === true, retryWait: Boolean(value.retryWait), + workspaceFailure: value.workspaceFailure?.message ?? null, observedRunId: value.observedRunId ?? null, userAction: value.userAction ?? null }; +} + +/** Rest = no live run and nothing the system itself is still doing. */ +export function isResting(sample: TaskSample): boolean { + if (!sample.reachable || !sample.serverReady) return false; + if ((sample.runs ?? []).some(run => ACTIVE_RUN_STATUSES.has(run.status))) return false; + if (sample.scheduledRetry) return false; + // A board-owned recovery waits on a person: that is a resting state. + if (sample.activeRecoveryAction && sample.recoveryOwner !== "board") return false; + const interruption = sample.interruption; + if (interruption && (interruption.state === "working" || interruption.state === "retry_scheduled" || interruption.retryWait)) return false; + // Unexplained idle work is a possible stall, not a rest: keep watching it so + // the task can still resume and R5 can measure the stall. + if (isStalledSample(sample)) return false; + return true; +} + +/** Stop watching an unexplained idle task once it exceeds the R5 stall limit. */ +export const STALL_GIVE_UP_MS = STALL_LIMIT_MS + 15_000; + +export class TaskObserver { + readonly record: TaskRecord; + issue: Json | null = null; + private restCandidateSince: number | null = null; + private stallSince: number | null = null; + private readonly faultSchedule: ScheduledFault[]; + private nextFault = 0; + done = false; + + constructor(readonly plan: TaskPlan, private readonly api: ChaosApi, private readonly lab: ChaosLab, + private readonly timeline: Timeline, private readonly supervisor: { serverReady: boolean; faultInProgress: boolean }, + /** Paths oracle commands may not read: the operator's home, the instance root (secrets key, database) and results. */ + private readonly protectedRoots: readonly string[] = [os.homedir()]) { + this.record = { plan, issueId: null, identifier: null, createdAt: null, launchError: null, samples: [], retry: null, + restAt: null, finalStatus: null, deadlineAt: null, timedOut: false, followUp: null, faultIds: [], sweep: null, + preFaultFiles: null, lostFiles: null }; + this.faultSchedule = taskFaultSchedule(plan); + } + + get definition() { return workloadDefinitions[this.plan.workload]; } + + async launch(): Promise { + const definition = this.definition; + try { + const issue = await this.api.post(`/api/companies/${this.lab.companyId}/issues`, { + title: definition.title(this.plan.nonce), + description: definition.description(this.plan.nonce), + status: "todo", + priority: "medium", + assigneeAgentId: this.lab.agents[this.plan.adapter].id, + ...(definition.usesRepository ? { projectId: this.lab.projectId } : {}), + }); + this.record.issueId = String(issue.id); + this.record.identifier = issue.identifier ?? null; + this.record.createdAt = Date.now(); + this.record.deadlineAt = this.record.createdAt + definition.budgetMs; + this.timeline.add("task_launched", { taskId: this.plan.taskId, issueId: issue.id, identifier: issue.identifier, + workload: this.plan.workload, adapter: this.plan.adapter, faults: this.faultSchedule }); + } catch (error) { + this.record.launchError = error instanceof Error ? error.message : String(error); + this.done = true; + this.timeline.add("task_launch_failed", { taskId: this.plan.taskId, error: this.record.launchError }); + } + } + + /** The next scheduled fault once it is due and the task has not rested yet. */ + faultDue(now: number): ScheduledFault | null { + if (this.done || this.record.createdAt === null) return null; + const fault = this.faultSchedule[this.nextFault]; + if (!fault || now < this.record.createdAt + fault.atMs) return null; + this.nextFault += 1; + return fault; + } + + /** + * B5: read bench.sh's progress file while the task is live. The harness sees + * the values change over minutes, which a computed answer cannot produce. + */ + private async observeBenchProgress(): Promise { + if (this.plan.workload !== "B5") return; + const cwd = (this.issue?.currentExecutionWorkspace as Json | null | undefined)?.cwd; + if (typeof cwd !== "string") return; + const value = (await readFile(path.join(cwd, ".bench", "progress"), "utf8").catch(() => "")).trim(); + if (!value) return; + const seen = this.record.benchProgressSeen ??= []; + if (!seen.includes(value)) seen.push(value); + } + + /** R2: remember which workspace files existed before the first fault hit this task. */ + async snapshotBeforeFault(): Promise { + if (this.record.preFaultFiles !== null || this.done) return; + // One task's snapshot failure must not end the wave; R2 fails with the reason. + try { + const workspace = await this.workspacePath(); + if (workspace) this.record.preFaultFiles = await listWorkspaceFiles(workspace); + } catch (error) { + this.record.preFaultSnapshotError = error instanceof Error ? error.message : String(error); + this.timeline.add("pre_fault_snapshot_failed", { taskId: this.plan.taskId, error: this.record.preFaultSnapshotError }); + } + } + + noteFault(faultId: string) { if (!this.record.faultIds.includes(faultId)) this.record.faultIds.push(faultId); } + + /** Outages extend the deadline: R1 allows budget plus outage time. */ + extendDeadline(ms: number) { if (this.record.deadlineAt !== null) this.record.deadlineAt += ms; } + + async poll(now = Date.now()): Promise { + if (this.done || !this.record.issueId) return; + const sample = await this.sample(now); + this.record.samples.push(sample); + await this.observeBenchProgress(); + if (await this.maybeFollowUp(sample, now)) { + // The comment response is not the agent's answer: require fresh samples. + this.restCandidateSince = null; + return; + } + this.stallSince = isStalledSample(sample) ? (this.stallSince ?? now) : null; + if (this.stallSince !== null && now - this.stallSince > STALL_GIVE_UP_MS) { + this.record.restAt = this.stallSince; + this.record.finalStatus = sample.issueStatus ?? null; + this.done = true; + this.timeline.add("task_stalled", { taskId: this.plan.taskId, status: sample.issueStatus, idleMs: now - this.stallSince }); + return; + } + if (!isResting(sample) || this.supervisor.faultInProgress) { + this.restCandidateSince = null; + } else { + this.restCandidateSince ??= now; + if (now - this.restCandidateSince >= REST_CONFIRM_MS) { + await this.onRest(sample, now); + return; + } + } + if (this.record.deadlineAt !== null && now > this.record.deadlineAt && !this.supervisor.faultInProgress) { + this.record.timedOut = true; + this.record.finalStatus = sample.issueStatus ?? this.record.finalStatus; + this.done = true; + this.timeline.add("task_timed_out", { taskId: this.plan.taskId, status: sample.issueStatus }); + } + } + + private async sample(now: number): Promise { + const serverReady = this.supervisor.serverReady && !this.supervisor.faultInProgress; + try { + const [issue, runs] = await Promise.all([ + this.api.get(`/api/issues/${this.record.issueId}`, 5_000), + this.api.get(`/api/issues/${this.record.issueId}/runs`, 5_000), + ]); + this.issue = issue; + return { at: now, reachable: true, serverReady, issueStatus: String(issue.status), + interruption: toInterruption(issue.interruption), activeRecoveryAction: Boolean(issue.activeRecoveryAction), + recoveryOwner: issue.activeRecoveryAction?.ownerType ?? null, + executionBlocker: Boolean(issue.executionBlocker), scheduledRetry: Boolean(issue.scheduledRetry), + runs: (runs ?? []).map(toRunSample) }; + } catch { + return { at: now, reachable: false, serverReady: false }; + } + } + + private async maybeFollowUp(sample: TaskSample, now: number): Promise { + const definition = this.definition; + if (!definition.followUp || this.record.followUp || this.plan.followUpAfterMs === null || !sample.reachable) return false; + if (now < this.record.createdAt! + this.plan.followUpAfterMs) return false; + // C1 targets a live run (B8); wait for one unless the task already rests. + const running = (sample.runs ?? []).some(run => run.status === "running"); + if (!running && !isResting(sample)) return false; + try { + const comment = await this.api.post(`/api/issues/${this.record.issueId}/comments`, { body: definition.followUp(this.plan.nonce) }); + this.record.followUp = { at: now, commentId: comment?.id ?? null, error: null }; + } catch (error) { + this.record.followUp = { at: now, commentId: null, error: error instanceof Error ? error.message : String(error) }; + } + this.timeline.add("task_follow_up", { taskId: this.plan.taskId, duringRun: running, ...this.record.followUp }); + return true; + } + + private async onRest(sample: TaskSample, now: number) { + const status = sample.issueStatus ?? null; + // C1: a follow-up posted after rest must be served before the task counts as resting. + if (this.definition.followUp && !this.record.followUp) return; + if (status !== "done" && !this.record.retry && sample.interruption?.canRetry) { + this.record.retry = await this.retry(sample, now); + this.restCandidateSince = null; + if (this.record.retry.accepted && this.record.deadlineAt !== null) this.record.deadlineAt = Math.max(this.record.deadlineAt, now + RETRY_BUDGET_MS); + return; + } + this.record.restAt = this.restCandidateSince ?? now; + this.record.finalStatus = status; + this.done = true; + this.timeline.add("task_rested", { taskId: this.plan.taskId, status, interruption: sample.interruption?.state ?? null, + canRetry: sample.interruption?.canRetry ?? false, retried: Boolean(this.record.retry) }); + } + + /** Exactly one click, with the same body ui/src/hooks/useTaskRetry.ts sends. */ + private async retry(sample: TaskSample, now: number): Promise { + const observedRunId = sample.interruption?.observedRunId ?? (this.issue?.executionBlocker?.runId as string | undefined) ?? null; + const attempt: RetryAttempt = { at: now, observedRunId, httpStatus: null, accepted: null, error: null }; + try { + const result = await this.api.post<{ accepted?: boolean }>(`/api/issues/${this.record.issueId}/retry`, + { idempotencyKey: randomUUID(), observedRunId }); + attempt.httpStatus = 200; + attempt.accepted = result?.accepted === true; + } catch (error) { + attempt.httpStatus = error instanceof ChaosApiError ? error.status : null; + attempt.error = error instanceof Error ? error.message : String(error); + } + this.timeline.add("task_retry_clicked", { taskId: this.plan.taskId, ...attempt }); + return attempt; + } + + async workspacePath(): Promise { + if (!this.definition.usesRepository) return null; + const current = this.issue?.currentExecutionWorkspace as Json | null | undefined; + if (typeof current?.cwd === "string") return current.cwd; + const workspaceId = this.issue?.executionWorkspaceId; + if (typeof workspaceId === "string") { + const workspace = await this.api.get(`/api/execution-workspaces/${workspaceId}`).catch(() => null); + if (typeof workspace?.cwd === "string") return workspace.cwd; + } + return null; + } + + async collectArtifacts(): Promise { + const definition = this.definition; + const issueId = this.record.issueId!; + const issue = await this.api.get(`/api/issues/${issueId}`).catch(() => this.issue); + if (issue) this.issue = issue; + const documents: Record = {}; + for (const key of definition.collect.documents) { + const document = await this.api.get(`/api/issues/${issueId}/documents/${key}`).catch(() => null); + documents[key] = typeof document?.body === "string" ? document.body : null; + } + const comments = await this.api.get(`/api/issues/${issueId}/comments`).catch(() => []); + const workspacePath = await this.workspacePath(); + if (workspacePath && this.record.preFaultFiles) { + const present = new Set(await listWorkspaceFiles(workspacePath)); + this.record.lostFiles = this.record.preFaultFiles.filter(file => !present.has(file)); + } + const files: Record = {}; + const commands: OracleArtifacts["commands"] = {}; + if (workspacePath) { + for (const relative of definition.collect.files(this.plan.nonce)) { + files[relative] = await readFile(path.join(workspacePath, relative), "utf8").catch(() => null); + } + // Agent-editable code: run it sandboxed, away from the operator's home directory. + for (const command of definition.collect.commands) commands[command.name] = await runOracleCommand(command.argv, workspacePath, { protectedRoots: this.protectedRoots }); + } + return { + issueStatus: issue?.status ?? this.record.finalStatus, + documents, + agentComments: (comments ?? []).filter(comment => comment.authorAgentId).map(comment => String(comment.body ?? "")), + files, commands, workspacePath, + ...(this.plan.workload === "B5" ? { benchProgressObserved: this.record.benchProgressSeen?.length ?? 0 } : {}), + }; + } + + /** R6 sweep: nothing still pending once the wave is quiet. */ + async sweep(): Promise { + if (!this.record.issueId) return { activeRuns: [], runningWorkspaceOperations: [], activeRecoveryAction: false, error: null }; + try { + const [issue, runs] = await Promise.all([ + this.api.get(`/api/issues/${this.record.issueId}`), + this.api.get(`/api/issues/${this.record.issueId}/runs`), + ]); + const samples = runs.map(toRunSample); + const operations: string[] = []; + for (const run of samples) { + // A failed read must not look like a clean rest: let it reach sweep.error. + const rows = await this.api.get(`/api/heartbeat-runs/${run.id}/workspace-operations`); + for (const row of rows ?? []) if (row.status === "running") operations.push(String(row.id)); + } + return { activeRuns: samples.filter(run => ACTIVE_RUN_STATUSES.has(run.status)).map(run => run.id), + runningWorkspaceOperations: operations, activeRecoveryAction: Boolean(issue.activeRecoveryAction), error: null }; + } catch (error) { + return { activeRuns: [], runningWorkspaceOperations: [], activeRecoveryAction: false, error: error instanceof Error ? error.message : String(error) }; + } + } +} diff --git a/tests/restart-chaos/oracle-sandbox.test.ts b/tests/restart-chaos/oracle-sandbox.test.ts new file mode 100644 index 00000000000..22c2d09da02 --- /dev/null +++ b/tests/restart-chaos/oracle-sandbox.test.ts @@ -0,0 +1,49 @@ +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { oracleSandboxProfile, runOracleCommand } from "./oracle-sandbox.js"; +import { createFixtureRepository } from "./seed.js"; + +const directories: string[] = []; +afterEach(async () => { await Promise.all(directories.splice(0).map(directory => rm(directory, { recursive: true, force: true }))); }); +const temp = async (prefix: string) => { const directory = await mkdtemp(path.join(os.tmpdir(), prefix)); directories.push(directory); return directory; }; + +describe("oracle sandbox", () => { + it("denies the protected roots and re-allows the workspace", () => { + const profile = oracleSandboxProfile({ workspace: "/w", scratch: "/s", protectedRoots: ["/Users/me"] }); + expect(profile).toContain('(deny file-read* (subpath "/Users/me"))'); + expect(profile).toContain("(deny network*)"); + expect(profile.indexOf('(allow file-read* (subpath "/w")')).toBeGreaterThan(profile.indexOf("(deny file-read*")); + }); + + it("refuses to run unsandboxed on other platforms unless allowed", async () => { + const result = await runOracleCommand(["true"], "/", { platform: "linux" }); + expect(result.exitCode).toBeNull(); + expect(result.stderr).toMatch(/refused/); + }); + + it.runIf(process.platform === "darwin")("protects an instance root that contains the workspace, except the workspace itself", async () => { + const instanceRoot = await temp("restart-chaos-instance-"); + await writeFile(path.join(instanceRoot, "master.key"), "canary-master-key\n"); + const workspace = path.join(instanceRoot, "fixtures", "chaos-sandbox"); + await createFixtureRepository(workspace, 1); + const leak = await runOracleCommand(["/bin/cat", path.join(instanceRoot, "master.key")], workspace, { protectedRoots: [instanceRoot] }); + expect(leak.stdout).not.toContain("canary-master-key"); + const greet = await runOracleCommand(["python3", "-m", "chaoscli", "greet", "chaos"], workspace, { protectedRoots: [instanceRoot] }); + expect(greet.stdout.trim()).toBe("hello, chaos"); + }, 60_000); + + it.runIf(process.platform === "darwin")("keeps agent code away from protected files but runs the fixture tests", async () => { + const secretRoot = await temp("restart-chaos-secret-"); + await writeFile(path.join(secretRoot, ".secrets"), "OPENAI_API_KEY=canary-value\n"); + const workspace = await temp("restart-chaos-ws-"); + await createFixtureRepository(workspace, 1); + const leak = await runOracleCommand(["/bin/cat", path.join(secretRoot, ".secrets")], workspace, { protectedRoots: [secretRoot] }); + expect(leak.exitCode).not.toBe(0); + expect(leak.stdout).not.toContain("canary-value"); + const greet = await runOracleCommand(["python3", "-m", "chaoscli", "greet", "chaos"], workspace, { protectedRoots: [secretRoot] }); + expect(greet).toMatchObject({ exitCode: 0 }); + expect(greet.stdout.trim()).toBe("hello, chaos"); + }, 60_000); +}); diff --git a/tests/restart-chaos/oracle-sandbox.ts b/tests/restart-chaos/oracle-sandbox.ts new file mode 100644 index 00000000000..50ae362b199 --- /dev/null +++ b/tests/restart-chaos/oracle-sandbox.ts @@ -0,0 +1,65 @@ +import { execFile } from "node:child_process"; +import { mkdtemp, realpath, rm } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; + +const execFileAsync = promisify(execFile); +const OUTPUT_LIMIT = 4_000; + +export interface OracleCommandResult { exitCode: number | null; stdout: string; stderr: string } + +/** Quote a path for an SBPL string literal. */ +function sbplString(value: string) { return `"${value.replaceAll("\\", "\\\\").replaceAll("\"", "\\\"")}"`; } + +/** + * macOS Seatbelt profile for a T-oracle command. The command runs code the + * agent could edit, so it may not read the operator's home directory (for + * example ~/.secrets or ~/.ssh), write anywhere except a scratch directory, or + * use the network. The workspace stays readable. + */ +export function oracleSandboxProfile(input: { workspace: string; scratch: string; protectedRoots: readonly string[] }): string { + const lines = ["(version 1)", "(allow default)", "(deny network*)", "(deny file-write*)", + `(allow file-write* (subpath ${sbplString(input.scratch)}) (literal "/dev/null") (literal "/dev/tty"))`]; + for (const root of input.protectedRoots) lines.push(`(deny file-read* (subpath ${sbplString(root)}))`); + // Later rules win: re-allow the workspace and scratch even if they sit under a protected root. + lines.push(`(allow file-read* (subpath ${sbplString(input.workspace)}) (subpath ${sbplString(input.scratch)}))`); + return lines.join("\n"); +} + +/** + * Run one oracle command against an agent workspace with a minimal environment, + * a scratch HOME, and (on macOS) a Seatbelt sandbox that hides the operator's + * home directory. Elsewhere it refuses unless explicitly allowed, because a + * temporary HOME alone does not stop absolute-path reads. + */ +export async function runOracleCommand(argv: readonly string[], workspace: string, options: { + protectedRoots?: readonly string[]; timeoutMs?: number; platform?: NodeJS.Platform; allowUnsandboxed?: boolean; +} = {}): Promise { + const platform = options.platform ?? process.platform; + const sandboxed = platform === "darwin"; + if (!sandboxed && !options.allowUnsandboxed && process.env.RESTART_CHAOS_ALLOW_UNSANDBOXED_ORACLE !== "1") { + return { exitCode: null, stdout: "", stderr: "oracle command refused: no sandbox on this platform (set RESTART_CHAOS_ALLOW_UNSANDBOXED_ORACLE=1 inside a disposable VM)" }; + } + // Seatbelt matches resolved paths (/var is /private/var on macOS). + const scratch = await realpath(await mkdtemp(path.join(os.tmpdir(), "restart-chaos-oracle-"))); + const resolve = (value: string) => realpath(value).catch(() => value); + try { + const env = { PATH: "/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin:/usr/local/bin", HOME: scratch, TMPDIR: scratch, + PYTHONDONTWRITEBYTECODE: "1", PYTHONNOUSERSITE: "1", LANG: "C.UTF-8" }; + const [command, args] = sandboxed + ? ["/usr/bin/sandbox-exec", ["-p", oracleSandboxProfile({ workspace: await resolve(workspace), scratch, + protectedRoots: await Promise.all((options.protectedRoots ?? [os.homedir()]).map(resolve)) }), ...argv]] + : [argv[0]!, argv.slice(1)]; + try { + const { stdout, stderr } = await execFileAsync(command, args, { cwd: workspace, env, timeout: options.timeoutMs ?? 60_000, maxBuffer: 1024 * 1024 }); + return { exitCode: 0, stdout: stdout.slice(-OUTPUT_LIMIT), stderr: stderr.slice(-OUTPUT_LIMIT) }; + } catch (error) { + const failure = error as { code?: number | string; stdout?: string; stderr?: string }; + return { exitCode: typeof failure.code === "number" ? failure.code : null, stdout: (failure.stdout ?? "").slice(-OUTPUT_LIMIT), + stderr: (failure.stderr ?? String(error)).slice(-OUTPUT_LIMIT) }; + } + } finally { + await rm(scratch, { recursive: true, force: true }); + } +} diff --git a/tests/restart-chaos/scheduler.test.ts b/tests/restart-chaos/scheduler.test.ts new file mode 100644 index 00000000000..491d689760d --- /dev/null +++ b/tests/restart-chaos/scheduler.test.ts @@ -0,0 +1,116 @@ +import { describe, expect, it } from "vitest"; +import { parseCommand, parseReplayRef, waveIdFor } from "./config.js"; +import { K4_HOLD_RANGE_MS, createRng, observedFaultSchedule, planTasks, planWave, replayWave, taskFaultSchedule } from "./scheduler.js"; +import type { WaveOptions } from "./types.js"; +import { workloadDefinitions } from "./workloads.js"; + +const options: WaveOptions = { env: "local", adapters: ["codex", "claude", "opencode"], workloads: ["A1", "B1", "B5", "C1"], + faults: ["K1", "K2", "K4"], concurrency: 4, tasks: 8, seed: 42, budgetCents: 3_000, pollMs: 3_000 }; + +describe("scheduler determinism", () => { + it("produces the same plan for the same seed", () => { + expect(planTasks(options)).toEqual(planTasks(options)); + }); + + it("produces a different plan for a different seed", () => { + expect(planTasks({ ...options, seed: 43 })).not.toEqual(planTasks(options)); + }); + + it("keeps fire times, holds and follow-ups inside each workload's window", () => { + for (const task of planTasks({ ...options, tasks: 60 })) { + const definition = workloadDefinitions[task.workload]; + expect(task.fireAfterMs).toBeGreaterThanOrEqual(definition.fireWindowMs[0]); + expect(task.fireAfterMs).toBeLessThanOrEqual(definition.fireWindowMs[1]); + if (task.fault === "K4") { + expect(task.holdMs).toBeGreaterThanOrEqual(K4_HOLD_RANGE_MS[0]); + expect(task.holdMs).toBeLessThanOrEqual(K4_HOLD_RANGE_MS[1]); + } else expect(task.holdMs).toBe(0); + expect(task.followUpAfterMs === null).toBe(task.workload !== "C1"); + expect(task.nonce).toMatch(/^[0-9a-f]{8}$/); + } + }); + + it("covers every workload × adapter × fault cell once per cycle", () => { + const cells = options.workloads.length * options.adapters.length * options.faults.length; + const tasks = planTasks({ ...options, tasks: cells }); + expect(new Set(tasks.map(task => `${task.workload}/${task.adapter}/${task.fault}`)).size).toBe(cells); + expect(tasks.map(task => task.taskId).slice(0, 3)).toEqual(["t01", "t02", "t03"]); + }); + + it("respects narrowed selections", () => { + const tasks = planTasks({ ...options, adapters: ["codex"], workloads: ["A1"], faults: ["K1"], tasks: 3 }); + expect(tasks.every(task => task.adapter === "codex" && task.workload === "A1" && task.fault === "K1")).toBe(true); + }); + + it("has a stable RNG sequence", () => { + const first = createRng(7); + const second = createRng(7); + expect(Array.from({ length: 5 }, () => first.next())).toEqual(Array.from({ length: 5 }, () => second.next())); + }); +}); + +describe("replay", () => { + const plan = planWave(waveIdFor(42, new Date("2026-11-06T10:11:12Z")), options, new Date("2026-11-06T10:11:12Z")); + + it("names waves with the timestamp and seed", () => { + expect(plan.waveId).toBe("w-20261106T101112-s42"); + }); + + it("reproduces exactly one saved task", () => { + const replay = replayWave(plan, { waveId: plan.waveId, taskId: "t03" }, 1_700_000_000_000); + expect(replay.tasks).toEqual([plan.tasks[2]]); + expect(replay.options.concurrency).toBe(1); + expect(replay.replayOf).toEqual({ waveId: plan.waveId, taskId: "t03" }); + expect(replay.waveId).toBe(`${plan.waveId}-replay-t03-1700000000000`); + // A replay of a replay names the original wave, and its id parses again. + const again = replayWave(replay, parseReplayRef(`${replay.waveId}/t03`), 1_700_000_000_001); + expect(again.waveId).toBe(`${plan.waveId}-replay-t03-1700000000001`); + }); + + it("replays the recorded fault sequence that crossed the task, other tasks' faults included", () => { + const createdAt = 1_000_000; + const faults = [ + { faultId: "f01", kind: "K1" as const, holdMs: 0, startedAt: createdAt + 42_000 }, + { faultId: "f02", kind: "K4" as const, holdMs: 150_000, startedAt: createdAt + 60_000 }, + { faultId: "f03", kind: "K2" as const, holdMs: 0, startedAt: createdAt + 900_000 }, + ]; + const observed = observedFaultSchedule(createdAt, ["f02", "f01"], faults); + expect(observed).toEqual([{ kind: "K1", holdMs: 0, atMs: 42_000 }, { kind: "K4", holdMs: 150_000, atMs: 60_000 }]); + const replay = replayWave(plan, { waveId: plan.waveId, taskId: "t02" }, 1, {}, observed); + expect(taskFaultSchedule(replay.tasks[0]!)).toEqual(observed); + // Without a recorded sequence the replay falls back to the planned fault. + const planned = replayWave(plan, { waveId: plan.waveId, taskId: "t02" }, 1); + expect(taskFaultSchedule(planned.tasks[0]!)).toEqual([{ kind: plan.tasks[1]!.fault, holdMs: plan.tasks[1]!.holdMs, atMs: plan.tasks[1]!.fireAfterMs }]); + }); + + it("rejects unknown tasks and mismatched waves", () => { + expect(() => replayWave(plan, { waveId: plan.waveId, taskId: "t99" })).toThrow(/no task t99/); + expect(() => replayWave(plan, { waveId: "w-20261106T101112-s1", taskId: "t01" })).toThrow(/not w-/); + }); +}); + +describe("CLI parsing", () => { + it("parses a wave command including pnpm's forwarded --", () => { + const parsed = parseCommand(["--", "--env", "local", "--adapters", "codex,claude,opencode", "--concurrency", "4", "--tasks", "8", "--faults", "K1,K2,K4", "--seed", "17"]); + expect(parsed).toMatchObject({ mode: "wave", options: { env: "local", adapters: ["codex", "claude", "opencode"], concurrency: 4, tasks: 8, faults: ["K1", "K2", "K4"], seed: 17 } }); + }); + + it("parses --replay", () => { + expect(parseCommand(["--replay", "w-20261106T101112-s42/t03"])).toMatchObject({ mode: "replay", ref: { waveId: "w-20261106T101112-s42", taskId: "t03" } }); + }); + + it("rejects replay mixed with plan options and malformed refs", () => { + expect(() => parseCommand(["--replay", "w-20261106T101112-s42/t03", "--seed", "1"])).toThrow(/reproduces the saved plan/); + expect(() => parseReplayRef("w-20261106T101112-s42")).toThrow(/\//); + expect(() => parseReplayRef("..%2Fetc/t01")).toThrow(/Invalid wave id/); + expect(() => parseReplayRef("../etc/t01")).toThrow(/\//); + expect(() => parseReplayRef("w-20261106T101112-s42/t1")).toThrow(/Invalid task id/); + }); + + it("rejects unknown faults, adapters and out-of-range numbers", () => { + expect(() => parseCommand(["--faults", "K9"])).toThrow(/Unknown faults/); + expect(() => parseCommand(["--adapters", "gemini"])).toThrow(/Unknown adapters/); + expect(() => parseCommand(["--concurrency", "0"])).toThrow(/between 1 and 20/); + expect(() => parseCommand(["--poll-ms", "500"])).toThrow(/between 2000 and 5000/); + }); +}); diff --git a/tests/restart-chaos/scheduler.ts b/tests/restart-chaos/scheduler.ts new file mode 100644 index 00000000000..a4ceab2f5ae --- /dev/null +++ b/tests/restart-chaos/scheduler.ts @@ -0,0 +1,100 @@ +import { workloadDefinitions } from "./workloads.js"; +import type { AdapterId, FaultKind, ScheduledFault, TaskPlan, WavePlan, WaveOptions, WorkloadId } from "./types.js"; +import type { ReplayRef } from "./config.js"; + +/** mulberry32: small, fast and fully determined by its 32-bit seed. */ +export function createRng(seed: number) { + let state = seed >>> 0; + const next = () => { + state = (state + 0x6d2b79f5) >>> 0; + let t = state; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4_294_967_296; + }; + return { + next, + int(min: number, max: number) { return min + Math.floor(next() * (max - min + 1)); }, + hex(length: number) { let out = ""; while (out.length < length) out += Math.floor(next() * 16).toString(16); return out; }, + shuffle(values: readonly T[]): T[] { + const copy = [...values]; + for (let index = copy.length - 1; index > 0; index -= 1) { + const swap = Math.floor(next() * (index + 1)); + [copy[index], copy[swap]] = [copy[swap]!, copy[index]!]; + } + return copy; + }, + }; +} + +export const K4_HOLD_RANGE_MS: readonly [number, number] = [120_000, 300_000]; + +/** + * Pairwise-friendly sampling: cycle a seeded shuffle of the full + * workload × adapter × fault product so small waves still spread across cells. + */ +export function planTasks(options: Pick): TaskPlan[] { + const rng = createRng(options.seed); + const cells: Array<{ workload: WorkloadId; adapter: AdapterId; fault: FaultKind }> = []; + for (const workload of options.workloads) for (const adapter of options.adapters) for (const fault of options.faults) cells.push({ workload, adapter, fault }); + const tasks: TaskPlan[] = []; + let order = rng.shuffle(cells); + for (let index = 0; index < options.tasks; index += 1) { + if (index > 0 && index % cells.length === 0) order = rng.shuffle(cells); + const cell = order[index % cells.length]!; + const definition = workloadDefinitions[cell.workload]; + const [fireMin, fireMax] = definition.fireWindowMs; + tasks.push({ + taskId: `t${String(index + 1).padStart(2, "0")}`, + index, + ...cell, + fireAfterMs: rng.int(fireMin, fireMax), + holdMs: cell.fault === "K4" ? rng.int(K4_HOLD_RANGE_MS[0], K4_HOLD_RANGE_MS[1]) : 0, + followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null, + nonce: rng.hex(8), + }); + } + return tasks; +} + +export function planWave(waveId: string, options: WaveOptions, createdAt = new Date()): WavePlan { + return { schema: "paperclip.restart-chaos.wave-plan/v1", waveId, seed: options.seed, + createdAt: createdAt.toISOString(), options, tasks: planTasks(options) }; +} + +/** Faults that actually crossed a task, as offsets from its launch. */ +export function observedFaultSchedule(createdAt: number | null, faultIds: readonly string[], + faults: ReadonlyArray<{ faultId: string; kind: FaultKind; holdMs: number; startedAt: number }>): ScheduledFault[] { + if (createdAt === null) return []; + return faults.filter(fault => faultIds.includes(fault.faultId)) + .map(fault => ({ kind: fault.kind, holdMs: fault.holdMs, atMs: Math.max(0, fault.startedAt - createdAt) })) + .sort((a, b) => a.atMs - b.atMs); +} + +/** The faults a task schedules: its replayed sequence, or its one planned fault. */ +export function taskFaultSchedule(task: TaskPlan): ScheduledFault[] { + return task.replayFaults?.length ? task.replayFaults : [{ kind: task.fault, holdMs: task.holdMs, atMs: task.fireAfterMs }]; +} + +/** + * A replay keeps the saved cell, hold and nonce and runs alone. When the source + * wave recorded which faults crossed the task (including other tasks' faults), + * the replay fires that same sequence at the same offsets from launch. + */ +export function replayWave(source: WavePlan, ref: ReplayRef, now = Date.now(), + overrides: Partial> = {}, observed: ScheduledFault[] = []): WavePlan { + if (source.schema !== "paperclip.restart-chaos.wave-plan/v1") throw new Error("Unsupported wave plan schema"); + if (source.waveId !== ref.waveId) throw new Error(`Plan is for ${source.waveId}, not ${ref.waveId}`); + const task = source.tasks.find(candidate => candidate.taskId === ref.taskId); + if (!task) throw new Error(`Wave ${ref.waveId} has no task ${ref.taskId}`); + const base = source.waveId.replace(/-replay-t[0-9]{2,3}-[0-9]+$/, ""); + return { + schema: source.schema, + waveId: `${base}-replay-${task.taskId}-${now}`, + seed: source.seed, + createdAt: new Date(now).toISOString(), + options: { ...source.options, ...overrides, tasks: 1, concurrency: 1 }, + tasks: [{ ...task, ...(observed.length ? { replayFaults: observed.map(fault => ({ ...fault })) } : {}) }], + replayOf: { ...ref }, + }; +} diff --git a/tests/restart-chaos/seed.ts b/tests/restart-chaos/seed.ts new file mode 100644 index 00000000000..b9ee5f26c93 --- /dev/null +++ b/tests/restart-chaos/seed.ts @@ -0,0 +1,119 @@ +import { execFileSync } from "node:child_process"; +import { chmod, mkdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { runnerProfiles } from "../runner-e2e/catalog.js"; +import type { SecretReference, SecretReferenceMap } from "../runner-e2e/types.js"; +import type { ChaosApi } from "./api.js"; +import { CREDENTIAL_FOR_ADAPTER, PROFILE_FOR_ADAPTER } from "./config.js"; +import type { Timeline } from "./timeline.js"; +import type { AdapterId, EnvironmentKind } from "./types.js"; +import { fixtureRepositoryFiles } from "./workloads.js"; + +export interface ChaosLab { + companyId: string; + environmentId: string; + projectId: string; + fixtureRepositoryPath: string; + agents: Record; +} + +/** Create the `chaos-sandbox` git repository: python CLI, one failing test, bench.sh. */ +export async function createFixtureRepository(directory: string, benchSeconds?: number): Promise { + await mkdir(directory, { recursive: true }); + for (const [relative, file] of Object.entries(fixtureRepositoryFiles(benchSeconds))) { + const target = path.join(directory, relative); + await mkdir(path.dirname(target), { recursive: true }); + await writeFile(target, file.content); + if (file.mode) await chmod(target, file.mode); + } + const git = (...args: string[]) => execFileSync("git", args, { cwd: directory, stdio: "pipe", + env: { ...process.env, GIT_AUTHOR_NAME: "Chaos Lab", GIT_AUTHOR_EMAIL: "chaos@example.invalid", + GIT_COMMITTER_NAME: "Chaos Lab", GIT_COMMITTER_EMAIL: "chaos@example.invalid" } }); + git("init", "-q", "-b", "main"); + git("add", "-A"); + git("commit", "-q", "-m", "chaos-sandbox fixture"); + return directory; +} + +/** + * Seeds the isolated instance. Credentials cross into the server only as + * encrypted company secrets, as in tests/runner-e2e/live-fixtures.ts, and the + * agent payloads come from the shared runner-e2e catalog profiles. + */ +export async function seedChaosLab(input: { + api: ChaosApi; + env: EnvironmentKind; + adapters: readonly AdapterId[]; + concurrency: number; + budgetCents: number; + fixtureRoot: string; + waveId: string; + timeline: Timeline; +}): Promise { + const { api, timeline } = input; + if (input.env === "daytona") { + // TODO(restart-chaos phase 1b): create the Daytona sandbox environment from + // the prepared image (see tests/runner-e2e/catalog.ts runnerEnvironments + // and live-fixtures.ts installFixtureDaytonaProvider) and add the R6 + // sandbox sweep. The image and env are being prepared separately. + throw new Error("--env daytona is not wired yet; use --env local"); + } + const fixtureRepositoryPath = await createFixtureRepository(path.join(input.fixtureRoot, "chaos-sandbox")); + const company = await api.post<{ id: string }>("/api/companies", { + name: "Chaos Lab", + description: `Restart chaos harness wave ${input.waveId}`, + budgetMonthlyCents: input.budgetCents, + }); + timeline.add("seed_company", { companyId: company.id }); + + const secretRefs: SecretReferenceMap = {}; + for (const adapter of input.adapters) { + const key = CREDENTIAL_FOR_ADAPTER[adapter]; + if (secretRefs[key]) continue; + const value = process.env[key]; + if (!value) throw new Error(`Missing ${key} for adapter ${adapter}; load ~/.secrets into the environment first`); + const created = await api.post<{ id: string }>(`/api/companies/${company.id}/secrets`, { + name: `Chaos ${key}`, key, value, description: "Restart chaos harness provider credential", + }); + secretRefs[key] = { type: "secret_ref", secretId: created.id, version: "latest" } satisfies SecretReference; + } + timeline.add("seed_secrets", { keys: Object.keys(secretRefs) }); + + const environments = await api.get>(`/api/companies/${company.id}/environments?driver=local`); + const environment = environments.find(candidate => candidate.driver === "local"); + if (!environment) throw new Error("Isolated instance has no managed local environment"); + + const agents = {} as ChaosLab["agents"]; + for (const adapter of input.adapters) { + const profile = runnerProfiles.find(candidate => candidate.id === PROFILE_FOR_ADAPTER[adapter]); + if (!profile) throw new Error(`runner-e2e catalog has no profile ${PROFILE_FOR_ADAPTER[adapter]}`); + const payload = profile.buildAgent({ environmentId: environment.id, environmentFixtureId: "local", + workspacePath: fixtureRepositoryPath, secretRefs, executionId: input.waveId }); + const name = `Chaos ${adapter}`; + const runtimeConfig = (payload.runtimeConfig ?? {}) as Record; + const created = await api.post<{ id: string; name: string }>(`/api/companies/${company.id}/agents`, { + ...payload, + name, + title: "Restart chaos worker", + runtimeConfig: { ...runtimeConfig, heartbeat: { ...((runtimeConfig.heartbeat ?? {}) as Record), + enabled: false, maxConcurrentRuns: Math.max(1, input.concurrency) } }, + }); + agents[adapter] = { id: created.id, name }; + timeline.add("seed_agent", { adapter, agentId: created.id, profile: profile.id, model: profile.model }); + } + + const project = await api.post<{ id: string }>(`/api/companies/${company.id}/projects`, { + name: "chaos-sandbox", + description: "Fixture repository for restart chaos workloads", + executionWorkspacePolicy: { + enabled: true, + defaultMode: "isolated_workspace", + allowIssueOverride: false, + environmentId: environment.id, + workspaceStrategy: { type: "git_worktree" }, + }, + workspace: { name: "Primary", sourceType: "local_path", cwd: fixtureRepositoryPath, isPrimary: true }, + }); + timeline.add("seed_project", { projectId: project.id, fixtureRepositoryPath }); + return { companyId: company.id, environmentId: environment.id, projectId: project.id, fixtureRepositoryPath, agents }; +} diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts new file mode 100644 index 00000000000..2cb1ccd5d58 --- /dev/null +++ b/tests/restart-chaos/supervisor.ts @@ -0,0 +1,299 @@ +/** + * Owns one isolated Paperclip control plane and injects K faults. + * + * Reuses the runner-e2e process code instead of copying it: the config seed + * (server-config.ts), port reservation (ports.ts), environment isolation + * (harness-env.ts), wrapper-owned Postgres (controller-database.ts), the + * process-tree stopper (server-stop.ts + process-tree-owner.ts), the source + * server entry (server-entry.ts) and the stop → unhealthy → hold → start → + * ready receipt in controller-outage.ts `replaceController`. + * + * Postgres runs beside, not under, the server process tree, so K1 kills only + * the controller and its owned descendants (local runners included), never the + * database (that is K9). The controller runtime cache is kept: discarding it is + * the separate K11 fault. + */ +import { spawn, type ChildProcess } from "node:child_process"; +import { randomBytes } from "node:crypto"; +import { createWriteStream, statSync, type WriteStream } from "node:fs"; +import { mkdir, readFile, symlink } from "node:fs/promises"; +import { createRequire } from "node:module"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { prepareControllerDatabase } from "../runner-e2e/controller-database.js"; +import { replaceController, type ControllerOutageReceipt } from "../runner-e2e/controller-outage.js"; +import { + assertIsolatedServerEnvironment, buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment, + resolvePaperclipRunnerBinaryForHarness, +} from "../runner-e2e/harness-env.js"; +import { reserveRunnerE2EServerPort } from "../runner-e2e/ports.js"; +import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "../runner-e2e/server-config.js"; +import { createRunnerE2EServerStopper, runnerE2EServerDetached } from "../runner-e2e/server-stop.js"; +import { prepareSourceServerDependencies } from "../runner-e2e/source-startup.js"; +import { runnerE2ETypeScriptProcessArgs } from "../runner-e2e/web-server-command.js"; +import type { MatrixExecution } from "../runner-e2e/types.js"; +import type { Timeline } from "./timeline.js"; +import type { FaultKind, FaultReceipt } from "./types.js"; + +const HEALTH_TIMEOUT_MS = 180_000; +const READY_TIMEOUT_MS = 300_000; +const STOP_TIMEOUT_MS = 30_000; + +export const FAULT_SIGNAL: Record = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL" }; + +export interface SupervisorPaths { temporaryRoot: string; home: string; configPath: string; serverLogPath: string } + +const secret = () => randomBytes(48).toString("base64url"); + +/** Same provider PATH shape as the runner-e2e launcher: pinned Codex first. */ +async function prepareProviderPath(repositoryRoot: string, temporaryRoot: string, inherited: string | undefined) { + const toolBin = path.join(temporaryRoot, "provider-bin"); + await mkdir(toolBin, { recursive: true }); + const runnerRequire = createRequire(path.join(repositoryRoot, "packages/paperclip-runner/package.json")); + const codexAcp = runnerRequire.resolve("@agentclientprotocol/codex-acp/package.json"); + const codexPackage = createRequire(codexAcp).resolve("@openai/codex/package.json"); + const manifest = JSON.parse(await readFile(codexPackage, "utf8")) as { bin?: string | Record }; + const codexBin = typeof manifest.bin === "string" ? manifest.bin : manifest.bin?.codex; + if (!codexBin) throw new Error("Pinned Codex executable not found"); + await symlink(path.resolve(path.dirname(codexPackage), codexBin), path.join(toolBin, "codex")); + return [toolBin, path.join(repositoryRoot, "packages/paperclip-runner/node_modules/.bin"), inherited].filter(Boolean).join(path.delimiter); +} + +export class ChaosSupervisor { + readonly baseUrl: string; + private child: ChildProcess | null = null; + private readonly expectedStops = new WeakSet(); + private readonly spawnErrors = new WeakMap(); + private readonly log: WriteStream; + private readonly stopper: ReturnType; + private database: { cleanup(): Promise } | undefined; + private environment: Record = {}; + private faultChain: Promise = Promise.resolve(); + private faultCount = 0; + private shuttingDown = false; + /** False from fault start until /api/ready reports `ready` again. */ + serverReady = false; + faultInProgress = false; + unexpectedExits = 0; + + private constructor(readonly repositoryRoot: string, readonly port: number, readonly paths: SupervisorPaths, + private readonly timeline: Timeline) { + this.baseUrl = `http://127.0.0.1:${port}`; + this.log = createWriteStream(paths.serverLogPath, { flags: "a", mode: 0o600 }); + this.stopper = createRunnerE2EServerStopper({ + gracefulTimeoutMs: STOP_TIMEOUT_MS, forcedTimeoutMs: 5_000, + hasSpawnError: candidate => this.spawnErrors.has(candidate), + markExpectedStop: candidate => { this.expectedStops.add(candidate); }, + log: message => { this.log.write(message); }, + }); + } + + static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline }): Promise { + const temporaryRoot = await createRunnerE2ETemporaryRoot(); + const home = path.join(temporaryRoot, "home"); + const paths: SupervisorPaths = { temporaryRoot, home, configPath: path.join(home, "instances", "default", "config.json"), + serverLogPath: path.join(input.waveDir, "server.log") }; + const port = await reserveRunnerE2EServerPort(); + const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline); + input.timeline.add("supervisor_created", { temporaryRoot, port }); + return supervisor; + } + + serverLogOffset(): number { + try { return statSync(this.paths.serverLogPath).size + this.log.writableLength; } catch { return 0; } + } + + async start(): Promise { + const { temporaryRoot, home, configPath } = this.paths; + const reservation = await prepareRunnerE2EServerConfig({ temporaryRoot, configPath, serverPort: this.port }); + await reservation?.close(); + // Wrapper-owned Postgres: survives K1/K2/K4 like a managed database would. + this.database = await prepareControllerDatabase(configPath, temporaryRoot); + this.timeline.add("database_started", {}); + const runnerBinary = resolvePaperclipRunnerBinaryForHarness( + [{ profile: { generation: "native" } } as unknown as MatrixExecution], this.repositoryRoot); + const source = { + ...buildRunnerE2EProcessEnvironment(process.env, []), + PATH: await prepareProviderPath(this.repositoryRoot, temporaryRoot, process.env.PATH), + PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: "[]", + PAPERCLIP_RUNNER_BINARY: runnerBinary, + PAPERCLIP_VITE_CACHE_DIR: path.join(temporaryRoot, "vite-cache"), + }; + const overrides = { + NODE_ENV: "test", PORT: String(this.port), XDG_CACHE_HOME: path.join(temporaryRoot, "xdg-cache"), + PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, PAPERCLIP_INSTANCE_ID: "default", + PAPERCLIP_AGENT_JWT_SECRET: secret(), PAPERCLIP_DECISION_SIGNING_SECRET: secret(), + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: secret(), BETTER_AUTH_SECRET: secret(), + PAPERCLIP_BIND: "loopback", PAPERCLIP_BIND_HOST: "127.0.0.1", PAPERCLIP_DEPLOYMENT_MODE: "local_trusted", + PAPERCLIP_DEPLOYMENT_EXPOSURE: "private", SERVE_UI: "true", PAPERCLIP_STORAGE_PROVIDER: "local_disk", + PAPERCLIP_STORAGE_LOCAL_DIR: path.join(temporaryRoot, "storage"), PAPERCLIP_SECRETS_PROVIDER: "local_encrypted", + PAPERCLIP_SECRETS_STRICT_MODE: "true", PAPERCLIP_DB_BACKUP_ENABLED: "false", + PAPERCLIP_DB_BACKUP_DIR: path.join(temporaryRoot, "backups"), PAPERCLIP_OPEN_ON_LISTEN: "false", + }; + const environment = buildPaperclipServerEnvironment(source, overrides); + assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath }); + this.environment = Object.fromEntries(Object.entries(environment).filter((entry): entry is [string, string] => entry[1] !== undefined)); + await prepareSourceServerDependencies(this.repositoryRoot, this.environment, false); + const child = await this.spawnServer(); + await this.waitForHealth(child); + await this.waitForReadyPhase([]); + this.timeline.add("server_ready", { pid: child.pid }); + } + + private async spawnServer(): Promise { + if (this.shuttingDown) throw new Error("Refusing to start Paperclip during harness shutdown"); + const entry = path.join(this.repositoryRoot, "tests/runner-e2e/server-entry.ts"); + const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(this.repositoryRoot, entry, ["onboard", "--yes", "--run"]), + { cwd: this.repositoryRoot, env: this.environment, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached }); + this.child = child; + child.stdout?.on("data", chunk => this.log.write(chunk)); + child.stderr?.on("data", chunk => this.log.write(chunk)); + child.once("error", error => { this.spawnErrors.set(child, error); }); + child.once("exit", (code, signal) => { + this.log.write(`\n[restart-chaos] server pid=${child.pid} exited code=${code} signal=${signal}\n`); + if (!this.expectedStops.has(child) && !this.shuttingDown) { + this.unexpectedExits += 1; + this.serverReady = false; + this.timeline.add("server_exited_unexpectedly", { pid: child.pid, code, signal }); + } + }); + await this.stopper.watch(child); + this.timeline.add("server_spawned", { pid: child.pid }); + return child; + } + + private exited(child: ChildProcess) { return child.exitCode !== null || child.signalCode !== null || this.spawnErrors.has(child); } + + private async readPhase(): Promise { + try { + const response = await fetch(`${this.baseUrl}/api/ready`, { signal: AbortSignal.timeout(1_000), cache: "no-store" }); + return String(((await response.json()) as { status?: string }).status ?? response.status); + } catch { return "unreachable"; } + } + + private async waitForHealth(child: ChildProcess, phases?: FaultReceipt["phases"]): Promise { + const deadline = Date.now() + HEALTH_TIMEOUT_MS; + while (Date.now() < deadline) { + if (this.shuttingDown) throw new Error("Harness shutdown interrupted startup"); + if (this.exited(child)) throw new Error(`Paperclip server exited during startup (code=${child.exitCode} signal=${child.signalCode})`); + if (phases) { + const phase = await this.readPhase(); + if (phase !== phases[phases.length - 1]?.phase) phases.push({ phase, at: Date.now() }); + } + try { + const response = await fetch(`${this.baseUrl}/api/health`, { signal: AbortSignal.timeout(1_000) }); + await response.body?.cancel(); + if (response.ok) return Date.now(); + } catch { /* still booting */ } + await delay(250); + } + throw new Error(`Paperclip server did not become healthy within ${HEALTH_TIMEOUT_MS}ms`); + } + + private async waitForHealthToStop(): Promise { + const deadline = Date.now() + STOP_TIMEOUT_MS; + while (Date.now() < deadline) { + try { const response = await fetch(`${this.baseUrl}/api/health`, { signal: AbortSignal.timeout(500) }); await response.body?.cancel(); } + catch { return; } + await delay(100); + } + throw new Error("Old Paperclip server stayed healthy after stop"); + } + + /** Records starting → recovering → ready from /api/ready (liveness gate). */ + private async waitForReadyPhase(phases: FaultReceipt["phases"]): Promise { + const deadline = Date.now() + READY_TIMEOUT_MS; + let last: string | null = phases[phases.length - 1]?.phase ?? null; + while (Date.now() < deadline) { + if (this.shuttingDown) throw new Error("Harness shutdown interrupted readiness wait"); + const phase = await this.readPhase(); + if (phase !== last) { phases.push({ phase, at: Date.now() }); last = phase; } + if (phase === "ready") { + const liveness = await fetch(`${this.baseUrl}/api/liveness`, { signal: AbortSignal.timeout(1_000) }).catch(() => null); + await liveness?.body?.cancel(); + if (liveness?.ok) { this.serverReady = true; return Date.now(); } + } + await delay(250); + } + throw new Error(`Paperclip did not report ready within ${READY_TIMEOUT_MS}ms`); + } + + /** Faults are serialized: one control plane, one outage at a time. */ + fault(kind: FaultKind, holdMs: number, requestedBy: string): Promise { + const requestedAt = Date.now(); + const run = this.faultChain.then(() => this.executeFault(kind, holdMs, requestedBy, requestedAt)); + this.faultChain = run.catch(() => undefined); + return run; + } + + private async executeFault(kind: FaultKind, holdMs: number, requestedBy: string, requestedAt: number): Promise { + const previous = this.child; + this.faultCount += 1; + const receipt: FaultReceipt = { + faultId: `f${String(this.faultCount).padStart(2, "0")}`, kind, requestedBy, signal: FAULT_SIGNAL[kind], holdMs, + requestedAt, startedAt: Date.now(), stoppedAt: null, restartedAt: null, healthyAt: null, readyAt: null, + phases: [], serverLogOffsetAtStart: this.serverLogOffset(), serverLogOffsetAtReady: null, error: null, + }; + this.faultInProgress = true; + this.serverReady = false; + this.timeline.add("fault_started", { faultId: receipt.faultId, kind, signal: receipt.signal, holdMs, requestedBy, previousPid: previous?.pid }); + this.log.write(`\n[restart-chaos] ${receipt.faultId} ${kind} ${receipt.signal} hold=${holdMs}ms requestedBy=${requestedBy} at ${new Date().toISOString()}\n`); + try { + if (!previous?.pid || this.exited(previous)) throw new Error("No live Paperclip server to fault"); + let outage: ControllerOutageReceipt | undefined; + outage = await replaceController({ + requestId: `chaos-${receipt.faultId}-${requestedBy}`, + previousPid: previous.pid, + outage: { signal: receipt.signal, holdMs }, + stop: async signal => { + if (signal === "SIGKILL") await this.stopper.crashForRestart(previous); + else await this.stopper(previous, "SIGTERM"); + if (this.child === previous) this.child = null; + }, + waitForUnhealthy: async () => { await this.waitForHealthToStop(); receipt.stoppedAt = Date.now(); this.timeline.add("fault_server_down", { faultId: receipt.faultId }); }, + // K1/K2/K4 keep the controller runtime cache; K11 will discard it. + discardCache: async () => {}, + start: async () => { receipt.restartedAt = Date.now(); this.timeline.add("fault_restarting", { faultId: receipt.faultId }); return this.spawnServer(); }, + waitForReady: async replacement => { receipt.healthyAt = await this.waitForHealth(replacement, receipt.phases); }, + cancelled: () => this.shuttingDown, + record: async progress => { outage = progress; }, + }); + receipt.readyAt = await this.waitForReadyPhase(receipt.phases); + receipt.serverLogOffsetAtReady = this.serverLogOffset(); + this.timeline.add("fault_ready", { faultId: receipt.faultId, replacementPid: outage?.replacementPid, + downMs: (receipt.restartedAt ?? 0) - (receipt.stoppedAt ?? 0), readyAfterMs: receipt.readyAt - receipt.startedAt, phases: receipt.phases }); + } catch (error) { + receipt.error = error instanceof Error ? error.message : String(error); + this.timeline.add("fault_failed", { faultId: receipt.faultId, error: receipt.error }); + throw Object.assign(new Error(`Fault ${receipt.faultId} failed: ${receipt.error}`), { receipt }); + } finally { + this.faultInProgress = false; + } + return receipt; + } + + /** + * Bounded recovery after a failed fault or a crash the harness did not cause: + * wait for a live server to become ready, or start a new one. Throws when the + * control plane cannot be brought back, so the wave fails instead of hanging. + */ + async ensureServing(): Promise { + if (this.serverReady || this.faultInProgress) return; + const live = this.child && !this.exited(this.child) ? this.child : null; + this.timeline.add("server_recovery_attempt", { livePid: live?.pid ?? null }); + const child = live ?? await this.spawnServer(); + await this.waitForHealth(child); + await this.waitForReadyPhase([]); + this.timeline.add("server_recovered", { pid: child.pid }); + } + + async stop(): Promise { + this.shuttingDown = true; + const errors: string[] = []; + try { await this.stopper.stopAll(); } catch (error) { errors.push(error instanceof Error ? error.message : String(error)); } + try { await this.database?.cleanup(); } catch (error) { errors.push(error instanceof Error ? error.message : String(error)); } + await new Promise(resolve => this.log.end(resolve)); + this.timeline.add("supervisor_stopped", { errors }); + if (errors.length) throw new Error(`Supervisor cleanup incomplete: ${errors.join("; ")}`); + } +} diff --git a/tests/restart-chaos/timeline.ts b/tests/restart-chaos/timeline.ts new file mode 100644 index 00000000000..95d73d1ead3 --- /dev/null +++ b/tests/restart-chaos/timeline.ts @@ -0,0 +1,18 @@ +import { appendFileSync } from "node:fs"; + +export interface TimelineEntry { at: string; t: number; kind: string; [key: string]: unknown } + +/** Append-only action timeline. Every supervisor and launcher action lands here. */ +export class Timeline { + readonly entries: TimelineEntry[] = []; + constructor(private readonly path: string | null, readonly origin = Date.now()) {} + + add(kind: string, data: Record = {}): TimelineEntry { + const now = Date.now(); + const entry: TimelineEntry = { at: new Date(now).toISOString(), t: now - this.origin, kind, ...data }; + this.entries.push(entry); + if (this.path) appendFileSync(this.path, `${JSON.stringify(entry)}\n`, { mode: 0o600 }); + if (process.env.RESTART_CHAOS_QUIET !== "1") console.info(`[chaos +${Math.round(entry.t / 1000)}s] ${kind} ${JSON.stringify(data)}`); + return entry; + } +} diff --git a/tests/restart-chaos/tsconfig.json b/tests/restart-chaos/tsconfig.json new file mode 100644 index 00000000000..03eb032646e --- /dev/null +++ b/tests/restart-chaos/tsconfig.json @@ -0,0 +1,13 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "noEmit": true, + "rootDir": "../..", + "types": ["node"], + "typeRoots": [ + "../../cli/node_modules/@types", + "../../server/node_modules/@types" + ] + }, + "include": ["./**/*.ts", "../../server/src/types/express.d.ts"] +} diff --git a/tests/restart-chaos/tsconfig.layers.json b/tests/restart-chaos/tsconfig.layers.json new file mode 100644 index 00000000000..f1c68632282 --- /dev/null +++ b/tests/restart-chaos/tsconfig.layers.json @@ -0,0 +1,8 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "declaration": false, + "declarationMap": false + }, + "references": [{ "path": "../../server/node_modules/.cache/paperclip-typecheck/tsconfig.declarations.json" }] +} diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts new file mode 100644 index 00000000000..ce2203f878a --- /dev/null +++ b/tests/restart-chaos/types.ts @@ -0,0 +1,182 @@ +/** Shared types for the control-plane restart chaos harness (plan §5). */ + +export const FAULT_KINDS = ["K1", "K2", "K4"] as const; +export type FaultKind = (typeof FAULT_KINDS)[number]; + +export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1"] as const; +export type WorkloadId = (typeof WORKLOAD_IDS)[number]; + +export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const; +export type AdapterId = (typeof ADAPTER_IDS)[number]; + +export const ENVIRONMENT_KINDS = ["local", "daytona"] as const; +export type EnvironmentKind = (typeof ENVIRONMENT_KINDS)[number]; + +export interface WaveOptions { + env: EnvironmentKind; + adapters: AdapterId[]; + workloads: WorkloadId[]; + faults: FaultKind[]; + concurrency: number; + tasks: number; + seed: number; + /** Company budget hard cap in cents; the product hard-stops at this value. */ + budgetCents: number; + /** Poll interval for the observer, 2–5 s per plan §5. */ + pollMs: number; +} + +export interface TaskPlan { + taskId: string; + index: number; + workload: WorkloadId; + adapter: AdapterId; + fault: FaultKind; + /** Fire the fault this long after the task's issue is created. */ + fireAfterMs: number; + /** Outage hold between stop and restart. 0 for K1/K2. */ + holdMs: number; + /** C1 only: post the follow-up comment this long after creation. */ + followUpAfterMs: number | null; + /** Unique marker used by the deterministic T-oracle. */ + nonce: string; + /** + * Replay only: every fault that crossed this task in the source wave, with + * its offset from the task's launch. Replaces the single planned fault. + */ + replayFaults?: ScheduledFault[]; +} + +export interface ScheduledFault { kind: FaultKind; holdMs: number; atMs: number } + +export interface WavePlan { + schema: "paperclip.restart-chaos.wave-plan/v1"; + waveId: string; + seed: number; + createdAt: string; + options: WaveOptions; + tasks: TaskPlan[]; + /** Present when this wave replays one task of an earlier wave. */ + replayOf?: { waveId: string; taskId: string }; +} + +export type Verdict = "pass" | "fail" | "n/a"; +export interface RuleResult { verdict: Verdict; detail: string } +export type RuleId = "R1" | "R2" | "R3" | "R4" | "R5" | "R6"; + +export interface RunSample { + id: string; + status: string; + createdAt: string | null; + startedAt: string | null; + finishedAt: string | null; + errorCode: string | null; + retryOfRunId: string | null; + wakeCommentIds: string[]; + livenessState: string | null; +} + +export interface InterruptionSample { + state: string; + canRetry: boolean; + retryWait: boolean; + workspaceFailure: string | null; + observedRunId: string | null; + userAction: string | null; +} + +/** One observer poll. Unreachable polls keep only the timestamp. */ +export interface TaskSample { + at: number; + reachable: boolean; + serverReady: boolean; + issueStatus?: string; + interruption?: InterruptionSample | null; + activeRecoveryAction?: boolean; + /** Owner of the active recovery action; "board" means it waits on a person. */ + recoveryOwner?: string | null; + executionBlocker?: boolean; + scheduledRetry?: boolean; + runs?: RunSample[]; +} + +export interface RetryAttempt { + at: number; + observedRunId: string | null; + httpStatus: number | null; + accepted: boolean | null; + error: string | null; +} + +export interface FaultReceipt { + faultId: string; + kind: FaultKind; + /** Task that scheduled the fault; every other running task is also exposed. */ + requestedBy: string; + signal: "SIGKILL" | "SIGTERM"; + holdMs: number; + requestedAt: number; + startedAt: number; + stoppedAt: number | null; + restartedAt: number | null; + healthyAt: number | null; + readyAt: number | null; + phases: Array<{ phase: string; at: number }>; + serverLogOffsetAtStart: number; + serverLogOffsetAtReady: number | null; + error: string | null; +} + +export interface OracleArtifacts { + issueStatus: string | null; + documents: Record; + agentComments: string[]; + files: Record; + commands: Record; + workspacePath: string | null; + /** B5: distinct `.bench/progress` values the harness itself saw while the task was live. */ + benchProgressObserved?: number; +} + +export interface OracleResult { verdict: Exclude; checks: Array<{ name: string; ok: boolean; detail?: string }> } + +export interface RestSweep { + activeRuns: string[]; + runningWorkspaceOperations: string[]; + activeRecoveryAction: boolean; + error: string | null; +} + +export interface TaskRecord { + plan: TaskPlan; + issueId: string | null; + identifier: string | null; + createdAt: number | null; + launchError: string | null; + samples: TaskSample[]; + retry: RetryAttempt | null; + restAt: number | null; + finalStatus: string | null; + deadlineAt: number | null; + timedOut: boolean; + followUp: { at: number; commentId: string | null; error: string | null } | null; + faultIds: string[]; + sweep: RestSweep | null; + /** Workspace files present just before the first fault that hit this task (R2). */ + preFaultFiles: string[] | null; + /** Set when the pre-fault snapshot could not read the workspace; R2 then fails visibly. */ + preFaultSnapshotError?: string | null; + /** Pre-fault files missing at rest; null when not checked. */ + lostFiles: string[] | null; + /** B5: distinct bench progress values the observer read during polling. */ + benchProgressSeen?: string[]; +} + +export interface TaskGrade { + crossed: boolean; + crossingFaults: string[]; + rules: Record; + T: OracleResult; + pass: boolean; + signature: string | null; +} diff --git a/tests/restart-chaos/vitest.config.ts b/tests/restart-chaos/vitest.config.ts new file mode 100644 index 00000000000..aeafb2963b9 --- /dev/null +++ b/tests/restart-chaos/vitest.config.ts @@ -0,0 +1,12 @@ +import { defineConfig } from "vitest/config"; + +/** Credential-free unit tests only; the harness itself is run by main.ts. */ +export default defineConfig({ + root: import.meta.dirname, + test: { + name: "restart-chaos-unit", + include: ["**/*.test.ts"], + exclude: ["results/**", "node_modules/**"], + testTimeout: 30_000, + }, +}); diff --git a/tests/restart-chaos/workloads.ts b/tests/restart-chaos/workloads.ts new file mode 100644 index 00000000000..a9f85a26fb2 --- /dev/null +++ b/tests/restart-chaos/workloads.ts @@ -0,0 +1,241 @@ +import type { OracleArtifacts, OracleResult, WorkloadId } from "./types.js"; + +/** Deterministic bench output: sum of squares 1..240. */ +export const BENCH_STEPS = 240; +export const BENCH_EXPECTED = `BENCH_RESULT=${(BENCH_STEPS * (BENCH_STEPS + 1) * (2 * BENCH_STEPS + 1)) / 6}`; + +export interface WorkloadDefinition { + id: WorkloadId; + label: string; + usesRepository: boolean; + /** Fault fire offset from issue creation (seeded uniform). */ + fireWindowMs: readonly [number, number]; + followUpWindowMs?: readonly [number, number]; + /** Time budget for the task excluding outage time (R1). */ + budgetMs: number; + title(nonce: string): string; + description(nonce: string): string; + followUp?(nonce: string): string; + /** Files and commands the observer must collect for the T-oracle. */ + collect: { documents: string[]; files(nonce: string): string[]; commands: Array<{ name: string; argv: string[] }> }; + grade(nonce: string, artifacts: OracleArtifacts): OracleResult; +} + +const briefMarker = (nonce: string) => `CHAOS-${nonce}`; +const followUpMarker = (nonce: string) => `FOLLOWUP-${nonce}`; +const notesFile = (nonce: string) => `NOTES-${nonce}.md`; +const benchFile = (nonce: string) => `bench-result-${nonce}.txt`; +const benchRecord = (nonce: string) => `.bench/run-${nonce}.txt`; +/** bench.sh must have actually run: at least this long between start and finish. */ +export const BENCH_MIN_SECONDS = 200; +/** B5: distinct progress values the harness must have observed itself while polling. */ +export const BENCH_MIN_OBSERVED_PROGRESS = 3; + +/** Parse bench.sh's own completion record: `started= finished= result=`. */ +export function parseBenchRecord(text: string | null | undefined) { + const fields = Object.fromEntries((text ?? "").trim().split(/\s+/).map(part => part.split("=")).filter(pair => pair.length >= 2) + .map(([key, ...rest]) => [key!, rest.join("=")])); + const started = Number(fields.started), finished = Number(fields.finished); + return Number.isFinite(started) && Number.isFinite(finished) && fields.result + ? { seconds: finished - started, result: fields.result } : null; +} + +function result(checks: OracleResult["checks"]): OracleResult { + return { verdict: checks.every(check => check.ok) ? "pass" : "fail", checks }; +} + +function statusCheck(artifacts: OracleArtifacts) { + return { name: "issue_done", ok: artifacts.issueStatus === "done", detail: `status=${artifacts.issueStatus ?? "unknown"}` }; +} + +function briefChecks(body: string | null | undefined, markers: string[]) { + const text = body ?? ""; + const bullets = text.split(/\r?\n/).filter(line => /^\s*[-*] \S/.test(line)).length; + return [ + { name: "brief_document_present", ok: body != null && text.trim().length > 0 }, + ...markers.map(marker => ({ name: `brief_contains_${marker.split("-")[0]!.toLowerCase()}_marker`, ok: text.includes(marker), detail: marker })), + { name: "brief_has_three_bullets", ok: bullets >= 3, detail: `bullets=${bullets}` }, + ]; +} + +const briefDescription = (nonce: string) => [ + "Write a short brief on why idempotent retries matter for background job queues.", + "Save it as this issue's document with key `brief` (title \"Brief\"), in markdown.", + `The document must contain the line \`${briefMarker(nonce)}\` exactly, on its own line,`, + "and at least three bullet points that start with \"- \".", + "Do not create files or other tasks. When the document is saved, mark this task done.", +].join("\n"); + +export const workloadDefinitions: Record = { + A1: { + id: "A1", + label: "Brief to issue document", + usesRepository: false, + fireWindowMs: [10_000, 45_000], + budgetMs: 10 * 60_000, + title: nonce => `Chaos A1 brief ${nonce}`, + description: briefDescription, + collect: { documents: ["brief"], files: () => [], commands: [] }, + grade(nonce, artifacts) { + return result([statusCheck(artifacts), ...briefChecks(artifacts.documents.brief, [briefMarker(nonce)])]); + }, + }, + C1: { + id: "C1", + label: "Comment while running", + usesRepository: false, + fireWindowMs: [20_000, 60_000], + followUpWindowMs: [8_000, 25_000], + budgetMs: 12 * 60_000, + title: nonce => `Chaos C1 brief with follow-up ${nonce}`, + description: briefDescription, + followUp: nonce => `Additional requirement for this task: also include the line \`${followUpMarker(nonce)}\` on its own line in the \`brief\` document before you mark the task done.`, + collect: { documents: ["brief"], files: () => [], commands: [] }, + grade(nonce, artifacts) { + return result([statusCheck(artifacts), ...briefChecks(artifacts.documents.brief, [briefMarker(nonce), followUpMarker(nonce)])]); + }, + }, + B1: { + id: "B1", + label: "Python CLI with tests", + usesRepository: true, + fireWindowMs: [20_000, 90_000], + budgetMs: 15 * 60_000, + title: nonce => `Chaos B1 slugify command ${nonce}`, + description: nonce => [ + "Work in this task's repository workspace (a small Python CLI called chaoscli).", + "Implement the `slugify` subcommand in `chaoscli/cli.py` so that", + "`python3 -m chaoscli slugify \"Hello Chaos World\"` prints `hello-chaos-world`", + "(lowercase, runs of non-alphanumeric characters become a single `-`, no leading or trailing `-`).", + "The currently failing test must pass: run `python3 -m unittest discover -s tests -v`.", + `Also create \`${notesFile(nonce)}\` in the repository root containing exactly the line \`B1-${nonce}\`.`, + "Use only the Python standard library. Do not run ./bench.sh. When the tests pass, mark this task done.", + ].join("\n"), + collect: { + documents: [], + files: nonce => [notesFile(nonce), "chaoscli/cli.py"], + commands: [ + { name: "unittest", argv: ["python3", "-m", "unittest", "discover", "-s", "tests"] }, + { name: "slugify", argv: ["python3", "-m", "chaoscli", "slugify", "Hello Chaos World"] }, + ], + }, + grade(nonce, artifacts) { + const tests = artifacts.commands.unittest; + const slug = artifacts.commands.slugify; + return result([ + statusCheck(artifacts), + { name: "workspace_located", ok: artifacts.workspacePath !== null }, + { name: "unittest_passes", ok: tests?.exitCode === 0, detail: `exit=${tests?.exitCode ?? "not run"}` }, + { name: "slugify_output", ok: slug?.exitCode === 0 && slug.stdout.trim() === "hello-chaos-world", detail: JSON.stringify(slug?.stdout.trim() ?? null) }, + { name: "notes_file", ok: artifacts.files[notesFile(nonce)]?.trim() === `B1-${nonce}` }, + ]); + }, + }, + B5: { + id: "B5", + label: "Long bench.sh run", + usesRepository: true, + fireWindowMs: [60_000, 180_000], + budgetMs: 20 * 60_000, + title: nonce => `Chaos B5 bench ${nonce}`, + description: nonce => [ + "Work in this task's repository workspace.", + `Run \`./bench.sh ${nonce}\` from the repository root in the foreground and wait for it to finish.`, + "It takes about four minutes; if your shell tool has a timeout, set it to at least 10 minutes. Do not background or interrupt it.", + "It prints one line of the form `BENCH_RESULT=`.", + `Write exactly that line into a new file \`${benchFile(nonce)}\` in the repository root.`, + "Do not compute the result yourself and do not edit bench.sh. Then mark this task done.", + ].join("\n"), + collect: { documents: [], files: nonce => [benchFile(nonce), benchRecord(nonce)], commands: [] }, + grade(nonce, artifacts) { + const content = artifacts.files[benchFile(nonce)]?.trim() ?? null; + const record = parseBenchRecord(artifacts.files[benchRecord(nonce)]); + return result([ + statusCheck(artifacts), + { name: "workspace_located", ok: artifacts.workspacePath !== null }, + { name: "bench_result_file", ok: content === BENCH_EXPECTED, detail: JSON.stringify(content) }, + { name: "bench_actually_ran", ok: record !== null && record.result === BENCH_EXPECTED && record.seconds >= BENCH_MIN_SECONDS, + detail: record ? `ran ${record.seconds}s, ${record.result}` : "no completion record from bench.sh" }, + // The record is agent-writable; the harness's own progress observations are not. + { name: "bench_progress_observed", ok: (artifacts.benchProgressObserved ?? 0) >= BENCH_MIN_OBSERVED_PROGRESS, + detail: `observed ${artifacts.benchProgressObserved ?? 0} progress values` }, + ]); + }, + }, +}; + +/** Files committed into the `chaos-sandbox` fixture repository. */ +export function fixtureRepositoryFiles(benchSeconds = 240): Record { + return { + "README.md": { content: "# chaos-sandbox\n\nFixture repository for the Paperclip restart chaos harness.\n" }, + "chaoscli/__init__.py": { content: "" }, + "chaoscli/__main__.py": { content: "from chaoscli.cli import main\n\nraise SystemExit(main())\n" }, + "chaoscli/cli.py": { content: [ + "import argparse", + "import sys", + "", + "", + "def greet(name: str) -> str:", + " return f\"hello, {name}\"", + "", + "", + "def slugify(text: str) -> str:", + " raise NotImplementedError(\"slugify is not implemented yet\")", + "", + "", + "def main(argv=None) -> int:", + " parser = argparse.ArgumentParser(prog=\"chaoscli\")", + " commands = parser.add_subparsers(dest=\"command\", required=True)", + " greet_parser = commands.add_parser(\"greet\")", + " greet_parser.add_argument(\"name\")", + " args = parser.parse_args(argv)", + " if args.command == \"greet\":", + " print(greet(args.name))", + " return 0", + " return 2", + "", + ].join("\n") }, + "tests/__init__.py": { content: "" }, + "tests/test_cli.py": { content: [ + "import unittest", + "", + "from chaoscli.cli import greet, slugify", + "", + "", + "class CliTest(unittest.TestCase):", + " def test_greet(self):", + " self.assertEqual(greet(\"chaos\"), \"hello, chaos\")", + "", + " def test_slugify(self):", + " self.assertEqual(slugify(\"Hello Chaos World\"), \"hello-chaos-world\")", + " self.assertEqual(slugify(\" A--b__C!! \"), \"a-b-c\")", + "", + "", + "if __name__ == \"__main__\":", + " unittest.main()", + "", + ].join("\n") }, + "bench.sh": { mode: 0o755, content: [ + "#!/bin/sh", + "# Deterministic long-running benchmark for restart chaos boundary B6.", + "set -eu", + `steps=${BENCH_STEPS}`, + `seconds="\${BENCH_SECONDS:-${benchSeconds}}"`, + "tag=\"${1:-untagged}\"", + "mkdir -p .bench", + "started=$(date +%s)", + "total=0", + "i=1", + "while [ \"$i\" -le \"$steps\" ]; do", + " total=$((total + i * i))", + " if [ $((i % 24)) -eq 0 ]; then echo \"bench progress $i/$steps\"; echo \"$i\" > .bench/progress; fi", + " sleep \"$(awk -v s=\"$seconds\" -v n=\"$steps\" 'BEGIN { printf \"%.3f\", s / n }')\"", + " i=$((i + 1))", + "done", + "echo \"started=$started finished=$(date +%s) result=BENCH_RESULT=$total\" > \".bench/run-$tag.txt\"", + "echo \"BENCH_RESULT=$total\"", + "", + ].join("\n") }, + ".gitignore": { content: ".bench/\n__pycache__/\n" }, + }; +} From 21a1e7885259eec1c83e24d18da90a282adbb83d Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 08:52:43 -0800 Subject: [PATCH 09/13] feat(environments): add managed plugin environment rows (#19490) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document > - That document cannot declare a plugin environment yet, so a managed instance cannot get a plugin-provided environment without manual setup > - The full change is split into three pull requests. This first one adds the plugin environment rows and their write floor, and nothing calls them yet > - The second pull request lets the managed config declare plugin entries and provisions them at boot. The third sets the instance default once and advertises the new managed-config capability > - The benefit is a smaller review for each step, and no instance changes behavior until the last step lands ## Linked Issues or Issue Description **Subsystem affected** Server environments (managed-config environment provisioning, environment routes) and the board UI's adapter Test target. **Problem or motivation** A managed instance cannot get a plugin environment from its managed config. Company settings cannot create one either. So each managed instance that needs one requires manual setup. **Proposed solution** Over three pull requests, let an `environments` entry in the managed config declare `driver: "plugin"`. Boot then provisions one locked row for each plugin entry, and makes one of them the instance default once on new instances. This pull request adds only the row layer: create or adopt a marked row, archive it, find it, and keep tenants from editing it. **Alternatives considered** Manual setup on each instance, and a new control-plane endpoint. Both need a step outside the deploy. The managed config already delivers the sandbox entry. **Roadmap alignment** This extends the existing managed-config `environments` section, which already provisions the managed sandbox. It adds no new product surface. ## What Changed - New `server/src/services/managed-plugin-environments.ts`, used through new `environmentService` methods (`ensureManagedPluginEnvironment`, `archiveManagedPluginEnvironment`, `findManagedPluginEnvironment`): - The ensure validates the driver config through `normalizeEnvironmentConfig` and `validatePluginEnvironmentDriverConfig`, the same path an operator-created plugin environment takes. - It marks the row `managedByPaperclip: true` and `managedPluginEnvironment: ":"`. - It adopts an existing unmarked row for the same plugin and driver instead of adding a second one, so a hand-made row keeps its agents. - A row that already matches gets no write. Changes write activity log entries. - Archive sets an active marked row to `archived`. The next ensure reactivates it. - Migration `0437`: unique index `environments_managed_plugin_environment_idx` on `metadata ->> 'managedPluginEnvironment'`. This is the only schema change of the three pull requests. Snapshot `0432` is pruned to keep the five newest. - `server/src/routes/environments.ts`: `isPlatformProvisionedEnvironment` also covers the new marker (write floor and redaction). `managedPluginEnvironment` is a reserved marker key, so a client cannot stamp it on its own row. Nothing provisions marked rows yet, so the marker-clear escape hatch stays open for them. The next pull request refuses it while the managed config declares the row's entry. - `ui/src/lib/adapter-test-environment.ts`: `resolveManagedSandboxEnvironmentId` requires `driver === "sandbox"`. The plugin row also carries `managedByPaperclip`, so the old "not local" check could take it for the managed sandbox. ## Verification - `cd server && npx vitest run src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres): - The ensure creates the marked row through the plugin validator. A second ensure writes nothing and logs nothing. - A hand-made row for the same plugin and driver is adopted. It keeps its id and its agents. - Archive, then ensure, reactivates the row. - The ensure refuses while the plugin has no running worker, and refuses a name that another environment holds. - The unique index allows one row per marker. - `npx vitest run src/__tests__/environment-routes.test.ts`: on a cloud-managed instance, edit, env-var edit, status change and delete of a marked plugin row are refused, including for instance admins. The marker alone keeps the floor. A client cannot stamp the marker. The marker clear is allowed. - `npx vitest run src/__tests__/environment-service.test.ts`: passes. - `cd ui && npx vitest run src/lib/adapter-test-environment.test.ts`: the plugin row is never taken for the managed sandbox. Under managed-sandbox-only, an agent on the sandbox tests in the sandbox, and an agent on the plugin environment tests there. - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck` and `pnpm --filter @paperclipai/db check:migrations` pass. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. `pnpm verify` asks for broader verification because of the migration. ## Risks - Migration: one new partial unique index on a small table. No row carries the marker before this change, so the index cannot fail on existing data. - No caller uses the new ensure yet, so no instance changes behavior. - The UI change affects only lists that hold a non-sandbox row with `managedByPaperclip`. Today that is only the local row, which the old check already excluded. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/migrations/0437_workable_oracle.sql | 1 + ...{0432_snapshot.json => 0437_snapshot.json} | 111 ++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/environments.ts | 6 + .../src/__tests__/environment-routes.test.ts | 69 +++++ .../managed-plugin-environments.test.ts | 178 +++++++++++ server/src/routes/environments.ts | 9 +- server/src/services/environments.ts | 36 +++ .../services/managed-plugin-environments.ts | 287 ++++++++++++++++++ ui/src/lib/adapter-test-environment.test.ts | 50 +++ ui/src/lib/adapter-test-environment.ts | 8 +- 11 files changed, 754 insertions(+), 8 deletions(-) create mode 100644 packages/db/src/migrations/0437_workable_oracle.sql rename packages/db/src/migrations/meta/{0432_snapshot.json => 0437_snapshot.json} (99%) create mode 100644 server/src/__tests__/managed-plugin-environments.test.ts create mode 100644 server/src/services/managed-plugin-environments.ts diff --git a/packages/db/src/migrations/0437_workable_oracle.sql b/packages/db/src/migrations/0437_workable_oracle.sql new file mode 100644 index 00000000000..71542e50c1c --- /dev/null +++ b/packages/db/src/migrations/0437_workable_oracle.sql @@ -0,0 +1 @@ +CREATE UNIQUE INDEX "environments_managed_plugin_environment_idx" ON "environments" USING btree (("metadata" ->> 'managedPluginEnvironment')) WHERE jsonb_typeof("environments"."metadata" -> 'managedPluginEnvironment') = 'string'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0432_snapshot.json b/packages/db/src/migrations/meta/0437_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0432_snapshot.json rename to packages/db/src/migrations/meta/0437_snapshot.json index d44daa9c781..96b47269974 100644 --- a/packages/db/src/migrations/meta/0432_snapshot.json +++ b/packages/db/src/migrations/meta/0437_snapshot.json @@ -1,6 +1,6 @@ { - "id": "a87278a8-15b3-4187-94a9-f4bb79d8640d", - "prevId": "9fcf29d1-a9a6-4ead-8d43-7e087a2b2b24", + "id": "2c0cb580-67fc-45d3-9ca5-cce4148dd1a4", + "prevId": "a80d63fe-5111-49e8-a39a-7a5589b3b31a", "version": "7", "dialect": "postgresql", "tables": { @@ -4058,6 +4058,28 @@ "method": "btree", "with": {} }, + "agent_wakeup_requests_tree_hold_resume_uq": { + "name": "agent_wakeup_requests_tree_hold_resume_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "idempotency_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'tree-hold-resume:%' AND \"agent_wakeup_requests\".\"status\" <> 'skipped'", + "concurrently": true, + "method": "btree", + "with": {} + }, "agent_wakeup_requests_chat_completion_uq": { "name": "agent_wakeup_requests_chat_completion_uq", "columns": [ @@ -28962,6 +28984,22 @@ "method": "btree", "with": {} }, + "environments_managed_plugin_environment_idx": { + "name": "environments_managed_plugin_environment_idx", + "columns": [ + { + "expression": "(\"metadata\" ->> 'managedPluginEnvironment')", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "jsonb_typeof(\"environments\".\"metadata\" -> 'managedPluginEnvironment') = 'string'", + "concurrently": false, + "method": "btree", + "with": {} + }, "environments_name_idx": { "name": "environments_name_idx", "columns": [ @@ -35053,6 +35091,28 @@ "method": "btree", "with": {} }, + "heartbeat_runs_tree_hold_suppression_idx": { + "name": "heartbeat_runs_tree_hold_suppression_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "(\"result_json\"->'treeHoldSuppression'->>'holdId')", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"status\" = 'cancelled' and \"heartbeat_runs\".\"result_json\" ? 'treeHoldSuppression'", + "concurrently": false, + "method": "btree", + "with": {} + }, "heartbeat_runs_cost_accounting_pending_idx": { "name": "heartbeat_runs_cost_accounting_pending_idx", "columns": [ @@ -42492,6 +42552,12 @@ "primaryKey": false, "notNull": false }, + "origin_issue_id": { + "name": "origin_issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, "origin_identity_context_id": { "name": "origin_identity_context_id", "type": "uuid", @@ -42935,6 +43001,27 @@ "method": "btree", "with": {} }, + "issues_company_origin_issue_idx": { + "name": "issues_company_origin_issue_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "origin_issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "issues_company_project_workspace_idx": { "name": "issues_company_project_workspace_idx", "columns": [ @@ -43534,6 +43621,19 @@ "onDelete": "no action", "onUpdate": "no action" }, + "issues_origin_issue_id_issues_id_fk": { + "name": "issues_origin_issue_id_issues_id_fk", + "tableFrom": "issues", + "tableTo": "issues", + "columnsFrom": [ + "origin_issue_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, "issues_execution_workspace_id_execution_workspaces_id_fk": { "name": "issues_execution_workspace_id_execution_workspaces_id_fk", "tableFrom": "issues", @@ -61884,6 +61984,13 @@ "type": "boolean", "primaryKey": false, "notNull": true, + "default": true + }, + "classic_workspace_enabled": { + "name": "classic_workspace_enabled", + "type": "boolean", + "primaryKey": false, + "notNull": true, "default": false }, "created_at": { diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index be513a9a67f..74008905afd 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -3046,6 +3046,13 @@ "when": 1793952565204, "tag": "0436_legal_reavers", "breakpoints": true + }, + { + "idx": 437, + "version": "7", + "when": 1793982073367, + "tag": "0437_workable_oracle", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/environments.ts b/packages/db/src/schema/environments.ts index a61c2a99ae1..2f7b4e0549a 100644 --- a/packages/db/src/schema/environments.ts +++ b/packages/db/src/schema/environments.ts @@ -25,6 +25,12 @@ export const environments = pgTable( .where( sql`${table.driver} = 'sandbox' AND (${table.metadata} ->> 'managedByPaperclip')::boolean = true`, ), + // One row per managed-config plugin entry: the marker is the entry's + // ":" pair (`ensureManagedPluginEnvironment`). + // Only string markers count; a cleared marker (`null` / `false`) is not one. + managedPluginEnvironmentIdx: uniqueIndex("environments_managed_plugin_environment_idx") + .on(sql`(${table.metadata} ->> 'managedPluginEnvironment')`) + .where(sql`jsonb_typeof(${table.metadata} -> 'managedPluginEnvironment') = 'string'`), nameIdx: uniqueIndex("environments_name_idx").on(table.name), }), ); diff --git a/server/src/__tests__/environment-routes.test.ts b/server/src/__tests__/environment-routes.test.ts index 51e82de3844..786e97d6ea2 100644 --- a/server/src/__tests__/environment-routes.test.ts +++ b/server/src/__tests__/environment-routes.test.ts @@ -1020,6 +1020,75 @@ describe("environment routes", () => { expect(mockEnvironmentService.create).not.toHaveBeenCalled(); }); + describe("managed plugin environment rows", () => { + const PLUGIN_KEY = "example.remote-host-plugin"; + const pluginRow = () => ({ + ...createPlatformSandboxEnvironment(), + id: "env-plugin-1", + name: "Remote Host", + driver: "plugin", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + envVars: {}, + metadata: { managedByPaperclip: true, managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + }); + + it("refuses edits, env-var edits, and deletes, including for instance admins", async () => { + mockEnvironmentService.getById.mockResolvedValue(pluginRow()); + const app = createApp(ownerAdminActor); + + for (const body of [{ name: "Renamed" }, { envVars: { A: "1" } }, { status: "archived" }]) { + const res = await request(app).patch("/api/environments/env-plugin-1").send(body); + expect(res.status).toBe(403); + expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); + } + const deleted = await request(app).delete("/api/environments/env-plugin-1"); + expect(deleted.status).toBe(403); + expect(deleted.body.details).toMatchObject({ code: "environment_platform_managed" }); + expect(mockEnvironmentService.update).not.toHaveBeenCalled(); + expect(mockEnvironmentService.removeIfDeletable).not.toHaveBeenCalled(); + }); + + it("keeps the row behind the floor on the plugin marker alone", async () => { + mockEnvironmentService.getById.mockResolvedValue({ + ...pluginRow(), + metadata: { managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + }); + const res = await request(createApp(ownerAdminActor)).patch("/api/environments/env-plugin-1").send({ name: "Renamed" }); + expect(res.status).toBe(403); + expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); + }); + + it("allows the marker clear while no managed-config entry can own the row", async () => { + // Nothing provisions marked plugin rows yet, so a marker is stale and + // stays recoverable. The managed-config plugin entry PR refuses this + // while the document declares the row's entry. + const row = pluginRow(); + mockEnvironmentService.getById.mockResolvedValue(row); + mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} }); + + const res = await request(createApp(ownerAdminActor)) + .patch("/api/environments/env-plugin-1") + .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }); + + expect(res.status).toBe(200); + expect(mockEnvironmentService.update).toHaveBeenCalled(); + }); + + it("refuses a client that stamps the plugin marker on its own row", async () => { + const res = await request(createApp(ownerAdminActor)) + .post("/api/companies/company-1/environments") + .send({ + name: "Fake remote host", + driver: "plugin", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + }); + expect(res.status).toBe(422); + expect(res.body.details).toMatchObject({ code: "environment_platform_marker_reserved" }); + expect(mockEnvironmentService.create).not.toHaveBeenCalled(); + }); + }); + it("rejects tenant patches that stamp platform markers so the row cannot become locked", async () => { const tenantEnvironment = { ...createPlatformSandboxEnvironment(), diff --git a/server/src/__tests__/managed-plugin-environments.test.ts b/server/src/__tests__/managed-plugin-environments.test.ts new file mode 100644 index 00000000000..fc8496b236d --- /dev/null +++ b/server/src/__tests__/managed-plugin-environments.test.ts @@ -0,0 +1,178 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { + activityLog, + agents, + companies, + createDb, + environments, + plugins, + type Db, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { environmentService } from "../services/environments.js"; + +const PLUGIN_KEY = "example.remote-host-plugin"; +const MARKER = `${PLUGIN_KEY}:tasks`; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("managed plugin environment rows", () => { + let database: Awaited>; + let db: Db; + let pluginId: string; + let running: boolean; + const call = vi.fn(async (_id: string, method: string, input: { config: Record }) => + method === "environmentValidateConfig" ? { ok: true, normalizedConfig: { ...input.config, normalized: true } } : null); + const workerManager = { isRunning: vi.fn(() => running), call }; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("paperclip-managed-plugin-environment-rows-"); + db = createDb(database.connectionString); + }, 90_000); + afterAll(async () => { await database?.cleanup(); }); + beforeEach(async () => { + running = true; + call.mockClear(); + await db.delete(activityLog); + await db.delete(agents); + await db.delete(environments); + await db.delete(plugins); + await db.delete(companies); + pluginId = randomUUID(); + await db.insert(plugins).values({ + id: pluginId, + pluginKey: PLUGIN_KEY, + packageName: "@example/remote-host-plugin", + version: "1.0.0", + status: "ready", + manifestJson: { + capabilities: ["environment.drivers.register"], + environmentDrivers: [{ driverKey: "tasks", requiresAgent: true, supportsTasks: true }], + } as never, + }); + }); + + const ensure = (overrides: Record = {}) => + environmentService(db).ensureManagedPluginEnvironment({ + name: "Remote Host", + description: "Runs the agent on a remote host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + workerManager, + ...overrides, + }); + const pluginRows = () => db.select().from(environments).where(eq(environments.driver, "plugin")); + const company = async () => { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Managed", issuePrefix: `M${id.slice(0, 7)}` }); + return id; + }; + + it("creates the marked row through the plugin validator, then changes nothing", async () => { + await company(); + const first = await ensure(); + expect(first.action).toBe("added"); + expect(first.environment).toMatchObject({ + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + status: "active", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: { normalized: true } }, + metadata: { managedByPaperclip: true, managedPluginEnvironment: MARKER }, + }); + expect(call).toHaveBeenCalledWith(pluginId, "environmentValidateConfig", { driverKey: "tasks", config: {} }); + expect(await db.select().from(activityLog)).toHaveLength(1); + + const second = await ensure(); + const rows = await pluginRows(); + expect(second.action).toBe("unchanged"); + expect(rows).toHaveLength(1); + expect(rows[0]!.updatedAt.getTime()).toBe(first.environment.updatedAt.getTime()); + expect(await db.select().from(activityLog)).toHaveLength(1); + }); + + it("adopts a hand-made row for the same plugin and driver, keeping its agents", async () => { + const companyId = await company(); + const [handMade] = await db.insert(environments).values({ + name: "Remote Host (manual)", + driver: "plugin", + status: "active", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { note: "made by hand" }, + }).returning(); + const agentId = randomUUID(); + await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "process", defaultEnvironmentId: handMade!.id }); + + const result = await ensure(); + const rows = await pluginRows(); + expect(result.action).toBe("adopted"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + id: handMade!.id, + name: "Remote Host", + metadata: { note: "made by hand", managedByPaperclip: true, managedPluginEnvironment: MARKER }, + }); + const [kept] = await db.select().from(agents).where(eq(agents.id, agentId)); + expect(kept!.defaultEnvironmentId).toBe(handMade!.id); + }); + + it("adopts a row again after its markers were cleared", async () => { + const first = await ensure(); + // What the route's marker-clear request stores. + await db.update(environments) + .set({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }) + .where(eq(environments.id, first.environment.id)); + + const again = await ensure(); + expect(again.action).toBe("adopted"); + expect(again.environment.id).toBe(first.environment.id); + expect(await pluginRows()).toHaveLength(1); + }); + + it("lets several rows hold a cleared marker", async () => { + const config = { pluginKey: PLUGIN_KEY, driverKey: "other", driverConfig: {} }; + await db.insert(environments).values([ + { name: "Cleared one", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: false } }, + { name: "Cleared two", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: false } }, + { name: "Cleared three", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: null } }, + ]); + expect(await pluginRows()).toHaveLength(3); + }); + + it("archives the active marked row and reactivates it on the next ensure", async () => { + await ensure(); + const archived = await environmentService(db).archiveManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }); + expect(archived?.status).toBe("archived"); + expect(await environmentService(db).archiveManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" })).toBeNull(); + expect((await environmentService(db).findManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }))?.status) + .toBe("archived"); + + const reactivated = await ensure(); + expect(reactivated.action).toBe("updated"); + expect(reactivated.environment.status).toBe("active"); + }); + + it("refuses to ensure while the plugin has no running worker", async () => { + running = false; + await expect(ensure()).rejects.toThrow(/has no running worker/); + expect(await pluginRows()).toHaveLength(0); + }); + + it("refuses to take a name another environment already holds", async () => { + await db.insert(environments).values({ name: "Remote Host", driver: "ssh", status: "active", config: {} }); + await expect(ensure()).rejects.toThrow(/already exists with driver "ssh"/); + }); + + it("allows at most one row per marker", async () => { + await ensure(); + await expect(db.insert(environments).values({ + name: "Second remote host", + driver: "plugin", + status: "active", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { managedPluginEnvironment: MARKER }, + })).rejects.toThrow(); + }); +}); diff --git a/server/src/routes/environments.ts b/server/src/routes/environments.ts index eed8b648d46..9a33f1b66f9 100644 --- a/server/src/routes/environments.ts +++ b/server/src/routes/environments.ts @@ -19,6 +19,7 @@ import { import { conflict, forbidden, unprocessable } from "../errors.js"; import { isCloudManagedInstance } from "../services/cloud-instance.js"; import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js"; +import { MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY } from "../services/managed-plugin-environments.js"; import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js"; import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js"; import { validate } from "../middleware/validate.js"; @@ -69,8 +70,10 @@ function isPlainRecord(value: unknown): value is Record { /** * Whether this environment row was provisioned by the platform: the - * managed-config environment provisioner marker, or the legacy managed - * Kubernetes wrapper marker (rows created by builds that predate the + * managed-config environment provisioner marker, the managed plugin + * environment marker (`":"`, stamped next to + * `managedByPaperclip` on a managed-config plugin entry's row), or the legacy + * managed Kubernetes wrapper marker (rows created by builds that predate the * generalized provisioner and have not been adopted yet). */ export function isPlatformProvisionedEnvironment(environment: { @@ -78,6 +81,7 @@ export function isPlatformProvisionedEnvironment(environment: { }): boolean { return ( environment.metadata?.managedByPaperclip === true || + typeof environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY] === "string" || environment.metadata?.managedKubernetesSandbox === true ); } @@ -151,6 +155,7 @@ function isTenantEditableManagedSandbox(environment: { const PLATFORM_PROVISIONED_MARKER_KEYS = [ "managedByPaperclip", + MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY, "managedKubernetesSandbox", ] as const; diff --git a/server/src/services/environments.ts b/server/src/services/environments.ts index ec93b17a184..26c6cae9d99 100644 --- a/server/src/services/environments.ts +++ b/server/src/services/environments.ts @@ -50,6 +50,13 @@ import { stockHash, type ManagedResourceStockStatus, } from "./managed-resource-drift.js"; +import { + archiveManagedPluginEnvironment as archiveManagedPluginEnvironmentRow, + ensureManagedPluginEnvironment as ensureManagedPluginEnvironmentRow, + findManagedPluginEnvironment as findManagedPluginEnvironmentRow, + type ManagedPluginEnvironmentInput, + type ManagedPluginEnvironmentReconcileAction, +} from "./managed-plugin-environments.js"; export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null }, attach: (tx: Db) => Promise): Promise { @@ -1062,6 +1069,35 @@ export function environmentService(db: Db) { archiveManagedSandboxEnvironment, + /** + * Ensure the instance-level row of one managed-config plugin entry, + * validating its driver config through the plugin environment validator. + * Adopts an unmarked row for the same plugin and driver (see + * `managed-plugin-environments.ts`). + */ + ensureManagedPluginEnvironment: async ( + input: ManagedPluginEnvironmentInput, + ): Promise<{ environment: Environment; action: ManagedPluginEnvironmentReconcileAction }> => { + const result = await ensureManagedPluginEnvironmentRow(db, input); + return { environment: toEnvironment(result.row), action: result.action }; + }, + + /** Archive the active row of a managed plugin entry whose plugin is not running. */ + archiveManagedPluginEnvironment: async ( + input: { pluginKey: string; driverKey: string }, + ): Promise => { + const row = await archiveManagedPluginEnvironmentRow(db, input); + return row ? toEnvironment(row) : null; + }, + + /** The marked row of a managed plugin entry, archived or not. */ + findManagedPluginEnvironment: async ( + input: { pluginKey: string; driverKey: string }, + ): Promise => { + const row = await findManagedPluginEnvironmentRow(db, input); + return row ? toEnvironment(row) : null; + }, + /** * Idempotently ensure a managed Kubernetes sandbox environment exists for * an instance, configured from instance/operator-supplied config. A thin diff --git a/server/src/services/managed-plugin-environments.ts b/server/src/services/managed-plugin-environments.ts new file mode 100644 index 00000000000..a3450722f9a --- /dev/null +++ b/server/src/services/managed-plugin-environments.ts @@ -0,0 +1,287 @@ +/** + * Instance-level plugin environment rows declared by the managed-config + * `environments` section (`driver: "plugin"` entries, see `managed-config.ts`). + * + * Each declared entry owns exactly one row, identified by the marker + * `metadata.managedPluginEnvironment = ":"` (unique + * index `environments_managed_plugin_environment_idx`) next to + * `managedByPaperclip: true`, which puts the row behind the environments-route + * write floor on cloud-managed instances. + * + * - The first ensure adopts an existing unmarked row for the same plugin and + * driver instead of adding a second one, so a hand-made row keeps the agents + * that point at it. + * - Later ensures converge the row to the declaration and write nothing when + * it already matches. + * - Archiving is only for a declared plugin that is not running; the next + * successful ensure reactivates the row. Nothing here archives a row + * because its entry left the document. + * + * The driver config always goes through the plugin environment validator + * (`normalizeEnvironmentConfig` + `validatePluginEnvironmentDriverConfig`), + * the same path an operator-created plugin environment takes. + */ + +import { and, eq, sql } from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { companies, environments } from "@paperclipai/db"; +import type { PluginEnvironmentConfig } from "@paperclipai/shared"; +import { logActivity, publishActivity, type ActivityPublication } from "./activity-log.js"; +import { normalizeEnvironmentConfig } from "./environment-config.js"; +import { validatePluginEnvironmentDriverConfig } from "./plugin-environment-driver.js"; +import type { PluginWorkerManager } from "./plugin-worker-manager.js"; + +export const MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY = "managedPluginEnvironment"; + +export function managedPluginEnvironmentMarker(input: { pluginKey: string; driverKey: string }): string { + return `${input.pluginKey}:${input.driverKey}`; +} + +export interface ManagedPluginEnvironmentInput { + name: string; + description?: string; + pluginKey: string; + driverKey: string; + driverConfig?: Record; + /** Runs the plugin's own config validation (`environmentValidateConfig`). */ + workerManager: Pick; +} + +export type ManagedPluginEnvironmentReconcileAction = "added" | "adopted" | "updated" | "unchanged"; + +export type EnvironmentRow = typeof environments.$inferSelect; + +export interface ManagedPluginEnvironmentRowResult { + row: EnvironmentRow; + action: ManagedPluginEnvironmentReconcileAction; +} + +function metadataOf(row: Pick): Record { + return (row.metadata ?? {}) as Record; +} + +function stableJson(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`; + if (value && typeof value === "object") { + return `{${Object.keys(value as Record) + .sort() + .map((key) => `${JSON.stringify(key)}:${stableJson((value as Record)[key])}`) + .join(",")}}`; + } + return JSON.stringify(value ?? null); +} + +function servesPair(row: EnvironmentRow, input: { pluginKey: string; driverKey: string }): boolean { + const config = (row.config ?? {}) as Record; + return config.pluginKey === input.pluginKey && config.driverKey === input.driverKey; +} + +async function validatedDriverConfig(db: Db, input: ManagedPluginEnvironmentInput): Promise { + const normalized = normalizeEnvironmentConfig({ + driver: "plugin", + config: { pluginKey: input.pluginKey, driverKey: input.driverKey, driverConfig: input.driverConfig ?? {} }, + }) as unknown as PluginEnvironmentConfig; + return await validatePluginEnvironmentDriverConfig({ + db, + workerManager: input.workerManager as PluginWorkerManager, + config: normalized, + }); +} + +/** Marker value of a row, or null when it has none (absent, or cleared to `null` / `false`). */ +function markerOf(row: Pick): string | null { + const value = metadataOf(row)[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]; + return typeof value === "string" ? value : null; +} + +/** + * Record activity for every company inside the caller's transaction, so the + * entries commit (or roll back) with the row change. Live events go to + * `publications` and are published by the caller after commit. + */ +async function logForEveryCompany( + tx: Db, + publications: ActivityPublication[], + input: { action: string; environmentId: string; details: Record }, +): Promise { + const companyIds = await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id)); + for (const companyId of companyIds) { + await logActivity(tx, { + companyId, + actorType: "system", + actorId: "managed-environment-reconciler", + action: input.action, + entityType: "environment", + entityId: input.environmentId, + details: input.details, + }, publications); + } +} + +/** + * Ensure the row for one declared plugin entry. Idempotent: a row that + * already matches the declaration is returned without a write. + */ +export async function ensureManagedPluginEnvironment( + db: Db, + input: ManagedPluginEnvironmentInput, +): Promise { + const marker = managedPluginEnvironmentMarker(input); + const config = await validatedDriverConfig(db, input); + const desiredConfig = { ...config } as Record; + const description = input.description ?? null; + + const publications: ActivityPublication[] = []; + const result = await db.transaction(async (tx): Promise => { + const reconciled = await reconcile(tx); + if (reconciled.action !== "unchanged") { + await logForEveryCompany(tx as unknown as Db, publications, { + action: `environment.managed_plugin_${reconciled.action}`, + environmentId: reconciled.row.id, + details: { pluginKey: input.pluginKey, driverKey: input.driverKey, reconciliationAction: reconciled.action }, + }); + } + return reconciled; + }); + publications.forEach(publishActivity); + return result; + + async function reconcile(tx: Parameters[0]>[0]): Promise { + // Lock every plugin row: concurrent ensures (the boot pass and a worker + // recovery, or two builds in a rolling deploy) serialize here, so only one + // of them can adopt or insert. + const pluginRows = await tx.select().from(environments).where(eq(environments.driver, "plugin")).for("update"); + let row = pluginRows.find((candidate) => markerOf(candidate) === marker) ?? null; + let adopted = false; + if (!row) { + const candidates = pluginRows + // A cleared marker (`null` / `false`, from the marker-clear request) + // counts as unmarked, so a cleared row is adopted again. + .filter((candidate) => servesPair(candidate, input) && markerOf(candidate) === null) + .sort((left, right) => + Number(right.name === input.name) - Number(left.name === input.name) + || Number(right.status === "active") - Number(left.status === "active") + || left.createdAt.getTime() - right.createdAt.getTime()); + row = candidates[0] ?? null; + adopted = row !== null; + } + + const nameOwner = await tx + .select({ id: environments.id, driver: environments.driver }) + .from(environments) + .where(eq(environments.name, input.name)) + .then((rows) => rows[0] ?? null); + + if (!row) { + if (nameOwner) { + throw new Error( + `Failed to ensure managed plugin environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`, + ); + } + const now = new Date(); + const inserted = await tx + .insert(environments) + .values({ + name: input.name, + description, + driver: "plugin", + status: "active", + config: desiredConfig, + envVars: {}, + metadata: { managedByPaperclip: true, [MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]: marker }, + createdAt: now, + updatedAt: now, + }) + .onConflictDoNothing() + .returning() + .then((rows) => rows[0] ?? null); + if (inserted) return { row: inserted, action: "added" }; + // A concurrent pass won the marker index; converge on its row. + row = await tx + .select() + .from(environments) + .where(sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${marker}`) + .for("update") + .then((rows) => rows[0] ?? null); + if (!row) throw new Error("Failed to ensure managed plugin environment"); + } + + // Keep the row's own name when another row already holds the declared one. + const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name; + const currentMetadata = metadataOf(row); + const desiredMetadata = { + ...currentMetadata, + managedByPaperclip: true, + [MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]: marker, + }; + const matches = + row.name === desiredName && + (row.description ?? null) === description && + row.status === "active" && + stableJson(row.config) === stableJson(desiredConfig) && + stableJson(currentMetadata) === stableJson(desiredMetadata); + if (matches) return { row, action: "unchanged" }; + + const updated = await tx + .update(environments) + .set({ + name: desiredName, + description, + status: "active", + config: desiredConfig, + metadata: desiredMetadata, + updatedAt: new Date(), + }) + .where(eq(environments.id, row.id)) + .returning() + .then((rows) => rows[0] ?? null); + if (!updated) throw new Error("Managed plugin environment changed during reconciliation"); + return { row: updated, action: adopted ? "adopted" : "updated" }; + } +} + +/** + * Archive the marked row of a declared plugin that is not ready or has no + * running worker, so run scheduling stops selecting it. Returns null when + * there is no active marked row. + */ +export async function archiveManagedPluginEnvironment( + db: Db, + input: { pluginKey: string; driverKey: string }, +): Promise { + const marker = managedPluginEnvironmentMarker(input); + const publications: ActivityPublication[] = []; + const archived = await db.transaction(async (tx) => { + const row = await tx + .update(environments) + .set({ status: "archived", updatedAt: new Date() }) + .where(and( + eq(environments.driver, "plugin"), + eq(environments.status, "active"), + sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${marker}`, + )) + .returning() + .then((rows) => rows[0] ?? null); + if (!row) return null; + await logForEveryCompany(tx as unknown as Db, publications, { + action: "environment.managed_provider_unavailable_archived", + environmentId: row.id, + details: { pluginKey: input.pluginKey, driverKey: input.driverKey }, + }); + return row; + }); + publications.forEach(publishActivity); + return archived; +} + +/** The marked row for one plugin entry, archived or not. */ +export async function findManagedPluginEnvironment( + db: Db, + input: { pluginKey: string; driverKey: string }, +): Promise { + return await db + .select() + .from(environments) + .where(sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${managedPluginEnvironmentMarker(input)}`) + .then((rows) => rows[0] ?? null); +} diff --git a/ui/src/lib/adapter-test-environment.test.ts b/ui/src/lib/adapter-test-environment.test.ts index 3422e34378e..8243815f65a 100644 --- a/ui/src/lib/adapter-test-environment.test.ts +++ b/ui/src/lib/adapter-test-environment.test.ts @@ -275,6 +275,56 @@ describe("resolveManagedSandboxEnvironmentId", () => { }); }); +describe("managed plugin environments under managed-sandbox-only", () => { + const local = makeEnvironment({ + id: "local-env", + driver: "local", + metadata: { managedByPaperclip: true, defaultForInstance: true }, + }); + // Listed first, so a lookup that only excluded `local` would pick it. + const pluginEnvironment = makeEnvironment({ + id: "plugin-env", + name: "Remote Host", + driver: "plugin", + config: { pluginKey: "example.remote-host-plugin", driverKey: "tasks", driverConfig: {} }, + metadata: { + managedByPaperclip: true, + managedPluginEnvironment: "example.remote-host-plugin:tasks", + }, + }); + const sandbox = makeEnvironment({ + id: "sandbox-env", + name: "Paperclip Sandbox", + metadata: { managedByPaperclip: true, managedSandboxProvider: "daytona" }, + }); + const environments = [local, pluginEnvironment, sandbox]; + const resolve = (agentDefaultEnvironmentId: string | null, instanceDefaultEnvironmentId: string | null) => + resolveAdapterTestEnvironmentId({ + agentDefaultEnvironmentId, + instanceDefaultEnvironmentId, + localDefaultEnvironmentId: resolveLocalDefaultEnvironmentId(environments), + managedSandboxOnly: true, + managedSandboxEnvironmentId: resolveManagedSandboxEnvironmentId(environments), + visibleEnvironmentIds: [pluginEnvironment.id, sandbox.id], + }); + + it("never mistakes the plugin row for the managed sandbox", () => { + expect(resolveManagedSandboxEnvironmentId(environments)).toBe("sandbox-env"); + expect(resolveManagedSandboxEnvironmentId([local, pluginEnvironment])).toBeNull(); + }); + + it("tests an agent without the plugin environment in the sandbox", () => { + expect(resolve("sandbox-env", "plugin-env")).toBe("sandbox-env"); + // An instance that keeps local as its default: the redirect lands on the sandbox. + expect(resolve(null, null)).toBe("sandbox-env"); + }); + + it("tests an agent on the plugin environment there", () => { + expect(resolve("plugin-env", null)).toBe("plugin-env"); + expect(resolve(null, "plugin-env")).toBe("plugin-env"); + }); +}); + describe("resolveLocalDefaultEnvironmentId", () => { it("finds the local-driver instance-default environment", () => { const environments = [ diff --git a/ui/src/lib/adapter-test-environment.ts b/ui/src/lib/adapter-test-environment.ts index 7c6d57e3b45..9628be51b10 100644 --- a/ui/src/lib/adapter-test-environment.ts +++ b/ui/src/lib/adapter-test-environment.ts @@ -86,9 +86,9 @@ export function resolveAdapterTestEnvironmentId(input: { * Find the active managed sandbox environment id in an environment list. The * platform provisioner stamps the managed environment with * `metadata.managedByPaperclip` (see `isPlatformManagedEnvironment`). The - * local-default environment also carries that stamp, so this function excludes - * the `local` driver and returns the non-local managed environment. The function - * selects only an `active` environment, never an `archived` one, so the Test + * local-default environment and managed plugin environment rows also carry + * that stamp, so this function requires the `sandbox` driver and never + * mistakes a plugin row for the managed sandbox. The function selects only an `active` environment, never an `archived` one, so the Test * resolution matches the server run-time resolver that requires an active * managed sandbox (see `findManagedSandboxEnvironment`). The Test resolution * uses this id as the redirect target under the managed-sandbox-only policy. The @@ -101,7 +101,7 @@ export function resolveManagedSandboxEnvironmentId( if (!environments) return null; const managed = environments.find( (environment) => - environment.driver !== "local" && + environment.driver === "sandbox" && environment.status === "active" && environment.metadata?.managedByPaperclip === true, ); From 55ea9cfa1bc7a5469ab0cf76ed2c487803238711 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 08:54:09 -0800 Subject: [PATCH 10/13] fix(runner): match retained cleanup and Codex close types to their contracts (#19495) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner runs native agent sessions, such as Codex, behind typed contracts > - #18839 started to typecheck the Runner tests, and it listed contract types that do not match their runtime use > - First, retained runnerd cleanup accepts any backend kind, but cleanup works only when that kind is `"runner"` > - Second, `CodexHarnessSession.close` makes its input optional, but the `HarnessSession` contract requires a reason > - This pull request narrows both types to match their contracts > - The benefit is that the compiler now rejects a cleanup proof that would silently release nothing ## Linked Issues or Issue Description No issue exists. #18839 listed both items under its product questions and did not change them. **What existing behavior does this improve?** 1. `completeRetainedNativeSessionCleanup` builds its release key from `proof.backend.kind`. The quarantine key comes from the backend descriptor's `kind`, and every production backend uses `"runner"`. If the two kinds differ, the function returns 0 and the quarantine stays. `RetainedRunnerdMaintenanceInput` and `RetainedRunnerdCleanupProof` typed `backend.kind` as any string. 2. `HarnessSession.close` takes `{ reason: string; force?: boolean }`. `CodexHarnessSession.close` took `input?: { reason: string }`. **Current behavior** The only production caller passes `backend: { kind: "runner", name: "codex_app_server" }`. All production `close` calls pass a reason. So nothing fails today. **Proposed behavior** The types state the same rules as the contracts. **Reason and benefit** A future caller that passes another kind gets a compile error, not a quarantine that stays without notice. **Breaking changes** None for production callers. Both types are narrower, and every current caller already passes the narrow values. ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: `backend.kind` is `"runner"` in `RetainedRunnerdMaintenanceInput` and `RetainedRunnerdCleanupProof`. `name` stays a string. - `packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts`: `close` takes `{ reason: string; force?: boolean }`, as the contract does. The body still reads `input?.reason`, because a generated test script calls `close()` with no arguments. Codex closes the same way with or without `force`. - `packages/paperclip-runner/src/live/runnerd-contract-types.test.ts`: new type test. The Runner test typecheck fails if either type accepts another backend kind, or if the Codex `close` parameters differ from the contract. With the master types, this file has 4 type errors. Not in this pull request: #18839 also listed that `PrpEvent.normalizedSessionId` is required in the type but optional in the event schemas. A separate pull request handles it in `validatePrpEvent`. ## Verification - `pnpm exec vitest run src/live/runnerd-contract-types.test.ts` (in `packages/paperclip-runner`): passes 2 tests. - `tsc --noEmit` passes in `packages/paperclip-runner` for `tsconfig.json`, `tsconfig.surfaces.json`, `tsconfig.test.json` and `tsconfig.surfaces.test.json`. - `cd server && pnpm typecheck`: passes. The server builds the maintenance input with the `"runner"` literal. - In `packages/paperclip-runner`, after `pnpm run test:typescript:prep`: `pnpm exec vitest run src/live/runnerd-codex-transport-settlement.test.ts src/drivers/codex/codex-protocol-cleanup.test.ts src/drivers/codex/codex-app-server-driver.lifecycle.test.ts` passes 3 files and 43 tests. - Not run locally: the full Runner `pnpm test` (including Rust tests) and the root `pnpm build`. Required CI covers them. ## Risks - Low risk. The change affects types only, plus one comment in the Codex session. Runtime behavior does not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../drivers/codex/codex-harness-session.ts | 4 +++- .../src/live/runnerd-codex-transport.ts | 5 +++-- .../src/live/runnerd-contract-types.test.ts | 22 +++++++++++++++++++ 3 files changed, 28 insertions(+), 3 deletions(-) create mode 100644 packages/paperclip-runner/src/live/runnerd-contract-types.test.ts diff --git a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts index fdde2d14d90..6bbbe3e7585 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts @@ -886,9 +886,11 @@ export class CodexHarnessSession }; } - async close(input?: { reason: string }): Promise { + // `force` is part of the contract. Codex closes the same way with or without it. + async close(input: { reason: string; force?: boolean }): Promise { this.cancelPendingRequests("session_closed"); this.eventQueue.close(); + // Untyped callers (for example generated scripts) may still omit the input. await this.transport.close(input?.reason); } diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 2f19d3aa461..503ef9cf878 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -2104,7 +2104,8 @@ export function readRunnerdArtifactBinding(runnerBinaryPath: string): { export interface RetainedRunnerdCleanupProof { readonly binding: Readonly; readonly identity: Readonly; - readonly backend: Readonly<{ kind: string; name: string }>; + /** Must match the quarantine domain, which every production backend builds with kind "runner". */ + readonly backend: Readonly<{ kind: "runner"; name: string }>; readonly providerSessionId: string; readonly activationDirectory: string; readonly sourceFingerprint: string; @@ -2400,7 +2401,7 @@ export interface RetainedRunnerdMaintenanceInput { requestId: string; binding: NativeRunIdentity; identity: DurableRecoveryIdentity; - backend: { kind: string; name: string }; + backend: { kind: "runner"; name: string }; stateDirectory: string; activationDirectory: string; sourceFingerprint: string; diff --git a/packages/paperclip-runner/src/live/runnerd-contract-types.test.ts b/packages/paperclip-runner/src/live/runnerd-contract-types.test.ts new file mode 100644 index 00000000000..09f52dacba0 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-contract-types.test.ts @@ -0,0 +1,22 @@ +import { describe, expectTypeOf, it } from "vitest"; +import type { HarnessSession } from "../contracts/harness-driver.js"; +import type { CodexHarnessSession } from "../drivers/codex/codex-harness-session.js"; +import type { RetainedRunnerdCleanupProof, RetainedRunnerdMaintenanceInput } from "./runnerd-codex-transport.js"; + +// The Runner test typecheck checks these assertions. Vitest only runs them. +describe("Runner contract types", () => { + it("accepts only the runner backend kind for retained cleanup", () => { + // The quarantine domain uses the descriptor kind, which is "runner" for every + // production backend. Another kind would release nothing. + expectTypeOf().toEqualTypeOf<"runner">(); + expectTypeOf().toEqualTypeOf<"runner">(); + + // @ts-expect-error: a backend kind other than "runner" is not allowed. + const remote: RetainedRunnerdMaintenanceInput["backend"] = { kind: "remote", name: "codex_app_server" }; + void remote; + }); + + it("matches the HarnessSession close contract in the Codex session", () => { + expectTypeOf>().toEqualTypeOf>(); + }); +}); From c6573962f803d582620c73660bcab1e4825dfbac Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:00:32 -0600 Subject: [PATCH 11/13] fix(workspaces): recover full sandboxes through read-only export (#19465) ## Thinking Path > - Paperclip preserves agent work across failed runs and server restarts. > - A remote workspace must have a durable save before its source can be released. > - Portable capture normally writes a Git mirror and a full archive inside the sandbox. > - A full sandbox can prevent capture before the provider's streaming download starts. > - Retrying that write cannot create the required free space. > - This change transfers a checked standalone source and seals its archive on the controller. > - The source remains retained unless the existing durable save completes. ## Linked Issues or Issue Description Related: #18644 reduced temporary capture space. #18172 added the provider's disk-free directory transfer. This repair connects portable capture to that transfer when the archive itself cannot be created. **What happened?** Workspace capture returns `remote_workspace_sandbox_disk_full` after provider work has finished. File recovery repeats the same sandbox write. **Expected behavior** Recover a stable standalone workspace without requiring another full archive on the source disk. Preserve staged changes, untracked work, confined links and the original source. **Steps to reproduce** 1. Create a standalone Git repository with a commit, staged changes and an untracked result. 2. Use the existing real-command transport fixture. Inject ENOSPC into the sandbox helper write or archive write. 3. Run portable capture. Before this change it rejects the helper failure. 4. With the change, restore the saved archive after removing the fixture source. Check the commit, staged content, untracked result and Git connectivity. ## What Changed - Attempt one controller-side capture after an exact sandbox disk-full failure. - Read a source inventory without installing a helper. Use the existing provider bulk transfer with confined links and runtime exclusions. - Compare the transferred inventory with source fingerprints before and after sealing. - Reject copied Git pointers, alternates and administrative symlinks. Parse config as data without includes; allow only portable keys before controller Git normalization. - Preserve socket exclusions. Retain the source and remove incomplete controller staging after failure. - Update execution semantics and focused regression coverage. No database migration or new execution authority. ## Verification - TDD red on base `2f35eb1080`: the new full-sandbox test failed with `remote_workspace_sandbox_disk_full` before production edits. - Green: 202 transport and archive tests passed before the additional Git configuration guards. At head `8541086cb5`, all 116 transport tests and the PostgreSQL durability acceptance pass. The final `e24e173ec9` exclusion guard passes those 116 cases plus its new Git-history regression across the full run and focused rerun. Environment: macOS arm64, Node 26.4.0; real Node/Git commands and filesystem transfer. - Review regressions were also red before their fixes: an internal Git config symlink bypassed the check, and `core.attributesFile` could select a controller path. Git administrative symlinks and nonportable config now fail before normalization. The focused safety selection passes 11 cases. - Cases cover helper and archive ENOSPC, commit/staged/untracked continuity after source removal, confined links, sockets, changed source, corrupt or extra copied files, unsafe Git metadata, retained diagnostic text and failed download. - Real PostgreSQL acceptance passes: normal workspace save recovers one sandbox ENOSPC, commits the encrypted snapshot, clears its writer, loses the original allocation, then independently reloads the saved result twice. The first acceptance attempt used an incorrect expected status (`available`); the documented state is `committed`. No production change was needed for that correction. - Review regressions for untracked Python/Node environments and excessive Git roots failed before their fixes. Dependency exclusions now match normal capture. Controller parsing starts only after whole-tree/repository budgets and layout checks; refspec/settings budgets and a shared two-minute verification/normalization deadline bound further work. The excessive-root test verifies zero Git subprocesses. - Real provider-boundary red/green: Daytona direct tar and capacity-fallback PTY transfers first dropped `._result.txt`. The new optional mapping contract preserves all user names and excludes only current transfer scratch. Kubernetes implements the same contract. Explicit caller exclusions and archive validation remain in force. - Standalone Linux qualification on Node 24.22.2/Debian bookworm: fresh shared/SDK build; Daytona and Kubernetes typecheck and build; Daytona file-sync/capacity/diagnostics 68/68 pass; Kubernetes file-sync 31/31 pass, including the actual `/proc` case. Qualification was refreshed again on the final `be6f7feacc` source tree: current shared/SDK rebuilt; all 815 staged source/config/manifest hashes match the delivered tree before and after testing. An initial container run as root bypassed a read-only file test; the same source passes as uid 1000. Installing zstd enabled eight initially skipped cases. - A further review regression proved `.git/pyvenv.cfg` could exclude Git history as a dependency. The inventory now handles `.git` before dependency filtering, matching normal capture. The full capture/restore test verifies the original commit and saved work. The first green attempt incorrectly compared command-result object identity; correcting the test to compare stdout passed with no production change. - Further red/green regressions cover read-only private-copy cleanup, a regular file replaced by a FIFO before open, case aliases of Git administrative metadata, and permission preservation under restrictive umask. Nonblocking no-follow file handles verify identity before and after reading. Cleanup changes only private controller directories. Provider extraction preserves ordinary permissions, refuses special mode bits before writing, and does not adopt source ownership, ACLs, attributes or flags. Final macOS Daytona tests pass 65/65; the final Linux qualification above was repeated on these exact provider sources. - Focused strict TypeScript compilation of the inventory and archive modules passes. - The first sandboxed archive run failed three Unix socket fixtures with EPERM. The permitted run outside that filesystem sandbox passed all tests. - Initial full local server typecheck could not resolve the unbuilt Runner package and produced cascading errors. Exact-head `e24e173ec9` CI typecheck, build, Runner and browser checks pass. That CI attempt failed two server shards: the unchanged retained-cleanup backlog test timed out at 15 seconds and passes locally unchanged; ten cache-reclaim fixture oracles failed because directory fallback consumed their sealed-archive counters/faults. Final commit `caed4e4f83` isolates the intended unavailable-directory boundary while preserving archive counts, recovery status, integrity and no-second-reclamation assertions. Those ten failures reproduced locally before the fixture edit. The full affected file passes 394/395; its sole other HTTP socket-hang-up case passes unchanged in an isolated rerun. This is combined coverage of all 395 cases, not a single complete green run. The final branch integrates master `53645a0104` at `98ea07225592930cbb0d18bcf250bc64f6bb13b4`, preserving upstream workspace phase timings. After integration: 117 transport cases, the PostgreSQL durability acceptance, the existing object-storage/timing regression, and 12 affected cache cases pass. The expanded recovery file now has 421 cases. Final-head `be6f7feacc45c1e32a26a2417e973f1e3752fd8d` CI passes all 55 applicable checks (three skipped, one neutral review), including the expanded recovery suite, typecheck, builds and Runner coverage. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42251345554). - Provider exclusion regressions were red through four real Linux transfer paths: Daytona direct archive, direct safe-link repair, capacity PTY repair, and Kubernetes streaming. A root socket and nested regular file share a basename; literal glob-character files/subtrees also survive. Preserve mode now anchors tar exclusions as literal root-relative paths and escapes `find -path` pruning. Default callers keep glob semantics. Receipt-count regressions were also red before correcting preserved-output counting. All 31 Kubernetes and 68 Daytona tests, builds and typechecks pass on the final source with 815 matching inputs. - The final semantics documentation explicitly records special mode bits, the stricter config allowlist (including innocuous unlisted keys), and provider qualification limited to Daytona/Kubernetes; these address the optional current-head review observations without broadening controller Git authority. - No Daytona credential is available in this session. Provider transfer is a local filesystem boundary in these tests. Hosted task recovery and rollout remain unverified. ## Risks - Recovery reads the source repeatedly and transfers the standalone tree. It can take longer than normal capture. - Linked worktrees, initialized submodules, Git administrative symlinks, nonportable Git configuration, special mode bits and alternate object stores remain retained if the normal capture cannot finish. This fallback does not follow their pointers onto the controller. - A provider without usable directory export, a changed source, or insufficient controller space cannot produce a receipt. The original source stays retained. - The server, SDK and affected provider plugins must be built from this revision together. An older provider can fail inventory verification on names it omits; it cannot publish an incomplete save. No merge, publication or deployment is included. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code editing, shell tools and automated tests. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 41 ++- .../src/sandbox-managed-runtime.ts | 9 + .../daytona/src/file-sync-capacity.test.ts | 104 +++++- .../daytona/src/file-sync.ts | 66 +++- .../kubernetes/src/file-sync.ts | 41 ++- .../kubernetes/test/unit/file-sync.test.ts | 72 ++++ packages/plugins/sdk/src/protocol.ts | 10 +- .../remote-workspace-readonly-inventory.ts | 135 ++++++++ .../remote-workspace-snapshot-archive.ts | 4 + .../remote-workspace-transport.test.ts | 314 +++++++++++++++++- .../services/remote-workspace-transport.ts | 73 +++- .../remote-workspaces.integration.test.ts | 46 +++ .../src/services/workspace-recovery.test.ts | 17 +- 13 files changed, 888 insertions(+), 44 deletions(-) create mode 100644 server/src/services/remote-workspace-readonly-inventory.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 14dec9fe6ca..289b51720c8 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1015,8 +1015,45 @@ does not repack an already dissociated mirror. Each streamed file must match its recorded hash, size and mode and retain its filesystem identity while read. Capture rechecks the complete source inventory and Git state before returning a receipt. A changed source or cancelled capture removes its incomplete archive; -an existing archive remains untouched. Temporary space must still hold the -normalized Git metadata and the complete archive together. +an existing archive remains untouched. Temporary sandbox space normally holds +the normalized Git metadata and the complete archive together. An exact disk-full +failure invokes one read-only export fallback. It fingerprints a standalone +source, transfers its files through the provider's existing bulk channel, and +checks the received inventory and the source again before sealing a portable +archive on the controller. It rechecks the source after sealing. No source files +or provider caches are removed. Linked Git worktrees, initialized submodules, alternate object stores, +Git administrative symlinks, setuid/setgid/sticky entries, and configuration outside the portable key contract +cannot use this fallback. Pure file-based config reads do not follow includes; +controller Git normalization starts only after every config passes that contract. +They remain retained if their normal archive cannot be saved. Transfer failures +or changed files retain the original disk-full error and do not produce a +preservation receipt. + +The fallback's `portableKey` allowlist in +`server/src/services/remote-workspace-readonly-inventory.ts` is authoritative and +intentionally stricter than normal capture's `portableGitSettings` sanitizer. +Normal capture drops unrelated settings; the fallback rejects every unlisted key +before controller Git can read the copied source configuration. This includes +otherwise ordinary settings such as `pull.rebase`, `commit.gpgsign`, `fetch.prune`, +`extensions.worktreeConfig`, and `remote.origin.mirror`. + +The directory transfer requests `preserveSourceEntries` to preserve ordinary +filenames and permissions. Its exclusions are literal root-relative paths, +including directory descendants; ordinary transfers keep glob exclusions. Daytona and +Kubernetes implement and qualify this mapping contract. Other providers have not +been qualified for this fallback; inventory verification still prevents a +provider that omits or alters entries from publishing an incomplete save. + +The fallback applies the normal untracked dependency exclusions before inspecting +links inside reinstallable environments. Tracked environment files remain part of +the save. Its outbound mapping requests `preserveSourceEntries`; providers retain +ordinary `._*` and scratch-like user filenames while excluding only the current +transfer's own scratch paths and the explicit source exclusions. Older providers +that omit files fail inventory verification rather than publish an incomplete save. +Fallback inventory is limited to 100,000 entries, 64 standalone repositories, +1 MiB of configuration, 1,024 settings and 256 refspec/merge entries. Controller +verification and archive normalization share a two-minute cancellation deadline. +These limits reject excessive work before publication and keep the source retained. Shallow and partial Git checkouts retain their available object files instead of asking a mirror clone to transfer promised or index-only objects. Capture diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.ts b/packages/adapter-utils/src/sandbox-managed-runtime.ts index 9159f855cdb..026b992786c 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.ts @@ -470,8 +470,17 @@ export interface SandboxSyncFileMapping { targetPath: string; kind: "file" | "directory"; mode?: number; + /** Directory glob patterns; literal root-relative paths when preserving source entries. */ exclude?: string[]; followSymlinks?: boolean; + /** + * Outbound directory transfers preserve source entry names, including `._*` + * and provider scratch-like names. In this mode `exclude` contains literal + * root-relative paths; providers omit only those paths and scratch paths + * created by this exact transfer. Archive member and symlink validation stay + * unchanged. Ignored for syncIn/files. + */ + preserveSourceEntries?: boolean; /** * Advisory read-write intent for the sandbox target. `"rw"` marks a target the * agent may change and keep; `"ro"` marks a read-only tree. An absent value diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync-capacity.test.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync-capacity.test.ts index a77e71f812a..883c90eda98 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync-capacity.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync-capacity.test.ts @@ -1,3 +1,4 @@ +import { createServer } from "node:net"; import { spawn, spawnSync } from "node:child_process"; import { promises as fs } from "node:fs"; import os from "node:os"; @@ -12,7 +13,8 @@ vi.mock("./plugin.js", async () => { import { performSyncOut } from "./file-sync.js"; const roots: string[] = []; -afterEach(async () => { await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); }); +const literalSocketCleanup: Array<() => Promise> = []; +afterEach(async () => { await Promise.all(literalSocketCleanup.splice(0).map(close => close())); vi.unstubAllEnvs(); await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); }); const executable = (names: string[], version: string) => names.map(name => { const resolved = spawnSync("/bin/sh", ["-c", 'command -v "$1"', "sh", name], { encoding: "utf8" }).stdout?.trim(); return resolved && spawnSync(resolved, ["--version"], { encoding: "utf8" }).stdout?.includes(version) ? resolved : undefined; @@ -49,7 +51,11 @@ while True: data = data[os.write(dst, data):] `; -it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "producer failure", "sparse archive over budget"])("preserves source bytes under archive capacity failure: %s", async scenario => { +it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "producer failure", "sparse archive over budget", "preserve source entries", "preserve entries direct archive", "preserve entries GNU host archive", "preserve modes native host", "preserve modes GNU host", "preserve special file native host", "preserve special directory GNU host", "preserve literal direct archive", "preserve literal direct repair", "preserve literal capacity repair"])("preserves source bytes under archive capacity failure: %s", async scenario => { + const preserveSourceEntries = scenario.startsWith("preserve"); + const literal = scenario.startsWith("preserve literal"); + const directArchive = scenario === "preserve entries direct archive" || scenario.startsWith("preserve literal direct"); + const unsafeLink = scenario !== "preserve literal direct archive"; const producerFails = scenario === "producer failure"; const sparseOverBudget = scenario === "sparse archive over budget"; const root = await fs.mkdtemp(path.join(os.tmpdir(), "daytona-capacity-")); @@ -63,28 +69,73 @@ it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "pr await fs.symlink(requiredTools[name]!, path.join(bin, name)); } const tar = requiredTools.tar!; + const specialMode = scenario.startsWith("preserve special"); + const modeRegression = scenario.startsWith("preserve modes") || specialMode; if (producerFails) { await fs.writeFile(path.join(bin, "tar"), `#!/bin/sh\n'${tar}' "$@"\nstatus=$?\ncase " $* " in *" -f - "*) exit 23 ;; esac\nexit "$status"\n`, { mode: 0o755 }); + } else if (modeRegression) { + await fs.writeFile(path.join(bin, "tar"), `#!/bin/sh\nexec '${tar}' --owner=12345 --group=12345 "$@"\n`, { mode: 0o755 }); } else { await fs.symlink(tar, path.join(bin, "tar")); } + if (specialMode) { + await fs.mkdir(output); + await fs.writeFile(path.join(output, "retained.txt"), "existing host work"); + } if (producerFails || sparseOverBudget) { await fs.mkdir(output); await fs.writeFile(path.join(output, "result.bin"), "newer host work"); } const env = { ...process.env, PATH: `${bin}:${process.env.PATH}` }; + if (scenario === "preserve entries GNU host archive") vi.stubEnv("PATH", env.PATH); + if (modeRegression) { + const hostBin = path.join(root, "host-bin"); + await fs.mkdir(hostBin); + const hostTar = scenario.endsWith("GNU host") ? tar : executable(["tar"], "bsdtar") ?? tar; + await fs.writeFile(path.join(hostBin, "tar"), `#!/bin/sh\numask 077\nexec '${hostTar}' "$@"\n`, { mode: 0o755 }); + vi.stubEnv("PATH", `${hostBin}:${process.env.PATH}`); + } const bytes = Buffer.alloc(5 * 1024 * 1024, sparseOverBudget ? 0 : "saved work\n"); await fs.writeFile(path.join(remote, "result.bin"), sparseOverBudget ? "" : bytes, { mode: 0o600 }); if (sparseOverBudget) await fs.truncate(path.join(remote, "result.bin"), bytes.length); - await fs.symlink("/etc/passwd", path.join(remote, "unsafe-link")); + if (unsafeLink) await fs.symlink("/etc/passwd", path.join(remote, "unsafe-link")); await fs.symlink("result.bin", path.join(remote, "safe-link")); + if (preserveSourceEntries) { + await fs.writeFile(path.join(remote, "excluded.txt"), "excluded by caller"); + await fs.mkdir(path.join(remote, "nested")); + if (modeRegression) await fs.chmod(path.join(remote, "nested"), 0o775); + for (const name of ["._result.txt", ".paperclip-upload-result", "nested/._result.txt", "nested/.paperclip-upload-result"]) { + await fs.writeFile(path.join(remote, name), `user work: ${name}`); + if (modeRegression) await fs.chmod(path.join(remote, name), 0o664); + } + } + if (literal) { + const socket = createServer(); + await new Promise((resolve, reject) => { socket.once("error", reject); socket.listen(path.join(remote, "ipc"), resolve); }); + // Closing unlinks the socket, so retain it throughout the actual export. + const close = () => new Promise(resolve => socket.close(() => resolve())); + literalSocketCleanup.push(close); + for (const name of ["nested/ipc", "nested/omit[ab]*?", "omitaZZx", "nested/skip[ab]*?/saved.txt"]) { + await fs.mkdir(path.dirname(path.join(remote, name)), { recursive: true }); + await fs.writeFile(path.join(remote, name), `literal work: ${name}`); + } + await fs.writeFile(path.join(remote, "omit[ab]*?"), "excluded literal file"); + await fs.mkdir(path.join(remote, "skip[ab]*?")); + await fs.writeFile(path.join(remote, "skip[ab]*?/excluded.txt"), "excluded subtree"); + } + if (specialMode) { + const specialPath = path.join(remote, scenario.includes("directory") ? "nested" : "._result.txt"); + const specialBits = scenario.includes("directory") ? 0o1775 : 0o1664; + await fs.chmod(specialPath, specialBits); + expect((await fs.stat(specialPath)).mode & 0o7777).toBe(specialBits); + } const failures: string[] = []; const sandbox = { // A miniature provider-reported quota makes the sparse-file case bounded. disk: sparseOverBudget ? 1 / 4096 : 1, process: { executeCommand: async (command: string) => { - const result = spawnSync("bash", ["-c", 'ulimit -f 128; exec sh -c "$1"', "bash", command], { env, encoding: "utf8" }); + const result = spawnSync("bash", ["-c", directArchive ? 'exec sh -c "$1"' : 'ulimit -f 128; exec sh -c "$1"', "bash", command], { env, encoding: "utf8" }); if (result.status !== 0) failures.push(result.stderr); return { exitCode: result.status ?? 1, result: result.stdout + result.stderr }; }, @@ -111,9 +162,18 @@ it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "pr deleteFile: (file: string) => fs.rm(file, { force: true }), }, }; - const sync = performSyncOut({ sandbox: sandbox as never, remoteDir: remote, timeoutSeconds: 30, - operations: [{ operationId: "export", files: [{ sourcePath: remote, targetPath: output, kind: "directory" }] }], + const sync = performSyncOut({ sandbox: sandbox as never, remoteDir: directArchive ? root : remote, timeoutSeconds: 30, + operations: [{ operationId: "export", files: [{ sourcePath: remote, targetPath: output, kind: "directory", preserveSourceEntries, exclude: literal ? ["ipc", "omit[ab]*?", "skip[ab]*?"] : preserveSourceEntries && !directArchive ? ["excluded.txt"] : undefined }] }], }); + if (specialMode) { + await expect(sync).rejects.toThrow("special mode bits"); + expect(await fs.readdir(output)).toEqual(["retained.txt"]); + expect(await fs.readFile(path.join(output, "retained.txt"), "utf8")).toBe("existing host work"); + expect((await fs.stat(path.join(remote, scenario.includes("directory") ? "nested" : "._result.txt"))).mode & 0o7777) + .toBe(scenario.includes("directory") ? 0o1775 : 0o1664); + expect((await fs.readFile(path.join(remote, "result.bin"))).equals(bytes)).toBe(true); + return; + } if (producerFails || sparseOverBudget) { await expect(sync).rejects.toThrow(sparseOverBudget ? "sandbox disk budget" : "without a successful archive receipt"); expect(await fs.readFile(path.join(output, "result.bin"), "utf8")).toBe("newer host work"); @@ -121,15 +181,39 @@ it.skipIf(Object.values(requiredTools).some(tool => !tool)).each(["success", "pr expect((await fs.readFile(path.join(remote, "result.bin"))).equals(bytes)).toBe(true); return; } - await sync; - expect(failures.join("\n")).toMatch(/File\s?size limit exceeded/i); + const result = await sync; + if (literal) expect(result.operations[0].filesTransferred).toBe(11); + if (!directArchive) expect(failures.join("\n")).toMatch(/File\s?size limit exceeded/i); expect((await fs.readFile(path.join(output, "result.bin"))).equals(bytes)).toBe(true); expect((await fs.readFile(path.join(remote, "result.bin"))).equals(bytes)).toBe(true); expect((await fs.stat(path.join(output, "result.bin"))).mode & 0o777).toBe(0o600); expect(await fs.readlink(path.join(output, "safe-link"))).toBe("result.bin"); expect(await fs.readdir(output)).not.toContain("unsafe-link"); - expect(await fs.readlink(path.join(remote, "unsafe-link"))).toBe("/etc/passwd"); - expect((await fs.readdir(remote)).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]); + if (unsafeLink) expect(await fs.readlink(path.join(remote, "unsafe-link"))).toBe("/etc/passwd"); + if (preserveSourceEntries) { + if (directArchive || literal) expect(await fs.readFile(path.join(output, "excluded.txt"), "utf8")).toBe("excluded by caller"); + else await expect(fs.stat(path.join(output, "excluded.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readFile(path.join(remote, "excluded.txt"), "utf8")).toBe("excluded by caller"); + for (const name of ["._result.txt", ".paperclip-upload-result", "nested/._result.txt", "nested/.paperclip-upload-result"]) { + expect(await fs.readFile(path.join(output, name), "utf8")).toBe(`user work: ${name}`); + expect(await fs.readFile(path.join(remote, name), "utf8")).toBe(`user work: ${name}`); + if (modeRegression) { + const restored = await fs.stat(path.join(output, name)); + expect(restored.mode & 0o777).toBe(0o664); + expect(restored.uid).toBe(process.getuid!()); + } + } + } + if (literal) { + for (const name of ["nested/ipc", "nested/omit[ab]*?", "omitaZZx", "nested/skip[ab]*?/saved.txt"]) { + expect(await fs.readFile(path.join(output, name), "utf8")).toBe(`literal work: ${name}`); + expect(await fs.readFile(path.join(remote, name), "utf8")).toBe(`literal work: ${name}`); + } + for (const name of ["ipc", "omit[ab]*?", "skip[ab]*?"]) await expect(fs.lstat(path.join(output, name))).rejects.toMatchObject({ code: "ENOENT" }); + expect((await fs.lstat(path.join(remote, "ipc"))).isSocket()).toBe(true); + } + if (modeRegression) expect((await fs.stat(path.join(output, "nested"))).mode & 0o777).toBe(0o775); + expect((await fs.readdir(remote)).filter(name => name.startsWith(".paperclip-upload"))).toEqual(preserveSourceEntries ? [".paperclip-upload-result"] : []); }, 40_000); it.skipIf(Object.values(requiredTools).some(tool => !tool)).each([ diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts index 90108c3a37f..fa1273c1a4b 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -254,12 +254,17 @@ export function splitLinkEntryOnce(field: string, delimiter: string): { name: st */ class UnsafeOutboundArchiveError extends Error {} -function assertTarListingLineConfined(line: string): void { +function assertTarListingLineConfined(line: string, rejectSpecialModes = false): void { if (line.trim().length === 0) return; const parsed = parseTarVerboseListingLine(line); if (!parsed) { throw new UnsafeOutboundArchiveError(`Daytona syncOut refusing tarball with an unparseable entry listing: ${line}`); } + // Source preservation promises ordinary 0777 permissions, not host special + // execution bits. Refuse before extraction; this must not trigger link repair. + if (rejectSpecialModes && /[sStT]/.test(line.trimStart().slice(0, 10))) { + throw new Error("Daytona syncOut source preservation does not support special mode bits"); + } const typeFlag = parsed.typeFlag; let name = parsed.rest; let linkTarget: string | null = null; @@ -302,12 +307,17 @@ const TAR_LISTING_TIMEOUT_MS = 120_000; export async function assertTarballEntriesConfined( archivePath: string, timeoutMs = TAR_LISTING_TIMEOUT_MS, + rejectSpecialModes = false, ): Promise { // A valid large workspace can exceed execFile's buffer. Stream within both // aggregate admission quotas and per-entry/diagnostic memory bounds, checking // every entry before extraction. Keep bytes until a full line to preserve // UTF-8 characters split across pipe chunks. - const child = spawn("tar", ["-tvf", archivePath], { + // AppleDouble interpretation must also be disabled while validating, or + // libarchive hides `._*` members that the preserving extractor will write. + const rawMacEntries = rejectSpecialModes && process.platform === "darwin" + && (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar"); + const child = spawn("tar", ["-tvf", archivePath, ...(rawMacEntries ? ["--options=!mac-ext"] : [])], { env: { ...process.env, COPYFILE_DISABLE: "1" }, stdio: ["ignore", "pipe", "pipe"], }); @@ -322,7 +332,7 @@ export async function assertTarballEntriesConfined( if (++entries > TAR_LISTING_MAX_ENTRIES) { throw recordEnvironmentSyncError(new Error("Daytona syncOut tar listing entry limit exceeded (250000)"), { transferFailureKind: "listing_entry_limit" }); } - try { assertTarListingLineConfined(line.toString("utf8")); } + try { assertTarListingLineConfined(line.toString("utf8"), rejectSpecialModes); } catch (error) { if (error instanceof UnsafeOutboundArchiveError) recordEnvironmentSyncError(error, { transferFailureKind: "unsafe_archive" }); throw error; @@ -386,13 +396,22 @@ export async function assertTarballEntriesConfined( } } -async function extractHostTarball(input: { archivePath: string; localDir: string }): Promise { +async function extractHostTarball(input: { archivePath: string; localDir: string; preserveSourceEntries?: boolean }): Promise { // The archive is sandbox-authored and untrusted: validate every member (and // link target) is confined before letting host-side tar write a single byte. - await withEnvironmentSyncTransferStep("archive_validate", () => assertTarballEntriesConfined(input.archivePath)); + await withEnvironmentSyncTransferStep("archive_validate", () => assertTarballEntriesConfined(input.archivePath, TAR_LISTING_TIMEOUT_MS, input.preserveSourceEntries)); await withEnvironmentSyncTransferStep("archive_extract", async () => { await fs.mkdir(input.localDir, { recursive: true }); - await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir], { + // macOS tar otherwise consumes source files named `._*` as AppleDouble + // metadata instead of writing their bytes. These are ordinary user entries. + // Preserve mode also ignores the host umask without adopting archive owners, + // ACLs, attributes or file flags from the sandbox. + const hostTarIsBsd = input.preserveSourceEntries + && (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar"); + await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir, + ...(input.preserveSourceEntries ? ["-p", "--no-same-owner", "--no-acls", "--no-xattrs"] : []), + ...(hostTarIsBsd ? ["--no-fflags"] : []), + ...(hostTarIsBsd && process.platform === "darwin" ? ["--no-mac-metadata", "--options=!mac-ext"] : [])], { env: { ...process.env, COPYFILE_DISABLE: "1" }, maxBuffer: 32 * 1024 * 1024, }); @@ -1346,11 +1365,20 @@ async function syncOutDirectoryMapping(input: { return withHostTempDir(async (tmp) => { const remoteTar = path.posix.join(remoteDir, scratchName(".tar")); const remoteList = path.posix.join(remoteDir, scratchName(".list")); - const excludes = ["._*", `${SCRATCH_PREFIX}*`, ...(mapping.exclude ?? [])]; - const excludeFlags = excludes - .flatMap((entry) => [entry, `${entry.replace(/\/$/, "")}/*`]) - .map((entry) => `--exclude ${shellQuote(entry)}`) - .join(" "); + const scratchPaths = [remoteTar, remoteList].flatMap(file => { + const relative = path.posix.relative(mapping.sourcePath, file); + return relative && !relative.startsWith("../") && !path.posix.isAbsolute(relative) + ? [relative, `./${relative}`] : []; + }); + const excludes = [...(mapping.preserveSourceEntries ? scratchPaths : ["._*", `${SCRATCH_PREFIX}*`]), ...(mapping.exclude ?? [])]; + // Preservation exclusions are literal paths relative to this source, not + // basename patterns. Both archive naming forms occur in the repair path. + const literalExcludes = [...new Set(excludes.map(entry => entry.replace(/^(?:\.\/)+/, "").replace(/\/+$/, "")))]; + const excludeFlags = (mapping.preserveSourceEntries ? "--anchored --no-wildcards " : "") + + (mapping.preserveSourceEntries + ? literalExcludes.flatMap(entry => [entry, `./${entry}`]) + : excludes.flatMap(entry => [entry, `${entry.replace(/\/$/, "")}/*`])) + .map(entry => `--exclude ${shellQuote(entry)}`).join(" "); // Tar the source in-sandbox (naming top-level entries so no "." self-entry is // embedded), reproducing the `followSymlinks` → `-h` mapping, then stream the // single archive back over the native bulk channel. @@ -1366,9 +1394,13 @@ async function syncOutDirectoryMapping(input: { // resolved targets remain inside this mapping. Nothing is deleted or // dereferenced. Host validation still checks the rebuilt archive, including // links changed by the sandbox between enumeration and tar creation. - const prunePaths = excludes.flatMap((entry) => [ - `-path ${shellQuote(`./${entry}`)}`, `-path ${shellQuote(`*/${entry}`)}`, - ]).join(" -o "); + // find -path also interprets glob characters, even inside shell quotes. + // Pruning a literal directory already removes its complete subtree. + const prunePaths = (mapping.preserveSourceEntries + ? literalExcludes.map(entry => `-path ${shellQuote(`./${entry.replace(/[\\*?[\]]/g, "\\$&")}`)}`) + : excludes.flatMap(entry => [ + `-path ${shellQuote(`./${entry}`)}`, `-path ${shellQuote(`*/${entry}`)}`, + ])).join(" -o "); const filterLinks = [ ...canonicalizerPreamble(shellQuote(mapping.sourcePath)), 'for _pc_link do', @@ -1381,7 +1413,7 @@ async function syncOutDirectoryMapping(input: { ].join("\n"); const confinedEntriesScript = (archive: string | null) => [ `cd ${shellQuote(mapping.sourcePath)}`, - `find . -mindepth 1 \\( ${prunePaths} \\) -prune -o -type l -exec sh -c ${shellQuote(filterLinks)} sh {} + -o \\( -type f -o -type d \\) -print0 ${archive ? `> ${shellQuote(remoteList)} && ` : "| "}` + + `find . -mindepth 1 ${prunePaths ? `\\( ${prunePaths} \\) -prune -o ` : ""}-type l -exec sh -c ${shellQuote(filterLinks)} sh {} + -o \\( -type f -o -type d \\) -print0 ${archive ? `> ${shellQuote(remoteList)} && ` : "| "}` + `tar -c --no-xattrs --hard-dereference --no-recursion --null ${excludeFlags} -f ${shellQuote(archive ?? "-")} -T ${shellQuote(archive ? remoteList : "-")}`, ].join(" && "); @@ -1431,7 +1463,7 @@ async function syncOutDirectoryMapping(input: { } bytesTransferred += await withEnvironmentSyncTransferStep("archive_download", async () => (await fs.stat(localTar)).size); try { - await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath }); + await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath, preserveSourceEntries: mapping.preserveSourceEntries }); break; } catch (error) { if (!(error instanceof UnsafeOutboundArchiveError) || attempt > 0) throw error; @@ -1444,7 +1476,7 @@ async function syncOutDirectoryMapping(input: { // cleaned by withHostTempDir. await Promise.all([remoteTar, remoteList].map((file) => sandbox.fs.deleteFile(file).catch(() => undefined))); } - const filesTransferred = await countHostFiles(mapping.targetPath, mapping.exclude); + const filesTransferred = await countHostFiles(mapping.targetPath, mapping.preserveSourceEntries ? undefined : mapping.exclude); return { filesTransferred, bytesTransferred }; }); } diff --git a/packages/plugins/sandbox-providers/kubernetes/src/file-sync.ts b/packages/plugins/sandbox-providers/kubernetes/src/file-sync.ts index 62109990165..317b5aefd8c 100644 --- a/packages/plugins/sandbox-providers/kubernetes/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/kubernetes/src/file-sync.ts @@ -369,8 +369,11 @@ export function splitLinkEntryOnce(field: string, delimiter: string): { name: st * escapes the tree. Legitimate in-tree relative links are preserved. Parses the * `-tvf` verbose listing; any unparseable line fails closed. */ -async function assertTarballEntriesConfined(archivePath: string): Promise { - const { stdout } = await execFileAsync("tar", ["-tvf", archivePath], { +async function assertTarballEntriesConfined(archivePath: string, rejectSpecialModes = false): Promise { + // Validate the same raw `._*` members the preserving extractor will write. + const rawMacEntries = rejectSpecialModes && process.platform === "darwin" + && (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar"); + const { stdout } = await execFileAsync("tar", ["-tvf", archivePath, ...(rawMacEntries ? ["--options=!mac-ext"] : [])], { env: { ...process.env, COPYFILE_DISABLE: "1" }, maxBuffer: 64 * 1024 * 1024, }); @@ -380,6 +383,11 @@ async function assertTarballEntriesConfined(archivePath: string): Promise if (!parsed) { throw new Error(`Kubernetes syncOut refusing tarball with an unparseable entry listing: ${line}`); } + // Source preservation promises 0777 modes only. Refuse special execution + // bits before the host extractor can apply them, retaining every source file. + if (rejectSpecialModes && /[sStT]/.test(line.trimStart().slice(0, 10))) { + throw new Error("Kubernetes syncOut source preservation does not support special mode bits"); + } const typeFlag = parsed.typeFlag; let name = parsed.rest; let linkTarget: string | null = null; @@ -409,12 +417,21 @@ async function assertTarballEntriesConfined(archivePath: string): Promise } } -async function extractHostTarball(input: { archivePath: string; localDir: string }): Promise { +async function extractHostTarball(input: { archivePath: string; localDir: string; preserveSourceEntries?: boolean }): Promise { // The archive is sandbox-authored and untrusted: validate every member (and // link target) is confined before letting host-side tar write a single byte. - await assertTarballEntriesConfined(input.archivePath); + await assertTarballEntriesConfined(input.archivePath, input.preserveSourceEntries); await fs.mkdir(input.localDir, { recursive: true }); - await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir], { + // macOS tar otherwise consumes source files named `._*` as AppleDouble + // metadata instead of writing their bytes. These are ordinary user entries. + // Preserve mode also ignores the host umask without adopting archive owners, + // ACLs, attributes or file flags from the sandbox. + const hostTarIsBsd = input.preserveSourceEntries + && (await execFileAsync("tar", ["--version"])).stdout.includes("bsdtar"); + await execFileAsync("tar", ["-xf", input.archivePath, "-C", input.localDir, + ...(input.preserveSourceEntries ? ["-p", "--no-same-owner", "--no-acls", "--no-xattrs"] : []), + ...(hostTarIsBsd ? ["--no-fflags"] : []), + ...(hostTarIsBsd && process.platform === "darwin" ? ["--no-mac-metadata", "--options=!mac-ext"] : [])], { env: { ...process.env, COPYFILE_DISABLE: "1" }, maxBuffer: 64 * 1024 * 1024, }); @@ -835,9 +852,13 @@ async function syncOutDirectoryMapping(input: { const { exec, mapping, remoteDir, timeoutMs, maxOutputBytes } = input; assertConfinedSandboxPath(remoteDir, mapping.sourcePath, "source"); return withHostTempDir(async (tmp) => { - const excludeFlags = ["._*", ...(mapping.exclude ?? [])] - .map((entry) => `--exclude ${shQuote(entry)}`) - .join(" "); + // Preservation exclusions name literal source-relative paths. GNU tar's + // default basename/glob matching could silently omit unrelated user files. + const excludes = [...(mapping.preserveSourceEntries ? [] : ["._*"]), ...(mapping.exclude ?? [])]; + const excludeFlags = (mapping.preserveSourceEntries ? "--anchored --no-wildcards " : "") + + (mapping.preserveSourceEntries + ? [...new Set(excludes.map(entry => entry.replace(/^(?:\.\/)+/, "").replace(/\/+$/, "")))].flatMap(entry => [entry, `./${entry}`]) + : excludes).map(entry => `--exclude ${shQuote(entry)}`).join(" "); const script = [ ...canonicalizerPreamble(shQuote(remoteDir)), // Confine the source dir through an open-then-verify pinned fd before taring. @@ -867,8 +888,8 @@ async function syncOutDirectoryMapping(input: { label: "syncOut directory transfer", }); const bytesTransferred = (await fs.stat(localTar)).size; - await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath }); - const filesTransferred = await countHostFiles(mapping.targetPath, mapping.exclude); + await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath, preserveSourceEntries: mapping.preserveSourceEntries }); + const filesTransferred = await countHostFiles(mapping.targetPath, mapping.preserveSourceEntries ? undefined : mapping.exclude); return { filesTransferred, bytesTransferred }; }); } diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/file-sync.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/file-sync.test.ts index 79104a9f6fc..c0bdfa2aeb0 100644 --- a/packages/plugins/sandbox-providers/kubernetes/test/unit/file-sync.test.ts +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/file-sync.test.ts @@ -1,3 +1,4 @@ +import { createServer } from "node:net"; import { afterEach, describe, expect, it } from "vitest"; import { promises as fs } from "node:fs"; import os from "node:os"; @@ -379,6 +380,77 @@ describe("kubernetes onEnvironmentSyncIn (native single-exec transfer)", () => { }); describe("kubernetes onEnvironmentSyncOut (native single-exec transfer)", () => { + // The pod's pinned descriptor protocol depends on Linux /proc/self/fd. + it.skipIf(process.platform !== "linux")("preserves source entry names when requested while retaining explicit exclusions", async () => { + const remoteDir = await makeTmp("k8s-source-entry-"), hostOut = await makeTmp("k8s-source-output-"); + await fs.mkdir(path.join(remoteDir, "nested")); + const names = ["._result.txt", ".paperclip-upload-result", "nested/._result.txt", "nested/.paperclip-upload-result"]; + await fs.chmod(path.join(remoteDir, "nested"), 0o775); + for (const name of names) { + await fs.writeFile(path.join(remoteDir, name), `user work: ${name}`); + await fs.chmod(path.join(remoteDir, name), 0o664); + } + await fs.writeFile(path.join(remoteDir, "excluded.txt"), "excluded by caller"); + const { exec } = makeRealExec(); + await performSyncOut({ exec, remoteDir, timeoutMs: 30_000, operations: [{ operationId: "preserve", files: [{ + kind: "directory", sourcePath: remoteDir, targetPath: hostOut, preserveSourceEntries: true, exclude: ["excluded.txt"], + }] }] }); + for (const name of names) { + expect(await fs.readFile(path.join(hostOut, name), "utf8")).toBe(`user work: ${name}`); + expect(await fs.readFile(path.join(remoteDir, name), "utf8")).toBe(`user work: ${name}`); + expect((await fs.stat(path.join(hostOut, name))).mode & 0o777).toBe(0o664); + } + expect((await fs.stat(path.join(hostOut, "nested"))).mode & 0o777).toBe(0o775); + await expect(fs.stat(path.join(hostOut, "excluded.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it.skipIf(process.platform !== "linux")("preserves literal excluded paths without excluding same basenames or glob matches", async () => { + const remoteDir = await makeTmp("k8s-literal-"), hostOut = await makeTmp("k8s-literal-out-"); + await fs.mkdir(path.join(remoteDir, "nested")); + const socket = createServer(); + await new Promise((resolve, reject) => { socket.once("error", reject); socket.listen(path.join(remoteDir, "ipc"), resolve); }); + try { + const names = ["nested/ipc", "nested/omit[ab]*?", "omitaZZx", "nested/skip[ab]*?/saved.txt"]; + for (const name of names) { + await fs.mkdir(path.dirname(path.join(remoteDir, name)), { recursive: true }); + await fs.writeFile(path.join(remoteDir, name), `literal work: ${name}`); + } + await fs.writeFile(path.join(remoteDir, "omit[ab]*?"), "excluded literal file"); + await fs.mkdir(path.join(remoteDir, "skip[ab]*?")); + await fs.writeFile(path.join(remoteDir, "skip[ab]*?/excluded.txt"), "excluded subtree"); + const { exec } = makeRealExec(); + const result = await performSyncOut({ exec, remoteDir, timeoutMs: 30_000, operations: [{ operationId: "literal", files: [{ + kind: "directory", sourcePath: remoteDir, targetPath: hostOut, preserveSourceEntries: true, exclude: ["ipc", "omit[ab]*?", "skip[ab]*?"], + }] }] }); + expect(result.operations[0].filesTransferred).toBe(4); + for (const name of names) { + expect(await fs.readFile(path.join(hostOut, name), "utf8")).toBe(`literal work: ${name}`); + expect(await fs.readFile(path.join(remoteDir, name), "utf8")).toBe(`literal work: ${name}`); + } + for (const name of ["ipc", "omit[ab]*?", "skip[ab]*?"]) await expect(fs.lstat(path.join(hostOut, name))).rejects.toMatchObject({ code: "ENOENT" }); + expect((await fs.lstat(path.join(remoteDir, "ipc"))).isSocket()).toBe(true); + } finally { await new Promise(resolve => socket.close(() => resolve())); } + }); + + it.skipIf(process.platform !== "linux").each(["file", "directory"])("refuses preserved %s special modes before host writes", async kind => { + const remoteDir = await makeTmp("k8s-special-source-"), hostOut = await makeTmp("k8s-special-output-"); + await fs.mkdir(path.join(remoteDir, "nested")); + await fs.writeFile(path.join(remoteDir, "result.txt"), "source work"); + await fs.writeFile(path.join(hostOut, "retained.txt"), "existing host work"); + const source = path.join(remoteDir, kind === "directory" ? "nested" : "result.txt"); + const mode = kind === "directory" ? 0o1775 : 0o1664; + await fs.chmod(source, mode); + expect((await fs.stat(source)).mode & 0o7777).toBe(mode); + const { exec } = makeRealExec(); + await expect(performSyncOut({ exec, remoteDir, timeoutMs: 30_000, operations: [{ operationId: "preserve", files: [{ + kind: "directory", sourcePath: remoteDir, targetPath: hostOut, preserveSourceEntries: true, + }] }] })).rejects.toThrow("special mode bits"); + expect(await fs.readdir(hostOut)).toEqual(["retained.txt"]); + expect(await fs.readFile(path.join(hostOut, "retained.txt"), "utf8")).toBe("existing host work"); + expect((await fs.stat(source)).mode & 0o7777).toBe(mode); + expect(await fs.readFile(path.join(remoteDir, "result.txt"), "utf8")).toBe("source work"); + }); + it("streams all file mappings back over a SINGLE exec and reassembles them at host targets, preserving mode and per-operation counts", async () => { const remoteDir = await makeTmp("k8s-sandbox-"); const hostOut = await makeTmp("k8s-hostout-"); diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.ts index 3d70c65bd3d..0ad0814b2fd 100644 --- a/packages/plugins/sdk/src/protocol.ts +++ b/packages/plugins/sdk/src/protocol.ts @@ -810,13 +810,21 @@ export interface PluginSyncFileMapping { * shows the bytes only to code that already runs in that sandbox. */ mode?: number; - /** Glob patterns to exclude when `kind` is `"directory"`. */ + /** Directory glob patterns; literal root-relative paths when preserving source entries. */ exclude?: string[]; /** * Symlink handling for `kind: "directory"` transfers. Falsy preserves symlinks * as links; `true` dereferences them to their target bytes. Mirrors tar's `-h`. */ followSymlinks?: boolean; + /** + * Outbound directory transfers preserve source entry names, including `._*` + * and provider scratch-like names. In this mode `exclude` contains literal + * root-relative paths; providers omit only those paths and scratch paths + * created by this exact transfer. Archive member and symlink validation stay + * unchanged. Ignored for syncIn/files. + */ + preserveSourceEntries?: boolean; /** * Advisory read-write intent for the sandbox target. `"rw"` means the author * expects the agent to change the bytes at the target and keep the change. diff --git a/server/src/services/remote-workspace-readonly-inventory.ts b/server/src/services/remote-workspace-readonly-inventory.ts new file mode 100644 index 00000000000..2969e601c02 --- /dev/null +++ b/server/src/services/remote-workspace-readonly-inventory.ts @@ -0,0 +1,135 @@ +/** Runs unchanged in the sandbox and on its transferred, private controller copy. */ +export async function readWorkspaceInventory(root: string, controllerCopy = false, signal?: AbortSignal) { + const fs = await import("node:fs/promises"), path = await import("node:path"), crypto = await import("node:crypto"); + const fail = (): never => { throw new Error("remote_workspace_readonly_capture_unsafe"); }; + const budget = (): never => { throw new Error("remote_workspace_readonly_capture_budget"); }; + const check = () => signal?.throwIfAborted(); + check(); + if (!path.isAbsolute(root) || await fs.realpath(root) !== root || !(await fs.lstat(root)).isDirectory()) fail(); + const digest = crypto.createHash("sha256"), exclude = [".paperclip-runtime"]; + const repositories: string[] = [], configs: string[] = []; + let entries = 0, scannedEntries = 0, byteSize = 0, configBytes = 0; + const gitRead = async (cwd: string, args: string[]) => { + check(); + const { execFile } = await import("node:child_process"); + return new Promise((resolve, reject) => execFile("git", ["-c", "core.fsmonitor=false", "-c", "core.hooksPath=/dev/null", "-C", cwd, ...args], { + signal, timeout: 10_000, maxBuffer: 64 * 1024 * 1024, + env: { PATH: process.env.PATH, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: "/dev/null", GIT_CONFIG_COUNT: "0", + GIT_OPTIONAL_LOCKS: "0", GIT_NO_LAZY_FETCH: "1" }, + }, (error, result) => error ? reject(new Error("remote_workspace_readonly_capture_unsafe")) : resolve(result))); + }; + const visit = async (relative: string, parentRepository: string | null = null) => { + check(); + const directory = path.join(root, relative), before = await fs.lstat(directory); + if (!before.isDirectory() || before.isSymbolicLink() || await fs.realpath(directory) !== directory) fail(); + const names: string[] = []; + for await (const entry of await fs.opendir(directory)) { + check(); if (++scannedEntries > 100_000) budget(); names.push(entry.name); + } + const inGit = relative.split("/").includes(".git"); + const repository = !inGit && names.includes(".git") ? directory : parentRepository; + if (!inGit && names.includes(".git")) { + const entry = await fs.lstat(path.join(directory, ".git")); + if (!entry.isDirectory() || entry.isSymbolicLink()) fail(); + repositories.push(directory); if (repositories.length > 64) budget(); + // The sandbox uses the same tracked-dependency rule as normal capture. + // Controller verification never invokes repository-aware Git here. + if (!controllerCopy && await fs.realpath((await gitRead(directory, ["rev-parse", "--show-toplevel"])).trim()) !== directory) fail(); + } + for (const name of names.sort()) { + check(); + const next = relative ? `${relative}/${name}` : name; + if (next === ".paperclip-runtime") continue; + if (!name || name === "." || name === ".." || /[\\\u0000-\u001f\u007f]/u.test(name)) fail(); + const file = path.join(root, next), stat = await fs.lstat(file); + // A copied worktree pointer or alternate could resolve against the + // controller's filesystem. Only standalone, self-contained Git metadata + // can use this fallback; the sandbox remains retained otherwise. + const parts = next.split("/"), git = parts.indexOf(".git"); + if (name === ".git" && !stat.isDirectory()) fail(); + // Git follows configuration and administrative links itself. A link + // inside this tree must not bypass the controller config-file check. + if (git >= 0 && stat.isSymbolicLink()) fail(); + const administrationPath = parts.slice(git + 1).join("/"); + const foldedAdministrationPath = administrationPath.toLowerCase(); + // A case-sensitive sandbox may be copied onto a case-insensitive host. + // Reject aliases before Git can interpret them as controller metadata. + if (git >= 0 && ((foldedAdministrationPath === "config" && administrationPath !== "config") + || ["commondir", "config.worktree", "objects/info/alternates", "objects/info/http-alternates"] + .includes(foldedAdministrationPath))) fail(); + if (stat.isSocket()) { exclude.push(next); continue; } + if (stat.isDirectory()) { + if (!controllerCopy && !inGit && name !== ".git") { + const marker = await fs.lstat(name === "node_modules" ? path.join(directory, "package.json") + : path.join(file, "pyvenv.cfg")).catch(() => null); + if (marker?.isFile() && !marker.isSymbolicLink() + && (!repository || !(await gitRead(directory, ["ls-files", "-z", "--", name])))) { + exclude.push(next); continue; + } + } + digest.update(JSON.stringify([next, "directory", stat.mode & 0o777])); entries++; + await visit(next, repository); + } else if (stat.isSymbolicLink()) { + const target = await fs.readlink(file), resolved = path.posix.normalize(path.posix.join(path.posix.dirname(next), target)); + if (!target || path.posix.isAbsolute(target) || /[\\\u0000-\u001f\u007f]/u.test(target) + || resolved === ".." || resolved.startsWith("../")) fail(); + digest.update(JSON.stringify([next, "symlink", target])); entries++; + } else if (stat.isFile()) { + if (git >= 0 && parts.slice(git + 1).join("/") === "config") { + configs.push(file); configBytes += stat.size; + if (stat.size > 65_536 || configBytes > 1024 * 1024) budget(); + } + const original = await fs.lstat(file, { bigint: true }); + const handle = await fs.open(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | fs.constants.O_NONBLOCK); + const hash = crypto.createHash("sha256"); let size = 0; + try { + const before = await handle.stat({ bigint: true }); + const fields = ["dev", "ino", "size", "mode", "mtimeNs", "ctimeNs"] as const; + if (!before.isFile() || !original.isFile() || Number(before.dev) !== stat.dev || Number(before.ino) !== stat.ino + || Number(before.size) !== stat.size || Number(before.mode) !== stat.mode + || fields.some(key => original[key] !== before[key]) || await fs.realpath(file) !== file) fail(); + for await (const bytes of handle.createReadStream({ autoClose: false, signal })) { check(); size += bytes.length; hash.update(bytes); } + const after = await handle.stat({ bigint: true }), current = await fs.lstat(file, { bigint: true }); + if (!current.isFile() || size !== Number(before.size) || await fs.realpath(file) !== file + || fields.some(key => before[key] !== after[key] || after[key] !== current[key])) fail(); + } finally { await handle.close(); } + byteSize += size; + if (!Number.isSafeInteger(byteSize)) fail(); + digest.update(JSON.stringify([next, "file", stat.mode & 0o777, size, hash.digest("hex")])); entries++; + } else fail(); + } + const after = await fs.lstat(directory); + if (before.dev !== after.dev || before.ino !== after.ino || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) fail(); + }; + await visit(""); + if (Buffer.byteLength(JSON.stringify(exclude)) > 65_536) fail(); + if (controllerCopy) { + // Bound the complete tree and verify every repository layout before the + // first controller subprocess. Files named config inside Git metadata are + // ordinary data unless they are the exact repository-root configuration. + for (const repository of repositories) { + check(); + const directory = path.join(repository, ".git"), head = path.join(directory, "HEAD"); + if ((await fs.stat(head)).size > 4096 || !/^(ref: refs\/\S+|[0-9a-f]{40}|[0-9a-f]{64})\s*$/u.test(await fs.readFile(head, "utf8"))) fail(); + if (!(await fs.lstat(path.join(directory, "objects"))).isDirectory() + || !(await fs.lstat(path.join(directory, "refs")).catch(() => fs.lstat(path.join(directory, "reftable")))).isDirectory()) fail(); + } + let settings = 0, refspecs = 0; + for (const config of configs) { + check(); + // Parse this file as data. Never discover a repository or follow includes. + const keys = (await gitRead(path.dirname(root), ["config", "--file", config, "--no-includes", "--null", "--name-only", "--list"])) + .split("\0").filter(Boolean); + settings += keys.length; refspecs += keys.filter(key => /^remote\..+\.fetch$|^branch\..+\.merge$/u.test(key)).length; + if (settings > 1024 || refspecs > 256) budget(); + const portableKey = /^(?:core\.(?:repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode|symlinks|sparsecheckout|sparsecheckoutcone)|index\.sparse|extensions\.(?:objectformat|refstorage|partialclone)|remote\..+\.(?:url|pushurl|fetch|promisor|partialclonefilter)|branch\..+\.(?:remote|merge|rebase)|user\.(?:name|email))$/u; + if (!keys.every(key => portableKey.test(key))) fail(); + } + } + return { sha256: digest.digest("hex"), byteSize, entries, exclude }; +} + +export function readonlyWorkspaceInventoryScript() { + const source = readWorkspaceInventory.toString().replaceAll("__vite_ssr_dynamic_import__", "import"); + return `const __name=value=>value;const inspect=${source};inspect(process.argv[1]).then(value=>process.stdout.write(JSON.stringify(value))).catch(()=>{process.stderr.write('remote_workspace_readonly_capture_unsafe');process.exitCode=1});`; +} diff --git a/server/src/services/remote-workspace-snapshot-archive.ts b/server/src/services/remote-workspace-snapshot-archive.ts index eeb2f0cbc0b..d4928e8bb0a 100644 --- a/server/src/services/remote-workspace-snapshot-archive.ts +++ b/server/src/services/remote-workspace-snapshot-archive.ts @@ -306,6 +306,10 @@ async function workspaceArchive(input: { }; const directories: string[] = []; const excludedPaths = new Set(); + for (const relative of input.exclude ?? []) { + if (!safeRelative(relative)) fail("unsupported_filename"); + excludedPaths.add(relative); + } const excluded = (next: string) => input.exclude?.some(item => next === item || next.startsWith(`${item}/`)) || next === ".paperclip-runtime" || next.startsWith(".paperclip-runtime/"); const walk = async (root: string, skipGit = false) => { diff --git a/server/src/services/remote-workspace-transport.test.ts b/server/src/services/remote-workspace-transport.test.ts index 6e5b093a354..cdab59635ca 100644 --- a/server/src/services/remote-workspace-transport.test.ts +++ b/server/src/services/remote-workspace-transport.test.ts @@ -6,6 +6,7 @@ import path from "node:path"; import { pathToFileURL } from "node:url"; import { promisify } from "node:util"; import { deflateSync } from "node:zlib"; +import net from "node:net"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; import { @@ -18,6 +19,8 @@ import { type RemoteWorkspaceTarget, } from "./remote-workspace-transport.js"; import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspace-failure.js"; +import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js"; +import { readWorkspaceInventory, readonlyWorkspaceInventoryScript } from "./remote-workspace-readonly-inventory.js"; const exec = promisify(execFile); @@ -468,6 +471,58 @@ describe("remote workspace transport safety with real sandbox commands", () => { .rejects.toThrow(/^remote_workspace_operation_timed_out: archive_helper \(configured timeout 300000ms\)$/); }); + it.each(["helper", "archive"])("saves a full sandbox through read-only transfer when the %s cannot write", async failureStage => { + await git(workspace, "init", "--initial-branch=main"); + await fs.writeFile(path.join(workspace, "result.txt"), "committed result\n"); + await git(workspace, "add", "."); await git(workspace, "commit", "-m", "Saved work"); + const head = (await git(workspace, "rev-parse", "HEAD")).stdout; + await fs.writeFile(path.join(workspace, "result.txt"), "staged result\n"); + await git(workspace, "add", "result.txt"); + await fs.writeFile(path.join(workspace, "notes.txt"), "untracked result\n"); + await fs.symlink("notes.txt", path.join(workspace, "notes-link")); + const preload = path.join(scratch, "full-sandbox.cjs"); + const failing = failureStage === "helper" ? String.raw`/archive\.cjs\.[^/]+\.tmp$/` : String.raw`/workspace\.pcws$/`; + await fs.writeFile(preload, String.raw`const fs=require('node:fs/promises'),failing=${failing}; + const fail=()=>{throw Object.assign(new Error('No space left on device'),{code:'ENOSPC'});}; + const writeFile=fs.writeFile,open=fs.open;fs.writeFile=async(file,...args)=>{ + if(failing.test(String(file)))fail();return writeFile(file,...args);}; + fs.open=async(file,...args)=>{const handle=await open(file,...args);if(failing.test(String(file)))handle.writeFile=async()=>fail();return handle;}; + require('node:module').syncBuiltinESMExports();`); + const { remote, execute } = target(); + const run = execute.getMockImplementation()!; + execute.mockImplementation(input => run({ ...input, env: { ...input.env, NODE_OPTIONS: `--require ${preload}` } })); + const transfer = remote.runner!.syncOut!; + remote.runner!.syncOut = async (operations, options) => { + for (const operation of operations) for (const file of operation.files) { + if (file.kind === "directory") { + expect(file.followSymlinks).toBe(false); + expect(file.exclude).toContain(".paperclip-runtime"); + await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true, + filter: source => !file.exclude!.some(excluded => { + const relative = path.relative(file.sourcePath, source).split(path.sep).join("/"); + return relative === excluded || relative.startsWith(`${excluded}/`); + }) }); + } else return transfer(operations, options); + } + return { operations: operations.map(operation => ({ operationId: operation.operationId, + filesTransferred: operation.files.length, bytesTransferred: 0 })) }; + }; + const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }); + try { + expect(await fs.readFile(path.join(workspace, "result.txt"), "utf8")).toBe("staged result\n"); + expect(await fs.readFile(path.join(workspace, "notes.txt"), "utf8")).toBe("untracked result\n"); + await fs.rm(workspace, { recursive: true }); + const restored = path.join(scratch, "restored"); + await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, + expectedManifestDigest: saved.manifestDigest }); + expect((await git(restored, "rev-parse", "HEAD")).stdout).toBe(head); + expect((await git(restored, "show", ":result.txt")).stdout).toBe("staged result\n"); + expect(await fs.readFile(path.join(restored, "notes.txt"), "utf8")).toBe("untracked result\n"); + expect(await fs.readlink(path.join(restored, "notes-link"))).toBe("notes.txt"); + await git(restored, "fsck", "--full"); + } finally { await saved.cleanup(); } + }); + it.each([ { step: "helper install", failing: String.raw`/archive\.cjs\.[^/]+\.tmp$/`, outputMode: "split" as const }, { step: "archive write", failing: String.raw`/workspace\.pcws$/`, outputMode: "split" as const }, @@ -489,6 +544,7 @@ describe("remote workspace transport safety with real sandbox commands", () => { const { remote, execute } = target({}, outputMode); const run = execute.getMockImplementation()!; execute.mockImplementation(input => run({ ...input, env: { ...input.env, NODE_OPTIONS: `--require ${preload}` } })); + remote.runner!.syncOut = async () => { throw new Error("synthetic transfer unavailable"); }; const operationId = randomUUID(); await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId })) .rejects.toThrow(/^remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace$/); @@ -498,6 +554,260 @@ describe("remote workspace transport safety with real sandbox commands", () => { expect(await fs.readFile(path.join(workspace, "result.txt"), "utf8")).toBe("synthetic saved result\n"); }); + it("preserves work beside a socket and records its exclusion during full-disk recovery", async () => { + await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n"); + const socket = path.join(workspace, "ipc"), server = net.createServer(); + await new Promise((resolve, reject) => { server.once("error", reject); server.listen(socket, resolve); }); + let saved: Awaited> | undefined; + try { + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + remote.runner!.syncOut = async operations => { + const mapping = operations[0]!.files[0]!; + await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true, + filter: file => !mapping.exclude!.includes(path.relative(mapping.sourcePath, file)) }); + return { operations: [] }; + }; + saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }); + const bytes = await fs.readFile(saved.archivePath), length = bytes.readUInt32BE(6); + expect(JSON.parse(bytes.subarray(10, 10 + length).toString()).excluded).toContain("ipc"); + const restored = path.join(scratch, "restored"); + await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest }); + expect(await fs.readdir(restored)).toEqual(["saved.txt"]); + expect((await fs.lstat(socket)).isSocket()).toBe(true); + } finally { + await saved?.cleanup(); + await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + } + }); + + it.each(["source changed", "copy changed", "extra file"])("rejects a read-only fallback when %s", async fault => { + await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n"); + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + let copiedRoot: string | undefined; + remote.runner!.syncOut = async operations => { + const file = operations[0]!.files[0]!; copiedRoot = path.dirname(file.targetPath); + await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true }); + if (fault === "source changed") await fs.writeFile(path.join(workspace, "saved.txt"), "newer work\n"); + else await fs.writeFile(path.join(file.targetPath, fault === "extra file" ? "extra.txt" : "saved.txt"), "wrong copy\n"); + return { operations: [] }; + }; + await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() })) + .rejects.toMatchObject({ message: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + cause: { message: "remote_workspace_readonly_capture_failed: stage=verify reason=source_changed transport=provider_error" } }); + expect(await fs.readFile(path.join(workspace, "saved.txt"), "utf8")).toBe(fault === "source changed" ? "newer work\n" : "retained work\n"); + expect(copiedRoot).toBeDefined(); + await expect(fs.access(copiedRoot!)).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it.each(["python", "node"])("recovers a full sandbox without copying an untracked %s environment", async kind => { + await git(workspace, "init", "--initial-branch=main"); + await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n"); + await git(workspace, "add", "saved.txt"); await git(workspace, "commit", "-m", "Saved work"); + const dependency = kind === "python" ? ".venv" : "node_modules"; + await fs.mkdir(path.join(workspace, dependency, "bin"), { recursive: true }); + await fs.writeFile(path.join(workspace, kind === "python" ? ".venv/pyvenv.cfg" : "package.json"), kind === "python" ? "home = /usr/bin\n" : "{}\n"); + await fs.symlink(process.execPath, path.join(workspace, dependency, "bin", "runtime")); + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + remote.runner!.syncOut = async operations => { + const mapping = operations[0]!.files[0]!; + expect(mapping.exclude).toContain(dependency); + await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true, + filter: file => !(mapping.exclude ?? []).some(excluded => { + const relative = path.relative(mapping.sourcePath, file).split(path.sep).join("/"); + return relative === excluded || relative.startsWith(`${excluded}/`); + }) }); + return { operations: [] }; + }; + const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }); + try { + const restored = path.join(scratch, "restored"); + await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest }); + expect(await fs.readFile(path.join(restored, "saved.txt"), "utf8")).toBe("retained work\n"); + await expect(fs.access(path.join(restored, dependency))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readlink(path.join(workspace, dependency, "bin", "runtime"))).toBe(process.execPath); + } finally { await saved.cleanup(); } + }); + + it("never excludes Git history because its metadata contains a virtualenv marker", async () => { + await git(workspace, "init", "--initial-branch=main"); + await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n"); + await git(workspace, "add", "saved.txt"); await git(workspace, "commit", "-m", "Saved work"); + const commit = (await git(workspace, "rev-parse", "HEAD")).stdout; + await fs.writeFile(path.join(workspace, ".git", "pyvenv.cfg"), "home = /usr/bin\n"); + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + remote.runner!.syncOut = async operations => { + const mapping = operations[0]!.files[0]!; + expect(mapping.exclude).not.toContain(".git"); + await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true }); + return { operations: [] }; + }; + const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }); + try { + const restored = path.join(scratch, "restored"); + await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest }); + expect((await git(restored, "rev-parse", "HEAD")).stdout).toBe(commit); + expect(await fs.readFile(path.join(restored, "saved.txt"), "utf8")).toBe("retained work\n"); + } finally { await saved.cleanup(); } + }); + + it("rejects excessive Git roots before parsing controller configuration", async () => { + for (let index = 0; index < 65; index++) { + const directory = path.join(workspace, `repo-${index}`, ".git"); await fs.mkdir(directory, { recursive: true }); + await fs.writeFile(path.join(directory, "config"), ""); + } + const bin = path.join(scratch, "bin"), invoked = path.join(scratch, "git-invoked"); await fs.mkdir(bin); + await fs.writeFile(path.join(bin, "git"), `#!/bin/sh\nprintf invoked > '${invoked}'\nexit 1\n`, { mode: 0o700 }); + vi.stubEnv("PATH", `${bin}${path.delimiter}${process.env.PATH}`); + try { + await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_budget"); + await expect(fs.access(invoked)).rejects.toMatchObject({ code: "ENOENT" }); + } finally { vi.unstubAllEnvs(); } + }); + + it("bounds refspec work before controller normalization", async () => { + await git(workspace, "init", "--initial-branch=main"); + await fs.appendFile(path.join(workspace, ".git", "config"), `[remote "origin"]\n${"fetch = +refs/heads/*:refs/remotes/origin/*\n".repeat(257)}`); + await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_budget"); + }); + + it("cancels controller inventory before reading its source", async () => { + const reason = new Error("verification deadline"); + await expect(readWorkspaceInventory(path.join(scratch, "missing"), true, AbortSignal.abort(reason))).rejects.toBe(reason); + }); + + it("retains tracked environment files during full-disk recovery", async () => { + await git(workspace, "init", "--initial-branch=main"); + await fs.mkdir(path.join(workspace, ".venv")); + await fs.writeFile(path.join(workspace, ".venv", "pyvenv.cfg"), "tracked user data\n"); + await fs.writeFile(path.join(workspace, ".venv", "result.txt"), "must survive\n"); + await git(workspace, "add", ".venv"); await git(workspace, "commit", "-m", "Tracked environment data"); + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + const saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }); + try { + const restored = path.join(scratch, "restored"); + await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest }); + expect(await fs.readFile(path.join(restored, ".venv", "result.txt"), "utf8")).toBe("must survive\n"); + } finally { await saved.cleanup(); } + }); + + it.skipIf(process.platform === "win32")("rejects a file replaced by a FIFO before inventory open without blocking", async () => { + const file = path.join(workspace, "saved.txt"), fifo = path.join(scratch, "fifo"), preload = path.join(scratch, "replace-with-fifo.cjs"); + await fs.writeFile(file, "retained work\n"); await exec("mkfifo", [fifo]); + await fs.writeFile(preload, `const fs=require('node:fs/promises');const lstat=fs.lstat;let replaced=false; + fs.lstat=async(file,...rest)=>{const stat=await lstat(file,...rest); + if(!replaced&&String(file)===${JSON.stringify(file)}&&stat.isFile()){replaced=true; + await fs.rename(file,file+'.retained');await fs.rename(${JSON.stringify(fifo)},file);}return stat;}; + require('node:module').syncBuiltinESMExports();`); + await expect(exec(process.execPath, ["--require", preload, "-e", readonlyWorkspaceInventoryScript(), workspace], { timeout: 3000 })) + .rejects.toMatchObject({ code: 1, stderr: "remote_workspace_readonly_capture_unsafe" }); + expect(await fs.readFile(`${file}.retained`, "utf8")).toBe("retained work\n"); + }); + + it("rejects internal Git config symlinks before controller normalization can follow an include", async () => { + await fs.mkdir(path.join(workspace, ".git")); + const outside = path.join(scratch, "outside-config"); + await fs.writeFile(outside, "[user]\nname = outside\n"); + await fs.writeFile(path.join(workspace, "payload-config"), `[include]\npath = ${outside}\n`); + await fs.symlink("../payload-config", path.join(workspace, ".git", "config")); + await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_unsafe"); + expect(await fs.readFile(outside, "utf8")).toBe("[user]\nname = outside\n"); + }); + + it("saves read-only directories and removes only the private controller copy", async () => { + const source = path.join(workspace, "readonly"), restored = path.join(scratch, "restored"); + await fs.mkdir(source); await fs.writeFile(path.join(source, "saved.txt"), "read-only work\n"); await fs.chmod(source, 0o555); + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + const transfer = remote.runner!.syncOut!; let copy: string | undefined; + remote.runner!.syncOut = (operations, options) => { copy = operations[0]!.files[0]!.targetPath; return transfer(operations, options); }; + let saved: Awaited> | undefined; + try { + saved = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }); + expect((await fs.stat(source)).mode & 0o777).toBe(0o555); + expect(await fs.readFile(path.join(source, "saved.txt"), "utf8")).toBe("read-only work\n"); + await expect(fs.access(copy!)).rejects.toMatchObject({ code: "ENOENT" }); + await restoreRemoteWorkspaceArchive({ archivePath: saved.archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest }); + expect((await fs.stat(path.join(restored, "readonly"))).mode & 0o777).toBe(0o555); + expect(await fs.readFile(path.join(restored, "readonly", "saved.txt"), "utf8")).toBe("read-only work\n"); + } finally { + for (const directory of [source, path.join(restored, "readonly"), ...(copy ? [path.join(copy, "readonly")] : [])]) { + await fs.chmod(directory, 0o700).catch(() => undefined); + } + await saved?.cleanup(); + if (copy) await fs.rm(path.dirname(copy), { recursive: true, force: true }); + } + }); + + it("retains initialized submodule workspaces when a full sandbox cannot normalize their Git pointers", async () => { + const origin = await repository(); + await git(workspace, "init", "--initial-branch=main"); + await fs.writeFile(path.join(workspace, "result.txt"), "retain parent work\n"); + await git(workspace, "add", "result.txt"); await git(workspace, "commit", "-m", "Parent"); + await git(workspace, "-c", "protocol.file.allow=always", "submodule", "add", "--", origin, "sub"); + const before = (await git(workspace, "status", "--porcelain")).stdout; + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + const syncOut = vi.fn(remote.runner!.syncOut!); remote.runner!.syncOut = syncOut; + await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() })) + .rejects.toThrow(/^remote_workspace_sandbox_disk_full:/); + expect(syncOut).not.toHaveBeenCalled(); + expect((await git(workspace, "status", "--porcelain")).stdout).toBe(before); + expect(await fs.readFile(path.join(workspace, "sub", "source.txt"), "utf8")).toBe("initial synthetic source\n"); + }); + + it.each(["Config", "objects/info/Alternates"])("rejects case aliases of Git administration paths before controller Git reads: %s", async relative => { + await git(workspace, "init", "--initial-branch=main"); + const outside = path.join(scratch, "outside"); await fs.mkdir(outside); + if (relative === "Config") { + await fs.rename(path.join(workspace, ".git", "config"), path.join(workspace, ".git", "Config")); + await fs.writeFile(path.join(workspace, ".git", "Config"), `[include]\npath = ${path.join(outside, "config")}\n`); + } else await fs.writeFile(path.join(workspace, ".git", relative), `${outside}\n`); + await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_unsafe"); + expect(await fs.readdir(outside)).toEqual([]); + }); + + it("rejects nonportable Git config before Git can read a controller attributes file", async () => { + await git(workspace, "init", "--initial-branch=main"); + await git(workspace, "config", "core.attributesFile", path.join(scratch, "outside-attributes")); + await expect(readWorkspaceInventory(workspace, true)).rejects.toThrow("remote_workspace_readonly_capture_unsafe"); + }); + + it.each(["worktree pointer", "alternate store", "config include", "core worktree"])("retains a full sandbox with a %s instead of resolving controller paths", async fault => { + const outside = path.join(scratch, "outside"); await fs.mkdir(outside); + await fs.writeFile(path.join(outside, "sentinel"), "must stay outside\n"); + await fs.writeFile(path.join(workspace, "saved.txt"), "retained work\n"); + if (fault === "worktree pointer") await fs.writeFile(path.join(workspace, ".git"), `gitdir: ${outside}\n`); + else { + await git(workspace, "init", "--initial-branch=main"); + if (fault === "alternate store") await fs.writeFile(path.join(workspace, ".git", "objects", "info", "alternates"), `${outside}\n`); + if (fault === "config include") await git(workspace, "config", "include.path", path.join(outside, "sentinel")); + if (fault === "core worktree") await git(workspace, "config", "core.worktree", outside); + } + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + const transfer = remote.runner!.syncOut!; + const syncOut = vi.fn(transfer); remote.runner!.syncOut = syncOut; + await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() })) + .rejects.toThrow(/^remote_workspace_sandbox_disk_full:/); + if (["worktree pointer", "alternate store"].includes(fault)) expect(syncOut).not.toHaveBeenCalled(); + expect(await fs.readFile(path.join(workspace, "saved.txt"), "utf8")).toBe("retained work\n"); + expect(await fs.readdir(outside)).toEqual(["sentinel"]); + expect(await fs.readFile(path.join(outside, "sentinel"), "utf8")).toBe("must stay outside\n"); + }); + it("keeps the full-disk diagnostic exact", async () => { const { remote, execute } = target(); execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", timedOut: false, signal: null, pid: process.pid, @@ -507,13 +817,14 @@ describe("remote workspace transport safety with real sandbox commands", () => { .rejects.toThrow(/^remote_workspace_operation_failed: operation=[a-z_]+ exit=1$/); }); - it.each(["split", "one-shot"] as const)("preserves a Git child disk failure with %s output without its private text", async outputMode => { + it.each(["split", "one-shot"] as const)("preserves a Git child disk failure with %s output when read-only download is unavailable", async outputMode => { await git(workspace, "init", "--initial-branch=main"); await fs.writeFile(path.join(workspace, "saved.txt"), "retained source bytes\n"); const bin = path.join(scratch, "controlled-bin"); await fs.mkdir(bin); const gitWrapper = path.join(bin, "git"); await fs.writeFile(gitWrapper, "#!/bin/sh\nprintf '%s' 'No space left on device: SYNTHETIC_PRIVATE_GIT_PATH' >&2\nexit 2\n", { mode: 0o700 }); const { remote } = target({ PATH: `${bin}${path.delimiter}${process.env.PATH}` }, outputMode); + remote.runner!.syncOut = async () => { throw new Error("synthetic transfer unavailable"); }; await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() })) .rejects.toThrow(/^remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace$/); expect(await fs.readFile(path.join(workspace, "saved.txt"), "utf8")).toBe("retained source bytes\n"); @@ -567,6 +878,7 @@ describe("remote workspace transport safety with real sandbox commands", () => { const diskFull = "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace"; execute.mockResolvedValueOnce({ exitCode: 1, stdout: diskFull, stderr: "remote_workspace_archive_fs_enoent", timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + remote.runner!.syncOut = async () => { throw new Error("synthetic transfer unavailable"); }; await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() })) .rejects.toThrow(diskFull); }); diff --git a/server/src/services/remote-workspace-transport.ts b/server/src/services/remote-workspace-transport.ts index 59df98adf52..b1e9e1ec32f 100644 --- a/server/src/services/remote-workspace-transport.ts +++ b/server/src/services/remote-workspace-transport.ts @@ -1,11 +1,13 @@ import { createHash, randomUUID } from "node:crypto"; import { createReadStream } from "node:fs"; -import { mkdtemp, rm, stat } from "node:fs/promises"; +import { mkdtemp, realpath, rm, stat } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { RemoteWorkspaceSource } from "@paperclipai/shared"; -import { remoteWorkspaceArchiveScript, remoteWorkspaceCaptureFailureMessage, remoteWorkspaceFailureDetail } from "./remote-workspace-snapshot-archive.js"; +import { captureRemoteWorkspaceDirectory, remoteWorkspaceArchiveScript, remoteWorkspaceCaptureFailureMessage, remoteWorkspaceFailureDetail } from "./remote-workspace-snapshot-archive.js"; +import { readWorkspaceInventory, readonlyWorkspaceInventoryScript } from "./remote-workspace-readonly-inventory.js"; +import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; import type { GitRemoteAuthProvider } from "./git-credentials.js"; import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge"; import { MAX_SANDBOX_SYNC_TIMEOUT_MS, type SandboxSyncOperation } from "@paperclipai/adapter-utils/sandbox-managed-runtime"; @@ -418,6 +420,73 @@ const elapsedSince = (started: number) => Math.max(0, Math.round(performance.now export async function captureRemoteWorkspaceArchive(input: { target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string; timing?: RemoteWorkspaceSavePhaseTiming; +}) { + try { return await captureSandboxWorkspaceArchive(input); } + catch (error) { + if (!(error instanceof Error) || error.message !== REMOTE_WORKSPACE_SANDBOX_DISK_FULL) throw error; + // Disk exhaustion cannot be repaired by writing another archive to the + // same sandbox. Transfer a verified standalone source without writing to + // it, then normalize and seal it on the controller. Failed verification + // retains the original disk-full failure and never permits source cleanup. + try { return await captureReadOnlyWorkspaceArchive(input); } + catch (cause) { throw new Error(error.message, { cause }); } + } +} + +async function captureReadOnlyWorkspaceArchive(input: { + target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string; +}) { + if (!/^[a-f0-9-]{36}$/.test(input.operationId)) throw new Error("remote_workspace_operation_identity_invalid"); + const inspect = async (timeoutMs = 300_000) => { + const value = JSON.parse(await command(input.target, "capture", ["-e", readonlyWorkspaceInventoryScript(), input.remoteCwd], undefined, timeoutMs)) as Awaited>; + if (!/^[a-f0-9]{64}$/.test(value.sha256) || !Number.isSafeInteger(value.byteSize) || value.byteSize < 0 + || !Number.isSafeInteger(value.entries) || value.entries < 0 || !Array.isArray(value.exclude) + || value.exclude.length > 4096 || !value.exclude.every(item => typeof item === "string" && item.length > 0 + && !/[\\\u0000-\u001f\u007f]/u.test(item) && !path.posix.isAbsolute(item) + && item.split("/").every(part => part !== "" && part !== "." && part !== "..")) + || !value.exclude.includes(".paperclip-runtime")) throw new Error("remote_workspace_readonly_capture_unsafe"); + return value; + }; + let root: string | undefined; + let stage: "inspect" | "transfer" | "verify" | "archive" = "inspect"; + try { + const before = await inspect(); + root = await mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-readonly-")); + root = await realpath(root); + const copy = path.join(root, "source"), archivePath = path.join(root, "workspace.pcws"); + stage = "transfer"; + await transferArchive(input.target, "syncOut", [{ operationId: input.operationId, + files: [{ kind: "directory", sourcePath: input.remoteCwd, targetPath: copy, + followSymlinks: false, preserveSourceEntries: true, exclude: before.exclude }] }], Math.max(1, before.byteSize)); + stage = "verify"; + const verificationStarted = performance.now(); + const verificationSignal = AbortSignal.timeout(120_000); + const remaining = () => { verificationSignal.throwIfAborted(); return Math.max(1, Math.ceil(120_000 - (performance.now() - verificationStarted))); }; + const received = await readWorkspaceInventory(copy, true, verificationSignal); + const same = (observed: typeof before) => observed.sha256 === before.sha256 + && observed.byteSize === before.byteSize && observed.entries === before.entries; + if (!same(received) || !same(await inspect(remaining()))) throw new Error("remote_workspace_readonly_capture_changed"); + stage = "archive"; + const receipt = await captureRemoteWorkspaceDirectory({ cwd: copy, archivePath, exclude: before.exclude, signal: verificationSignal }); + stage = "verify"; + if (!same(await inspect(remaining()))) throw new Error("remote_workspace_readonly_capture_changed"); + verificationSignal.throwIfAborted(); + // Only the sealed portable archive remains a candidate for durable upload. + await removeTaskDirectoryTree(copy); + const savedRoot = root; + return { ...receipt, archivePath, cleanup: () => removeTaskDirectoryTree(savedRoot) }; + } catch (error) { + if (root) await removeTaskDirectoryTree(root); + const reason = error instanceof Error && error.message === "remote_workspace_readonly_capture_unsafe" ? "unsafe_source" + : error instanceof Error && error.message === "remote_workspace_readonly_capture_changed" ? "source_changed" + : error instanceof Error && error.message === "remote_workspace_readonly_capture_budget" ? "budget_exceeded" : "operation_failed"; + const { label, code } = sandboxProviderFailureLabel(error); + throw new Error(`remote_workspace_readonly_capture_failed: stage=${stage} reason=${reason} transport=${label}${code ? ` code=${code}` : ""}`); + } +} + +async function captureSandboxWorkspaceArchive(input: { + target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string; timing?: RemoteWorkspaceSavePhaseTiming; }) { const timing = input.timing ?? {}; const captureStarted = performance.now(); diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 8c46f3edb43..912416f1457 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -1105,6 +1105,52 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem expect(operation?.issueId).toBe(secondIssue); }, 60_000); + it("durably saves a full sandbox and restores its work after losing the original allocation", async () => { + const reserved = await reserve({ kind: "empty" }), remote = await target("full-sandbox"); + const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: remote }); + await fs.writeFile(path.join(prepared.remoteRoot, "saved.txt"), "completed work before restart\n"); + const execute = remote.runner!.execute.bind(remote.runner); + let fullDiskFailures = 0; + vi.spyOn(remote.runner!, "execute").mockImplementation(input => { + if (input.args?.some(arg => arg.endsWith("/archive.cjs"))) { + fullDiskFailures++; + return Promise.resolve({ exitCode: 1, stdout: "", stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace", + timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() }); + } + return execute(input); + }); + const syncOut = remote.runner!.syncOut!; + remote.runner!.syncOut = async (operations, options) => { + if (!operations.some(operation => operation.files.some(file => file.kind === "directory"))) return syncOut(operations, options); + for (const operation of operations) for (const mapping of operation.files) { + await fs.cp(mapping.sourcePath, mapping.targetPath, { recursive: true, verbatimSymlinks: true, + filter: source => !(mapping.exclude ?? []).some(excluded => { + const relative = path.relative(mapping.sourcePath, source).split(path.sep).join("/"); + return relative === excluded || relative.startsWith(`${excluded}/`); + }) }); + } + return { operations: [] }; + }; + const saved = await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId: ids.run, + lease: prepared.lease, target: remote, boundary: "final", assertOwned: async () => undefined }); + expect(fullDiskFailures).toBe(1); + expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata)).toMatchObject({ + snapshotId: saved.snapshotId, dirty: false, writerRunId: null, + }); + const [snapshot] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, saved.snapshotId)); + expect(snapshot).toMatchObject({ status: "committed" }); + await fs.rm(remote.remoteCwd, { recursive: true, force: true }); + const load = createSavedWorkspaceLoader(db, storage); + for (let reconnect = 0; reconnect < 2; reconnect++) { + const restored = await load({ companyId: ids.company, workspaceId: reserved.id, snapshotId: saved.snapshotId }); + try { expect(await fs.readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("completed work before restart\n"); } + finally { await restored.cleanup(); } + } + const operations = await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, ids.run), + eq(workspaceOperations.phase, "remote_workspace_snapshot"))); + expect(operations.at(-1)).toMatchObject({ status: "succeeded" }); + }, 60_000); + it("retains the writer and previous snapshot when object storage fails", async () => { const reserved = await reserve({ kind: "empty" }), first = await target("outage-sandbox"); const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first }); diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index 1a8b87ca3a9..6aec8b9be18 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -1118,6 +1118,7 @@ describe("automatic retained workspace recovery", () => { } }).where(eq(heartbeatRuns.id, f.runId)); const target = targets.get(f.lease.id)!, syncOut = target.runner!.syncOut!, execute = target.runner!.execute; let transfers = 0, sandboxWriteFailures = 0; + const readonlyExports: Array[0][number]["files"][number]> = []; if (pressure === "sandbox-write") target.runner!.execute = async input => { const fault = `(()=>{const fs=require('node:fs/promises'),sync=require('node:fs'); const full=()=>${JSON.stringify([path.join(pack, "node_modules"), path.join(pack, "provider-assets")])}.some(p=>sync.existsSync(p)); @@ -1135,13 +1136,21 @@ describe("automatic retained workspace recovery", () => { return response; }; target.runner!.syncOut = async operations => { + const mappings = operations.flatMap(operation => operation.files); + if (mappings.some(mapping => mapping.kind === "directory")) { + // These regressions isolate journaled cache reclamation after BOTH + // export paths fail. Directory-export recovery has its own real-provider + // coverage; this boundary explicitly cannot perform that alternative. + readonlyExports.push(...mappings); + throw new Error("cache-reclaim fixture directory export unavailable"); + } transfers++; if (pressure === "transfer" && (await Promise.all(["node_modules", "provider-assets"].map(name => lstat(path.join(pack, name)).then(() => true, () => false)))).some(Boolean)) throw new Error(REMOTE_WORKSPACE_SANDBOX_DISK_FULL); return syncOut(operations); }; - return { ...f, runner, pack, sessions, digest, transfers: () => transfers, + return { ...f, runner, pack, sessions, digest, transfers: () => transfers, readonlyExports, sandboxWriteFailures: () => sandboxWriteFailures }; } @@ -1189,6 +1198,9 @@ describe("automatic retained workspace recovery", () => { await recoverRetainedWorkspace(f.input); expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered"); expect(f.transfers()).toBe(2); + expect(f.readonlyExports).toEqual([{ kind: "directory", sourcePath: await realpath(f.remote), + targetPath: expect.stringMatching(/[/]source$/), followSymlinks: false, preserveSourceEntries: true, + exclude: [".paperclip-runtime"] }]); await expect(lstat(path.join(f.pack, "node_modules"))).rejects.toMatchObject({ code: "ENOENT" }); await expect(lstat(path.join(f.pack, "provider-assets"))).rejects.toMatchObject({ code: "ENOENT" }); expect(JSON.parse(await readFile(path.join(f.pack, "provider-pack.json"), "utf8")).digest).toBe(f.digest); @@ -1264,6 +1276,9 @@ describe("automatic retained workspace recovery", () => { const f = await pressuredToolCache(), target = targets.get(f.lease.id)!, syncOut = target.runner!.syncOut!; let transfers = 0; target.runner!.syncOut = async operations => { + // Fail the second sealed-archive transfer, after cache cleanup, rather + // than consuming this fault on the earlier read-only directory fallback. + if (operations.some(operation => operation.files.some(mapping => mapping.kind === "directory"))) return syncOut(operations); if (++transfers === 2) throw new Error("controlled later transfer failure"); return syncOut(operations); }; From 952ecdbb39b8464c9b4bf32ee8fa0727521232c1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:01:02 -0600 Subject: [PATCH 12/13] feat(ui): show assignee avatars in task lists (#18607) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People use task lists to see who owns each task. > - The shared task table shows an assignee name without a picture. > - A picture helps people find an agent or person when they scan the list. > - This pull request shows the existing avatar beside the assignee name. > - The table uses the same image and fallback components as other identity views. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Assignee column in the shared task table. **Subsystem affected** `ui/` — the task table and its page data. **Current behavior** The column shows only an assignee name. People must read each name to find an owner. **Proposed behavior** Show the agent avatar or human profile image beside the name when the Assignee column is visible. Use the existing fallback if an image is missing. Show no avatar for an unassigned task. **Reason and benefit** People can use the same identity pictures throughout the app. This helps them scan task ownership. **Breaking changes** None. The API, task assignment, name sorting, and filters keep their existing contracts. **Additional context** Related work: #2422 adds agent avatar URL support. It does not add avatars to this table. #13319 aligns icons in task properties. This change uses the existing identity components. ## What Changed - Pass agent appearance and human profile images from existing company data to the task table. - Keep the avatar visible beside a long name as the name scrolls on hover or focus. - Respect the Assignee column setting and keep unassigned rows without avatars. - Load human photos only from Paperclip asset URLs. Show initials for external profile URLs without an image request. - Add tests with the real task row for agent and human images, image fallbacks, unassigned tasks, and the Columns control. - Use the shared human label when a directory entry is missing. The local operator shows Board. Other missing users show the short user ID. - Update Storybook data, design guidance, and the task-list verification steps. ## Verification - Current head: `c49fee5c9b`. The source-tested merge is `586873fe47921b250ab0de18371d25632f46cff0`, which integrates master `53645a0104`. The last commit removes only the inherited avatar paragraph from the feature map under the current map-scope rule. Map checks pass again. No history rewrite or duplicate feature PR. - Reproduced the old voice-report CI failure before integration. The unchanged test on `7f1d137a` passed on an unloaded local machine. With a controlled 3,250 ms delay in its first local provider fixture and a 15-second wait, the old polling failed: expected 105 report requests, received 315. This confirms the fake clock crossed refresh deadlines during database work. - Master already fixes this exact failure through #18628 (`d26628e35a`). The fixed polling passes with the identical delay and wait budget. The exact session IDs, 105 requests, one attempt per report, bounded reconstruction, restart, and quiet assertions remain. The temporary delay is removed. - Final committed code: all 15 voice-report tests and 46 scheduler/coordinator tests pass against disposable embedded PostgreSQL on macOS arm64. No external provider call is used. - All 154 tests pass across eight focused UI suites: TaskList, TaskList.assignees, CoordinatorTasks, IssueRow, AgentAvatar, useTaskListCollection, useTaskListColumnPreferences, and TaskChatBubble. - Three merge conflicts retained both current behavior and the avatar feature: current sorting and column preferences, unread state, assignee photos, and shared human labels. The focused tests cover those contracts. - UI TypeScript, UI build, token gates, map structure, Storybook catalog generation, and whitespace checks pass. Storybook output is generated and ignored. - The original server shard failure remains recorded: [voice-report CI failure](https://github.com/paperclipai/paperclip/actions/runs/41472540522/job/125369899753), expected zero idle holds but received two. Older attempts also had attention-retention and sandbox-runtime timeouts; their causes were not established in those attempts. Fresh CI must qualify the merged head. - Previous review findings were fixed with red/green coverage: missing avatar tests, shared human labels, and arbitrary human image URLs. Current code still limits human photos to Paperclip asset URLs; external URLs use initials and start no image preload. - Prior browser checks covered agent and human images, fallbacks, unassigned rows, hidden columns, and long-name keyboard focus. Those checks used production components with fixture data and do not prove live backend behavior. No new browser or hosted task action was performed for this CI repair. - Current head `c49fee5c9b`: Greptile and Paperclip review both give 5/5 with no findings. The initial CI attempt passed the original server shard and all other completed test suites. The workspaces-a 2/2 job was interrupted by a runner shutdown signal ([job log](https://github.com/paperclipai/paperclip/actions/runs/42250845315/job/128006963628)); it has no complete test result. A single bounded retry of that shard and its dependent verification has been requested after the initial workflow ended. No source change is based on that interruption. - Final-head qualification: `c49fee5c9b75b695d2894933bbcbe0896aab9148` passes all 55 applicable CI checks (three skipped, one neutral review). [CI result](https://github.com/paperclipai/paperclip/actions/runs/42250845315). Greptile gives 5/5 with no open findings. Hosted task recovery remains pending verified rollout. ## Risks - Low risk. The change adds identity pictures through existing UI components and existing company data. - Human profile images hosted outside Paperclip show initials in this table. This matches the existing task-message avatar rule. - A long name has less width beside its avatar. The existing hover and focus reveal keeps the name readable. - Current master changes table sorting and preferences. Merge resolution retains these contracts and their tests. Fresh CI and review must pass before merge. ## Model Used OpenAI GPT-6 through Codex assisted with code and verification. The runtime did not expose an exact model ID or context-window size. The agent used code execution and repository tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — current head `c49fee5c9b`; the map-scope finding is fixed and its thread is resolved - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- DESIGN.md | 9 +- ui/src/components/TaskList.assignees.test.tsx | 122 ++++++++++++++++++ ui/src/components/TaskList.tsx | 18 ++- ui/src/components/TaskListColumns.tsx | 10 +- ui/src/pages/CoordinatorTasks.test.tsx | 13 +- ui/src/pages/CoordinatorTasks.tsx | 13 +- ui/src/pages/DesignSandboxTasks.tsx | 13 +- .../content | Bin 0 -> 1450 bytes .../task-row-design.stories.tsx | 2 +- .../design-proposals/task-list-data.ts | 5 +- .../design-proposals/task-page-data.ts | 2 +- 11 files changed, 186 insertions(+), 21 deletions(-) create mode 100644 ui/src/components/TaskList.assignees.test.tsx create mode 100644 ui/storybook/public/api/assets/33333333-3333-4333-8333-333333333333/content diff --git a/DESIGN.md b/DESIGN.md index 026e1f6b633..cff31c9a04c 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -407,9 +407,12 @@ Project, Parent Task, Tags, Worktree, Last updated**. The reusable width tokens and configurable column order. Review the complete composition in **Design proposals → Components → Task list → All columns**. -- Created by is the recorded creator (human or agent); Assignee is the current - assignee. Both show names without avatars. Do not infer either from tree depth - or task status. +- Created by is the recorded creator (human or agent) and shows a name. Assignee + shows the current agent's avatar or human's profile photo beside their name, + with the shared avatar fallback when an image is unavailable. Human photos load + only from Paperclip asset URLs; external profile URLs use initials. The avatar stays + visible while a long name reveals on hover or focus. Do not infer either + identity from tree depth or task status. - Task combines the status icon and title under one header and one Columns-menu choice. Sorting Task uses the title. Status sorting is unavailable for now. - Parent Task refers to the structural parent; it can show its ID and title. diff --git a/ui/src/components/TaskList.assignees.test.tsx b/ui/src/components/TaskList.assignees.test.tsx new file mode 100644 index 00000000000..e418cdca5e9 --- /dev/null +++ b/ui/src/components/TaskList.assignees.test.tsx @@ -0,0 +1,122 @@ +// @vitest-environment jsdom +import { act, type ComponentProps } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { appearanceForPalette } from "@paperclipai/shared"; +import { createIssue } from "../../storybook/fixtures/paperclipData"; +import { TaskList, type TaskListItem } from "./TaskList"; +import { TooltipProvider } from "./ui/tooltip"; + +vi.mock("@/lib/router", () => ({ + Link: ({ children, to, state: _state, disableIssueQuicklook: _quicklook, issuePrefetch: _prefetch, ...props }: + ComponentProps<"a"> & { to?: string; state?: unknown; disableIssueQuicklook?: boolean; issuePrefetch?: unknown }) => + {children}, +})); + +Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true }); + +const agentItem: TaskListItem = { + issue: createIssue({ id: "agent-task", title: "Agent task", assigneeAgentId: "agent-1", assigneeUserId: null }), + assignee: "Research Agent", + assigneeAgent: { + id: "agent-1", + name: "Research Agent", + appearance: { ...appearanceForPalette("deep-tide"), customAvatarAssetId: "22222222-2222-4222-8222-222222222222" }, + }, +}; +const humanItem: TaskListItem = { + issue: createIssue({ id: "human-task", title: "Human task", assigneeAgentId: null, assigneeUserId: "user-1" }), + assignee: "Alex Smith", + assigneeUserAvatarUrl: "/api/assets/33333333-3333-4333-8333-333333333333/content", +}; +const brokenHumanAvatar = "/api/assets/44444444-4444-4444-8444-444444444444/content"; +const imageRequests = vi.fn<(src: string) => void>(); + +describe("TaskList assignee identities", () => { + let host: HTMLDivElement; + let root: Root; + + beforeEach(() => { + imageRequests.mockClear(); + host = document.createElement("div"); + document.body.append(host); + root = createRoot(host); + // Radix preloads profile pictures through Image. jsdom does not load them. + vi.stubGlobal("Image", class extends EventTarget { + complete = false; + naturalWidth = 0; + set src(value: string) { + imageRequests(value); + queueMicrotask(() => { + this.complete = true; + this.naturalWidth = value === brokenHumanAvatar ? 0 : 100; + this.dispatchEvent(new Event(this.naturalWidth ? "load" : "error")); + }); + } + }); + }); + + afterEach(async () => { + await act(async () => root.unmount()); + host.remove(); + vi.unstubAllGlobals(); + }); + + async function render(items: TaskListItem[]) { + await act(async () => root.render( + , + )); + } + + const cell = (title: string) => host.querySelector(`[role="rowgroup"][aria-label="${title} task group"] [role="cell"][data-column="assignee"]`)!; + + it("shows agent appearance and human profile pictures beside their names", async () => { + await render([agentItem, humanItem]); + expect(cell("Agent task").textContent).toBe("Research Agent"); + expect(cell("Agent task").querySelector("img")?.getAttribute("src")).toBe("/api/assets/22222222-2222-4222-8222-222222222222/content"); + expect(cell("Human task").textContent).toBe("Alex Smith"); + expect(cell("Human task").querySelector("img")?.getAttribute("src")).toBe(humanItem.assigneeUserAvatarUrl); + }); + + it.each([ + "https://example.com/track.png", + "http://example.com/track.png", + "//example.com/track.png", + "/profile/alex.png", + ])("uses initials without loading an untrusted human avatar: %s", async (source) => { + await render([{ ...humanItem, assigneeUserAvatarUrl: source }]); + expect(cell("Human task").querySelector('[data-slot="avatar-fallback"]')?.textContent).toBe("AS"); + expect(cell("Human task").querySelector("img")).toBeNull(); + expect(imageRequests).not.toHaveBeenCalled(); + }); + + it("uses initials for missing or failed pictures and leaves unassigned rows without an avatar", async () => { + const unassigned: TaskListItem = { + issue: createIssue({ id: "unassigned-task", title: "Unassigned task", assigneeAgentId: null, assigneeUserId: null }), + }; + await render([agentItem, { ...humanItem, assigneeUserAvatarUrl: null }, unassigned]); + expect(cell("Human task").querySelector('[data-slot="avatar-fallback"]')?.textContent).toBe("AS"); + expect(cell("Unassigned task").textContent).toBe("Unassigned"); + expect(cell("Unassigned task").querySelector("img, [data-slot=avatar], [data-slot=agent-avatar]")).toBeNull(); + + await act(async () => cell("Agent task").querySelector("img")!.dispatchEvent(new Event("error"))); + expect(cell("Agent task").querySelector('[data-slot="agent-avatar"]')?.textContent).toBe("RA"); + await render([{ ...humanItem, assigneeUserAvatarUrl: brokenHumanAvatar }]); + expect(cell("Human task").querySelector('[data-slot="avatar-fallback"]')?.textContent).toBe("AS"); + expect(cell("Human task").querySelector("img")).toBeNull(); + }); + + it("removes and restores assignee avatars through the Columns control", async () => { + await render([agentItem, humanItem]); + const columns = host.querySelector('button[aria-label="Columns"]')!; + await act(async () => columns.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true }))); + const assignee = [...document.querySelectorAll('[role="menuitemcheckbox"]')].find(item => item.textContent === "Assignee")!; + expect(assignee).toBeDefined(); + await act(async () => assignee.click()); + expect(host.querySelector('[role="cell"][data-column="assignee"]')).toBeNull(); + expect(host.querySelector("img")).toBeNull(); + await act(async () => assignee.click()); + expect(cell("Agent task").querySelector("img")?.getAttribute("src")).toBe("/api/assets/22222222-2222-4222-8222-222222222222/content"); + expect(cell("Human task").querySelector("img")?.getAttribute("src")).toBe(humanItem.assigneeUserAvatarUrl); + }); +}); diff --git a/ui/src/components/TaskList.tsx b/ui/src/components/TaskList.tsx index b653de661b9..791219d0f37 100644 --- a/ui/src/components/TaskList.tsx +++ b/ui/src/components/TaskList.tsx @@ -3,6 +3,7 @@ import type { Issue } from "@paperclipai/shared"; import { ChevronDown, ChevronRight, GitBranch } from "lucide-react"; import { buildIssueTree } from "../lib/issue-tree"; import { issueFilterLabel } from "../lib/issue-filters"; +import { paperclipHumanAvatarUrl } from "../lib/issue-chat-human-author"; import { defaultTaskListColumns, defaultTaskListSort, isTaskListColumnSort, taskListOrderToIssueSort, type TaskListColumnId, type TaskListSort, type TaskListOrder } from "../lib/task-list"; import { sortIssues } from "../lib/issue-sort"; import { timeAgo } from "../lib/timeAgo"; @@ -13,6 +14,9 @@ import { TaskRowGroup } from "./TaskRowGroup"; import { Badge } from "./ui/badge"; import { Button } from "./ui/button"; import { useTaskRowBudget } from "../hooks/useTaskRowBudget"; +import { AgentAvatar, type AvatarAgent } from "./AgentAvatar"; +import { deriveInitials } from "./Identity"; +import { Avatar, AvatarFallback, AvatarImage } from "./ui/avatar"; import { useTaskListColumnPreferences } from "../hooks/useTaskListColumnPreferences"; import type { TaskCollectionPreferenceLocation } from "../lib/task-collection-preferences"; @@ -21,6 +25,8 @@ export type TaskListItem = { unread?: boolean; createdBy?: string | null; assignee?: string | null; + assigneeAgent?: AvatarAgent | null; + assigneeUserAvatarUrl?: string | null; }; export type TaskListGrouping = "none" | "status" | "project"; @@ -94,6 +100,8 @@ export function TaskList({ items, allItems = items, initialColumns = defaultTask const expanded = !collapsed.has(issue.id); const parent = issue.parentId ? byId.get(issue.parentId)?.issue : null; const branch = issue.currentExecutionWorkspace?.branchName; + const assigneeName = item.assignee ?? (issue.assigneeAgentId ? "Unknown agent" : issue.assigneeUserId ? "User" : "Unassigned"); + const assigneeUserAvatarUrl = paperclipHumanAvatarUrl(item.assigneeUserAvatarUrl); return ( + ) : issue.assigneeUserId ? ( + + ) : null, project: issue.project?.name, parentTask: parent ? {parent.identifier} {parent.title} : undefined, tags: issue.labels?.length ? {issue.labels.map(label => {label.name})} : undefined, diff --git a/ui/src/components/TaskListColumns.tsx b/ui/src/components/TaskListColumns.tsx index d1dd128f528..2815818cf7f 100644 --- a/ui/src/components/TaskListColumns.tsx +++ b/ui/src/components/TaskListColumns.tsx @@ -56,10 +56,11 @@ export function TaskListViewport({ children, visibleColumns = defaultTaskListCol const detailColumns = taskListColumns.filter(column => column.id !== "task"); type DetailColumn = Exclude<(typeof detailColumns)[number]["id"], "taskId">; -/** Values are resolved by the collection; identity columns intentionally accept names only. */ +/** Values are resolved by the collection; names remain separate from avatar presentation. */ export type TaskRowColumnValues = Partial, ReactNode>> & { createdBy?: string | null; assignee?: string | null; + assigneeAvatar?: ReactNode; }; function SortableColumnHeader({ id, label, start, sort, onSortChange, onColumnOrderChange, helpId }: { @@ -184,7 +185,12 @@ export function TaskRowColumnCells({ identifier, values }: { identifier: string; {columns.filter(id => id !== "task").map(id => ( - {id === "taskId" ? identifier : values[id] ?? "—"} + {id === "assignee" && values.assigneeAvatar ? ( + + {values.assigneeAvatar} + {values.assignee ?? "—"} + + ) : {id === "taskId" ? identifier : values[id] ?? "—"}} ))} diff --git a/ui/src/pages/CoordinatorTasks.test.tsx b/ui/src/pages/CoordinatorTasks.test.tsx index 1ad5885cde5..3a61befe4ec 100644 --- a/ui/src/pages/CoordinatorTasks.test.tsx +++ b/ui/src/pages/CoordinatorTasks.test.tsx @@ -45,8 +45,8 @@ vi.mock("@/components/TaskListToolbar", () => ({ TaskListToolbar: ({ filters, vi })); vi.mock("@/components/TaskList", async importOriginal => ({ ...await importOriginal(), - TaskList: ({ items, emptyState, onSortChange }: { onSortChange?: (sort: TaskListSort) => void; items: { issue: { id: string; title: string } }[]; emptyState: React.ReactNode }) => -
{useLocation().search}{items.length ? items.map(item =>

{item.issue.title}

) : emptyState}
, + TaskList: ({ items, emptyState, onSortChange }: { onSortChange?: (sort: TaskListSort) => void; items: { issue: { id: string; title: string }; assignee?: string | null }[]; emptyState: React.ReactNode }) => +
{useLocation().search}{items.length ? items.map(item =>

{item.issue.title}

) : emptyState}
, })); describe("Coordinator tasks using the proposed table", () => { @@ -152,6 +152,15 @@ describe("Coordinator tasks using the proposed table", () => { expect(mocks.get).not.toHaveBeenCalled(); }); + it.each([ + ["local-board", "Board"], + ["missing-user", "missi"], + ])("keeps the shared human identity label when %s is absent from the directory", async (assigneeUserId, label) => { + mocks.list.mockResolvedValue([createIssue({ id: "human-task", companyId: "company", parentId: null, assigneeAgentId: null, assigneeUserId })]); + await mount("/issues"); + expect(host.querySelector('[data-task-id="human-task"]')?.getAttribute("data-assignee")).toBe(label); + }); + it("opens routine task links, filters before paging, and lets the checkbox change the collection", async () => { const routine = createIssue({ id: "routine-task", companyId: "company", parentId: null, originKind: "routine_execution", title: "Routine report" }); const manual = createIssue({ id: "manual-task", companyId: "company", parentId: null, originKind: "manual", title: "Manual report" }); diff --git a/ui/src/pages/CoordinatorTasks.tsx b/ui/src/pages/CoordinatorTasks.tsx index 2e38c33b197..151668e54d5 100644 --- a/ui/src/pages/CoordinatorTasks.tsx +++ b/ui/src/pages/CoordinatorTasks.tsx @@ -24,7 +24,8 @@ import { useIssueRowDetails } from "@/hooks/useIssueRowDetails"; import { useCoordinatorWorkspaceEnabled } from "@/hooks/useCoordinatorWorkspaceEnabled"; import { usePublishSharedQueryData, useSharedPollingQuery } from "@/hooks/useSharedPolling"; import { collectLiveIssueIds } from "@/lib/liveIssueIds"; -import { buildCompanyUserLabelMap } from "@/lib/company-members"; +import { formatUserLabel } from "@/lib/assignees"; +import { buildCompanyUserProfileMap } from "@/lib/company-members"; import { coordinatorAttentionTasksQueryKey } from "@/lib/coordinator-attention"; import { createIssueDetailLocationState } from "@/lib/issueDetailBreadcrumb"; import { defaultIssueFilterState, type IssueFilterState } from "@/lib/issue-filters"; @@ -155,8 +156,8 @@ export function CoordinatorTasks({ viewControl }: { viewControl?: ReactNode } = const sharedLiveRuns = useSharedPollingQuery({ companyId, resourceKey: "live-runs", queryKey: queryKeys.liveRuns(companyId!), enabled: !!companyId, refetchInterval: false, leaderOnly: true }); const liveRuns = useQuery({ queryKey: queryKeys.liveRuns(companyId!), queryFn: () => heartbeatsApi.liveRunsForCompany(companyId!), enabled: sharedLiveRuns.enabled, refetchInterval: sharedLiveRuns.refetchInterval }); usePublishSharedQueryData(sharedLiveRuns, liveRuns.data, liveRuns.dataUpdatedAt); - const users = buildCompanyUserLabelMap(directory.data?.users); - const agentNames = new Map(agents.data?.map(agent => [agent.id, agent.name])); + const users = buildCompanyUserProfileMap(directory.data?.users); + const agentById = new Map(agents.data?.map(agent => [agent.id, agent])); const projectById = new Map(projects.data?.map(project => [project.id, project])); const records = (needsAttention ? attentionTasks.flatMap(task => task.data ?? []) : tasks.data?.pages.flatMap(page => page.items) ?? []) as Issue[]; const unreadIds = new Set(view === "unread" ? records.map(issue => issue.id) : []); @@ -190,8 +191,10 @@ export function CoordinatorTasks({ viewControl }: { viewControl?: ReactNode } = const items: TaskListItem[] = [...new Map(scopedRecords.map(issue => [issue.id, issue])).values()].map(issue => ({ issue: { ...issue, project: issue.project ?? projectById.get(issue.projectId ?? "") }, unread: unreadIds.has(issue.id) || issue.isUnreadForMe === true, - createdBy: issue.createdByAgentId ? agentNames.get(issue.createdByAgentId) ?? "Unknown agent" : users.get(issue.createdByUserId ?? "") ?? "Board", - assignee: issue.assigneeAgentId ? agentNames.get(issue.assigneeAgentId) ?? "Unknown agent" : users.get(issue.assigneeUserId ?? "") ?? null, + createdBy: issue.createdByAgentId ? agentById.get(issue.createdByAgentId)?.name ?? "Unknown agent" : users.get(issue.createdByUserId ?? "")?.label ?? "Board", + assignee: issue.assigneeAgentId ? agentById.get(issue.assigneeAgentId)?.name ?? "Unknown agent" : users.get(issue.assigneeUserId ?? "")?.label ?? formatUserLabel(issue.assigneeUserId), + assigneeAgent: agentById.get(issue.assigneeAgentId ?? ""), + assigneeUserAvatarUrl: users.get(issue.assigneeUserId ?? "")?.image, })); const collection = useTaskListCollection({ items, companyId, userId, query, onQueryChange: setQuery, view: needsAttention ? "all" : view, layout, onLayoutChange: setLayout, diff --git a/ui/src/pages/DesignSandboxTasks.tsx b/ui/src/pages/DesignSandboxTasks.tsx index ca6ae1fc9c2..3c56c479cf4 100644 --- a/ui/src/pages/DesignSandboxTasks.tsx +++ b/ui/src/pages/DesignSandboxTasks.tsx @@ -16,7 +16,8 @@ import { useBreadcrumbs } from "@/context/BreadcrumbContext"; import { useCompany } from "@/context/CompanyContext"; import { useDialogActions } from "@/context/DialogContext"; import { useTaskListCollection } from "@/hooks/useTaskListCollection"; -import { buildCompanyUserLabelMap } from "@/lib/company-members"; +import { formatUserLabel } from "@/lib/assignees"; +import { buildCompanyUserProfileMap } from "@/lib/company-members"; import { queryKeys } from "@/lib/queryKeys"; import { taskListViewStorageKey } from "@/lib/task-list-views"; import type { Issue } from "@paperclipai/shared"; @@ -66,14 +67,16 @@ export function DesignSandboxTasks() { }, enabled: !!companyId && !!userId, }); - const users = buildCompanyUserLabelMap(directory.data?.users); - const agentNames = new Map(agents.data?.map(agent => [agent.id, agent.name])); + const users = buildCompanyUserProfileMap(directory.data?.users); + const agentById = new Map(agents.data?.map(agent => [agent.id, agent])); const projectById = new Map(projects.data?.map(project => [project.id, project])); const items: TaskListItem[] = [...new Map((tasks.data?.pages.flat() ?? []).map(issue => [issue.id, issue])).values()].map(issue => ({ issue: { ...issue, project: issue.project ?? projectById.get(issue.projectId ?? "") }, unread: unread.data?.has(issue.id) ?? false, - createdBy: issue.createdByAgentId ? agentNames.get(issue.createdByAgentId) ?? "Unknown agent" : users.get(issue.createdByUserId ?? "") ?? (issue.createdByUserId === "local-board" ? "Board" : "Unknown user"), - assignee: issue.assigneeAgentId ? agentNames.get(issue.assigneeAgentId) ?? "Unknown agent" : users.get(issue.assigneeUserId ?? "") ?? (issue.assigneeUserId === "local-board" ? "Board" : null), + createdBy: issue.createdByAgentId ? agentById.get(issue.createdByAgentId)?.name ?? "Unknown agent" : users.get(issue.createdByUserId ?? "")?.label ?? (issue.createdByUserId === "local-board" ? "Board" : "Unknown user"), + assignee: issue.assigneeAgentId ? agentById.get(issue.assigneeAgentId)?.name ?? "Unknown agent" : users.get(issue.assigneeUserId ?? "")?.label ?? formatUserLabel(issue.assigneeUserId), + assigneeAgent: agentById.get(issue.assigneeAgentId ?? ""), + assigneeUserAvatarUrl: users.get(issue.assigneeUserId ?? "")?.image, })); const collection = useTaskListCollection({ items, companyId, userId, viewStorageKey: taskListViewStorageKey("design-integration-sandbox", userId ?? "local-board"), diff --git a/ui/storybook/public/api/assets/33333333-3333-4333-8333-333333333333/content b/ui/storybook/public/api/assets/33333333-3333-4333-8333-333333333333/content new file mode 100644 index 0000000000000000000000000000000000000000..7e63dcc804a36c6dbeb79776c2d29497d2ce2e87 GIT binary patch literal 1450 zcmV;b1y%ZqP);2?jK)5KPOwQ(t_+lYs zgmPJCS=O4IG7prBXDJgyei2_nh`}PWd_zvilM=CsG8yxm^b%olk23MHHsY3Lt)W~t z`b{*W5KPpgheTna^(gGmev`{0Bt9}tb1fcuF1!6E<-!xrV2DsanLOe*8P{Oqn5S@{ zt03kbc*1U10Y{ZogbO1S>5R5n$Mowx1_RE$!Lfel+q4eED@B zk^nNYlD1;Wk#Hc}-xPxjGO5tz%&;{r@cOIAg!d5tG zJUCrJH4JYG8W_y%Usi?F45frX7&^eeunhxtHMOFI0<^<>_M#3(k~~2EsZD@HU*L5h z0y9Ki&qqK29U+hm7-qN$2XOm6&F^b_uKz-SXh?#ofC1;lOIMtEL+q{^8T(dM)a=yy zOi4*lpl*W+1WH489)Jn&>|8W!510){)rEmes;;rswxvY`CPob%XMRns?P~niZM0qB z<_{C9asLMz1dyEuTwcO5Sha1Z8X3ERUIQdBa^;4q#0(84YQO_FV;E~}X;V|vVQu6S zfP}j5UzBLw3bno8Br~C|v1PfpKW&;-kEay?7G;vAGtf79*>{4OaK3*i zy$7(+>xP3>0C#Uw|Ck4>U3GQ2#>`wqmAqNCgzJlbM^|LhU6R_p_ki!@oZ`KPlrUg2 zmdzz8d!?QjLGA4~F{k+CRQKW&AUBJ^zp6YiGB|wIcVbR4Fg%*{_C=lHVHFT2I&`1; z={iG%OVgfjj+>^r7A{NwHZ|irF{k+T_h~41Pov!!w=C=j)Br1Ap9^B$n`ArQ~!0F_UI@%%2Ju(rW}0YA|I@|Tetn*aa+07*qoM6N<$ Ef&r7JBme*a literal 0 HcmV?d00001 diff --git a/ui/storybook/stories/design-proposals/task-row-design.stories.tsx b/ui/storybook/stories/design-proposals/task-row-design.stories.tsx index 7c1b1309c56..cf91f8a5d38 100644 --- a/ui/storybook/stories/design-proposals/task-row-design.stories.tsx +++ b/ui/storybook/stories/design-proposals/task-row-design.stories.tsx @@ -224,7 +224,7 @@ const meta = { exception: "Staged design-system refresh awaiting visual review and app adoption; kept separate from production references.", implementation: "production", boundary: "fixtures", - notes: "Library specimen of refreshed IssueRow, TaskRowGroup, and TaskListColumnHeader. Task combines status and title under one header and visibility choice; Task sorting uses titles, with no status sort. All Columns starts with every column and demonstrates sibling hierarchy, sorting, pointer/keyboard column reordering, pinned column settings, visibility and reset using local fixture state. Last updated is last by default; the flyout follows the current column order. Reset restores that order and shows only Task, Task ID, and Last updated. The Columns flyout follows the Paper reference with compact 32px items, regular labels, and right-aligned checks. Sorted headers retain their background. Table typography uses sm with a 16px line height; every truncated cell uses a local marquee. The entire proposal canvas uses Paper's background token. Status tooltips use shared labels. The composition is under review and does not establish production API, preference persistence, or full accessibility acceptance.", + notes: "Library specimen of refreshed IssueRow, TaskRowGroup, and TaskListColumnHeader. Task combines status and title under one header and visibility choice; Task sorting uses titles, with no status sort. All Columns shows agent avatars, a human profile image and human fallback avatars beside assignee names. Created by remains a name. Unassigned rows have no avatar. Hiding Assignee removes its avatars. All Columns starts with every column and demonstrates sibling hierarchy, sorting, pointer/keyboard column reordering, pinned column settings, visibility and reset using local fixture state. Last updated is last by default; the flyout follows the current column order. Reset restores that order and shows only Task, Task ID, and Last updated. The Columns flyout follows the Paper reference with compact 32px items, regular labels, and right-aligned checks. Sorted headers retain their background. Table typography uses sm with a 16px line height; every truncated cell uses a local marquee. The entire proposal canvas uses Paper's background token. Status tooltips use shared labels. The composition is under review and does not establish production API, preference persistence, or full accessibility acceptance.", }, }, } satisfies Meta; diff --git a/ui/storybook/support/design-proposals/task-list-data.ts b/ui/storybook/support/design-proposals/task-list-data.ts index af5d11fb7ca..c064349d42e 100644 --- a/ui/storybook/support/design-proposals/task-list-data.ts +++ b/ui/storybook/support/design-proposals/task-list-data.ts @@ -59,6 +59,9 @@ export function columnData(issue: Issue, owner: string) { parent: treeTasks.find(candidate => candidate.id === issue.parentId), creator: rowIssue.createdByAgentId ? agent.name : "Scott", assignee: rowIssue.assigneeAgentId ? agent.name : rowIssue.assigneeUserId ? "Design" : null, + assigneeAgent: rowIssue.assigneeAgentId ? agent : null, + // Static Storybook fixture served at the same asset path as an uploaded photo. + assigneeUserAvatarUrl: rowIssue.assigneeUserId && issue.id === healthTask.id ? "/api/assets/33333333-3333-4333-8333-333333333333/content" : null, }; } @@ -79,5 +82,5 @@ export function sortValue(issue: Issue, column: TaskListColumnId) { export const reviewTaskItems: TaskListItem[] = treeTasks.map(issue => { const data = columnData(issue, ownerFor(issue)); - return { issue: data.issue, createdBy: data.creator, assignee: data.assignee, unread: issue === connectorParent || issue === healthTask }; + return { issue: data.issue, createdBy: data.creator, assignee: data.assignee, assigneeAgent: data.assigneeAgent, assigneeUserAvatarUrl: data.assigneeUserAvatarUrl, unread: issue === connectorParent || issue === healthTask }; }); diff --git a/ui/storybook/support/design-proposals/task-page-data.ts b/ui/storybook/support/design-proposals/task-page-data.ts index 9b198f6f683..9e666804752 100644 --- a/ui/storybook/support/design-proposals/task-page-data.ts +++ b/ui/storybook/support/design-proposals/task-page-data.ts @@ -7,7 +7,7 @@ const extraTasks: TaskListItem[] = [ task(460, "Collect feedback on the refreshed task list", "todo"), ].map((issue, index) => { const data = columnData(issue, index === 2 ? "Design" : "Codex"); - return { issue: data.issue, createdBy: data.creator, assignee: data.assignee, unread: index === 0 }; + return { issue: data.issue, createdBy: data.creator, assignee: data.assignee, assigneeAgent: data.assigneeAgent, assigneeUserAvatarUrl: data.assigneeUserAvatarUrl, unread: index === 0 }; }); export const initialItems = [...reviewTaskItems, ...extraTasks]; export const recentTasks = [...initialItems] From 0d03610d2f73b7cd799547e43929bb2303319024 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:01:25 -0600 Subject: [PATCH 13/13] fix(opencode): export bounded tool argument diagnostics (#18699) ## Thinking Path > - Paperclip manages work across agent runtimes. > - Company observability sends allowed records to the operator's OTLP receiver. > - Native OpenCode exports tool results but omits submitted argument diagnostics. > - Exact argument values are excluded by the current capture policy. > - Safe type information can distinguish malformed envelopes without copying private values. > - This change exports a fixed diagnostic shape from authenticated dispatch and schema-rejection events. > - Operators can compare rejected and accepted calls under the existing capture controls. ## Linked Issues or Issue Description **What happened?** OpenCode tool spans omit argument diagnostics. The exporter reads native tool events. The authenticated MCP boundary has the submitted arguments, but its dispatch item events do not create tool spans. Calls rejected by its schema check do not emit dispatch items. **Expected behavior** With tool details and tool content enabled, Paperclip tool spans include the method, diagnostic schema version and allowed argument field types. Exact values and arbitrary field names stay excluded. **Steps to reproduce** 1. Configure a company OTLP receiver and enable native OpenCode export. 2. Enable tool details and tool content. 3. Submit a rejected Paperclip tool call and then a valid call. 4. Inspect the input diagnostics and outcome of each span. Related work: #17989 proposes metadata-only native export. #18098 added the normalized content exporter. This change repairs argument diagnostics in that exporter. ## What Changed - Export `paperclip.opencode.tool-diagnostics.v1` as `braintrust.input` for authenticated Paperclip tool calls. - Include `tools/call`, the argument type and types for a fixed list of envelope fields. - Use dispatch IDs for Paperclip tool spans. Keep separately named `opencode tool ...` provider records for failures before dispatch. Mark each span source. - Preserve schema-rejected calls as failed items without invoking the controller handler. Keep authentication and the closed tool catalog. - Bound schema-rejection error text in stored events and transcripts with the existing redaction and 8 KiB limit. Keep the provider response unchanged. - Preserve the controller opt-in, tool-detail and tool-content switches, redaction, limits and run identity. - Add MCP-to-receiver regression coverage and update the capture policy and capability maps. ### Hypothesis and thesis for a fix The exporter drops the dispatch diagnostic signal. Raw input omission is intentional. The repair belongs in the OpenCode driver and exporter. Shared protocol, tool permissions and task completion stay unchanged. The regression sends schema-rejected, handler-rejected and valid argument envelopes through a fixture OpenCode subprocess and the authenticated MCP bridge. It checks the downstream rejection and receipt, completion event and actual local OTLP request. The baseline has no dispatch spans. The candidate has separate spans with safe type diagnostics. Two warm runs and disabled capture use the same path. Source: [OpenCode diagnostic pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/f172172e-9c65-453f-bb3a-7c6ba4f5a400). ## Verification - Baseline: `a92f616cd0a457c8ba85d994b24e1f92c2e4d1de`. The new integration regression fails because tool spans are absent. The same regression passes with this change. - Exporter and MCP bridge suites: 37 tests pass. Three final driver controls pass. Ten selected controls also passed before the final provider-record repair. Runner TypeScript typecheck and build pass after the review repair. - The review regression failed with two spans before the repair. It passes with three spans after the repair. Unauthenticated and unassigned calls do not reach the rejection observer. A broken observer cannot change the schema rejection. - The full OpenCode/exporter/MCP run passes 94 tests and fails one existing final-answer selection test. The same failure occurs in an untouched baseline checkout. It is not claimed as a pass. - A further filtered run reached the default five-second timeout in two existing continuation tests during concurrent repository checks. All five selected dispatch, warm-session and continuation controls pass with a 15-second test deadline. The earlier timeouts remain recorded. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Runner TypeScript checks and build pass. Full repository typecheck and build stop at Rust because `cargo` is not installed. The full repository test command was stopped before workspace handoff. No full-suite pass is claimed. Local source commits: `c48a8af1c186e415c802fa0ac95209b16b4a3122` `6744991f6`, and `a8ccf4693`. The remote branch is based on newer master `4814d1ab691c176170510a93a175dafdc9e021c7`. All initial eight changed files match between those bases before the patch. - The integration provider and controller are fixtures. The OTLP receiver is a real local HTTP server. A separate real OpenCode 1.18.34 probe opened a session. The remote free model returned `Free usage limit reached`. No tool call or receipt was produced. No live recovery is claimed. - Review follow-up: a real MCP request with 12,000 invalid array elements produced a 264,908-character event error before the repair. The same regression now stores 8,192 characters in both events and transcript. The MCP error receipt stays unchanged and no handler runs. All 37 exporter/MCP tests and 12 selected driver/exporter/MCP controls pass. Runner TypeScript checks and build pass. Local follow-up commit: `64d4fa583`. - The previous remote head passed Runner, typecheck, build and E2E. Its unchanged attention-retention shard failed twice with 15-second timeouts, then cleanup deadlock and duplicate fixture IDs. Both affected retention suites passed all 70 tests locally unchanged. The new head needs fresh CI and review; no full CI pass is claimed. - Current-master qualification (2026-11-06, merge `2959ecdb8d`): normally merge master `53645a0104`, including the existing test-only retention-fixture repair from #18774 reused by #18770. Keep the OpenCode diagnostics feature and upstream durable MCP and completion-feedback behavior. No test deadline or assertion changed. The three old CI timeouts did not reproduce locally: the four completed-blocker-history cases passed on the old head in 29.52 seconds, then passed on the merged head in 13.23 seconds. The fixture rollback/provenance regression also passed. All 116 exporter, MCP bridge and OpenCode driver fixture tests passed. Runner TypeScript checks and both TypeScript compilation targets passed; ACPX profiles match the release manifest and protocol-contract checks pass. These checks use local PostgreSQL, HTTP and process fixtures. No hosted provider call or deployed recovery is claimed. Fresh CI and review are pending for this merge. - Map-scope review follow-up (`61e662e179`): remove only the inherited diagnostics paragraphs from the company capability maps. This change adds logging detail within an existing capability. `node scripts/check-maps.mjs` and `git diff --check` pass. Production and test inputs are unchanged from the qualified merge; fresh CI and review are pending for the final head. - Test compiler follow-up (`6675b46377`): reproduce CI TS2722 with `tsc -p packages/paperclip-runner/tsconfig.test.json --noEmit`. The harness contract makes transcript retrieval optional. The OpenCode bounds regression now fails explicitly if that capability is absent, then inspects the narrowed method. The same complete Runner test compiler project passes, and the actual local schema-rejection fixture passes with all receipt and 8 KiB assertions retained. Production inputs are unchanged. Fresh final-head CI and review are pending. - Final-head qualification: `6675b4637743aa269e5214756b158e1e0a4561a5` passes all 56 applicable CI checks (three skipped, one neutral review). [CI result](https://github.com/paperclipai/paperclip/actions/runs/42253119577). Greptile gives 5/5 with no open findings. Hosted task recovery remains pending verified rollout. ## Risks - Authenticated spans use dispatch IDs. Provider records use separate names and a source attribute. Consumers must choose the source they need to avoid counting two observations of the same tool action. - Safe types cannot reconstruct nested arguments or explain every value-level validation failure. - Export remains best effort and bounded. Lost native frames no longer remove dispatch diagnostics, but abrupt process termination can still lose a turn. - Full local Rust checks and a deployed product journey are not yet verified. The pull request is unmerged and not deployed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The exact runtime variant and context-window size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/agent-observability.md | 20 +++- .../opencode/company-observability.test.ts | 49 ++++++++++ .../drivers/opencode/company-observability.ts | 47 +++++++-- .../src/drivers/opencode/mcp-bridge.test.ts | 34 +++++++ .../src/drivers/opencode/mcp-bridge.ts | 8 +- .../opencode/opencode-server-driver.test.ts | 97 +++++++++++++++++++ .../opencode/opencode-server-driver.ts | 19 ++++ .../test/fixtures/fake-opencode-server.mjs | 30 ++++++ 8 files changed, 291 insertions(+), 13 deletions(-) diff --git a/doc/agent-observability.md b/doc/agent-observability.md index c3b63085ba0..b6eec6c3df6 100644 --- a/doc/agent-observability.md +++ b/doc/agent-observability.md @@ -23,13 +23,27 @@ Metrics, raw API bodies and Codex Guardian assessments are disabled. Native OpenCode 1.18.34 uses a runner-owned HTTP/JSON exporter. Each completed, failed or cancelled turn produces a trace with Paperclip company, agent, run and -turn IDs. Tool outcomes and message-level token/cost receipts are child spans; +turn IDs. Authenticated Paperclip tool dispatch outcomes, native tool outcomes and +message-level token/cost receipts are child spans; repeated usage updates replace the same message receipt. Missing prices remain unknown. This projection excludes hidden reasoning, raw provider payloads and system instructions. Prompt, response, tool-detail and tool-content switches apply before export. Content is bounded to 8 KiB per field and 64 tools plus 64 -usage records per turn; truncation is marked. Tool inputs and full model request -bodies are not captured. OpenCode's performance-only OTLP sink stays separate. +usage records per turn; truncation is marked. With both tool-detail and tool-content +capture enabled, Paperclip tool spans include `braintrust.input` diagnostics +(`paperclip.opencode.tool-diagnostics.v1`): the upstream `tools/call` method, +argument JSON type and types for a fixed list of envelope fields. These are +shape diagnostics, not a copy of the arguments. Arbitrary field names, argument +values, nested payloads and full model request bodies are not captured. +Authenticated schema rejections also produce failed tool spans without invoking +the controller handler. Conflicting request IDs produce separate failed items, +so a rejection cannot overwrite a previous dispatch outcome. +Paperclip tool spans use authenticated dispatch IDs and the name `tool `. +Provider SSE records for those declared tools use `opencode tool ` and +remain available when failure occurs before dispatch, such as an oversized +request. These are separate observations, not correlated calls. The +`paperclip.tool.source` attribute distinguishes dispatch and provider records. +Native tools retain their provider spans without submitted-argument diagnostics. OpenCode's performance-only OTLP sink stays separate. The exporter sends once at turn completion with a three-second deadline, at most four requests in flight and no idle polling or persistent queue. Closing drains diff --git a/packages/paperclip-runner/src/drivers/opencode/company-observability.test.ts b/packages/paperclip-runner/src/drivers/opencode/company-observability.test.ts index e6729350f9f..54120e67f2e 100644 --- a/packages/paperclip-runner/src/drivers/opencode/company-observability.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/company-observability.test.ts @@ -117,3 +117,52 @@ it("does not block event delivery when a receiver stalls and cancels the request expect(aborted).toBe(true); expect(diagnostics).toEqual(["Company observability export unavailable (export_failed)."]); }, 5000); + +it("bounds dispatch diagnostics without traversing arbitrary keys or nested values", async () => { + const { observer, bodies } = setup(); + observer.observe(event("turn.submitted")); + const args = { name: "provider-secret", arguments: { secret: "x".repeat(100000) }, ["private-key".repeat(10000)]: true }; + observer.observe(event("item.started", { kind: "dynamicToolCall", item: { name: "paperclip_call_assigned_tool", arguments: args } }, "run-one", "call")); + observer.observe(event("item.completed", { kind: "dynamicToolCall", item: { result: { message: "provider-secret secret-canary" } } }, "run-one", "call")); + observer.observe(event("turn.completed")); + await observer.close(); + const spans = bodies[0].resourceSpans[0].scopeSpans[0].spans; + const attributes = spanAttributes(spans[1]); + expect(JSON.parse(String(attributes["braintrust.input"]))).toEqual({ schema: "paperclip.opencode.tool-diagnostics.v1", method: "tools/call", argumentType: "object", fields: { name: "string", arguments: "object" } }); + expect(attributes["braintrust.output"]).toBe('{"message":"[REDACTED] [REDACTED]"}'); + expect(Buffer.byteLength(String(attributes["braintrust.input"]))).toBeLessThan(1024); + expect(JSON.stringify(bodies)).not.toContain("private-key"); +}); +it.each([null, [], "private-string"])("exports only the type for non-object arguments %j", async args => { + const { observer, bodies } = setup(); + observer.observe(event("turn.submitted")); + observer.observe(event("item.started", { kind: "dynamicToolCall", item: { name: "tool", arguments: args } }, "run-one", "call")); + observer.observe(event("turn.completed")); + await observer.close(); + const spans = bodies[0].resourceSpans[0].scopeSpans[0].spans; + expect(JSON.parse(String(spanAttributes(spans[1])["braintrust.input"])).fields).toEqual({}); + expect(JSON.stringify(bodies)).not.toContain("private-string"); +}); +it("does not export dispatch spans when tool details are disabled", async () => { + const { observer, bodies } = setup({ ...environment, OTEL_LOG_TOOL_DETAILS: "0" }); + observer.observe(event("turn.submitted")); + observer.observe(event("item.started", { kind: "dynamicToolCall", item: { name: "tool", arguments: { name: "private-name" } } }, "run-one", "call")); + observer.observe(event("turn.completed")); + await observer.close(); + expect(bodies[0].resourceSpans[0].scopeSpans[0].spans).toHaveLength(1); + expect(JSON.stringify(bodies)).not.toContain("private-name"); +}); + +it("retains provider failures for declared tools even before any dispatch", async () => { + const bodies: any[] = []; + const observer = createOpenCodeCompanyObservability({ environment, model: "test", sensitiveValues: [], managedToolNames: ["paperclip_call_assigned_tool"], + fetch: async (_url, init) => { bodies.push(JSON.parse(String(init?.body))); return new Response("{}"); } })!; + observer.observe(event("turn.submitted")); + observer.observe(event("tool.execution.completed", { name: "paperclip_call_assigned_tool", status: "failed", output: "request exceeded limit" }, "run-one", "native-rejection")); + observer.observe(event("turn.completed")); + await observer.close(); + const spans = bodies[0].resourceSpans[0].scopeSpans[0].spans; + expect(spans[1].name).toBe("opencode tool paperclip_call_assigned_tool"); + expect(spans[1].status.code).toBe(2); + expect(spanAttributes(spans[1])).toMatchObject({ "paperclip.tool.source": "provider", "braintrust.output": "request exceeded limit" }); +}); diff --git a/packages/paperclip-runner/src/drivers/opencode/company-observability.ts b/packages/paperclip-runner/src/drivers/opencode/company-observability.ts index 4ea7cda93c6..479dc2cdcd1 100644 --- a/packages/paperclip-runner/src/drivers/opencode/company-observability.ts +++ b/packages/paperclip-runner/src/drivers/opencode/company-observability.ts @@ -23,9 +23,27 @@ function pairs(value: string | undefined): Record { })); } +function argumentDiagnostics(value: unknown): string { + const type = (entry: unknown) => entry === null ? "null" : Array.isArray(entry) ? "array" : typeof entry; + const fields = record(value); + return JSON.stringify({ + schema: "paperclip.opencode.tool-diagnostics.v1", method: "tools/call", + argumentType: type(value), + fields: Object.fromEntries([ + "name", "arguments", "schema", "reportedWorkDisposition", "summary", + "completionClaim", "verification", "evidence", "artifacts", "continuation", "blocker", + ].filter(key => Object.hasOwn(fields, key)).map(key => [key, type(fields[key])])), + }); +} +function outputText(value: unknown): string { + if (typeof value === "string") return value; + try { return JSON.stringify(value) ?? ""; } catch { return ""; } +} + /** A controller opt-in, not OpenCode's raw OTLP. Projected PRP events never include reasoning or raw provider payloads. */ export function createOpenCodeCompanyObservability(input: { environment: NodeJS.ProcessEnv | undefined; model: string; sensitiveValues: readonly string[]; + managedToolNames?: readonly string[]; fetch?: typeof globalThis.fetch; onDiagnostic?: (message: string) => void; }) { const env = input.environment; @@ -42,6 +60,8 @@ export function createOpenCodeCompanyObservability(input: { ["paperclip.company.id", "paperclip.agent.id", "paperclip.observability.revision"] .filter(key => supplied[key]).map(key => [key, supplied[key]])) }; } catch { diagnostic("configuration_invalid"); return undefined; } + const managedTools = new Set(["paperclip_finish", "paperclip_block", ...(input.managedToolNames ?? [])] + .flatMap(name => [name, `paperclip_${name}`])); const capture = (key: string) => env[key] === "1"; const clean = (value: unknown, limit = maxTextBytes): string => { if (typeof value !== "string") return ""; @@ -123,19 +143,28 @@ export function createOpenCodeCompanyObservability(input: { if (event.eventType === "item.completed" && payload.kind === "agentMessage" && capture("OTEL_LOG_ASSISTANT_RESPONSES")) { turn.root.attributes["braintrust.output"] = clean(payload.text); } - if (event.eventType.startsWith("tool.execution.") && event.itemId && capture("OTEL_LOG_TOOL_DETAILS")) { - let span = turn.tools.get(event.itemId); + const dynamic = payload.kind === "dynamicToolCall" && ["item.started", "item.completed"].includes(event.eventType); + const item = record(payload.item); + const canonicalTool = event.eventType.startsWith("tool.execution."); + if ((dynamic || canonicalTool) && event.itemId && capture("OTEL_LOG_TOOL_DETAILS")) { + const key = `${dynamic ? "dispatch" : "native"}:${event.itemId}`; + let span = turn.tools.get(key); if (!span && turn.tools.size < maxRows) { - const name = clean(payload.name, 128); - span = makeSpan(`tool ${name || "execution"}`, time, { - "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name }); - turn.tools.set(event.itemId, span); + const name = clean(dynamic ? item.name : payload.name, 128); + span = makeSpan(`${!dynamic && managedTools.has(String(payload.name)) ? "opencode tool" : "tool"} ${name || "execution"}`, time, { + "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name, + "paperclip.tool.source": dynamic ? "dispatch" : "provider" }); + turn.tools.set(key, span); } if (span) { - span.end = time; span.failed = payload.status === "failed"; + span.end = time; span.failed = dynamic ? item.is_error === true : payload.status === "failed"; if (capture("OTEL_LOG_TOOL_CONTENT")) { - span.attributes["braintrust.output"] = clean(payload.output); - if (payload.outputTruncated === true) turn.truncated = true; + if (dynamic && event.eventType === "item.started") { + span.attributes["braintrust.input"] = argumentDiagnostics(item.arguments); + } else { + span.attributes["braintrust.output"] = clean(dynamic ? outputText(item.result ?? item.error) : payload.output); + } + if (!dynamic && payload.outputTruncated === true) turn.truncated = true; } } else turn.truncated = true; } diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts index 94bc3056fbd..3511eb73cd7 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts @@ -326,3 +326,37 @@ describe("OpenCode MCP bridge", () => { }); }); }); + +it("reports admitted schema rejections without invoking handlers or widening authentication", async () => { + const handler = vi.fn(async () => ({ accepted: true })); + const rejected = vi.fn(); + const bridge = await startOpenCodeMcpBridge({ + tools: [{ name: "probe", inputSchema: { type: "object" } }], handler, onRejectedCall: rejected, + }); + bridges.push(bridge); + const body = { id: "bad", method: "tools/call", params: { name: "probe", arguments: [] } }; + expect((await rpc(bridge, body, "wrong")).status).toBe(401); + await rpc(bridge, { ...body, params: { name: "unassigned", arguments: [] } }); + expect(rejected).not.toHaveBeenCalled(); + expect(await (await rpc(bridge, body)).json()).toMatchObject({ result: { isError: true } }); + expect(rejected).toHaveBeenCalledExactlyOnceWith({ tool: "probe", callId: "bad", arguments: [] }, expect.stringContaining("Invalid tool input")); + expect(handler).not.toHaveBeenCalled(); +}); +it("keeps schema rejection intact when its observer fails", async () => { + const handler = vi.fn(async () => ({})); + const bridge = await startOpenCodeMcpBridge({ handler, onRejectedCall: () => { throw new Error("observer unavailable"); } }); + bridges.push(bridge); + expect(await (await rpc(bridge, { id: 1, method: "tools/call", params: { name: "paperclip_finish", arguments: {} } })).json()).toMatchObject({ result: { isError: true } }); + expect(handler).not.toHaveBeenCalled(); +}); +it("reports conflicting request IDs without repeating the accepted action", async () => { + const handler = vi.fn(async () => ({ accepted: true })); + const rejected = vi.fn(); + const bridge = await startOpenCodeMcpBridge({ tools: [{ name: "probe", inputSchema: { type: "object" } }], handler, onRejectedCall: rejected }); + bridges.push(bridge); + const body = { id: "same", method: "tools/call", params: { name: "probe", arguments: { name: "first" } } }; + await rpc(bridge, body); + expect(await (await rpc(bridge, { ...body, params: { name: "probe", arguments: { name: "second" } } })).json()).toMatchObject({ result: { isError: true } }); + expect(handler).toHaveBeenCalledOnce(); + expect(rejected).toHaveBeenCalledExactlyOnceWith({ tool: "probe", callId: "same", arguments: { name: "second" } }, "Duplicate call identity conflict."); +}); diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts index 419943a8dfd..81e854d2a83 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts @@ -29,6 +29,7 @@ export interface OpenCodeMcpBridgeOptions { /** Runner-owned operations callable only by a trusted harness extension and never advertised to the model. */ privateTools?: readonly Readonly>[]; handler: (call: OpenCodeMcpCall) => Promise; + onRejectedCall?: (call: Pick, error: string) => void; /** Null leaves durable requests with the owning Runner lease and bridge close. */ timeoutMs?: number | null; /** Human-gated private operations may remain pending much longer than model tool calls. */ @@ -103,6 +104,7 @@ export async function startOpenCodeMcpBridge( calls, controllers, handler: options.handler, + onRejectedCall: options.onRejectedCall, diagnoseFinishInput, timeoutMs: options.timeoutMs === null ? null : options.timeoutMs ?? DEFAULT_TIMEOUT_MS, privateToolTimeoutMs: options.privateToolTimeoutMs ?? options.timeoutMs ?? DEFAULT_TIMEOUT_MS, @@ -186,6 +188,7 @@ async function handleRequest( calls: Map }>; controllers: Map; handler: OpenCodeMcpBridgeOptions["handler"]; + onRejectedCall: OpenCodeMcpBridgeOptions["onRejectedCall"]; timeoutMs: number | null; privateToolTimeoutMs: number; privateToolNames: ReadonlySet; @@ -283,15 +286,18 @@ async function handleRequest( context.diagnoseFinishInput({ input: params.arguments, callId, validationCategory }); } const detail = validator?.errors?.map((error) => `${error.instancePath || "/"} ${error.message ?? "is invalid"}`).join("; "); + const error = `Invalid tool input${detail ? `: ${detail}` : "."}`; + try { context.onRejectedCall?.({ tool, callId, arguments: args }, error); } catch {} writeRpc(response, id, { isError: true, - content: [{ type: "text", text: `Invalid tool input${detail ? `: ${detail}` : "."}` }], + content: [{ type: "text", text: error }], }); return; } const fingerprint = canonicalJson({ tool, args }); const existing = context.calls.get(callId); if (existing && existing.fingerprint !== fingerprint) { + try { context.onRejectedCall?.({ tool, callId, arguments: args }, "Duplicate call identity conflict."); } catch {} writeRpc(response, id, { isError: true, content: [{ type: "text", text: "Duplicate call identity conflict." }] }); return; } diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index 0613569216a..840d36f97c7 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -154,6 +154,103 @@ afterAll(async () => { }); describe("OpenCodeServerDriver", () => { + it("bounds schema rejection errors in events and transcript without changing the MCP receipt", async () => { + await chmod(fixture, 0o755); + const root = await mkdtemp(join(tmpdir(), "paperclip-large-rejection-")); roots.push(root); + const handler = vi.fn(); + const driver = new OpenCodeServerDriver({ model: "openrouter/test-model", command: fixture, runtimeDirectory: root, + environment: { PATH: process.env.PATH }, + dynamicTools: [{ name: "large_call", inputSchema: { type: "array", items: { type: "string" } } }], + dynamicToolHandler: handler, + }); + const session = await driver.openSession({ runId: "large-rejection", normalizedSessionId: "large-rejection", workingDirectory: root }); + try { + await session.startTurn({ message: { role: "user", text: "oversized-schema-rejection" } }); + const events = await collectTurnEvents(session.events()); + expect(events.at(-1)?.eventType).toBe("turn.completed"); + const receipt = JSON.parse(await readFile(join(root, "fake-large-rejection-receipt.json"), "utf8")); + expect(receipt.result.isError).toBe(true); + expect(receipt.result.content[0].text.length).toBeGreaterThan(8_192); + expect(handler).not.toHaveBeenCalled(); + if (!session.transcript) throw new Error("OpenCode session must expose its transcript"); + for (const recorded of [events, (await session.transcript()).events]) { + const errors = recorded.filter(event => event.eventType === "item.completed") + .map(event => event.payload.item as { is_error?: boolean; error?: string }).filter(item => item?.is_error); + expect(errors).toHaveLength(1); + expect(errors[0]!.error).toContain("Invalid tool input"); + expect(errors[0]!.error).toHaveLength(8_192); + } + } finally { await session.close({ reason: "test" }); } + }); + + it.each(["1", "0"])("exports safe dispatch argument diagnostics with tool content=%s", async capture => { + await chmod(fixture, 0o755); + const bodies: any[] = []; + const collector = createHttpServer((request, response) => { + const chunks: Buffer[] = []; + request.on("data", chunk => chunks.push(chunk)); + request.on("end", () => { bodies.push(JSON.parse(Buffer.concat(chunks).toString("utf8"))); response.end("{}"); }); + }); + await new Promise(resolve => collector.listen(0, "127.0.0.1", resolve)); + const address = collector.address(); + if (!address || typeof address === "string") throw new Error("collector did not bind"); + const root = await mkdtemp(join(tmpdir(), "paperclip-argument-diagnostics-")); roots.push(root); + const calls: unknown[] = []; + const driver = new OpenCodeServerDriver({ model: "openrouter/test-model", command: fixture, runtimeDirectory: root, + environment: { PATH: process.env.PATH, PAPERCLIP_OBSERVABILITY_OPENCODE: "1", + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: `http://127.0.0.1:${address.port}/v1/traces`, + OTEL_LOG_TOOL_DETAILS: "1", OTEL_LOG_TOOL_CONTENT: capture }, + taskEnvelope: createCodexTaskEnvelope({ objective: "argument-diagnostics" }), + dynamicTools: [{ name: "paperclip_call_assigned_tool", inputSchema: { type: "object" } }], + dynamicToolHandler: async call => { + calls.push(call.arguments); + if (typeof (call.arguments as { name?: unknown }).name !== "string") throw new Error("name must be a string"); + return { receipt: "benign-receipt" }; + }, + }); + let session: Awaited> | undefined; + try { + session = await driver.openSession({ runId: "first", normalizedSessionId: "diagnostics", workingDirectory: root }); + for (const runId of ["first", "second"]) { + if (runId === "second") session.attachRun!({ runId }); + await session.startTurn({ message: { role: "user", text: JSON.stringify({ message: "argument-diagnostics", completionContract: createCodexTaskEnvelope({ objective: "argument-diagnostics" }).completionContract }) } }); + const events = await collectTurnEvents(session.events()); + expect(events.at(-1)?.eventType).toBe("turn.completed"); + expect(events.filter(event => event.eventType === "run.result.proposed")).toHaveLength(1); + const receipts = JSON.parse(await readFile(join(root, "fake-diagnostic-receipts.json"), "utf8")); + expect(receipts[0].result.isError).toBe(true); + expect(receipts[1].result.isError).toBe(true); + expect(receipts[2].result.content[0].text).toContain("benign-receipt"); + } + await session.close({ reason: "test" }); session = undefined; + expect(calls).toHaveLength(4); + expect(bodies).toHaveLength(2); + for (const [index, body] of bodies.entries()) { + const spans = body.resourceSpans[0].scopeSpans[0].spans; + const attrs = (span: any) => Object.fromEntries(span.attributes.map((a: any) => [a.key, a.value.stringValue ?? a.value.boolValue])); + const tools = spans.filter((span: any) => span.name === "tool paperclip_call_assigned_tool"); + expect(tools).toHaveLength(3); + const native = spans.filter((span: any) => span.name === "opencode tool paperclip_call_assigned_tool"); + expect(native).toHaveLength(1); + expect(attrs(native[0])["paperclip.tool.source"]).toBe("provider"); + expect(attrs(native[0])).not.toHaveProperty("braintrust.input"); + expect(attrs(tools[0])["paperclip.tool.source"]).toBe("dispatch"); + + expect(tools.map((span: any) => span.status.code)).toEqual([2, 2, 1]); + expect(attrs(tools[0])["paperclip.run.id"]).toBe(index === 0 ? "first" : "second"); + if (capture === "1") { + expect(JSON.parse(attrs(tools[0])["braintrust.input"])).toMatchObject({ argumentType: "array", fields: {} }); + expect(JSON.parse(attrs(tools[1])["braintrust.input"])).toMatchObject({ schema: "paperclip.opencode.tool-diagnostics.v1", method: "tools/call", argumentType: "object", fields: { name: "number", arguments: "object" } }); + expect(JSON.parse(attrs(tools[2])["braintrust.input"]).fields.name).toBe("string"); + } else for (const tool of tools) expect(attrs(tool)).not.toHaveProperty("braintrust.input"); + } + for (const excluded of ["private-argument-canary", "private-key-canary", "private-value-canary", "benign-operation"]) expect(JSON.stringify(bodies)).not.toContain(excluded); + } finally { + await session?.close({ reason: "test" }); + await new Promise(resolve => { collector.close(() => resolve()); collector.closeAllConnections(); }); + } + }); + it("forwards configured bindings to the provider shell and redacts their values", async () => { const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-configured-env-")); roots.push(root); diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index d0d992ffc4a..9010a7a9646 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -90,6 +90,7 @@ import { canonicalOpenCodeMcpToolName, startOpenCodeMcpBridge, type OpenCodeMcpBridge, + type OpenCodeMcpBridgeOptions, } from "./mcp-bridge.js"; import { nativeMcpLaunchBinding } from "../native-mcp.js"; import { @@ -399,6 +400,7 @@ export class OpenCodeServerDriver implements HarnessDriver { root, cwd, trace, + onRejectedCall: (call, error) => session?.observeRejectedToolCall(call, error), dispatch: (call) => { if (session === null) throw new Error("OpenCode session is not ready for tool calls"); @@ -450,6 +452,7 @@ export class OpenCodeServerDriver implements HarnessDriver { snapshot, now: this.#options.now ?? (() => new Date()), observabilityEnvironment: this.#options.environment, + managedToolNames: this.#options.dynamicTools?.map(tool => text(tool.name)).filter(Boolean), onDiagnostic: this.#options.onDiagnostic, }); session.startEventPump(); @@ -573,11 +576,13 @@ class OpenCodeHarnessSession implements HarnessSession { snapshot: PersistedHarnessSession | null; now: () => Date; observabilityEnvironment?: NodeJS.ProcessEnv; + managedToolNames?: readonly string[]; onDiagnostic?: (message: string) => void; }) { this.#runtime = input.runtime; this.#companyObservability = createOpenCodeCompanyObservability({ environment: input.observabilityEnvironment, model: input.model, sensitiveValues: input.runtime.sensitiveValues, + managedToolNames: input.managedToolNames, onDiagnostic: input.onDiagnostic ?? ((message) => console.warn(message)), }); this.#fetch = input.fetcher; @@ -1128,6 +1133,18 @@ class OpenCodeHarnessSession implements HarnessSession { ); } + observeRejectedToolCall(call: { tool: string; callId: string; arguments: unknown }, error: string): void { + const turnId = this.#activeTurnId; + if (!turnId || this.#closed) return; + const itemId = `${call.callId}:rejected:${randomBytes(12).toString("hex")}`; + this.#emit("item.started", { + kind: "dynamicToolCall", item: { type: "tool_call", id: itemId, name: call.tool, arguments: call.arguments }, + }, { turnId, itemId }); + this.#emit("item.completed", { + kind: "dynamicToolCall", item: { type: "tool_result", id: itemId, tool_use_id: itemId, is_error: true, error: redact(error) }, + }, { turnId, itemId }); + } + async dispatchTool(call: { tool: string; callId: string; @@ -2425,6 +2442,7 @@ async function startRuntime(input: { root: string; cwd: string; trace: ProviderTraceFileSink | null; + onRejectedCall?: OpenCodeMcpBridgeOptions["onRejectedCall"]; dispatch: (call: { tool: string; callId: string; @@ -2463,6 +2481,7 @@ async function startRuntime(input: { const bridge = await startOpenCodeMcpBridge({ tools: input.options.dynamicTools, handler: input.dispatch, + onRejectedCall: input.onRejectedCall, onDiagnostic: input.options.onDiagnostic, ...(input.options.durableToolRequests ? { timeoutMs: null } : {}), }).catch(async (error) => { diff --git a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs index 7ed45de8fc3..f344de74f2d 100644 --- a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs +++ b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs @@ -535,6 +535,36 @@ const server = createServer(async (request, response) => { emit({ type: "session.idle", properties: { sessionID: session.id } }); return; } + if (String(parsedPrompt.message ?? "").includes("oversized-schema-rejection")) { + const receipt = await mcpRequest("tools/call", { + name: "paperclip_large_call", arguments: Array(12_000).fill(0), + }); + await writeFile(join(process.cwd(), "fake-large-rejection-receipt.json"), JSON.stringify(receipt)); + await callTerminalTool(promptPayload); + emit({ type: "session.idle", properties: { sessionID: session.id } }); + return; + } + if (String(parsedPrompt.message ?? parsedPrompt.task?.prompt ?? "").includes("argument-diagnostics")) { + const receipts = [await mcpRequest("tools/call", { + name: "paperclip_paperclip_call_assigned_tool", arguments: [], + })]; + for (const name of [42, "benign-operation"]) { + receipts.push(await mcpRequest("tools/call", { + name: "paperclip_paperclip_call_assigned_tool", + arguments: { name, arguments: { token: "private-argument-canary" }, "private-key-canary": "private-value-canary" }, + })); + } + emit({ type: "message.updated", id: "diagnostic-message-event", properties: { sessionID: session.id, + info: { id: "diagnostic-message", sessionID: session.id, role: "assistant" } } }); + emit({ type: "message.part.updated", id: "diagnostic-part-event", properties: { sessionID: session.id, + part: { id: "diagnostic-provider-row", messageID: "diagnostic-message", sessionID: session.id, + type: "tool", tool: "paperclip_paperclip_call_assigned_tool", + callID: "unrelated-provider-call", state: { status: "completed", output: "provider-receipt" } } } }); + await writeFile(join(process.cwd(), "fake-diagnostic-receipts.json"), JSON.stringify(receipts)); + await callTerminalTool(promptPayload); + emit({ type: "session.idle", properties: { sessionID: session.id } }); + return; + } if (!String(parsedPrompt.message ?? parsedPrompt.task?.prompt ?? "").includes("current-wake-completion-feedback")) await callFirstPaperclipTool(); if (String(parsedPrompt.message ?? "").includes("managed-github-operation")) { const result = await new Promise(resolve => {