fix(workspaces): preserve the read-only capture phase budget (#19803)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote workspace saves must preserve the files produced by a
completed turn.
> - When a sandbox disk is full, the existing fallback transfers a
stable source and seals it on the controller.
> - The fallback gives metadata checks and the portable codec the same
two-minute timer.
> - Slow but successful metadata checks can leave the codec only
milliseconds, although normal sandbox capture allows five minutes.
> - This change keeps the metadata budget and gives the local codec the
existing finite capture budget.
> - All source checks, archive checks, ownership checks, and failure
cleanup remain required.

## Linked Issues or Issue Description

Refs #19465. Related: #19364 and #19619. The open diagnostic change in
#19619 records fallback causes; it does not separate these phase
budgets. No duplicate budget fix was found.

**What happened?**

A retained workspace failed five read-only exports at `stage=archive`
with `transport_timeout`; four warnings also named `ABORT_ERR`. The
fallback had already transferred the source and passed the received-tree
and source inventory checks. Its controller codec received the same
two-minute signal used by those checks.

**Expected behavior**

Metadata checks share a finite two-minute budget. Sealing uses the same
finite five-minute policy as normal sandbox capture. The final source
check uses only the unused metadata budget. A changed source, expired
budget, or cancelled codec must not produce a preservation receipt.

**Steps to reproduce**

1. Create a standalone Git fixture with staged changes, an untracked
result, and a confined symlink.
2. Make the initial sandbox helper report the exact disk-full error.
3. Transfer real files and run the actual inventory scripts. Delay a
successful second inventory reply until just before the native
two-minute deadline.
4. Observe that the real portable codec starts with a live signal, then
aborts when that shared timer expires.

## What Changed

- Name and reuse the existing 300-second command policy for controller
capture.
- Retain the aggregate 120-second metadata budget. Preserve its unused
remainder for the final source recheck.
- Keep the exact disk-full error, source comparisons, file/Git checks,
and cleanup behavior.
- Add six real-filesystem and real-codec controls. Document the separate
phase limits and the current absence of a caller abort signal on the
save interface.
- Repair the existing onboarding test's fake-environment mapping with
the same real-agent PATCH mapping used by its sibling. Verify the
persisted agent through a real GET. Production UI code is unchanged.

## Verification

- Native-timer RED on base `35d41f11bb`: a successful source reply
arrived at 119,981.62 ms. Its command took 119,854.47 ms within its
119,873 ms cap. The actual codec started at 119,982.70 ms with a live
signal and aborted at 120,001.84 ms. Its nested cause matched the
retained warning: `stage=archive reason=operation_failed
transport=transport_timeout code=ABORT_ERR`. The source inventory was
unchanged and private staging was removed.
- The real-archive control sealed 10,166,395 bytes and 533 entries in
555.55 ms. It restored HEAD, staged data, untracked data, and the
symlink, and passed Git fsck. The recorded delay is synthetic; it is not
a measurement of the failed hosted workspace or provider throughput.
- The native RED leaves about 18 ms for the final check too. The
repaired code must still refuse that exact exhausted-metadata case; it
is not claimed as a native-clock GREEN.
- The committed fast regressions control phase time and delegate to the
actual inventory commands and portable codec. They cover successful
restore after the old metadata signal expires during sealing,
pre-capture metadata expiry, codec deadline/cancellation, and final
metadata expiry after a real completed archive.
- From `server/`: `node ../node_modules/vitest/vitest.mjs run
src/services/remote-workspace-readonly-capture-budget.test.ts
src/services/remote-workspace-transport.test.ts
src/services/remote-workspace-readonly-inventory.test.ts
src/services/remote-workspace-transfer-timeout.test.ts --fsModuleCache`
— 156 tests passed in 42.30 seconds before removal of a redundant test
duration assertion. The final-head six-case regression also passes after
rebasing onto `fff70fe49e`; only unrelated Runner and E2E files changed
in that rebase.
- Review follow-up `ad565dd984` makes the phase clock fully controlled.
It checks the requested final 2,000 ms budget separately from the real
subprocess's 30-second test limit. All six controls pass in 11.94
seconds, and server TypeScript passes again. The earlier
real-time-plus-offset clock could falsely fail on a loaded machine; that
review finding is fixed.
- Server TypeScript and module boundary checks pass. The first
TypeScript attempt could not resolve the unbuilt plugin SDK. Building
that existing workspace prerequisite resolved the setup errors; no
source workaround was added.
- The first preflight attempt used the host pnpm shim and was stopped.
The second lacked sandbox network access and was stopped. The canonical
cached preflight then completed in 22.1 seconds with the Runner build
restored from cache. A later test launch used the wrong working
directory for a test rewrite; its wrapper was stopped before waiting
through a second native-timer test. None of these interrupted attempts
is a pass.
- Final-head CI first failed one unchanged Runner-support privacy test
at 30 seconds: [job
failure](https://github.com/paperclipai/paperclip/actions/runs/42359560195/job/128365192635).
The other 6,864 tests passed and four were skipped. The failing test and
its report owners match base `fff70fe49e`; the exact case passes locally
unchanged in 2.97 seconds (one pass, 40 unselected). Job cleanup
reported an orphan `unzip`; that does not establish the timeout cause.
GitHub rejected the scoped retry while the parent workflow was still
running, so no retry has started. No old-head retry was started because
the separately proven onboarding fixture repair requires a new head and
a fresh full pipeline. The original Runner failure remains recorded.
- The same CI run failed onboarding case 160 at 60 seconds; its retained
page showed `Environment not found`. The fixture injected an environment
only in browser responses, then sent that fake ID to the real agent
update. The original case passed locally in 37.44 seconds despite
rejected HTTP 422 updates. Adding a real update-success guard produced a
local RED in 26.83 seconds (expected 200, got 422). Reusing the sibling
fixture's fake-ID-to-local-default PATCH mapping produced GREEN in 22.52
seconds. A real GET verifies the same persisted agent ID, name, and
default environment. Existing sign-in, cancellation, and no-hire
assertions are unchanged. CI screenshot/error context and the local
persistence RED remain retained.
- All other product checks on `ad565dd984` passed, including build,
TypeScript, all general and serialized server shards, and native Runner
checks. Final head `58bb4ee2411dfefca462f0e9b5a701d84ce1326e` passes the
fresh pipeline: [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42360927780).
Runner support passes in 3m36s and onboarding shard 5/8 passes in 5m46s,
with no Runner-support source change. The original failures remain part
of this verification history.
- Environment: macOS arm64, Node 26.4.0, real Node/Git/filesystem
operations. No provider or recovery calls, replayed turn, source
mutation, or hosted recovery acceptance was performed. Final-head
required CI and review pass; the exact gate state is recorded below.

- Exact-head final gate: 57 successful checks, three skipped, two
neutral, and no failures. Both required gates (`ci / verify` and `ci /
e2e`) pass. Greptile reviewed this exact head at 5/5, and zero review
threads remain open. The non-required companion `review` job is still
queued with no steps; it has not passed, and GitHub therefore reports
`UNSTABLE` while the PR remains mergeable.

## Risks

- Controller work after transfer can now take up to two minutes for
metadata plus five minutes for sealing. Each phase remains bounded.
Existing entry, repository, configuration, and transfer limits stay in
force.
- The save interface has no caller abort signal. This change adds no
Stop guarantee and does not replace the recovery retry window with an
operation deadline. Existing physical ownership and publication fences
remain in force.
- This proves the timer mechanism with a synthetic slow inventory. It
does not measure the actual dirty workspace size or show that its
archive finishes within five minutes. A guarded file-only recovery and
independent durable readback remain required after normal delivery.
- The onboarding fixture maps only its exact synthetic environment ID.
It still calls the real agent update and read routes. It does not loosen
production environment validation.
- No schema, provider transport, task status, recovery policy, or source
cleanup authority changes.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, repository tools,
code execution, and regression tests. The exact serving model ID and
context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (both required gates and all
product checks pass; non-required companion review remains queued as
stated above)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-07 03:58:24 -06:00
1 parent 2468b3f907
commit 27170dca49
4 files changed
+200 -6

No files matched your search

+7 -1
View File
@@ -1075,7 +1075,13 @@ transfer's own scratch paths and the explicit source exclusions. Older providers
that omit files fail inventory verification rather than publish an incomplete save.
Fallback inventory is limited to 100,000 entries, 64 standalone repositories,
1 MiB of configuration, 1,024 settings and 256 refspec/merge entries. Controller
verification and archive normalization share a two-minute cancellation deadline.
verification shares a two-minute budget across the received-tree check and both
source rechecks. Controller archive normalization has its own five-minute
cancellation deadline, matching normal sandbox capture. Time spent sealing does
not consume the remaining metadata verification budget. The fallback does not
extend a caller deadline or create new execution authority. Its save interface
currently has no caller abort signal; the existing source ownership checks still
fence durable publication.
These limits reject excessive work before publication and keep the source retained.
Inventory rechecks cancellation after file identity validation and before opening
the read stream. An expired deadline rejects the capture without an unhandled
@@ -0,0 +1,166 @@
import { execFile } from "node:child_process";
import { createHash, randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
import * as archiveOwner from "./remote-workspace-snapshot-archive.js";
import { readWorkspaceInventory } from "./remote-workspace-readonly-inventory.js";
import { captureRemoteWorkspaceArchive, REMOTE_WORKSPACE_SANDBOX_DISK_FULL, type RemoteWorkspaceTarget } from "./remote-workspace-transport.js";
const exec = promisify(execFile);
let root: string;
let source: string;
async function git(cwd: string, ...args: string[]) {
return exec("git", ["-C", cwd, "-c", "user.name=Budget fixture", "-c", "user.email=fixture@example.invalid",
"-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", ...args], {
maxBuffer: 32 * 1024 ** 2,
env: { PATH: process.env.PATH, GIT_CONFIG_GLOBAL: "/dev/null", GIT_CONFIG_NOSYSTEM: "1", GIT_TERMINAL_PROMPT: "0" },
});
}
beforeEach(async () => {
root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "readonly-capture-budget-")));
source = path.join(root, "source");
await fs.mkdir(source);
await git(source, "init", "--initial-branch=main");
for (let directory = 0; directory < 10; directory++) {
const current = path.join(source, `part-${directory}`);
await fs.mkdir(current);
for (let file = 0; file < 40; file++) await fs.writeFile(path.join(current, `item-${file}.txt`), Buffer.alloc(4096, file));
}
await fs.writeFile(path.join(source, "result.bin"), Buffer.alloc(8 * 1024 ** 2, 97));
await git(source, "add", ".");
await git(source, "commit", "-m", "Synthetic source");
await fs.writeFile(path.join(source, "result.bin"), Buffer.alloc(8 * 1024 ** 2, 98));
await git(source, "add", "result.bin");
await fs.writeFile(path.join(source, "untracked.txt"), "final dirty workspace bytes\n");
await fs.symlink("untracked.txt", path.join(source, "result-link"));
});
afterEach(async () => {
vi.restoreAllMocks();
await fs.rm(root, { recursive: true, force: true });
});
it("seals and restores the actual standalone source within a fresh existing verification budget", async () => {
const before = await readWorkspaceInventory(source);
const archivePath = path.join(root, "control.pcws");
const saved = await archiveOwner.captureRemoteWorkspaceDirectory({ cwd: source, archivePath,
exclude: before.exclude, signal: AbortSignal.timeout(120_000) });
const restored = path.join(root, "restored");
await archiveOwner.restoreRemoteWorkspaceArchive({ archivePath, targetDir: restored, expectedManifestDigest: saved.manifestDigest });
expect((await git(restored, "rev-parse", "HEAD")).stdout).toBe((await git(source, "rev-parse", "HEAD")).stdout);
expect(createHash("sha256").update(await fs.readFile(path.join(restored, "result.bin"))).digest("hex"))
.toBe(createHash("sha256").update(await fs.readFile(path.join(source, "result.bin"))).digest("hex"));
expect((await git(restored, "show", ":result.bin")).stdout).toBe((await git(source, "show", ":result.bin")).stdout);
expect(await fs.readFile(path.join(restored, "untracked.txt"), "utf8")).toBe("final dirty workspace bytes\n");
expect(await fs.readlink(path.join(restored, "result-link"))).toBe("untracked.txt");
await git(restored, "fsck", "--full");
expect(await readWorkspaceInventory(source)).toEqual(before);
}, 30_000);
it.each(["separate_capture", "inventory_expired", "capture_deadline", "capture_cancelled", "final_inventory_expired"] as const)(
"keeps read-only capture phase budgets and source safety: %s", async scenario => {
const before = await readWorkspaceInventory(source);
let advancedMs = 0;
vi.spyOn(performance, "now").mockImplementation(() => advancedMs);
const deadlines: Array<{ ms: number; controller: AbortController }> = [];
vi.spyOn(AbortSignal, "timeout").mockImplementation(ms => {
const controller = new AbortController();
deadlines.push({ ms, controller });
return controller.signal;
});
const actualCapture = archiveOwner.captureRemoteWorkspaceDirectory;
let archiveCompleted = false;
const capture = vi.spyOn(archiveOwner, "captureRemoteWorkspaceDirectory").mockImplementation(async input => {
expect(deadlines[0]!.ms).toBe(120_000);
expect(deadlines[1]!.ms).toBe(300_000);
expect(input.signal).toBe(deadlines[1]!.controller.signal);
advancedMs += 5_000;
deadlines[0]!.controller.abort(new DOMException("Metadata timeout", "TimeoutError"));
let interruption: ReturnType<typeof setTimeout> | undefined;
if (scenario === "capture_deadline" || scenario === "capture_cancelled") {
interruption = setTimeout(() => {
advancedMs += 300_000;
deadlines[1]!.controller.abort(new DOMException("Capture interrupted",
scenario === "capture_deadline" ? "TimeoutError" : "AbortError"));
}, 10);
}
try {
const result = await actualCapture(input);
archiveCompleted = true;
return result;
} finally { if (interruption) clearTimeout(interruption); }
});
let inventoryCalls = 0;
let copiedRoot = "";
const execute: CommandManagedRuntimeRunner["execute"] = async input => {
if (!input.args?.[1]?.includes("const inspect=")) return { exitCode: 1, stdout: "",
stderr: REMOTE_WORKSPACE_SANDBOX_DISK_FULL, timedOut: false, signal: null,
pid: process.pid, startedAt: new Date().toISOString() };
const result = await exec(input.command, input.args, { cwd: input.cwd, timeout: 30_000,
maxBuffer: 4 * 1024 ** 2, env: { PATH: process.env.PATH, ...input.env } });
inventoryCalls++;
if (inventoryCalls === 2) {
advancedMs += 118_000;
if (scenario === "inventory_expired") deadlines[0]!.controller.abort(new DOMException("Metadata timeout", "TimeoutError"));
}
if (inventoryCalls === 3) {
expect(input.timeoutMs).toBe(2_000);
if (scenario === "final_inventory_expired") {
advancedMs += 2_000;
deadlines[2]!.controller.abort(new DOMException("Metadata timeout", "TimeoutError"));
}
}
return { ...result, exitCode: 0, timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() };
};
const syncOut: NonNullable<CommandManagedRuntimeRunner["syncOut"]> = async operations => {
for (const operation of operations) for (const file of operation.files) {
copiedRoot = file.targetPath;
await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true,
filter: current => !file.exclude?.some(excluded => {
const relative = path.relative(file.sourcePath, current).split(path.sep).join("/");
return relative === excluded || relative.startsWith(`${excluded}/`);
}) });
}
return { operations: operations.map(operation => ({ operationId: operation.operationId,
filesTransferred: operation.files.length, bytesTransferred: before.byteSize })) };
};
const target: RemoteWorkspaceTarget = { kind: "remote", transport: "sandbox", remoteCwd: source,
runner: { execute, syncOut, syncIn: syncOut } };
let failure: unknown;
const saved = await captureRemoteWorkspaceArchive({ target, remoteCwd: source, operationId: randomUUID() })
.catch(error => { failure = error; return null; });
try {
expect(await readWorkspaceInventory(source)).toEqual(before);
if (scenario === "separate_capture") {
expect(failure).toBeUndefined();
expect(saved).not.toBeNull();
expect(archiveCompleted).toBe(true);
expect(inventoryCalls).toBe(3);
expect(deadlines.map(item => item.ms).slice(0, 2)).toEqual([120_000, 300_000]);
const restored = path.join(root, "restored-fallback");
await archiveOwner.restoreRemoteWorkspaceArchive({ archivePath: saved!.archivePath,
targetDir: restored, expectedManifestDigest: saved!.manifestDigest });
expect(await fs.readFile(path.join(restored, "untracked.txt"), "utf8")).toBe("final dirty workspace bytes\n");
expect(await fs.readlink(path.join(restored, "result-link"))).toBe("untracked.txt");
expect((await git(restored, "rev-parse", "HEAD")).stdout).toBe((await git(source, "rev-parse", "HEAD")).stdout);
expect((await git(restored, "show", ":result.bin")).stdout).toBe((await git(source, "show", ":result.bin")).stdout);
await git(restored, "fsck", "--full");
} else {
expect(saved).toBeNull();
expect(failure).toBeInstanceOf(Error);
expect((failure as Error).message).toBe(REMOTE_WORKSPACE_SANDBOX_DISK_FULL);
const stage = scenario === "inventory_expired" || scenario === "final_inventory_expired" ? "verify" : "archive";
expect((failure as Error).cause).toMatchObject({ message: expect.stringContaining(`stage=${stage} reason=operation_failed`) });
expect(await fs.lstat(copiedRoot).catch(() => null)).toBeNull();
if (scenario === "inventory_expired") expect(capture).not.toHaveBeenCalled();
if (scenario === "final_inventory_expired") expect(archiveCompleted).toBe(true);
if (scenario.startsWith("capture_")) expect(archiveCompleted).toBe(false);
}
} finally { await saved?.cleanup(); }
}, 30_000);
@@ -203,8 +203,10 @@ export function sandboxProviderFailureLabel(error: unknown): { label: string; co
return { label, code, status };
}
const WORKSPACE_COMMAND_TIMEOUT_MS = 300_000;
async function command(target: RemoteWorkspaceTarget, operation: RemoteWorkspaceOperation,
args: string[], env?: Record<string, string>, timeoutMs = 300_000) {
args: string[], env?: Record<string, string>, timeoutMs = WORKSPACE_COMMAND_TIMEOUT_MS) {
const execute = () => target.runner!.execute({ command: "node", args, cwd: target.remoteCwd,
env: { NODE_OPTIONS: "", NODE_PATH: "", ...env }, timeoutMs, bypassSession: true });
// Copy-back and restore record why the provider refused the command.
@@ -466,7 +468,7 @@ async function captureReadOnlyWorkspaceArchive(input: {
target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string;
}) {
if (!/^[a-f0-9-]{36}$/.test(input.operationId)) throw new Error("remote_workspace_operation_identity_invalid");
const inspect = async (timeoutMs = 300_000) => {
const inspect = async (timeoutMs = WORKSPACE_COMMAND_TIMEOUT_MS) => {
const value = JSON.parse(await command(input.target, "capture", ["-e", readonlyWorkspaceInventoryScript(), input.remoteCwd], undefined, timeoutMs)) as Awaited<ReturnType<typeof readWorkspaceInventory>>;
if (!/^[a-f0-9]{64}$/.test(value.sha256) || !Number.isSafeInteger(value.byteSize) || value.byteSize < 0
|| !Number.isSafeInteger(value.entries) || value.entries < 0 || !Array.isArray(value.exclude)
@@ -495,11 +497,15 @@ async function captureReadOnlyWorkspaceArchive(input: {
const same = (observed: typeof before) => observed.sha256 === before.sha256
&& observed.byteSize === before.byteSize && observed.entries === before.entries;
if (!same(received) || !same(await inspect(remaining()))) throw new Error("remote_workspace_readonly_capture_changed");
const finalVerificationBudget = remaining();
stage = "archive";
const receipt = await captureRemoteWorkspaceDirectory({ cwd: copy, archivePath, exclude: before.exclude, signal: verificationSignal });
const captureSignal = AbortSignal.timeout(WORKSPACE_COMMAND_TIMEOUT_MS);
const receipt = await captureRemoteWorkspaceDirectory({ cwd: copy, archivePath, exclude: before.exclude, signal: captureSignal });
captureSignal.throwIfAborted();
stage = "verify";
if (!same(await inspect(remaining()))) throw new Error("remote_workspace_readonly_capture_changed");
verificationSignal.throwIfAborted();
const finalVerificationSignal = AbortSignal.timeout(finalVerificationBudget);
if (!same(await inspect(finalVerificationBudget))) throw new Error("remote_workspace_readonly_capture_changed");
finalVerificationSignal.throwIfAborted();
// Only the sealed portable archive remains a candidate for durable upload.
await removeTaskDirectoryTree(copy);
const savedRoot = root;
+16
View File
@@ -240,6 +240,18 @@ test.describe("Onboarding wizard", () => {
await route.fulfill({ response });
});
let persistedAgentId = "";
let persistedAgentStatus = 0;
await page.route("**/api/agents/*", async route => {
if (route.request().method() !== "PATCH") return route.continue();
const body = route.request().postDataJSON();
if (body.defaultEnvironmentId !== FAKE_SANDBOX_ENVIRONMENT_ID) return route.continue();
const response = await route.fetch({ postData: JSON.stringify({ ...body, defaultEnvironmentId: null }) });
persistedAgentId = new URL(route.request().url()).pathname.split("/").at(-1)!;
persistedAgentStatus = response.status();
await route.fulfill({ response });
});
// The sign-in Connect now starts. Stubbed so the card is deterministic:
// the session start answers, and the guarded prompt read hands back an
// authorization URL for the card's link row.
@@ -346,6 +358,10 @@ test.describe("Onboarding wizard", () => {
const cardLink = page.getByRole("link", { name: /^Sign in to / });
await expect(cardLink).toBeVisible({ timeout: 15_000 });
await expect(cardLink).toHaveAttribute("href", /claude\.ai\/oauth\/authorize/);
await expect.poll(() => persistedAgentStatus).toBe(200);
const persisted = await page.request.get(`/api/agents/${persistedAgentId}`);
expect(persisted.ok()).toBe(true);
expect(await persisted.json()).toMatchObject({ id: persistedAgentId, name: "Ada", defaultEnvironmentId: null });
await expect(
page.getByRole("button", { name: /^Sign in to / }),
).toBeEnabled({ timeout: 15_000 });