mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 19:28:46 +01:00
fix(workspaces): identify remote command timeouts (#18827)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces prepare repositories and save files through sandbox commands. > - Each command has a timeout and a known operation. > - The transport discarded that information when a command timed out. > - A failed clone then appeared as the same generic error as other failures. > - This pull request keeps the operation and timeout in a fixed diagnostic. > - Operators can identify the failed stage without exposing command output. ## Linked Issues or Issue Description **What happened?** A remote workspace setup command reached its 300-second transport limit before provider work started. The workspace layer reported only `remote_workspace_operation_failed`. That message concealed the timeout and the affected operation. A later full clone on the same sandbox image passed in 107 seconds, so the original clone delay remains unexplained. This PR repairs the loss of diagnostic information. It does not claim to prevent the underlying delay. **Expected behavior** A transport timeout must remain a failure. Its diagnostic must identify the operation and configured limit. It must exclude raw output and must not automatically repeat a command whose remote outcome is unknown. **Steps to reproduce** 1. Return `timedOut: true` from the sandbox command boundary during workspace preparation or cleanup. 2. Observe that the base implementation reports only `remote_workspace_operation_failed`. 3. Apply this change. Preparation reports `remote_workspace_operation_timed_out: prepare (configured timeout 300000ms)`. Cleanup retains its separate 10-second limit. **Paperclip version or commit** The regression was reproduced on master base `367f29478d14003cc5c0f161b7ce8f3164993634`. The fixed current head is `be45ce6594254bd6b94023a550d8b655dd704bb0`. **Deployment mode** Hosted Paperclip with native Runner and Daytona sandbox execution. Local acceptance also uses disposable PostgreSQL and filesystem commands. Related work: #18403 introduced remote workspace authority. #18705 added a fixed full-disk diagnostic. I searched recent clone and remote-workspace PRs and found no duplicate timeout-reporting repair. This is a defect fix and does not add a roadmap feature. ## What Changed - Give each internal workspace command a closed operation name: prepare, repository, archive helper, capture, restore, or cleanup. - Preserve that operation and the configured limit in a typed error when the transport reports a timeout. Do not claim a measured elapsed time. - Keep other failure diagnostics and exact safe-message checks unchanged. - Preserve typed cleanup timeouts in the operation cleanup journal while retaining generic diagnostics for untrusted errors. - Add preparation, cleanup, and real-PostgreSQL lifecycle regressions. Document the timeout and the need to reconcile saved receipts before recovery. ## Verification - Red: both transport regressions failed on the base implementation. A separate real-PostgreSQL cleanup regression reproduced the review finding: the operation journal replaced a typed timeout with a generic error. - Green: all 62 remote transport and archive tests and all 50 real-PostgreSQL workspace lifecycle cases passed. The lifecycle suite ran with an explicit disposable database; it was not skipped. Command: `pnpm exec vitest run --project @paperclipai/server server/src/services/remote-workspace-transport.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts`. - Safety: a timeout with exit code zero still fails. Raw stdout and stderr stay out of the error. A timeout cannot claim the fixed unfinished-Git diagnostic. No command retry occurs. Existing saved files remain unchanged. Existing archive, restore, receipt, symlink, credential-isolation, and full-disk controls pass. The cleanup lifecycle cases retain the acknowledged snapshot, preserve working files, avoid duplicate uploads, record only a safe typed timeout, and settle cleanup on the next normal reconciliation. An untrusted error that mimics the timeout prefix stays generic. - Live acceptance: production transport code ran against actual Daytona SDK 0.203.0 timeouts on the incident image, with 4 CPUs, 8 GiB RAM, and a 10 GiB disk. A controlled one-second SDK limit produced the generic baseline error and the fixed exact preparation timeout. The current `be45ce6594` code was rechecked for both preparation and cleanup. Each returned the typed error and its own configured limit (300 seconds and 10 seconds) while the controlled SDK fault remained one second. Each case made one command call. Each disposable sandbox was deleted before the next case. - The first live-probe attempt used an incorrect error-name check. It failed its acceptance check and remains recorded. The probe was corrected to use the same SDK timeout base class as production. Its first sandbox was deleted. This was a probe setup failure, not red evidence for the repair. - A separate same-image probe ran the real full repository preparation successfully in 107 seconds. The affected command path is unchanged except for error reporting. The original task was retried once through the product on deployed source `adb49a44346849e751548b2fb640693dbfad4b6f`. The requested memo and its first document revision persisted after reload and matched the downloaded attachment. The run succeeded, the task reached Done, its execution lock cleared, and final workspace save and result commit succeeded. The diagnostic PR is not deployed. - Recursive typecheck and build passed on `be45ce6594`. `git diff --check` passed. The first full-suite attempt stopped before tests because the offline install had skipped an embedded PostgreSQL native postinstall. The package's official symlink hydration fixed `initdb`. - The next Mac run passed 350 chat cases and hit one unchanged 15-second chat deadline. That case then passed in isolation, and all six full chat shards passed. The broader Mac matrix was later stopped as partial when CI results became available. No full single-command Mac pass is claimed. - Full Vitest file coverage uses the exact successful CI merge tree, `d264fd98e7fb6eff4a9e6b00d402f88894cc0e57`. Its canonical inventory has 2,899 files. Passing CI covered 2,057 files. The remaining 842 ran locally through the standard exact-file harness. The inventory and selection checks show no unassigned files. All four changed source/test files are byte-identical between the PR head and that merge tree. - The first remaining-file run passed 771 files. The database group passed 68 files and reported four deadline/readiness failures in three unchanged files. Those same three files passed all 12 cases when run sequentially on Node 24.22.2, matching CI's major version. Assertions, readiness limits, and test deadlines did not change. The first attempt used Node 26.4.0 with a parallel UI workload; this does not isolate the contribution of runtime version versus load. Both attempts remain recorded. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41699549115) is green on current head `be45ce6594`: `ci / verify`, typecheck, build, all selected server/chat/workspace groups, Runner checks, and all eight browser shards passed. Greptile is 5/5 with no open threads. The prior P2 was resolved by removing new code comments. - Structured Paperclip review remains queued because its own previous accepted result exhausted workspace export recovery. A fresh current-head request is saved, but normal Retry returns HTTP 409 (`accepted_result_without_new_input`). Its advisory cleanup finding is fixed in `be45ce6594` and covered by red/green lifecycle evidence. This check is informational. The branch requires `ci / verify` and `ci / e2e`, and both pass. The independent review-service limitation remains recorded; this PR is not deployed. ## Risks Timeout error text changes. Consumers that compare the old generic message may need to accept the new diagnostic. No such production comparison was found in this repository. API shapes, deadlines, retries, ownership, credentials, and model selection do not change. A timeout still does not prove that the remote process stopped or that a save failed before publication. Existing recovery must reconcile that state. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The host does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
1f55073f96
commit
e0276e0166
5 files changed
+87
-20
No files matched your search
@@ -1848,6 +1848,17 @@ host, sandbox, or Git remote. Files changed since the last save still depend on
|
||||
the live sandbox: losing that allocation can lose those edits. There is no
|
||||
periodic workspace backup in this release.
|
||||
|
||||
Remote workspace command timeouts report `remote_workspace_operation_timed_out`
|
||||
with the operation (`prepare`, `repository`, `archive_helper`, `capture`, `restore`,
|
||||
or `cleanup`) and its timeout in milliseconds. Command output is excluded from
|
||||
that diagnostic. A timeout does not prove the remote process stopped or that a
|
||||
save failed before publication. Reconcile the existing workspace and operation
|
||||
receipts before recovery; the transport does not automatically repeat the command.
|
||||
Export cleanup keeps this typed timeout diagnostic in the operation's
|
||||
`metadata.exportCleanup.lastError`. Other cleanup errors retain a generic message.
|
||||
The saved snapshot stays acknowledged and the cleanup obligation stays pending
|
||||
until the existing reconciliation path settles it.
|
||||
|
||||
Files reads use the current committed saved version when remote authority is
|
||||
clean and has no active writer, even while its old allocation remains recorded.
|
||||
The saved loader verifies the company, workspace, snapshot, and archive before
|
||||
|
||||
@@ -308,7 +308,7 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
.rejects.toThrow(/^remote_workspace_operation_failed$/);
|
||||
execute.mockResolvedValueOnce({ ...failure, timedOut: true, stderr: gitOperation });
|
||||
await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }))
|
||||
.rejects.toThrow(/^remote_workspace_operation_failed$/);
|
||||
.rejects.toThrow(/^remote_workspace_operation_timed_out: archive_helper \(configured timeout 300000ms\)$/);
|
||||
});
|
||||
|
||||
it.each([
|
||||
@@ -347,6 +347,35 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
.rejects.toThrow(/^remote_workspace_operation_failed$/);
|
||||
});
|
||||
|
||||
it("identifies a timed-out preparation without exposing transport output or retrying it", async () => {
|
||||
const { remote, execute } = target();
|
||||
const resolveGitAuth = vi.fn(async () => null);
|
||||
execute.mockResolvedValueOnce({ exitCode: 0, stdout: '{"cloneRequired":true}', stderr: "",
|
||||
timedOut: false, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
execute.mockResolvedValueOnce({ exitCode: null, stdout: "SYNTHETIC_PRIVATE_STDOUT", stderr: "SYNTHETIC_PRIVATE_STDERR",
|
||||
timedOut: true, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
|
||||
await expect(prepareRemoteWorkspaceDirectory({ target: remote, remoteCwd: path.join(workspace, "clone"),
|
||||
source: { kind: "git", repoUrl: "https://example.invalid/source.git", ref: null },
|
||||
preparationIdentity: "a".repeat(64), resolveGitAuth }))
|
||||
.rejects.toThrow(/^remote_workspace_operation_timed_out: prepare \(configured timeout 300000ms\)$/);
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
expect(resolveGitAuth).toHaveBeenCalledTimes(1);
|
||||
expect(await fs.readdir(workspace)).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports the cleanup deadline without treating a timeout as a completed operation", async () => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "preserve the workspace\n");
|
||||
const { remote, execute } = target();
|
||||
execute.mockResolvedValueOnce({ exitCode: 0, stdout: "SYNTHETIC_PRIVATE_STDOUT", stderr: "SYNTHETIC_PRIVATE_STDERR",
|
||||
timedOut: true, signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
|
||||
await expect(cleanupRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }))
|
||||
.rejects.toThrow(/^remote_workspace_operation_timed_out: cleanup \(configured timeout 10000ms\)$/);
|
||||
expect(execute).toHaveBeenCalledTimes(1);
|
||||
expect(await fs.readFile(path.join(workspace, "saved.txt"), "utf8")).toBe("preserve the workspace\n");
|
||||
});
|
||||
|
||||
it("never follows a preexisting snapshot receipt temporary symlink", async () => {
|
||||
await fs.writeFile(path.join(workspace, "result.txt"), "synthetic saved result\n");
|
||||
const operationId = randomUUID();
|
||||
|
||||
@@ -27,12 +27,23 @@ export const REMOTE_WORKSPACE_SANDBOX_DISK_FULL =
|
||||
// Node reports ENOSPC or EDQUOT. Git and tar child processes report the text.
|
||||
const DISK_FULL_CHECK = String.raw`const diskFull=e=>['ENOSPC','EDQUOT'].includes(e?.code)||/No space left on device|Disk quota exceeded/i.test(String(e?.stderr??'')+String(e?.message??''));`;
|
||||
|
||||
async function command(target: RemoteWorkspaceTarget, args: string[], env?: Record<string, string>, timeoutMs = 300_000) {
|
||||
type RemoteWorkspaceOperation = "prepare" | "repository" | "archive_helper" | "capture" | "restore" | "cleanup";
|
||||
|
||||
export class RemoteWorkspaceOperationTimeoutError extends Error {
|
||||
constructor(operation: RemoteWorkspaceOperation, timeoutMs: number) {
|
||||
super(`remote_workspace_operation_timed_out: ${operation} (configured timeout ${timeoutMs}ms)`);
|
||||
this.name = "RemoteWorkspaceOperationTimeoutError";
|
||||
}
|
||||
}
|
||||
|
||||
async function command(target: RemoteWorkspaceTarget, operation: RemoteWorkspaceOperation,
|
||||
args: string[], env?: Record<string, string>, timeoutMs = 300_000) {
|
||||
const result = await target.runner!.execute({ command: "node", args, cwd: target.remoteCwd,
|
||||
env: { NODE_OPTIONS: "", NODE_PATH: "", ...env }, timeoutMs, bypassSession: true });
|
||||
if (result.exitCode !== 0 || result.timedOut) {
|
||||
if (result.timedOut) throw new RemoteWorkspaceOperationTimeoutError(operation, timeoutMs);
|
||||
if (result.exitCode !== 0) {
|
||||
const gitOperation = "remote_workspace_archive_git_operation_in_progress: finish or abort the Git operation before saving the workspace";
|
||||
const reported = result.timedOut ? "" : result.stderr.trim();
|
||||
const reported = result.stderr.trim();
|
||||
throw new Error([gitOperation, REMOTE_WORKSPACE_SANDBOX_DISK_FULL].includes(reported) ? reported : "remote_workspace_operation_failed");
|
||||
}
|
||||
return result.stdout;
|
||||
@@ -95,14 +106,14 @@ export async function prepareRemoteWorkspaceDirectory(input: {
|
||||
preparationIdentity: string;
|
||||
}) {
|
||||
if (input.source.kind === "git") {
|
||||
const prior = JSON.parse(await command(input.target, ["-e", PREPARE, input.remoteCwd,
|
||||
const prior = JSON.parse(await command(input.target, "prepare", ["-e", PREPARE, input.remoteCwd,
|
||||
JSON.stringify({ ...input.source, preparationIdentity: input.preparationIdentity,
|
||||
authArgs: [], authEnvKeys: [], probeOnly: true })])) as { cwd?: string; cloneRequired?: boolean };
|
||||
if (prior.cwd === input.remoteCwd) return;
|
||||
if (prior.cloneRequired !== true) throw new Error("remote_workspace_root_changed");
|
||||
}
|
||||
const auth = input.source.kind === "git" ? await input.resolveGitAuth(input.source.repoUrl) : null;
|
||||
const output = await command(input.target, ["-e", PREPARE, input.remoteCwd,
|
||||
const output = await command(input.target, "prepare", ["-e", PREPARE, input.remoteCwd,
|
||||
JSON.stringify({ ...input.source, preparationIdentity: input.preparationIdentity,
|
||||
authArgs: auth?.configArgs ?? [], authEnvKeys: Object.keys(auth?.env ?? {}) })], auth?.env);
|
||||
const result = JSON.parse(output) as { cwd: string };
|
||||
@@ -150,7 +161,7 @@ export async function prepareRemoteRepository(input: {
|
||||
const branch=await git(['symbolic-ref','--short','-q','HEAD']).catch(()=>null);process.stdout.write(JSON.stringify({cwd:root,pinnedCommit:input.pinnedCommit||head,branchName:branch}));
|
||||
})().catch(()=>{process.stderr.write('remote_workspace_repository_failed');process.exitCode=1});`;
|
||||
const payload = { ...input, target: undefined, resolveGitAuth: undefined };
|
||||
const prior = JSON.parse(await command(input.target, ["-e", script, input.remoteCwd,
|
||||
const prior = JSON.parse(await command(input.target, "repository", ["-e", script, input.remoteCwd,
|
||||
JSON.stringify({ ...payload, authArgs: [], authEnvKeys: [], probeOnly: true })])) as {
|
||||
cwd?: string; pinnedCommit?: string; branchName?: string | null; cloneRequired?: boolean;
|
||||
};
|
||||
@@ -159,7 +170,7 @@ export async function prepareRemoteRepository(input: {
|
||||
return { cwd: prior.cwd, pinnedCommit: prior.pinnedCommit!, branchName: prior.branchName ?? null };
|
||||
}
|
||||
const auth = await input.resolveGitAuth(input.repoUrl);
|
||||
const output = await command(input.target, ["-e", script, input.remoteCwd,
|
||||
const output = await command(input.target, "repository", ["-e", script, input.remoteCwd,
|
||||
JSON.stringify({ ...payload, authArgs: auth?.configArgs ?? [], authEnvKeys: Object.keys(auth?.env ?? {}) })], auth?.env);
|
||||
const value = JSON.parse(output) as { cwd: string; pinnedCommit: string; branchName: string | null };
|
||||
if (value.cwd !== input.remoteCwd || !/^[a-f0-9]{40,64}$/.test(value.pinnedCommit)) throw new Error("remote_workspace_repository_receipt_invalid");
|
||||
@@ -180,7 +191,7 @@ async function installArchiveHelper(target: RemoteWorkspaceTarget, remoteCwd: st
|
||||
try{await fs.writeFile(temp,Buffer.from(body,'base64'),{mode:0o600,flag:'wx'});await fs.rename(temp,file);}
|
||||
catch(e){if(e?.code!=='EEXIST')await fs.rm(temp,{force:true}).catch(()=>{});throw e;}
|
||||
})().catch(error=>{if(diskFull(error))process.stderr.write(${JSON.stringify(REMOTE_WORKSPACE_SANDBOX_DISK_FULL)});process.exitCode=1});`;
|
||||
await command(target, ["-e", install, remoteCwd, directory, helper, Buffer.from(remoteWorkspaceArchiveScript()).toString("base64")]);
|
||||
await command(target, "archive_helper", ["-e", install, remoteCwd, directory, helper, Buffer.from(remoteWorkspaceArchiveScript()).toString("base64")]);
|
||||
return { directory, helper, archive: path.posix.join(directory, "workspace.pcws"), receipt: path.posix.join(directory, "receipt.json") };
|
||||
}
|
||||
|
||||
@@ -197,7 +208,7 @@ export async function cleanupRemoteWorkspaceArchive(input: {
|
||||
if(!entry.isDirectory()||entry.isSymbolicLink())throw Error('unsafe');}
|
||||
await fs.rm(dir,{recursive:true,force:true});
|
||||
})().catch(()=>{process.stderr.write('remote_workspace_export_cleanup_failed');process.exitCode=1});`;
|
||||
await command(input.target, ["-e", script, input.remoteCwd, input.operationId], undefined, 10_000);
|
||||
await command(input.target, "cleanup", ["-e", script, input.remoteCwd, input.operationId], undefined, 10_000);
|
||||
}
|
||||
|
||||
export async function captureRemoteWorkspaceArchive(input: {
|
||||
@@ -234,7 +245,7 @@ export async function captureRemoteWorkspaceArchive(input: {
|
||||
process.stderr.write(typeof error?.stderr==='string'&&error.stderr.trim()===gitOperation?gitOperation
|
||||
:diskFull(error)?${JSON.stringify(REMOTE_WORKSPACE_SANDBOX_DISK_FULL)}:'remote_workspace_capture_failed');process.exitCode=1;
|
||||
});`;
|
||||
const { layout, ...receipt } = JSON.parse(await command(input.target, ["-e", capture, staged.helper, input.remoteCwd, staged.directory])) as {
|
||||
const { layout, ...receipt } = JSON.parse(await command(input.target, "capture", ["-e", capture, staged.helper, input.remoteCwd, staged.directory])) as {
|
||||
manifestDigest: string; sha256: string; byteSize: number; entries: number; layout: "legacy" | "captured";
|
||||
};
|
||||
if (!["legacy", "captured"].includes(layout) || !/^[a-f0-9]{64}$/.test(receipt.manifestDigest) || !/^[a-f0-9]{64}$/.test(receipt.sha256)
|
||||
@@ -284,7 +295,7 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600});
|
||||
if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');await fs.rename(staged,root);
|
||||
})().catch(()=>{process.stderr.write('remote_workspace_restore_failed');process.exitCode=1});`;
|
||||
await command(input.target, ["-e", restore, staged.helper, staged.archive, input.remoteCwd, input.manifestDigest, input.restorationIdentity]);
|
||||
await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd, input.manifestDigest, input.restorationIdentity]);
|
||||
// Publish the restored location before cleanup can fail, stall, or lose its
|
||||
// reply. The owner journals this receipt and then reclaims disposable staging.
|
||||
return { operationId, stagingRoot: parent, cleanupPending: true };
|
||||
|
||||
@@ -919,17 +919,28 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
expect(await preserveRemoteWorkspacePreparationFailure(db, prepared.lease)).toBeNull();
|
||||
}, 60_000);
|
||||
|
||||
it("records and retries export cleanup after a lost reply without another upload", async () => {
|
||||
it.each([
|
||||
["a lost reply", "remote_workspace_export_cleanup_failed"],
|
||||
["a timeout", "remote_workspace_operation_timed_out: cleanup (configured timeout 10000ms)"],
|
||||
["an untrusted error", "remote_workspace_export_cleanup_failed"],
|
||||
] as const)("records and retries export cleanup after %s without another upload", async (failure, diagnostic) => {
|
||||
const reserved = await reserve({ kind: "empty" }), first = await target("cleanup-sandbox");
|
||||
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first });
|
||||
await fs.writeFile(path.join(prepared.remoteRoot, "saved.txt"), "keep working files\n");
|
||||
const sandbox = first as Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
|
||||
const execute = sandbox.runner!.execute.bind(sandbox.runner!);
|
||||
let lost = false;
|
||||
let lost = false, cleanupAttempts = 0;
|
||||
sandbox.runner!.execute = async input => {
|
||||
const result = await execute(input);
|
||||
if (!lost && input.args?.[1]?.includes("remote_workspace_export_cleanup_failed") && result.exitCode === 0) {
|
||||
lost = true; throw new Error("lost cleanup reply");
|
||||
if (input.args?.[1]?.includes("remote_workspace_export_cleanup_failed") && result.exitCode === 0) {
|
||||
cleanupAttempts++;
|
||||
if (lost) return result;
|
||||
lost = true;
|
||||
if (failure === "a timeout") return { ...result, timedOut: true,
|
||||
stdout: "SYNTHETIC_PRIVATE_STDOUT", stderr: "SYNTHETIC_PRIVATE_STDERR" };
|
||||
throw new Error(failure === "an untrusted error"
|
||||
? "remote_workspace_operation_timed_out: cleanup (configured timeout 10000ms)\nSYNTHETIC_PRIVATE_STDERR"
|
||||
: "lost cleanup reply");
|
||||
}
|
||||
return result;
|
||||
};
|
||||
@@ -940,11 +951,13 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
const [before] = await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, ids.run),
|
||||
eq(workspaceOperations.phase, "remote_workspace_snapshot"), eq(workspaceOperations.status, "succeeded"),
|
||||
eq(workspaceOperations.id, (await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, saved.snapshotId)))[0]!.sourceOperationId)));
|
||||
expect(before?.metadata?.exportCleanup).toMatchObject({ state: "pending", lastError: "remote_workspace_export_cleanup_failed" });
|
||||
expect(before?.metadata?.exportCleanup).toMatchObject({ state: "pending", lastError: diagnostic });
|
||||
expect(cleanupAttempts).toBe(1);
|
||||
expect(await saveRemoteWorkspace(input)).toMatchObject({ snapshotId: saved.snapshotId, generation: saved.generation });
|
||||
expect(upload).toHaveBeenCalledOnce();
|
||||
const [after] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, before!.id));
|
||||
expect(after?.metadata?.exportCleanup).toMatchObject({ state: "cleaned", lastError: null });
|
||||
expect(cleanupAttempts).toBe(2);
|
||||
expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")).toBe("keep working files\n");
|
||||
}, 60_000);
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ import { canonicalExecutionUuid } from "./execution-identity.js";
|
||||
import { persistActivity } from "./activity-log.js";
|
||||
import {
|
||||
captureRemoteWorkspaceArchive, cleanupRemoteWorkspaceArchive, prepareRemoteWorkspaceDirectory, prepareRemoteRepository, remoteWorkspaceRepositoryPath,
|
||||
requireRemoteWorkspaceTarget, restoreRemoteWorkspaceDirectory,
|
||||
RemoteWorkspaceOperationTimeoutError, requireRemoteWorkspaceTarget, restoreRemoteWorkspaceDirectory,
|
||||
} from "./remote-workspace-transport.js";
|
||||
import {
|
||||
beginWorkspaceSnapshot, getWorkspaceSnapshot, loadWorkspaceSnapshotArchive, uploadWorkspaceSnapshot,
|
||||
@@ -281,16 +281,19 @@ async function cleanupSavedExports(input: { db: Db; companyId: string; workspace
|
||||
for (const row of rows) {
|
||||
await input.assertOwned();
|
||||
let cleaned = false;
|
||||
let lastError: string | null = null;
|
||||
try {
|
||||
await cleanupRemoteWorkspaceArchive({ target: input.target,
|
||||
remoteCwd: row.phase === "remote_workspace_restore" ? path.posix.dirname(input.location.remoteCwd) : input.location.remoteCwd,
|
||||
operationId: row.id });
|
||||
cleaned = true;
|
||||
} catch { /* Keep a durable retry obligation; the acknowledged snapshot is safe. */ }
|
||||
} catch (error) {
|
||||
lastError = error instanceof RemoteWorkspaceOperationTimeoutError ? error.message : "remote_workspace_export_cleanup_failed";
|
||||
}
|
||||
await input.assertOwned();
|
||||
await input.db.update(workspaceOperations).set({ metadata: sql`jsonb_set(coalesce(${workspaceOperations.metadata}, '{}'::jsonb),
|
||||
'{exportCleanup}', coalesce(${workspaceOperations.metadata}->'exportCleanup', '{}'::jsonb) ||
|
||||
${JSON.stringify({ state: cleaned ? "cleaned" : "pending", lastError: cleaned ? null : "remote_workspace_export_cleanup_failed" })}::jsonb)`,
|
||||
${JSON.stringify({ state: cleaned ? "cleaned" : "pending", lastError })}::jsonb)`,
|
||||
updatedAt: new Date() })
|
||||
.where(and(eq(workspaceOperations.id, row.id), eq(workspaceOperations.companyId, input.companyId),
|
||||
inArray(workspaceOperations.status, ["succeeded", "failed"])));
|
||||
|
||||
Reference in new issue
Block a user