mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
fix(remote-workspaces): restore newer saved work over a retained sandbox's own retired root (#19231)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote workspaces save a task's files as snapshots and restore them on the sandbox that runs the next turn > - A parent task can delegate subtasks. The subtasks inherit the parent's workspace, run on their own sandboxes, and save newer snapshots > - When the parent wakes on its retained (warm, reused) sandbox, its earlier workspace root is still on that sandbox at the same published path > - The restore refused that root with `reason=occupied_root`, so the parent run failed at setup and the task became blocked with a board recovery action > - This pull request lets the restore replace that root only when the server proves it is the workspace's own earlier, fully saved allocation on this sandbox > - The benefit is that a delegating parent on a retained sandbox continues with the children's saved work, and unknown occupants are still refused ## Linked Issues or Issue Description **What happened?** A parent task on an agent with a retained sandbox delegates two subtasks to other agents. Both subtasks finish. The parent's wake run fails before the provider starts with `remote_workspace_restore_failed: step=sandbox_restore; ... stage=inspect_root reason=occupied_root`. The task becomes `blocked` with a stranded-work recovery action. Parents on agents with ephemeral sandboxes complete in the same wave. Root cause: the subtasks inherit the parent's execution workspace. Each subtask becomes the writer at a new epoch on its own sandbox, restores at the same published directory name, and saves a newer snapshot. The workspace location then points at the last subtask's sandbox. The parent's wake run gets a lease on its retained sandbox, so preparation is not "same allocation" and must restore the newer snapshot. The target path on the retained sandbox still holds the parent's epoch-1 root, which has a preparation receipt (`workspace-ready.json`) and no restore receipt. The restore script refuses any occupied root that is not the same restore, so it fails with `occupied_root`. A warm reuse cannot just keep the old root, because the snapshot is newer. **Expected behavior** The parent restores the newest saved work (its own files plus the children's results) into its published root and continues. A foreign or unknown occupant is still refused. **Steps to reproduce** 1. Prepare a remote workspace on sandbox A, write files, and save (final boundary). 2. Make a second run the writer on sandbox B. It restores, writes new files, and saves. 3. Make a third run the writer on sandbox A again (same provider lease id, new lease row) and prepare. 4. Before this change, step 3 fails with `stage=inspect_root reason=occupied_root`. **Paperclip version or commit** `master` at `47027e7ba5`. ## What Changed - `server/src/services/remote-workspaces.ts`: add `isRetiredOwnAllocation`. It reads this workspace's snapshot and restore operations recorded at the same environment, provider lease id, and root path. It returns true only when every such use is from an earlier epoch than the current writer, the last use ended in a succeeded `final` save with quiesced save work (`unknown=false`, no active invocations), no other save in that epoch is newer or unsettled, and the saved snapshot is committed or later. Preparation passes the result to the restore as `replaceRetiredRoot`, and only when there is no migration and the authority is clean. - `server/src/services/remote-workspace-transport.ts`: the restore script accepts `replaceRetiredRoot`. In that mode it replaces an occupied root only when the root carries a well-formed Paperclip receipt (`workspace-ready.json` or `workspace-restored.json` with a 64-hex identity). It parks the old root at `<root>.retired-<identity>`, publishes the restored folder, rolls the old root back if the publish rename fails, and then removes the parked copy. A repeated restore with the same identity adopts the published folder and removes any parked copy left by an interrupted attempt. Without a receipt, the script still fails with `reason=occupied_root`. `replaceRetiredRoot` and `allowEmptyRoot` cannot be combined. - `doc/execution-semantics.md`: document the rule. - `server/src/services/remote-workspaces.integration.test.ts`: add a red/green regression and the safety cases. ## Verification - Red before the fix: `npx vitest run src/services/remote-workspaces.integration.test.ts -t "retired root"` failed in the `own retired root` case with `remote_workspace_restore_failed: step=sandbox_restore; remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=inspect_root reason=occupied_root`. This is the same cause text as the staging failure. - Green after the fix: all six cases pass: - `own retired root`: the parent restores on its retained sandbox. The root has the parent's file and the child's file, the parked and staging folders are removed, and the authority records the parent's lease. A repeated preparation (lost reply) keeps the parent's new unsaved edits. - `own restored root`: an occupant with a restore receipt from a different preparation is also replaced. - `receipt missing` and `forged receipt`: the occupant is refused with `occupied_root`, and its files stay. - `later allocation`: a later use of that root on that sandbox without a final save is refused. - `never saved here`: a different sandbox with an unknown folder at that path is refused. - `npx vitest run src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts`: 247 passed. This includes the existing `occupied root` cold restore case, which is still refused. - `tsc --noEmit` shows no errors in the changed files. The local tree has unrelated errors from unbuilt workspace packages. - I did not run a live staging run. Staging is read-only for this work, and this PR does not deploy. ## Risks - The replaced root is removed after the publish. The server proof states that its content is in the saved lineage. If a process wrote to that root after its final save and outside a Paperclip run, those writes are lost. The same assumption applies when authority moves to another allocation today. - The proof uses operation records that have `exportCleanup` location metadata. Older records without that metadata do not qualify, and those cases still fail closed with `occupied_root`. - A warm runner process that kept an open working directory in the old root sees a removed folder. This is not verified live. The new turn is given the published path, which holds the restored files. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. It used tool use for code search, shell, test runs, and read-only staging inspection in the browser. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5d8d9bf83e
commit
e8163fc860
4 files changed
+182
-11
No files matched your search
@@ -1057,6 +1057,17 @@ folder), or an exhausted budget leaves a board-owned recovery action. Stranded-w
|
||||
recovery names this failure for the statuses it scans; for other statuses, such as
|
||||
a `blocked` task woken by a reply, the failure path records the action directly.
|
||||
|
||||
A retained sandbox can still hold a workspace's earlier root after the workspace
|
||||
moved to another allocation, for example when child tasks inherit a parent's
|
||||
workspace and save newer work on their own sandboxes. When the parent wakes on its
|
||||
retained sandbox, the restore may replace that occupied root only when the server
|
||||
proves it is the workspace's own earlier allocation on this sandbox: the last
|
||||
recorded use of that root on this sandbox is an earlier epoch that ended in a
|
||||
quiesced, committed final save, and the folder carries a Paperclip preparation or
|
||||
restore receipt. Its files are then already part of the saved lineage being
|
||||
restored. Any other occupant (no receipt, a later unsaved use, or a sandbox that
|
||||
never saved that root) is still refused with `reason=occupied_root`.
|
||||
|
||||
The provider's own idle auto-stop must not coincide with the warm runner idle
|
||||
window. The host forwards the run's effective warm idle window (environment
|
||||
`runnerIdleTimeoutMs`, else the agent's `idleTimeoutMs`) to the sandbox provider
|
||||
|
||||
@@ -454,7 +454,10 @@ export async function captureRemoteWorkspaceArchive(input: {
|
||||
export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
target: RemoteWorkspaceTarget; remoteCwd: string; archivePath: string; manifestDigest: string; restorationIdentity: string; allowEmptyRoot?: boolean;
|
||||
allocationFiles?: Array<{ relativePath: string; sha256: string }>;
|
||||
/** The server proved this sandbox's root is the workspace's own earlier allocation, fully saved. */
|
||||
replaceRetiredRoot?: boolean;
|
||||
}) {
|
||||
if (input.replaceRetiredRoot && input.allowEmptyRoot) throw new Error("remote_workspace_restore_mode_invalid");
|
||||
if (!/^[a-f0-9]{64}$/.test(input.restorationIdentity)) throw new Error("remote_workspace_restore_identity_invalid");
|
||||
const id = input.restorationIdentity;
|
||||
const operationId = `${id.slice(0, 8)}-${id.slice(8, 12)}-${id.slice(12, 16)}-${id.slice(16, 20)}-${id.slice(20, 32)}`;
|
||||
@@ -468,7 +471,7 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
// reply can then adopt the same files without repeating a destructive restore.
|
||||
const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process'),crypto=require('node:crypto');
|
||||
${DISK_FULL_CHECK}${CAPTURE_FAILURE_CHECK}${SETUP_FAILURE_CHECK}let stage='validate';
|
||||
(async()=>{const [helper,archive,root,digest,identity,allowEmpty,rawFiles]=process.argv.slice(1),files=JSON.parse(rawFiles);
|
||||
(async()=>{const [helper,archive,root,digest,identity,allowEmpty,rawFiles,replaceRetired]=process.argv.slice(1),files=JSON.parse(rawFiles);
|
||||
if(!/^[a-f0-9]{64}$/.test(identity)||!Array.isArray(files)||files.length>8)throw Error('invalid_identity');
|
||||
for(const file of files)if(!/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath)
|
||||
||file.relativePath==='.paperclip-runtime/workspace-restored.json'||!/^[a-f0-9]{64}$/.test(file.sha256))throw Error('invalid_allocation');
|
||||
@@ -487,7 +490,22 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
const content=await fs.readFile(name);if(crypto.createHash('sha256').update(content).digest('hex')!==file.sha256)throw Error('allocation_changed');
|
||||
return {name:path.basename(name),content};}));
|
||||
};
|
||||
const parked=root+'.allocation-'+identity;
|
||||
// A retired root holds a Paperclip preparation or restore receipt; anything else is unknown.
|
||||
const ownReceipt=async p=>{
|
||||
const runtime=path.join(p,'.paperclip-runtime');if(!await stat(runtime))return false;await directory(runtime);
|
||||
for(const [name,key] of [['workspace-restored.json','identity'],['workspace-ready.json','preparationIdentity']]){
|
||||
const file=path.join(runtime,name),s=await stat(file);if(!s)continue;
|
||||
if(!s.isFile()||s.isSymbolicLink()||s.nlink!==1||s.size>16384)throw Error('unsafe_receipt');
|
||||
if(/^[a-f0-9]{64}$/.test(String(JSON.parse(await fs.readFile(file,'utf8'))?.[key])))return true;}
|
||||
return false;
|
||||
};
|
||||
const parked=root+'.allocation-'+identity,retired=root+'.retired-'+identity;
|
||||
// Saved directory modes can be read-only. Make the parked tree's own directories
|
||||
// owner-writable without following symlinks, then remove it.
|
||||
const removeRetired=async()=>{if(!await stat(retired))return;
|
||||
const open=async d=>{const s=await fs.lstat(d);if(!s.isDirectory()||s.isSymbolicLink())return;
|
||||
await fs.chmod(d,(s.mode&0o7777)|0o700);for(const name of await fs.readdir(d))await open(path.join(d,name));};
|
||||
await open(retired);await fs.rm(retired,{recursive:true,force:true});};
|
||||
stage='allocation';if(await stat(parked)){if(allowEmpty!=='true')throw Error('unexpected_allocation');await allocation(parked);if(!await stat(root))await fs.rename(parked,root);}
|
||||
stage='inspect_root';const prior=await stat(root);if(prior)await directory(root);
|
||||
const marker=path.join(root,'.paperclip-runtime','workspace-restored.json');
|
||||
@@ -495,11 +513,17 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
stage='existing_receipt';await directory(path.dirname(marker));const file=await fs.lstat(marker);
|
||||
if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt');
|
||||
const receipt=JSON.parse(await fs.readFile(marker,'utf8'));
|
||||
if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed');
|
||||
if(await stat(parked)){await allocation(parked);await fs.rm(parked,{recursive:true});}return;
|
||||
if(receipt.identity===identity&&receipt.manifestDigest===digest){
|
||||
if(await stat(parked)){await allocation(parked);await fs.rm(parked,{recursive:true});}
|
||||
if(replaceRetired==='true')await removeRetired();return;}
|
||||
if(replaceRetired!=='true')throw Error('workspace_owner_changed');
|
||||
}
|
||||
if(prior&&allowEmpty!=='true')throw Error('occupied_root');
|
||||
stage='allocation';const retained=prior?await allocation(root):[];
|
||||
stage='inspect_root';
|
||||
// Replacing is allowed only for a root the server proved is this workspace's
|
||||
// own earlier allocation whose last use ended in a completed final save.
|
||||
const retire=Boolean(prior)&&replaceRetired==='true'&&await ownReceipt(root);
|
||||
if(prior&&!retire&&allowEmpty!=='true')throw Error('occupied_root');
|
||||
stage='allocation';const retained=prior&&!retire?await allocation(root):[];
|
||||
if(!prior&&files.length)throw Error('allocation_missing');
|
||||
stage='extract';const staged=await fs.mkdtemp(root+'.restoring-');
|
||||
try{
|
||||
@@ -508,13 +532,17 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
for(const file of retained)await fs.writeFile(path.join(runtime,file.name),file.content,{flag:'wx',mode:0o600});
|
||||
await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600});
|
||||
stage='publish';if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');
|
||||
if(prior){await allocation(root);if(await stat(parked))throw Error('occupied_parking');await fs.rename(root,parked);}
|
||||
try{await fs.rename(staged,root);}catch(e){if(prior&&!await stat(root))await fs.rename(parked,root);throw e;}
|
||||
stage='cleanup_parked';if(prior){await allocation(parked);await fs.rm(parked,{recursive:true});}
|
||||
const holding=retire?retired:parked;
|
||||
if(prior){if(retire){if(!await ownReceipt(root))throw Error('occupied_root');}else await allocation(root);
|
||||
if(await stat(holding))throw Error('occupied_parking');await fs.rename(root,holding);}
|
||||
try{await fs.rename(staged,root);}catch(e){if(prior&&!await stat(root))await fs.rename(holding,root);throw e;}
|
||||
stage='cleanup_parked';if(prior&&!retire){await allocation(parked);await fs.rm(parked,{recursive:true});}
|
||||
if(replaceRetired==='true')await removeRetired();
|
||||
}finally{await fs.rm(staged,{recursive:true,force:true});}
|
||||
})().catch(error=>{process.stderr.write(setupFailure('remote_workspace_restore_failed',stage,error));process.exitCode=1});`;
|
||||
await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd,
|
||||
input.manifestDigest, input.restorationIdentity, String(input.allowEmptyRoot === true), JSON.stringify(input.allocationFiles ?? [])]);
|
||||
input.manifestDigest, input.restorationIdentity, String(input.allowEmptyRoot === true), JSON.stringify(input.allocationFiles ?? []),
|
||||
String(input.replaceRetiredRoot === true)]);
|
||||
// Publish the restored location before cleanup can fail, stall, or lose its
|
||||
// reply. The owner journals this receipt and then reclaims disposable staging.
|
||||
return { operationId, stagingRoot: parent, cleanupPending: true };
|
||||
|
||||
@@ -470,6 +470,100 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
try { expect(await fs.readFile(path.join(browse.root, "report.md"), "utf8")).toContain("useful saved result"); } finally { await browse.cleanup(); }
|
||||
}, 60_000);
|
||||
|
||||
it.each(["own retired root", "own restored root", "read-only folder", "receipt missing", "forged receipt", "later allocation",
|
||||
"unsettled save", "newer checkpoint", "never saved here"] as const)(
|
||||
"restores a newer save over this workspace's own retired root on a reused sandbox (%s)", async mode => {
|
||||
// A parent prepares on its retained sandbox, then children inherit the
|
||||
// workspace on other sandboxes and save newer work. When the parent wakes on
|
||||
// the same sandbox, its old root is still present.
|
||||
const reserved = await reserve({ kind: "empty" }), parentSandbox = await target("sandbox-A");
|
||||
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: parentSandbox });
|
||||
await fs.writeFile(path.join(prepared.remoteRoot, "plan.md"), "parent plan\n");
|
||||
await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId: ids.run,
|
||||
lease: prepared.lease, target: parentSandbox, assertOwned: async () => undefined });
|
||||
const writer = async (providerLeaseId: string) => {
|
||||
const runId = randomUUID(), leaseId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId: ids.company, environmentId: ids.environment,
|
||||
heartbeatRunId: runId, issueId: ids.issue, provider: "daytona", providerLeaseId, status: "active" });
|
||||
const workspace = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, runId,
|
||||
agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null, existingWorkspaceId: reserved.id, source: { kind: "empty" } });
|
||||
const leased = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]! as EnvironmentLease;
|
||||
return { runId, workspace, lease: leased };
|
||||
};
|
||||
const child = await writer("allocation-child"), childSandbox = await target("sandbox-B");
|
||||
const childPrepared = await prepareRemoteWorkspace({ db, storage, workspace: child.workspace, runId: child.runId, lease: child.lease, target: childSandbox });
|
||||
await fs.writeFile(path.join(childPrepared.remoteRoot, "child-result.md"), "child result\n");
|
||||
const childSaved = await saveRemoteWorkspace({ db, storage, workspace: childPrepared.workspace, runId: child.runId,
|
||||
lease: childPrepared.lease, target: childSandbox, assertOwned: async () => undefined });
|
||||
if (mode === "receipt missing") {
|
||||
// An occupant without a Paperclip preparation or restore receipt is unknown.
|
||||
await fs.rm(path.join(prepared.remoteRoot, ".paperclip-runtime"), { recursive: true, force: true });
|
||||
}
|
||||
if (mode === "own restored root" || mode === "forged receipt") {
|
||||
// The retired root was itself restored by an earlier preparation of this workspace.
|
||||
const runtime = path.join(prepared.remoteRoot, ".paperclip-runtime");
|
||||
await fs.rm(path.join(runtime, "workspace-ready.json"));
|
||||
await fs.writeFile(path.join(runtime, "workspace-restored.json"), JSON.stringify(mode === "forged receipt"
|
||||
? { identity: "not-an-identity" } : { identity: "a".repeat(64), manifestDigest: "b".repeat(64) }));
|
||||
}
|
||||
if (mode === "read-only folder") {
|
||||
// A non-root sandbox user must still be able to remove the retired tree.
|
||||
const locked = path.join(prepared.remoteRoot, "locked");
|
||||
await fs.mkdir(path.join(locked, "inner"), { recursive: true });
|
||||
await fs.writeFile(path.join(locked, "inner", "file.txt"), "old\n");
|
||||
await fs.chmod(path.join(locked, "inner"), 0o555); await fs.chmod(locked, 0o555);
|
||||
}
|
||||
if (mode === "unsettled save" || mode === "newer checkpoint") {
|
||||
// The last use of this root ended in a save that is not a settled final save.
|
||||
await db.insert(workspaceOperations).values({ companyId: ids.company, executionWorkspaceId: reserved.id, heartbeatRunId: ids.run,
|
||||
issueId: ids.issue, phase: "remote_workspace_snapshot", status: mode === "unsettled save" ? "running" : "succeeded",
|
||||
metadata: { epoch: 1, boundary: "checkpoint", generation: 99,
|
||||
saveWork: { schema: "paperclip.remote-workspace-save-work.v1", activeIds: [], unknown: false, quiescedAt: new Date().toISOString() },
|
||||
exportCleanup: { ...readRemoteWorkspaceAuthority(prepared.workspace.metadata)!.location, state: "cleaned" } } });
|
||||
}
|
||||
if (mode === "later allocation") {
|
||||
// A later writer used this root on the parent's sandbox after its last final save.
|
||||
await db.insert(workspaceOperations).values({ companyId: ids.company, executionWorkspaceId: reserved.id, heartbeatRunId: child.runId,
|
||||
issueId: ids.issue, phase: "remote_workspace_restore", status: "succeeded",
|
||||
metadata: { epoch: 2, exportCleanup: { ...readRemoteWorkspaceAuthority(prepared.workspace.metadata)!.location, state: "cleaned" } } });
|
||||
}
|
||||
// Preparation narrows a target to its root; a later turn reattaches to the sandbox.
|
||||
const reattach = () => ({ ...parentSandbox, remoteCwd: path.join(root, "sandbox-A") });
|
||||
let parentTarget = reattach();
|
||||
if (mode === "never saved here") {
|
||||
// A different sandbox that holds an unknown folder at this path.
|
||||
parentTarget = await target("sandbox-C");
|
||||
await fs.cp(prepared.remoteRoot, path.join(parentTarget.remoteCwd, ".paperclip-workspaces", path.basename(prepared.remoteRoot)), { recursive: true });
|
||||
}
|
||||
const parent = await writer(mode === "never saved here" ? "allocation-unrelated" : prepared.lease.providerLeaseId!);
|
||||
const restoring = prepareRemoteWorkspace({ db, storage, workspace: parent.workspace, runId: parent.runId, lease: parent.lease, target: parentTarget });
|
||||
if (mode !== "own retired root" && mode !== "own restored root" && mode !== "read-only folder") {
|
||||
const refused = await restoring.then(() => null, (error: unknown) => error as RemoteWorkspaceRestoreError);
|
||||
expect(refused).toBeInstanceOf(RemoteWorkspaceRestoreError);
|
||||
expect(refused).toMatchObject({ step: "sandbox_restore", retryable: false });
|
||||
expect(refused!.message).toMatch(/stage=inspect_root reason=occupied_root$/);
|
||||
expect(await fs.readFile(path.join(prepared.remoteRoot, "plan.md"), "utf8")).toBe("parent plan\n");
|
||||
expect(await fs.lstat(path.join(prepared.remoteRoot, "child-result.md")).catch(() => null)).toBeNull();
|
||||
expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata)).toEqual(readRemoteWorkspaceAuthority(parent.workspace.metadata));
|
||||
return;
|
||||
}
|
||||
const restored = await restoring;
|
||||
expect(restored.remoteRoot).toBe(prepared.remoteRoot);
|
||||
expect(await fs.readFile(path.join(restored.remoteRoot, "plan.md"), "utf8")).toBe("parent plan\n");
|
||||
expect(await fs.readFile(path.join(restored.remoteRoot, "child-result.md"), "utf8")).toBe("child result\n");
|
||||
// The replaced root and the restore staging are gone; only saved files were replaced.
|
||||
expect((await fs.readdir(path.dirname(restored.remoteRoot))).filter(name => name.startsWith(path.basename(restored.remoteRoot))))
|
||||
.toEqual([path.basename(restored.remoteRoot)]);
|
||||
expect(readRemoteWorkspaceAuthority(restored.workspace.metadata)).toMatchObject({ snapshotId: childSaved.snapshotId, dirty: true,
|
||||
writerRunId: parent.runId, location: { providerLeaseId: prepared.lease.providerLeaseId, remoteCwd: prepared.remoteRoot } });
|
||||
// A lost reply repeats the same preparation without replacing the restored root again.
|
||||
await fs.writeFile(path.join(restored.remoteRoot, "parent-continued.md"), "unsaved parent edits\n");
|
||||
const repeated = await prepareRemoteWorkspace({ db, storage, workspace: parent.workspace, runId: parent.runId, lease: parent.lease, target: reattach() });
|
||||
expect(repeated.remoteRoot).toBe(prepared.remoteRoot);
|
||||
expect(await fs.readFile(path.join(restored.remoteRoot, "parent-continued.md"), "utf8")).toBe("unsaved parent edits\n");
|
||||
}, 60_000);
|
||||
|
||||
it.each(["per_turn", "warm", "occupied", "foreign-target", "foreign-allocation", "forged-receipt"])("preserves retained native workspace identity through migration: %s", async scenario => {
|
||||
const original = path.join(root, "old-controller", "workspace"); await fs.mkdir(original, { recursive: true });
|
||||
await fs.writeFile(path.join(original, "saved.txt"), "retained work");
|
||||
|
||||
@@ -370,6 +370,41 @@ function publishedWorkspaceDirectory(workspace: Workspace & Pick<typeof executio
|
||||
return directory;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the root at `remoteRoot` on this lease's sandbox is this workspace's
|
||||
* own earlier allocation whose last use ended in a completed final save. A
|
||||
* retained sandbox keeps that root after the workspace moves on, for example
|
||||
* when child tasks inherit the workspace and save newer work elsewhere. Its
|
||||
* files are then already part of the saved lineage being restored.
|
||||
*/
|
||||
async function isRetiredOwnAllocation(db: Db, workspace: Workspace, lease: EnvironmentLease, remoteRoot: string, epoch: number) {
|
||||
if (!lease.environmentId || !lease.providerLeaseId) return false;
|
||||
const rows = await db.select({ id: workspaceOperations.id, phase: workspaceOperations.phase, status: workspaceOperations.status,
|
||||
metadata: workspaceOperations.metadata }).from(workspaceOperations).where(and(
|
||||
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
|
||||
inArray(workspaceOperations.phase, ["remote_workspace_snapshot", "remote_workspace_restore"]),
|
||||
sql`${workspaceOperations.metadata}->'exportCleanup'->>'environmentId' = ${lease.environmentId}`,
|
||||
sql`${workspaceOperations.metadata}->'exportCleanup'->>'providerLeaseId' = ${lease.providerLeaseId}`,
|
||||
sql`${workspaceOperations.metadata}->'exportCleanup'->>'remoteCwd' = ${remoteRoot}`));
|
||||
const epochOf = (metadata: Record<string, unknown> | null) => Number(metadata?.epoch);
|
||||
if (!rows.length || rows.some(row => !Number.isSafeInteger(epochOf(row.metadata)))) return false;
|
||||
// Every later writer must have used another allocation.
|
||||
const last = Math.max(...rows.map(row => epochOf(row.metadata)));
|
||||
if (last >= epoch) return false;
|
||||
const lastUse = rows.filter(row => epochOf(row.metadata) === last);
|
||||
const saves = lastUse.filter(row => row.phase === "remote_workspace_snapshot" && object(row.metadata?.retiredSave).schema !== RETIRED_SAVE_SCHEMA);
|
||||
if (saves.some(row => row.status !== "succeeded" && row.status !== "failed")) return false;
|
||||
const generation = (row: typeof rows[number]) => Number(row.metadata?.generation);
|
||||
const final = saves.filter(row => row.status === "succeeded").sort((a, b) => generation(b) - generation(a))[0];
|
||||
const work = readSaveWork(final?.metadata);
|
||||
if (!final || final.metadata?.boundary !== "final" || !work || work.unknown || work.activeIds.length
|
||||
|| saves.some(row => row !== final && !(generation(row) < generation(final)))) return false;
|
||||
const [snapshot] = await db.select({ status: executionWorkspaceSnapshots.status }).from(executionWorkspaceSnapshots).where(and(
|
||||
eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id),
|
||||
eq(executionWorkspaceSnapshots.sourceOperationId, final.id)));
|
||||
return Boolean(snapshot && !["pending", "uploaded"].includes(snapshot.status));
|
||||
}
|
||||
|
||||
export async function prepareRemoteWorkspace(input: {
|
||||
db: Db; workspace: Workspace & Pick<typeof executionWorkspaces.$inferSelect, "providerRef">;
|
||||
runId: string; lease: EnvironmentLease; target: AdapterExecutionTarget | null;
|
||||
@@ -405,6 +440,8 @@ export async function prepareRemoteWorkspace(input: {
|
||||
try {
|
||||
const snapshot = await getWorkspaceSnapshot(input.db, { companyId: input.workspace.companyId, workspaceId: input.workspace.id, snapshotId: initial.snapshotId });
|
||||
if (!snapshot) throw new Error("remote_workspace_snapshot_missing");
|
||||
const replaceRetiredRoot = migration === null && !initial.dirty
|
||||
&& await isRetiredOwnAllocation(input.db, input.workspace, input.lease, remoteRoot, initial.epoch);
|
||||
const scratch = await mkdtemp(path.join(os.tmpdir(), "pc-workspace-restore-"));
|
||||
try {
|
||||
const archivePath = path.join(scratch, "workspace.pcws");
|
||||
@@ -413,7 +450,8 @@ export async function prepareRemoteWorkspace(input: {
|
||||
step = "sandbox_restore";
|
||||
restoredExport = await restoreRemoteWorkspaceDirectory({ target, remoteCwd: remoteRoot, archivePath, manifestDigest: snapshot.manifestDigest,
|
||||
restorationIdentity: preparationIdentity, allowEmptyRoot: migration !== null,
|
||||
allocationFiles: migration ? workspaceAllocationFiles(input.lease, remoteRoot) : [] });
|
||||
allocationFiles: migration ? workspaceAllocationFiles(input.lease, remoteRoot) : [],
|
||||
replaceRetiredRoot });
|
||||
} finally { await rm(scratch, { recursive: true, force: true }); }
|
||||
} catch (error) {
|
||||
// Cancellation keeps its own handling, as do lookup refusals.
|
||||
|
||||
Reference in new issue
Block a user