From 0e27369fa4d9d3ceafc87774bb38dcd7dbd2fe4f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 4 Nov 2026 17:45:00 -0600 Subject: [PATCH] fix(native): restart sandbox prep interrupted before its sync descriptor (#19054) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on sandbox environments sync the host workspace into the sandbox before the provider starts > - Runtime selection writes the native execution input first. It writes the workspace sync reference only after the sandbox is seeded, and before any provider dispatch > - A server restart between those two writes leaves a run with an input but no sync reference > - Restart recovery then fails the run during workspace preparation with `native_workspace_sync_descriptor_binding_missing`, although the sandbox never received any work from that run > - This pull request lets recovery re-admit the host workspace in that state, so sync preparation starts fresh on the same run > - The benefit is that a deploy or restart during sandbox preparation no longer turns queued work into a terminal `setup_failed` run ## Linked Issues or Issue Description **What happened?** After a server restart, six native sandbox runs failed with `setup_failed` and the error `native_workspace_sync_descriptor_binding_missing`. Each run timeline shows the same order: run started, `snapshot.git`, `snapshot.baseline`, then "Server shutdown suspended native controller ownership", then a restart recovery claim ("Restarting an incomplete native runner bootstrap" or "Resuming the durable native provider session"), then the error. No run had a `nativeWorkspaceSync` reference. Each run had a persisted `nativeExecutionInput`. **Expected behavior** When sandbox sync preparation did not finish, restart recovery prepares the sandbox workspace again from the host workspace and continues the run. **Steps to reproduce** 1. Start a native run on a sandbox environment. 2. Restart the server after runtime selection persists `nativeExecutionInput` but before `prepareNativeWorkspaceSync` persists its reference. 3. Restart recovery re-enters workspace preparation and fails with `native_workspace_sync_descriptor_binding_missing`. **Paperclip version or commit** Reproduced on `master` at be20d4ad97. **Relevant logs or output** `native_workspace_sync_descriptor_binding_missing` thrown from `server/src/services/heartbeat/workspace-preparation.ts`. Related: #17838 (preparation recovery ownership). No open PR covers this guard. ## What Changed - `server/src/services/heartbeat/workspace-preparation.ts`: remove the guard that failed sandbox recovery when the run has no workspace sync reference. The existing fallback admits `executionWorkspace.cwd`, which is the same host directory that `prepareNativeWorkspaceSync` then uses as `workspaceLocalDir`. A comment records the ordering invariant that makes this safe. - `server/src/services/heartbeat/workspace-preparation.test.ts`: the Daytona "absent descriptor" recovery case now expects preparation to succeed and to re-admit the host workspace. The "corrupt descriptor" case still fails closed with `native_workspace_sync_descriptor_digest_mismatch`. ## Verification - Red before the fix: `npx vitest run src/services/heartbeat/workspace-preparation.test.ts -t "validates the admitted host binding"` (from `server/`) failed for "daytona recovery with absent descriptor" with `native_workspace_sync_descriptor_binding_missing`, the same error as the failed runs. - Green after the fix: the full `src/services/heartbeat/workspace-preparation.test.ts` file passes, including the local, ssh, runner-task, valid-descriptor and corrupt-descriptor recovery cases. - Ordering check (code read): `runtime-selection.ts` persists `nativeExecutionInput` in the selection transaction, then calls `prepareNativeWorkspaceSync`, which persists the reference as its last step. `heartbeat.ts` calls `executeHeartbeatRuntime` only after runtime selection returns. No code path removes `nativeWorkspaceSync` from `runnerProfileJson`. - Not run: a live restart on a sandbox environment. The original staging runs are not recovered by this PR. ## Risks - Low risk. The change only affects native recovery on sandbox targets (not runner-task, ssh, remote-authority or controller-only runs) when the sync reference is absent. - A present but invalid descriptor still fails closed. - If a future change dispatches a provider before the sync reference is persisted, this path would re-seed over remote work. The comment at the call site records that invariant. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser reads of run records). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../services/heartbeat/workspace-preparation.test.ts | 11 +++++++---- .../src/services/heartbeat/workspace-preparation.ts | 9 ++++++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index 16fc8fd685d..f0d5d898ce6 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -1762,12 +1762,16 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace } const reserve = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryRecoveryBinding"); const prepare = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission"); - if (transport !== "daytona" || descriptor === "valid") { + if (descriptor !== "corrupt") { const result = await prepareHeartbeatWorkspace(db, input); expect(result.executionTarget).toEqual(target); if (transport === "ssh" || transport === "task") expect(reserve).not.toHaveBeenCalled(); else expect(reserve).toHaveBeenCalledWith({ companyId: issue.companyId, issueId: issue.id, - workspaceId: first.persistedExecutionWorkspace!.id, cwd: admittedRoot, agentId: input.agent.id, runId: input.run.id }); + workspaceId: first.persistedExecutionWorkspace!.id, + // Without a committed sync descriptor the sandbox never received this + // run's work, so recovery re-admits the host workspace it will seed from. + cwd: transport === "daytona" && descriptor === "absent" ? first.executionWorkspace.cwd : admittedRoot, + agentId: input.agent.id, runId: input.run.id }); if (transport !== "daytona") { await expect(prepareNativeWorkspaceSync({ db, runId: input.run.id, companyId: issue.companyId, workspaceId: first.persistedExecutionWorkspace!.id, workspaceLocalDir: first.executionWorkspace.cwd, @@ -1775,8 +1779,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace } expect(admittedRoot).not.toBe(first.executionWorkspace.cwd); } else { - await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(descriptor === "absent" - ? "native_workspace_sync_descriptor_binding_missing" : "native_workspace_sync_descriptor_digest_mismatch"); + await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_workspace_sync_descriptor_digest_mismatch"); expect(reserve).not.toHaveBeenCalled(); expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]); } diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index 239a05ebceb..5f3bc76406c 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -1447,10 +1447,13 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace && !(executionTarget?.kind === "remote" && (executionTarget.transport === "ssh" || (executionTarget.transport === "sandbox" && executionTarget.runnerTask)))) { const workspaceId = persistedNativeExecutionInput?.binding.executionWorkspaceId ?? nativeRecoveryExecutionWorkspaceId!; + // Runtime selection persists the native input before it prepares the + // sandbox sync, and commits this reference before any provider dispatch. + // An absent reference therefore means preparation was interrupted (for + // example by a server restart) before the sandbox received this run's work. + // There is no remote copy-back to protect, so recovery re-admits the host + // workspace that the fresh sync preparation will seed from. const reference = readNativeWorkspaceSyncReference(parseObject(run.runnerProfileJson).nativeWorkspaceSync); - if (!reference && executionTarget?.kind === "remote" && executionTarget.transport === "sandbox") { - throw new Error("native_workspace_sync_descriptor_binding_missing"); - } const cwd = reference ? await readNativeWorkspaceSyncLocalCwd({ companyId: agent.companyId, runId: run.id, workspaceId, reference }) : nativeExecutionUsesTaskWorkspace(persistedNativeExecutionInput)