fix(native): restart sandbox prep interrupted before its sync descriptor (#19054)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on sandbox environments sync the host workspace into the
sandbox before the provider starts
> - Runtime selection writes the native execution input first. It writes
the workspace sync reference only after the sandbox is seeded, and
before any provider dispatch
> - A server restart between those two writes leaves a run with an input
but no sync reference
> - Restart recovery then fails the run during workspace preparation
with `native_workspace_sync_descriptor_binding_missing`, although the
sandbox never received any work from that run
> - This pull request lets recovery re-admit the host workspace in that
state, so sync preparation starts fresh on the same run
> - The benefit is that a deploy or restart during sandbox preparation
no longer turns queued work into a terminal `setup_failed` run

## Linked Issues or Issue Description

**What happened?**
After a server restart, six native sandbox runs failed with
`setup_failed` and the error
`native_workspace_sync_descriptor_binding_missing`. Each run timeline
shows the same order: run started, `snapshot.git`, `snapshot.baseline`,
then "Server shutdown suspended native controller ownership", then a
restart recovery claim ("Restarting an incomplete native runner
bootstrap" or "Resuming the durable native provider session"), then the
error. No run had a `nativeWorkspaceSync` reference. Each run had a
persisted `nativeExecutionInput`.

**Expected behavior**
When sandbox sync preparation did not finish, restart recovery prepares
the sandbox workspace again from the host workspace and continues the
run.

**Steps to reproduce**
1. Start a native run on a sandbox environment.
2. Restart the server after runtime selection persists
`nativeExecutionInput` but before `prepareNativeWorkspaceSync` persists
its reference.
3. Restart recovery re-enters workspace preparation and fails with
`native_workspace_sync_descriptor_binding_missing`.

**Paperclip version or commit**
Reproduced on `master` at be20d4ad97.

**Relevant logs or output**
`native_workspace_sync_descriptor_binding_missing` thrown from
`server/src/services/heartbeat/workspace-preparation.ts`.

Related: #17838 (preparation recovery ownership). No open PR covers this
guard.

## What Changed

- `server/src/services/heartbeat/workspace-preparation.ts`: remove the
guard that failed sandbox recovery when the run has no workspace sync
reference. The existing fallback admits `executionWorkspace.cwd`, which
is the same host directory that `prepareNativeWorkspaceSync` then uses
as `workspaceLocalDir`. A comment records the ordering invariant that
makes this safe.
- `server/src/services/heartbeat/workspace-preparation.test.ts`: the
Daytona "absent descriptor" recovery case now expects preparation to
succeed and to re-admit the host workspace. The "corrupt descriptor"
case still fails closed with
`native_workspace_sync_descriptor_digest_mismatch`.

## Verification

- Red before the fix: `npx vitest run
src/services/heartbeat/workspace-preparation.test.ts -t "validates the
admitted host binding"` (from `server/`) failed for "daytona recovery
with absent descriptor" with
`native_workspace_sync_descriptor_binding_missing`, the same error as
the failed runs.
- Green after the fix: the full
`src/services/heartbeat/workspace-preparation.test.ts` file passes,
including the local, ssh, runner-task, valid-descriptor and
corrupt-descriptor recovery cases.
- Ordering check (code read): `runtime-selection.ts` persists
`nativeExecutionInput` in the selection transaction, then calls
`prepareNativeWorkspaceSync`, which persists the reference as its last
step. `heartbeat.ts` calls `executeHeartbeatRuntime` only after runtime
selection returns. No code path removes `nativeWorkspaceSync` from
`runnerProfileJson`.
- Not run: a live restart on a sandbox environment. The original staging
runs are not recovered by this PR.

## Risks

- Low risk. The change only affects native recovery on sandbox targets
(not runner-task, ssh, remote-authority or controller-only runs) when
the sync reference is absent.
- A present but invalid descriptor still fails closed.
- If a future change dispatches a provider before the sync reference is
persisted, this path would re-seed over remote work. The comment at the
call site records that invariant.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser reads of run records).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-04 17:45:00 -06:00
1 parent ac57b183f0
commit 0e27369fa4
2 files changed
+13 -7

No files matched your search

@@ -1762,12 +1762,16 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
}
const reserve = vi.spyOn(input.services.executionWorkspacesSvc, "assertTaskRepositoryRecoveryBinding");
const prepare = vi.spyOn(input.services.executionWorkspacesSvc, "prepareTaskRepositoriesForAdmission");
if (transport !== "daytona" || descriptor === "valid") {
if (descriptor !== "corrupt") {
const result = await prepareHeartbeatWorkspace(db, input);
expect(result.executionTarget).toEqual(target);
if (transport === "ssh" || transport === "task") expect(reserve).not.toHaveBeenCalled();
else expect(reserve).toHaveBeenCalledWith({ companyId: issue.companyId, issueId: issue.id,
workspaceId: first.persistedExecutionWorkspace!.id, cwd: admittedRoot, agentId: input.agent.id, runId: input.run.id });
workspaceId: first.persistedExecutionWorkspace!.id,
// Without a committed sync descriptor the sandbox never received this
// run's work, so recovery re-admits the host workspace it will seed from.
cwd: transport === "daytona" && descriptor === "absent" ? first.executionWorkspace.cwd : admittedRoot,
agentId: input.agent.id, runId: input.run.id });
if (transport !== "daytona") {
await expect(prepareNativeWorkspaceSync({ db, runId: input.run.id, companyId: issue.companyId,
workspaceId: first.persistedExecutionWorkspace!.id, workspaceLocalDir: first.executionWorkspace.cwd,
@@ -1775,8 +1779,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
}
expect(admittedRoot).not.toBe(first.executionWorkspace.cwd);
} else {
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(descriptor === "absent"
? "native_workspace_sync_descriptor_binding_missing" : "native_workspace_sync_descriptor_digest_mismatch");
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("native_workspace_sync_descriptor_digest_mismatch");
expect(reserve).not.toHaveBeenCalled();
expect(await leases(input.run.id)).toMatchObject([{ status: "active" }]);
}
@@ -1447,10 +1447,13 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
&& !(executionTarget?.kind === "remote" && (executionTarget.transport === "ssh"
|| (executionTarget.transport === "sandbox" && executionTarget.runnerTask)))) {
const workspaceId = persistedNativeExecutionInput?.binding.executionWorkspaceId ?? nativeRecoveryExecutionWorkspaceId!;
// Runtime selection persists the native input before it prepares the
// sandbox sync, and commits this reference before any provider dispatch.
// An absent reference therefore means preparation was interrupted (for
// example by a server restart) before the sandbox received this run's work.
// There is no remote copy-back to protect, so recovery re-admits the host
// workspace that the fresh sync preparation will seed from.
const reference = readNativeWorkspaceSyncReference(parseObject(run.runnerProfileJson).nativeWorkspaceSync);
if (!reference && executionTarget?.kind === "remote" && executionTarget.transport === "sandbox") {
throw new Error("native_workspace_sync_descriptor_binding_missing");
}
const cwd = reference
? await readNativeWorkspaceSyncLocalCwd({ companyId: agent.companyId, runId: run.id, workspaceId, reference })
: nativeExecutionUsesTaskWorkspace(persistedNativeExecutionInput)