mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(native): verify a staged attachment restart on the claimed sandbox lease (#20136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up codex run on a per-turn Daytona sandbox gets a fresh sandbox, with harness state restored from the previous turn's checkpoint, and attaches the previous runner session to the new run through a staged attachment > - A graceful restart during that start suspends the run. The next controller reattaches to the surviving remote runner, after proving it is on the sandbox lease the restart claim names > - The staged-attachment path required the lease to have been acquired as `resumed`. A per-turn follow-up's lease is `created`, so every such restart was refused with `native_runner_staged_attachment_recovery_unproven` > - The run then needed a Retry > - This pull request binds the staged restart to the claimed lease the same way the ordinary remote reattach does: same provider lease, same remote directory, and not a replacement sandbox > - The benefit is that a per-turn follow-up interrupted by a restart continues on its own ## Linked Issues or Issue Description Refs #20103 **What happened?** Daytona per-turn restart-chaos wave w-20261108T225520-s9501 (master15b987d9b8, image cbbe8224). Two codex C1 follow-up runs, created during a K2 graceful drain at 23:11:45, were suspended mid-start: - `4d1d3563` (t07); - `71b43ea1` (t08). Two to three seconds after the controller came back, both failed with `native_runner_staged_attachment_recovery_unproven`, rested in `native_continuation_requires_reconciliation`, and needed a Retry. The stack shows the first guard in `verifyStagedAttachmentRestart`. Both runs' environment leases were acquired with `sandboxLeaseAcquisition.outcome: "created"`, because per-turn mode creates a fresh sandbox for each turn. The guard accepted only `"resumed"`. **Expected behavior** When the restart claim's provider lease and remote directory match the run's own lease, the staged attachment's restart is verified as before, and the run continues without a Retry. The rest of the staged proof is unchanged. A replacement sandbox, or a different provider lease or directory, is still refused. **Steps to reproduce** 1. Run a codex follow-up on a per-turn Daytona environment, so the lease is `created` and the start stages an attachment. 2. Gracefully restart the control plane during the start. 3. Before this change, the restarted run fails with `native_runner_staged_attachment_recovery_unproven`. **Paperclip version or commit**15b987d9b8(wave). Branch rebased on current master. ## What Changed - `verifyStagedAttachmentRestart` (`native-session-executor.ts`) no longer requires `sandboxLeaseAcquisition.outcome === "resumed"`. It requires the same lease binding as `verifyRemoteRunnerReattachment`: - the target's provider lease equals the claim's `remote.providerLeaseId`; - the target's remote directory equals the claim's `remoteCwd`; - the lease is not a `replacement` sandbox. - A new test case in `native-session-executor.test.ts` covers a created sandbox (admitted), a replacement sandbox, another provider lease, and another remote directory (all refused). ## Verification - Red/green: on master, the created-sandbox case fails with `native_runner_staged_attachment_recovery_unproven`. With the change it reaches the staged recheck. The three refusal cases are refused on both. - `vitest run` of `native-session-executor.test.ts` and `native-restart-recovery-isolation.test.ts`: 2 files, 1305 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk. The check now matches the ordinary remote reattach, which already admits created leases. Everything after it is unchanged: the encrypted staged header, the source run and result, the template digest, and the remote runner identity are all still proven before any RPC. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
4420a5adcf
commit
6451d2bfbf
2 files changed
+41
-1
No files matched your search
@@ -15634,6 +15634,44 @@ describe("runnerd provider runtime wiring", () => {
|
||||
} finally { restore.mockRestore(); header.mockRestore(); verify.mockRestore(); await controller.stop(); }
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ label: "a sandbox this run created", outcome: "created", providerLeaseId: "sandbox", remoteCwd: "/workspace", admitted: true },
|
||||
{ label: "a replacement sandbox", outcome: "replacement", providerLeaseId: "sandbox", remoteCwd: "/workspace", admitted: false },
|
||||
{ label: "another provider lease", outcome: "created", providerLeaseId: "other-sandbox", remoteCwd: "/workspace", admitted: false },
|
||||
{ label: "another remote directory", outcome: "created", providerLeaseId: "sandbox", remoteCwd: "/elsewhere", admitted: false },
|
||||
] as const)("binds a staged attachment restart to the claimed sandbox lease: $label", async ({ outcome, providerLeaseId, remoteCwd, admitted }) => {
|
||||
const current = scopedNativeExecution(`staged-attachment-lease-${outcome}-${providerLeaseId}-${remoteCwd.length}`);
|
||||
current.task.prompt = "Inspect attachment ref:original";
|
||||
const root = await initializeScopedRunnerRoot(current, "attachment-runner");
|
||||
const sourceIdentity = { runnerInstanceId: "attachment-runner", environmentLeaseId: "attachment-lease",
|
||||
runId: "attachment-source", normalizedSessionId: current.session.normalizedSessionId!, turnId: "source-turn", itemId: "source-item" };
|
||||
const targetIdentity = { ...sourceIdentity, runId: current.binding.runId, turnId: "target-turn", itemId: "target-item" };
|
||||
const controller = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), identity: sourceIdentity,
|
||||
expectedRunnerVersion: "fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
|
||||
controller.stageRunAttachment(targetIdentity, { provider: { kind: "codex", instructions: "prepared attachment grant" } });
|
||||
const saved = JSON.parse(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8"));
|
||||
state.stageNativeRunnerWakeAttachments.mockImplementationOnce(async () =>
|
||||
({ attachments: [{ id: "attachment-original" }], cleanup: async () => undefined }) as never);
|
||||
state.renderNativeRunnerStagedAttachmentPrompt.mockReturnValueOnce("Materialized attachment: .paperclip-inbound/current/original");
|
||||
const reached = new Error("verified staged admission reached recheck");
|
||||
const restore = vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint").mockResolvedValue(null);
|
||||
const header = vi.spyOn(runnerCheckpoint, "loadNativeRunnerCheckpointHeader").mockResolvedValue("staged header");
|
||||
const verify = vi.spyOn(stagedAttachmentRecovery, "resolveNativeStagedAttachmentRecovery")
|
||||
.mockResolvedValueOnce({ currentExecution: structuredClone(current), sourceIdentity, targetIdentity,
|
||||
templateDigest: saved.stagedRunAttachment.templateDigest } as never)
|
||||
.mockRejectedValueOnce(reached);
|
||||
try {
|
||||
const resume = executePaperclipNativeSession({ db: leaseDb(current), execution: current, useRunnerd: true,
|
||||
runnerInstanceId: "attachment-runner", restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId,
|
||||
leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null,
|
||||
remote: { kind: "sandbox", leaseId: "attachment-lease", providerLeaseId: "sandbox", remoteCwd: "/workspace" } } as never,
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", leaseId: "attachment-lease", remoteCwd,
|
||||
runner: { execute: vi.fn() }, sandboxLeaseAcquisition: { outcome, providerLeaseId } } as never });
|
||||
if (admitted) { await expect(resume).rejects.toBe(reached); expect(verify).toHaveBeenCalledTimes(2); }
|
||||
else { await expect(resume).rejects.toThrow("staged_attachment_recovery_unproven"); expect(verify).not.toHaveBeenCalled(); }
|
||||
} finally { restore.mockRestore(); header.mockRestore(); verify.mockRestore(); await controller.stop(); }
|
||||
});
|
||||
|
||||
it("passes the run checkpoint active turn into restart recovery", async () => {
|
||||
state.createBackend.mockClear();
|
||||
state.createTransport.mockClear();
|
||||
|
||||
@@ -6464,7 +6464,9 @@ async function verifyStagedAttachmentRestart(input: {
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (input.restartRecovery?.kind !== "reattach_remote_runner" || input.restartRecovery.remote.kind === "plugin_task" ||
|
||||
target?.kind !== "remote" || target.transport !== "sandbox" || target.runnerTask ||
|
||||
target.sandboxLeaseAcquisition?.outcome !== "resumed") return deny();
|
||||
target.sandboxLeaseAcquisition?.outcome === "replacement" ||
|
||||
input.restartRecovery.remote.providerLeaseId !== target.sandboxLeaseAcquisition?.providerLeaseId ||
|
||||
input.restartRecovery.remote.remoteCwd !== target.remoteCwd) return deny();
|
||||
const [current] = await input.db.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, input.execution.binding.runId), eq(heartbeatRuns.companyId, input.execution.binding.companyId),
|
||||
eq(heartbeatRuns.agentId, input.execution.binding.agentId), eq(heartbeatRuns.nativeIssueId, input.execution.binding.issueId)));
|
||||
|
||||
Reference in new issue
Block a user