mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(native): move an interrupted bootstrap's session aside on the run's own created sandbox lease (#20154)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When restart recovery proves a remote bootstrap reached no provider work, it bootstraps the same run again in the same sandbox. Before relaunching, it stops the interrupted runner (#20103) > - The relaunch starts a fresh controller record, whose command ids restart at `command_prp_00000001`. The sandbox still held the interrupted runner's journal, which had already recorded that command id with different data > - The relaunched runnerd rejected it ("commandId was reused with different command data") and exited while waiting for `run.prepare`. Every same-run retry collided the same way until the retry budget ran out and the task blocked > - The proven interrupted-bootstrap session move (#20103) ran only for a resumed lease. A same-run retry on a per-turn sandbox reuses the run's own created lease, so nothing moved the interrupted session aside > - This pull request applies that same proof-gated session move on the run's own created lease > - The benefit is that a same-run bootstrap retry starts its runner on a clean journal, and the old journal is kept as evidence ## Linked Issues or Issue Description Refs #20103 **What happened?** Daytona per-turn restart-chaos M1 rerun w-20261109T013733-s9501 (master8e3b64a257, image 34bc12bf): - **t10 (codex M1):** the codex subtask's run `f8a8f483` went through eleven same-run bootstrap retries between 01:56 and 02:07. - Each relaunched runnerd logged `paperclip-runnerd: commandId was reused with different command data` and exited about 12 s later while waiting for `run.prepare`. - The run then failed `native_session_retry_exhausted` ("Provider startup failed on 3 attempts before any provider work began"). The subtask blocked, and so did the parent. - **t02:** the same "runnerd exited while waiting for run.prepare" pattern appeared on retry run `2e13568f`. **Expected behavior** A same-run bootstrap retry moves the interrupted session aside, keeping it as a `.bootstrap-restart-<time>-<uuid>` sibling, and claims a fresh session root before relaunching. The relaunched runner then cannot see the interrupted attempt's command ids. This already happened on a resumed lease and now also happens on the run's own created lease. **Steps to reproduce** 1. Let a remote bootstrap reach the runner (it journals `command_prp_00000001`), then restart before any provider work. 2. Recovery bootstraps the same run again in the same sandbox. Before this change, the relaunched runnerd sees the old journal, rejects the reused command id and exits, and every retry repeats this. **Paperclip version or commit**8e3b64a257(wave). Branch rebased on current master. ## What Changed - `native-session-executor.ts`: the created-lease branch of remote harness preparation now applies the resumed branch's proven restart. It runs when all of these hold: - the session is `not_suspended` and belongs to this run; - there is no backup; - the claim is `bootstrap_incomplete`; - the stale-runner stop proved the interrupted runner gone (`hasRemoteBootstrapRestartProof`). It then runs `moveInterruptedBootstrapSessionAside` and `claimUntouchedRemoteSession`. If the move or the claim fails, it throws `runner_harness_state_mismatch: created_not_suspended` and changes nothing. - An earlier revision of this PR archived only the `runner` directory. Review showed that would leave a session the existing claim cannot reopen on a resumed lease, so that revision was replaced. - On a fresh command namespace: moving the session aside removes every id the relaunched runner could see, so a separate namespace isn't needed. It would also change the persisted command id format that saved controller state depends on. - `native-session-executor.test.ts`: the existing "bootstraps only an untouched provider session" matrix runs the real move and claim scripts against a temporary filesystem. It gains two created-lease cases: - an interrupted bootstrap: the old session is kept as a sibling and a fresh root is claimed; - a symlinked parent: refused, and nothing moves. ## Verification - Red/green: both created-lease cases fail on master and pass with the change. The resumed cases from #20103 are unchanged: 42 cases in the matrix pass. - `vitest run` over the restart-recovery and bootstrap test files and `native-session-executor.test.ts`: 7 files, 1540 tests passed (14 skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. - The collision itself happens in runnerd (the production log shows it on every retry). The live check is a Daytona per-turn K2/K4 rerun. ## Risks - Low risk. The move reuses the resumed path's proof gates, its symlink checks and its whole-session move, which keeps the session and never deletes it. The new branch acts only when every gate holds; otherwise preparation continues exactly as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
510746c682
commit
fa7950121d
2 files changed
+11
No files matched your search
@@ -19641,6 +19641,8 @@ describe("runnerd provider runtime wiring", () => {
|
|||||||
...["resumed", "replacement"].map(outcome => ({ outcome, scenario: "fresh", homeMode: "preconfigured" })),
|
...["resumed", "replacement"].map(outcome => ({ outcome, scenario: "fresh", homeMode: "preconfigured" })),
|
||||||
...["interrupted_bootstrap", "interrupted_bootstrap_other_run", "interrupted_bootstrap_unproven", "interrupted_bootstrap_symlink_parent"]
|
...["interrupted_bootstrap", "interrupted_bootstrap_other_run", "interrupted_bootstrap_unproven", "interrupted_bootstrap_symlink_parent"]
|
||||||
.map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })),
|
.map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })),
|
||||||
|
{ outcome: "created", scenario: "interrupted_bootstrap", homeMode: "isolated" },
|
||||||
|
{ outcome: "created", scenario: "interrupted_bootstrap_symlink_parent", homeMode: "isolated" },
|
||||||
])(
|
])(
|
||||||
"bootstraps only an untouched provider session in a $outcome workspace lease: $scenario ($homeMode)", async ({ outcome, scenario, homeMode }) => {
|
"bootstraps only an untouched provider session in a $outcome workspace lease: $scenario ($homeMode)", async ({ outcome, scenario, homeMode }) => {
|
||||||
const remoteCwd = join(isolatedStateDirectory, "remote");
|
const remoteCwd = join(isolatedStateDirectory, "remote");
|
||||||
|
|||||||
@@ -15674,6 +15674,15 @@ async function createRunnerdBackendWithinSessionClaim(
|
|||||||
// lease was not provider-confirmed lost. Never silently create a new
|
// lease was not provider-confirmed lost. Never silently create a new
|
||||||
// provider session from that ambiguous state.
|
// provider session from that ambiguous state.
|
||||||
throw new Error("runner_harness_state_mismatch: backup_without_reusable_lease");
|
throw new Error("runner_harness_state_mismatch: backup_without_reusable_lease");
|
||||||
|
} else if (state.incompleteReason === "not_suspended"
|
||||||
|
&& input.restartRecovery?.kind === "bootstrap_incomplete"
|
||||||
|
&& state.notSuspendedRunId === input.execution.binding.runId
|
||||||
|
&& hasRemoteBootstrapRestartProof(input.execution.binding.runId)) {
|
||||||
|
if (!(await moveInterruptedBootstrapSessionAside()
|
||||||
|
&& takeRemoteBootstrapRestartProof(input.execution.binding.runId)
|
||||||
|
&& await claimUntouchedRemoteSession())) {
|
||||||
|
throw new Error("runner_harness_state_mismatch: created_not_suspended");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else if (remoteTarget && remoteCommandRunner) {
|
} else if (remoteTarget && remoteCommandRunner) {
|
||||||
|
|||||||
Reference in new issue
Block a user