From fa7950121dbce5de9e84c63d8aa8f72a64479198 Mon Sep 17 00:00:00 2001
From: Dotta <34892728+cryppadotta@users.noreply.github.com>
Date: Sun, 8 Nov 2026 21:06:26 -0600
Subject: [PATCH] fix(native): move an interrupted bootstrap's session aside on
 the run's own created sandbox lease (#20154)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When restart recovery proves a remote bootstrap reached no provider
work, it bootstraps the same run again in the same sandbox. Before
relaunching, it stops the interrupted runner (#20103)
> - The relaunch starts a fresh controller record, whose command ids
restart at `command_prp_00000001`. The sandbox still held the
interrupted runner's journal, which had already recorded that command id
with different data
> - The relaunched runnerd rejected it ("commandId was reused with
different command data") and exited while waiting for `run.prepare`.
Every same-run retry collided the same way until the retry budget ran
out and the task blocked
> - The proven interrupted-bootstrap session move (#20103) ran only for
a resumed lease. A same-run retry on a per-turn sandbox reuses the run's
own created lease, so nothing moved the interrupted session aside
> - This pull request applies that same proof-gated session move on the
run's own created lease
> - The benefit is that a same-run bootstrap retry starts its runner on
a clean journal, and the old journal is kept as evidence

## Linked Issues or Issue Description

Refs #20103

**What happened?**
Daytona per-turn restart-chaos M1 rerun w-20261109T013733-s9501 (master
8e3b64a257, image 34bc12bf):
- **t10 (codex M1):** the codex subtask's run `f8a8f483` went through
eleven same-run bootstrap retries between 01:56 and 02:07.
- Each relaunched runnerd logged `paperclip-runnerd: commandId was
reused with different command data` and exited about 12 s later while
waiting for `run.prepare`.
- The run then failed `native_session_retry_exhausted` ("Provider
startup failed on 3 attempts before any provider work began"). The
subtask blocked, and so did the parent.
- **t02:** the same "runnerd exited while waiting for run.prepare"
pattern appeared on retry run `2e13568f`.

**Expected behavior**
A same-run bootstrap retry moves the interrupted session aside, keeping
it as a `.bootstrap-restart-<time>-<uuid>` sibling, and claims a fresh
session root before relaunching. The relaunched runner then cannot see
the interrupted attempt's command ids. This already happened on a
resumed lease and now also happens on the run's own created lease.

**Steps to reproduce**
1. Let a remote bootstrap reach the runner (it journals
`command_prp_00000001`), then restart before any provider work.
2. Recovery bootstraps the same run again in the same sandbox. Before
this change, the relaunched runnerd sees the old journal, rejects the
reused command id and exits, and every retry repeats this.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`: the created-lease branch of remote
harness preparation now applies the resumed branch's proven restart. It
runs when all of these hold:
  - the session is `not_suspended` and belongs to this run;
  - there is no backup;
  - the claim is `bootstrap_incomplete`;
- the stale-runner stop proved the interrupted runner gone
(`hasRemoteBootstrapRestartProof`).

It then runs `moveInterruptedBootstrapSessionAside` and
`claimUntouchedRemoteSession`. If the move or the claim fails, it throws
`runner_harness_state_mismatch: created_not_suspended` and changes
nothing.
- An earlier revision of this PR archived only the `runner` directory.
Review showed that would leave a session the existing claim cannot
reopen on a resumed lease, so that revision was replaced.
- On a fresh command namespace: moving the session aside removes every
id the relaunched runner could see, so a separate namespace isn't
needed. It would also change the persisted command id format that saved
controller state depends on.
- `native-session-executor.test.ts`: the existing "bootstraps only an
untouched provider session" matrix runs the real move and claim scripts
against a temporary filesystem. It gains two created-lease cases:
- an interrupted bootstrap: the old session is kept as a sibling and a
fresh root is claimed;
  - a symlinked parent: refused, and nothing moves.

## Verification

- Red/green: both created-lease cases fail on master and pass with the
change. The resumed cases from #20103 are unchanged: 42 cases in the
matrix pass.
- `vitest run` over the restart-recovery and bootstrap test files and
`native-session-executor.test.ts`: 7 files, 1540 tests passed (14
skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.
- The collision itself happens in runnerd (the production log shows it
on every retry). The live check is a Daytona per-turn K2/K4 rerun.

## Risks

- Low risk. The move reuses the resumed path's proof gates, its symlink
checks and its whole-session move, which keeps the session and never
deletes it. The new branch acts only when every gate holds; otherwise
preparation continues exactly as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../native-runtime/native-session-executor.test.ts       | 2 ++
 .../services/native-runtime/native-session-executor.ts   | 9 +++++++++
 2 files changed, 11 insertions(+)

diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts
index fb0650cd6ef..c1b67f1737f 100644
--- a/server/src/services/native-runtime/native-session-executor.test.ts
+++ b/server/src/services/native-runtime/native-session-executor.test.ts
@@ -19641,6 +19641,8 @@ describe("runnerd provider runtime wiring", () => {
     ...["resumed", "replacement"].map(outcome => ({ outcome, scenario: "fresh", homeMode: "preconfigured" })),
     ...["interrupted_bootstrap", "interrupted_bootstrap_other_run", "interrupted_bootstrap_unproven", "interrupted_bootstrap_symlink_parent"]
       .map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })),
+    { outcome: "created", scenario: "interrupted_bootstrap", homeMode: "isolated" },
+    { outcome: "created", scenario: "interrupted_bootstrap_symlink_parent", homeMode: "isolated" },
   ])(
     "bootstraps only an untouched provider session in a $outcome workspace lease: $scenario ($homeMode)", async ({ outcome, scenario, homeMode }) => {
     const remoteCwd = join(isolatedStateDirectory, "remote");
diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts
index 8e006540a3f..c2d683a6cbe 100644
--- a/server/src/services/native-runtime/native-session-executor.ts
+++ b/server/src/services/native-runtime/native-session-executor.ts
@@ -15674,6 +15674,15 @@ async function createRunnerdBackendWithinSessionClaim(
                       // lease was not provider-confirmed lost. Never silently create a new
                       // provider session from that ambiguous state.
                       throw new Error("runner_harness_state_mismatch: backup_without_reusable_lease");
+                    } else if (state.incompleteReason === "not_suspended"
+                      && input.restartRecovery?.kind === "bootstrap_incomplete"
+                      && state.notSuspendedRunId === input.execution.binding.runId
+                      && hasRemoteBootstrapRestartProof(input.execution.binding.runId)) {
+                      if (!(await moveInterruptedBootstrapSessionAside()
+                        && takeRemoteBootstrapRestartProof(input.execution.binding.runId)
+                        && await claimUntouchedRemoteSession())) {
+                        throw new Error("runner_harness_state_mismatch: created_not_suspended");
+                      }
                     }
                   }
                 } else if (remoteTarget && remoteCommandRunner) {