From cedf335c98b83a911e3f7ceff4bffeb67d92a445 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 20:28:12 -0600 Subject: [PATCH 01/12] fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records that turn as failed with `provider_turn_lost_on_restore` and continues on a same-conversation continuation turn > - Restart recovery blocks a run as `provider_checkpoint_permanently_failed` when its checkpoint carries a failed run terminal and no pending restart interruption > - While a continuation is running, the checkpoint keeps the lost turn's failed run terminal and names the continuation as the active turn. The pending-interruption check returns nothing once a turn is active > - So a second restart during the continuation read the run as permanently failed, blocked replay, and needed the board > - This pull request treats that failed terminal as the restart loss it records whenever an active turn follows a restart-lost terminal turn > - The benefit is that runs survive repeated restarts during their continuations, including M1 parents and their subtasks ## Linked Issues or Issue Description Refs #20148 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master 8e3b64a257), t05 (claude M1 parent): 1. Run `e1e37dab` lost turns to restarts at 00:41:32 and 00:47:53. Each time it continued, and the third continuation `turn_lab_92b03dc7…` was active. 2. At 00:48:29 the next restart blocked it: `native_restart_recovery_blocked` / `provider_checkpoint_permanently_failed`, with replay blocked. 3. Its saved checkpoint held `terminal.runTerminalState: "failed"` (from the lost turn), `activeTurnId` set to the continuation, and a last terminal turn fingerprinted `provider_turn_lost_on_restore`. Its codex/ACPX subtask's run `f9b10d06` was blocked in the same pass with the same shape, and its continuation had already proposed a semantic result. Both stayed blocked. The parent's Retry was held on the blocked child. **Expected behavior** A failed run terminal that records a restart loss, followed by an active continuation turn, is not a permanent failure. Recovery claims `resume_dead_runner`. The runtime then reconciles the active turn, and a held result settles a lost turn as completed. A real provider failure followed by a later turn still blocks. **Steps to reproduce** 1. Checkpoint a run whose last terminal turn is a restart loss, whose `terminal` is failed, and whose `activeTurnId` is a later continuation (optionally with a semantic result). 2. Claim restart recovery. Before this change it blocks with `provider_checkpoint_permanently_failed`. **Paperclip version or commit** 8e3b64a257 (wave). Branch rebased on current master. ## What Changed - `contracts/native-session-backend.ts` (runner): the new `nativeRestartContinuationInFlight` is true when all of these hold: - the checkpoint has an active turn; - its last terminal turn is a different turn; - that turn's fingerprint is a failed, result-less, recoverable `provider_turn_lost_on_restore`. - `native-session-runtime.ts`: recovery clears that stale failed terminal for a continuation in flight, as it already does for a checkpointed interruption. Before, it threw `provider_checkpoint_failed_terminal` before it reconciled the active turn. - `native-restart-recovery.ts` (server): `checkpointFailed` now also requires that no restart continuation be in flight. It uses the same contract function through the vendor boundary. - `native-session-runtime.test.ts`: an end-to-end recovery test. A checkpoint with the stale failed terminal and an active continuation resolves on the continuation's result without starting a new turn. - `native-runner-restart-recovery.integration.test.ts`: - a codex case of the stale-terminal shape as saved in production; - an ACPX case with a held result; - a held case where an active turn follows a real provider failure (`model_not_found`). ## Verification - Red/green: on master, the stale-terminal case is blocked with `provider_checkpoint_permanently_failed`. With the change it claims `resume_dead_runner`. The real-failure case stays blocked on both. - Red/green (runtime): without the runtime change, the end-to-end test throws `provider_checkpoint_failed_terminal`. With it, the run completes. - `vitest run` over the restart-recovery and failed-workspace-recovery test files in `server/src/services/native-runtime`: 5 files, 305 tests passed (14 skipped). - `vitest run` over the runner restart, recovery, continuation and terminal test files: 9 files, 385 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`) and `pnpm run typecheck:typescript` (`packages/paperclip-runner`): passed. ## Risks - Low risk. The exception applies only when the recorded failure is itself a recoverable restart loss and a later turn is in flight. Every other recovery gate is unchanged: the provider-attempt budget, controller takeover and process proofs. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/contracts/native-session-backend.ts | 14 ++++++ .../src/native-session-runtime.test.ts | 49 ++++++++++++++++++- .../src/native-session-runtime.ts | 3 +- .../native-runtime/native-restart-recovery.ts | 5 +- ...unner-restart-recovery.integration.test.ts | 29 +++++++++-- server/src/vendor/paperclip-runner/index.ts | 1 + 6 files changed, 94 insertions(+), 7 deletions(-) diff --git a/packages/paperclip-runner/src/contracts/native-session-backend.ts b/packages/paperclip-runner/src/contracts/native-session-backend.ts index 2057e8ee56b..3ac90be6d7d 100644 --- a/packages/paperclip-runner/src/contracts/native-session-backend.ts +++ b/packages/paperclip-runner/src/contracts/native-session-backend.ts @@ -118,6 +118,20 @@ export function nativeRestartInterruptedTurnId(snapshot: Pick): boolean { + const activeTurnId = snapshot.activeTurnId; + if (typeof activeTurnId !== "string" || !activeTurnId) return false; + const last = Array.isArray(snapshot.terminalTurns) ? snapshot.terminalTurns.at(-1) : null; + if (!last || typeof last.fingerprint !== "string" || !last.turnId || last.turnId === activeTurnId) return false; + try { + const fingerprint = JSON.parse(last.fingerprint); + return fingerprint?.terminalState === "failed" && fingerprint.result === null && + isNativeRestartInterruption(fingerprint.error); + } catch { return false; } +} + /** * Cancellation is a synchronous authority transition followed by passive * provider cleanup. Once `cancel` returns, the provider session must no longer diff --git a/packages/paperclip-runner/src/native-session-runtime.test.ts b/packages/paperclip-runner/src/native-session-runtime.test.ts index 73bff682ae6..9d40f756c99 100644 --- a/packages/paperclip-runner/src/native-session-runtime.test.ts +++ b/packages/paperclip-runner/src/native-session-runtime.test.ts @@ -22,7 +22,7 @@ import { NativeSessionCloseUnrecoverableError, NativeSessionCleanupQuarantinedError, NativeSessionProtocolIntegrityError, - nativeRestartInterruptedTurnId, + nativeRestartContinuationInFlight, nativeRestartInterruptedTurnId, } from "./contracts/native-session-backend.js"; import { PROVIDER_ATTACH_REFUSED, PROVIDER_SIDE_EFFECT_PENDING, PROVIDER_STATE_INCOMPATIBLE } from "./contracts/harness-driver.js"; import type { @@ -642,6 +642,53 @@ describe("executeNativeSession recovery", () => { expect(events.filter(event => event.eventType === "turn.failed").map(event => event.turnId)).toEqual(["turn-unrecorded-continuation"]); }); + it("recovers a restart continuation whose checkpoint still carries the lost turn's failed terminal", async () => { + const failure = { code: "provider_turn_lost_on_restore", recoverable: true }; + const checkpoint: PersistedNativeSession = { + backendKind: "runner", driverKind: "codex_app_server", sessionId: "thread-recovery", identity, + providerSessionId: "provider-recovery", cursor: "0", activeTurnId: "turn-continuation", + terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "failed", turnTerminalState: "failed", reportedWorkDisposition: "yielded" }, + terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }], + semanticResult: null, pendingRuntimeRequests: [], lineage: [], + dispositionOnlyRecoveryConsumed: true, dispositionOnlyRecoveryTurnId: "turn-continuation", + }; + const startTurn = vi.fn(async () => ({ turnId: "turn-unexpected" })); + const session: NativeSession = { + identity: () => identity, + async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; }, + async *events() { + yield { ...runnerEvent(1, "turn.completed", { status: "completed" }), turnId: "turn-continuation" }; + }, + startTurn, + async result() { return { result, terminal, turnId: "turn-continuation" }; }, + async snapshot() { return structuredClone(checkpoint); }, + async close() {}, + }; + const backend: NativeSessionBackend = { + async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; }, + async openSession() { throw new Error("must preserve the provider session"); }, + async recoverSession() { return { recovered: true, session }; }, + }; + const port: ControlPlanePort = { + async openRun() {}, + async checkpointSession() {}, + async appendEvent(event: PrpEvent) { return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; }, + async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; }, + async completeRun() {}, + }; + await expect(executeNativeSession({ + input, backend, controlPlane: port, persistedSession: checkpoint, + runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery", + resumeInterruptedTurn: true, + })).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } }); + expect(startTurn).not.toHaveBeenCalled(); + expect(nativeRestartContinuationInFlight(checkpoint)).toBe(true); + expect(nativeRestartContinuationInFlight({ ...checkpoint, activeTurnId: "turn-recovery" })).toBe(false); + expect(nativeRestartContinuationInFlight({ ...checkpoint, terminalTurns: [{ turnId: "turn-recovery", + fingerprint: JSON.stringify({ terminalState: "failed", error: { code: "model_not_found", recoverable: false }, result: null }) }] })) + .toBe(false); + }); + it.each(["acpx_runtime", "opencode_server"] as const)("continues a restart-lost %s turn once in the same conversation", async (driverKind) => { const qualified = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5"); const acpxInput: NativeExecutionInputV1 = (driverKind === "acpx_runtime" ? { diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.ts index 180521404b6..2e8bcb37074 100644 --- a/packages/paperclip-runner/src/native-session-runtime.ts +++ b/packages/paperclip-runner/src/native-session-runtime.ts @@ -33,6 +33,7 @@ import { NativeSessionCleanupQuarantinedError, NativeSessionProtocolIntegrityError, isNativeRestartInterruption, + nativeRestartContinuationInFlight, nativeRestartInterruptedTurnId, } from "./contracts/native-session-backend.js"; import { @@ -2327,7 +2328,7 @@ export async function executeNativeSession( } | null = null; let reconciledRecoveryCheckpoint: PersistedNativeSession | null = null; const checkpointedInterruption = persistedSession && nativeRestartInterruptedTurnId(persistedSession); - if (checkpointedInterruption && persistedSession) { + if (persistedSession && (checkpointedInterruption || nativeRestartContinuationInFlight(persistedSession))) { // The turn failed because its process vanished; the conversation is still // usable. Keep its terminal fingerprint so replay cannot execute it twice. persistedSession = { ...persistedSession, terminal: null }; diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index a5c799ff64a..bf8033add40 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -47,7 +47,7 @@ import { logger } from "../../middleware/logger.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js"; import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js"; -import { nativeRestartInterruptedTurnId, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { nativeRestartContinuationInFlight, nativeRestartInterruptedTurnId, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { retireSupersededNativeRetry } from "./native-superseded-retry.js"; import { legacyWorkspaceRecoveryNotMigrated } from "../legacy-workspace-recovery-migration.js"; import { requestNativeStartupCleanup, readNativeStartupContext, promoteNativeStartupCleanup } from "./native-startup-cleanup.js"; @@ -1436,7 +1436,8 @@ export async function claimNativeRestartRecoveries(input: { checkpointFailed: (checkpointRecord.terminal as Record | undefined) ?.runTerminalState === "failed" && - nativeRestartInterruptedTurnId(checkpointRecord as unknown as PersistedNativeSession) === null, + nativeRestartInterruptedTurnId(checkpointRecord as unknown as PersistedNativeSession) === null && + !nativeRestartContinuationInFlight(checkpointRecord as unknown as PersistedNativeSession), providerAttempt: row.coordinator.attempt, resumeChargesAttempt: !resumeKeepsAttempt, }); diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 31ef5944755..5dad4687c4e 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -2892,8 +2892,10 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); { attempt: 2, driverKind: "opencode_server" }, // A second restart that loses the first restart's continuation turn. { attempt: 2, driverKind: "codex_app_server", lostContinuation: true }, - ])("recovers a checkpointed $driverKind restart interruption within provider budget $attempt", async ({ attempt, driverKind, lostContinuation }) => { - const fixture = await seedRun(`INTERRUPTED-${driverKind}-${attempt}${lostContinuation ? "-CONTINUATION" : ""}`); + { attempt: 2, driverKind: "codex_app_server", lostContinuation: true, staleTerminal: true }, + { attempt: 2, driverKind: "acpx_runtime", lostContinuation: true, staleTerminal: true, heldResult: true }, + ])("recovers a checkpointed $driverKind restart interruption within provider budget $attempt", async ({ attempt, driverKind, lostContinuation, staleTerminal, heldResult }) => { + const fixture = await seedRun(`INTERRUPTED-${driverKind}-${attempt}${lostContinuation ? "-CONTINUATION" : ""}${staleTerminal ? "-STALE" : ""}${heldResult ? "-RESULT" : ""}`); await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { driverKind, sessionId: fixture.native.normalizedSessionId, @@ -2908,7 +2910,8 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); }) }], // A restart during the first continuation: its turn is still the // checkpointed active turn and the consumed marker names it. - ...(lostContinuation ? { activeTurnId: "continuation-turn", terminal: null, + ...(heldResult ? { semanticResult: { schema: "paperclip.run_result.v1", summary: "Saved before the restart." } } : {}), + ...(lostContinuation ? { activeTurnId: "continuation-turn", ...(staleTerminal ? {} : { terminal: null }), dispositionOnlyRecoveryConsumed: true, dispositionOnlyRecoveryTurnId: "continuation-turn" } : {}), } } }).where(eq(heartbeatRuns.id, fixture.runId)); await fixture.db.update(nativeRunFinalizations).set({ attempt }).where(eq(nativeRunFinalizations.runId, fixture.runId)); @@ -2923,6 +2926,26 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); : { kind: "blocked", runId: fixture.runId, reason: "execution_recovery_budget_exhausted" })]); }); + it("keeps a run whose active turn follows a real provider failure blocked as permanently failed", async () => { + const fixture = await seedRun("ACTIVE-AFTER-PROVIDER-FAILURE"); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { + driverKind: "codex_app_server", + sessionId: fixture.native.normalizedSessionId, + identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId }, + providerSessionId: "failed-provider", + activeTurnId: "later-turn", + semanticResult: null, + terminal: { runTerminalState: "failed", turnTerminalState: "failed" }, + terminalTurns: [{ turnId: "failed-turn", fingerprint: JSON.stringify({ + terminalState: "failed", result: null, error: { code: "model_not_found", recoverable: false }, + }) }], + } } }).where(eq(heartbeatRuns.id, fixture.runId)); + await fixture.db.update(nativeRunFinalizations).set({ attempt: 1 }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + await appendRunEvents(fixture, [{ eventType: "native.process_start_requested" }, { eventType: "turn.accepted", turnId: "later-turn" }]); + expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, restartKind: "hot", runIds: [fixture.runId] })) + .toEqual([expect.objectContaining({ kind: "blocked", runId: fixture.runId, reason: "provider_checkpoint_permanently_failed" })]); + }); + // RC-22b: controller crashes (no drain record) that each interrupted a resume // which had already done work exhausted the three-attempt budget. async function seedInterruptedCheckpoint(label: string) { diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 38108ec7a08..732c76e741d 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -193,6 +193,7 @@ export const NativeProviderStateIncompatibleError = runner.NativeProviderStateIn export const NativeProviderSideEffectPendingError = runner.NativeProviderSideEffectPendingError; export const NativeRunnerSourceReconciliationRequiredError = runner.NativeRunnerSourceReconciliationRequiredError; export const nativeRestartInterruptedTurnId = runner.nativeRestartInterruptedTurnId; +export const nativeRestartContinuationInFlight = runner.nativeRestartContinuationInFlight; export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup; export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup; From 3eea8b87bd61c5249063749cc70cbe20532a8675 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 20:47:05 -0600 Subject: [PATCH 02/12] fix(native): settle a finished owner whose journal holds a definitively answered command (#20148) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a per-turn follow-up starts a new runner, it retires the previous turn's finished owner. Retirement requires that owner's controller journal to be settled: every command answered, then a run terminal > - A restart can cut off a continuation turn. The recovered runner answers that `turn.start` with `failed` and the nested code `execution_indeterminate` ("it will not execute twice"), and the run continues on a later turn to an accepted result > - Retirement counted only `completed` commands as answered. A finished run whose journal held that answered `turn.start` never counted as settled, so the follow-up failed with `retirement_refused=journal_commands_unsettled(turn.start.failed)` and needed a Retry > - This pull request counts definitive `failed` and `rejected` answers as settled, except for runner lifecycle commands. An unknown-outcome `turn.start` counts as settled only once a later `turn.start` completed, which shows the run moved past it. Every other unknown or pending command stays unsettled > - The benefit is that a follow-up after a restart-interrupted, then finished, run continues on its own ## Linked Issues or Issue Description Refs #20140 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master 8e3b64a257), t07 (claude C1, K6): - The previous run `9cb2c0b3` lost a continuation turn to a restart. The runner answered that `turn.start` as `failed` with `result.result.code: "execution_indeterminate"`. - After the next restore it ran a new continuation to an accepted result, and the run terminal was committed at 00:48:18. - A hard kill then landed before the runner suspended, and recovery finalized the result. - Follow-up `372991c8` failed at 00:48:37 with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. **Expected behavior** A command the runner definitively answered (`failed` or `rejected`) leaves nothing outstanding, so it counts as settled. A `turn.start` with an unknown outcome is never executed again by the runner, so once a later `turn.start` completed, the run has moved past it and it also counts as settled. The follow-up then retires the finished owner and continues. The following still count as unsettled: - an unknown-outcome `turn.start` with no later completed turn; - any other unknown-outcome command; - a `pending` command; - a failed runner lifecycle command, such as a failed `runner.suspend` (the runner did not suspend). **Steps to reproduce** 1. Finish a per-turn run whose controller journal holds a `turn.start` answered `failed`, followed by a committed run terminal. 2. Start a follow-up. Before this change, retirement refuses with `journal_commands_unsettled(turn.start.failed)`. **Paperclip version or commit** 8e3b64a257 (wave). Branch rebased on current master. ## What Changed - `remote-warm-session-recovery.ts`: the new `unsettledControllerCommands` decides which commands are settled. `isSettledRemoteWarmControlState` uses it. - `completed` is settled. - An unknown outcome (raw `indeterminate`, or `failed` with `execution_indeterminate` at `result.code` or `result.result.code`) is settled only for a `turn.start` with a later completed `turn.start`. - Other `failed` or `rejected` answers are settled, except for `runner.suspend`, `runner.drain` and `runner.shutdown`. - `native-session-executor.ts`: the retirement refusal message lists unsettled commands with the same rule. - `native-recovered-owner-followup.test.ts`: a red/green test whose journals are in real lifecycle order. - Admitted: a `turn.start` answered `failed` or `rejected`; a superseded unknown-outcome `turn.start` in the runner's stored nested shape; a superseded raw `indeterminate`. - Held: the same unknown-outcome answer with no later completed turn; a failed `runner.suspend`. ## Verification - Red/green: on master, the admitted cases fail with exactly the t07 error. With the change they are admitted. The held cases stay held. If only `result.code` is checked, the unsuperseded case fails. - `vitest run` of `native-recovered-owner-followup.test.ts`, `remote-warm-session-recovery.test.ts` and `native-session-executor.test.ts`: 3 files, 1403 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk. Definitive answers to non-lifecycle commands, and unknown-outcome turn starts that a completed later turn superseded, become settled. - The run terminal requirement, post-terminal live-work checks, and every lifecycle and process proof are unchanged. - The same rule now also applies to remote warm recovery's settled check, where a definitively answered command is equally finished. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-recovered-owner-followup.test.ts | 34 +++++++++++++++++-- .../native-runtime/native-session-executor.ts | 4 +-- .../remote-warm-session-recovery.ts | 24 ++++++++++++- 3 files changed, 57 insertions(+), 5 deletions(-) diff --git a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts index 178faddc960..ae9be61febe 100644 --- a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts +++ b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts @@ -90,7 +90,9 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered * later committed its result and released the controller lease, but no * suspended Runner receipt or failover backup ever reached the controller. * Only the database controller checkpoint and recovery pointer survive. */ - async function seedRecoveredPriorRun(control: "settled" | "pending-command" | "unanswered-tool" = "settled", + async function seedRecoveredPriorRun(control: "settled" | "pending-command" | "unanswered-tool" + | "failed-turn-start" | "rejected-turn-start" | "indeterminate-turn-start" | "unknown-outcome-turn-start" + | "unsuperseded-unknown-outcome-turn-start" | "failed-suspend" = "settled", options: { controller?: { bootId: string; pid: number; processStartedAt: Date }; provider?: "codex" | "acpx" | "opencode" } = {}) { const issueId = randomUUID(), workspaceId = randomUUID(), runId = randomUUID(); await db.insert(issues).values({ id: issueId, companyId, title: "Continue after recovery", assigneeAgentId: agentId, status: "in_progress" }); @@ -127,10 +129,25 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered try { // Shape the file-backed journal first; the database checkpoint captures // it when persistence moves to the controller's external writer. - const commands = ["run.prepare", "session.open", "turn.start", "runner.suspend"].map(type => controller.queueCommand(type)); + const unknownOutcome = { status: "failed", result: { commandType: "turn.start", status: "failed", + result: { code: "execution_indeterminate", message: "runner recovered after journaling this command; it will not execute twice" } } }; + const answers = { + "failed-turn-start": { status: "failed", result: { status: "failed" } }, + "rejected-turn-start": { status: "rejected", result: { status: "rejected" } }, + "indeterminate-turn-start": { status: "indeterminate", result: { status: "indeterminate" } }, + "unknown-outcome-turn-start": unknownOutcome, + } as const; + const answered = answers as Record; + const commands = ["run.prepare", "session.open", ...(control in answered ? ["turn.start"] : []), "turn.start", "runner.suspend"] + .map(type => controller.queueCommand(type)); for (const command of control === "pending-command" ? commands.slice(0, -1) : commands) { Object.assign(command, { status: "completed", result: { status: "completed" } }); } + if (control in answered) Object.assign(commands[2]!, answers[control as keyof typeof answers]); + if (control === "unsuperseded-unknown-outcome-turn-start") { + Object.assign(commands.find(command => command.type === "turn.start")!, unknownOutcome); + } + if (control === "failed-suspend") Object.assign(commands.at(-1)!, { status: "failed", result: { status: "failed" } }); const event = (sourceSeq: number, sourceEventId: string, eventType: string) => ({ sourceSeq, sourceEventId, eventType, priority: 0 as const, envelope: { ...identity }, deliveryCount: 1, logicalEffectCount: 1 }); controller.store.state.committedEvents.push( @@ -628,6 +645,19 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered expect(await retry.start()).toBe("native_controller_recovery_unsupported:metadata_format"); }, 60_000); + it.each([ + { control: "failed-turn-start", outcome: ADMITTED }, + { control: "rejected-turn-start", outcome: ADMITTED }, + { control: "indeterminate-turn-start", outcome: ADMITTED }, + { control: "unknown-outcome-turn-start", outcome: ADMITTED }, + { control: "unsuperseded-unknown-outcome-turn-start", outcome: stranded("journal_commands_unsettled(turn.start.failed)") }, + { control: "failed-suspend", outcome: stranded("journal_commands_unsettled(runner.suspend.failed)") }, + ] as const)("retires a finished owner whose journal holds an answered $control command", async ({ control, outcome }) => { + const prior = await seedRecoveredPriorRun(control); + const followUp = await admitFollowUp(prior); + expect(await followUp.start()).toBe(outcome); + }, 60_000); + it("admits again when a retry rebuilds the same projection from the database", async () => { const prior = await seedRecoveredPriorRun(); const followUp = await admitFollowUp(prior); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c74977841dd..4782abec0a4 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -267,7 +267,7 @@ import { remoteRecoveryProbeReason, } from "./remote-runner-recovery-diagnostics.js"; import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js"; -import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery } from "./remote-warm-session-recovery.js"; +import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery, unsettledControllerCommands } from "./remote-warm-session-recovery.js"; import { assertNativeWarmIngressRetirement, nativeWarmIngressAllocation, nativeWarmIngressIdentity, type NativeWarmIngressClaim, type NativeWarmIngressOwner } from "./native-warm-ingress-ownership.js"; import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; import { emitStartupReuse, environmentStartupReuse } from "@paperclipai/adapter-utils/startup-reuse"; @@ -5458,7 +5458,7 @@ function readSettledRunnerdProjection(root: string, identity: Record command.status !== "completed") + const unsettled = unsettledControllerCommands(settledCommands) .map(command => `${retirementReasonName("commandType", command.type)}.${retirementReasonName("commandStatus", command.status)}`); return refuse(unsettled.length > 0 ? `journal_commands_unsettled(${[...new Set(unsettled)].slice(0, 4).join(",")})` : "journal_terminal_unsettled"); diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.ts b/server/src/services/native-runtime/remote-warm-session-recovery.ts index 40324702407..d1759466ed7 100644 --- a/server/src/services/native-runtime/remote-warm-session-recovery.ts +++ b/server/src/services/native-runtime/remote-warm-session-recovery.ts @@ -31,10 +31,32 @@ function postTerminalEvents(control: Record): unknown[] { /** A session is settled when its commands finished and nothing is live or waiting. * Events after the terminal describe the past, so refusing cannot undo them: they * are reported instead. Every event must still authenticate as this session's. */ +const RUNNER_LIFECYCLE_COMMAND_TYPES = new Set(["runner.suspend", "runner.drain", "runner.shutdown"]); + +function outcomeUnknown(command: Record): boolean { + return command.status === "indeterminate" || + (command.status === "failed" && (record(command.result).code === "execution_indeterminate" || + record(record(command.result).result).code === "execution_indeterminate")); +} + +export function unsettledControllerCommands(commands: readonly Record[]): Record[] { + const lastCompletedTurnStart = Math.max(-1, ...commands + .filter(command => command.type === "turn.start" && command.status === "completed") + .map(command => Number(command.controllerSeq))); + return commands.filter(command => { + if (command.status === "completed") return false; + if (outcomeUnknown(command)) { + return !(command.type === "turn.start" && Number(command.controllerSeq) < lastCompletedTurnStart); + } + return !((command.status === "failed" || command.status === "rejected") && + !RUNNER_LIFECYCLE_COMMAND_TYPES.has(String(command.type))); + }); +} + export function isSettledRemoteWarmControlState(control: Record, identity: Record): boolean { const commands = control.commands; const events = control.committedEvents; - if (!Array.isArray(commands) || commands.some(command => record(command).status !== "completed") || + if (!Array.isArray(commands) || unsettledControllerCommands(commands.map(record)).length > 0 || !Array.isArray(events) || Object.entries(identity).some(([key, value]) => record(control.identity)[key] !== value)) return false; const terminalIndex = events.findLastIndex(event => record(event).eventType === "run.terminal"); if (terminalIndex < 0) return false; From 763adbea09418be3c43ce1f296a3174da6ebc978 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 20:55:36 -0600 Subject: [PATCH 03/12] fix(execution): a pause's stop never blocks the task behind an incident with no control (#20143) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop pauses the agent, which stops its running work > - In the restart-chaos H1 scenario, the stopped task went `blocked` behind a board action that said "inspect and reconcile" and offered no button, and it stayed there after the board resumed the budget > - A pause is an intentional stop, not a failure, so it must not leave the task with no way forward > - This pull request treats a pause's stop as intentional: the task keeps its status and no incident opens, so the normal Retry works > - The benefit is that a budget or agent pause never strands a task behind a control-less incident ## Linked Issues or Issue Description Refs #20113, #20101 **What happened?** Restart-chaos wave w-20261108T210118-s9802 (H1). At the budget pause, task CHA-1 went `blocked` (21:02:27.870) with an active board `active_run_watchdog` action, cause `native_continuation_requires_reconciliation`, next action "Inspect the original failure and reconcile the previous execution before continuing." After raise_budget_and_resume it stayed blocked, `canRetry=false`, with no Retry. The run was cancelled by the agent lifecycle pause (`cancellation: { reason: "Agent lifecycle stop requested", initiator: system, expected: true }`). The wake-queue release (`recordNativeTerminalRecoveryIfNeeded`) blocks every cancelled native run that is not an acknowledged user Stop. **Expected behavior** A pause's stop keeps the task's status and opens no incident; the task can be continued. **Steps to reproduce** Pause an agent (budget hold or company budget pause) while its native run is active. Before this change, the task is blocked behind a board action with no control. ## What Changed - New `server/src/services/governance-pause-stop.ts`: `isGovernancePauseStop(run)` matches an expected system cancellation whose reason is the budget pause stop, or the agent lifecycle stop with a `pauseStop` stamp. - `wake-queue/adapters/postgres.ts` `recordNativeTerminalRecoveryIfNeeded`: returns early for such a run, so the release keeps the task's status and opens no reconciliation incident. - `agent-lifecycle-driver.ts`: the `pausing` host step stamps `resultJson.pauseStop = { holds, companyPauseReason }` on the agent's active runs before stopping them (one update, company pause reason by subquery). Termination and cleanup share the lifecycle stop reason but write no stamp, so they still open the incident. ## Verification - `wake-queue/adapters/postgres.test.ts`: new cases. A run stopped by the lifecycle pause or by the budget pause keeps the task `in_progress` with no incident (both red on master). Agent termination, and another system stop (company archive), still open the incident. - `agent-lifecycle/__tests__/lifecycle.test.ts`: "records what a pause stopped each active run for". `driver-scope.test.ts` updated for the stamp's single update. - `wake-queue` (107), `lifecycle` (68+), `driver-scope`, `companies-service`: 199 passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (after the follow-up continuation PR). ## Risks - A native run that a pause stopped no longer opens a reconciliation incident. Resuming it is the normal Retry, which continues the conversation. Termination, archive and failures still open one. - One extra update per pause attempt on the agent's active runs. It is idempotent (skips runs that already have the stamp) and not on any turn path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/driver-scope.test.ts | 9 +- .../__tests__/lifecycle.test.ts | 13 +++ .../wake-queue/adapters/postgres.test.ts | 90 ++++++++++++++++++- .../modules/wake-queue/adapters/postgres.ts | 13 +++ server/src/services/agent-lifecycle-driver.ts | 17 +++- server/src/services/governance-pause-stop.ts | 29 ++++++ 6 files changed, 165 insertions(+), 6 deletions(-) create mode 100644 server/src/services/governance-pause-stop.ts diff --git a/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts b/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts index 2f1999565b7..da514753201 100644 --- a/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts @@ -8,10 +8,13 @@ import { createLifecycleDriver } from "../../../services/agent-lifecycle-driver. describe("lifecycle host stop scope", () => { it("pauses only the requested agent without loading its company peers", async () => { - const db = { select: vi.fn(() => { throw new Error("No peer lookup expected"); }) } as unknown as Db; + // The pause stamps its cause on the agent's own runs (one update), never a peer lookup. + const update = vi.fn(() => ({ set: () => ({ where: async () => [] }) })); + const db = { select: vi.fn(() => { throw new Error("No peer lookup expected"); }), update } as unknown as Db; const driver = createLifecycleDriver(db, {} as never); - expect(await driver.runHost({ id: "target", companyId: "company", lifecycleState: "pausing", - lifecycleVersion: 2, lifecycleRequiredPluginIds: [], lifecycleError: null, lifecycleOperation: null })).toBe("complete"); + expect(await driver.runHost({ id: "target", companyId: "company", lifecycleState: "pausing", lifecycleHolds: ["manual"], + lifecycleVersion: 2, lifecycleRequiredPluginIds: [], lifecycleError: null, lifecycleOperation: null } as never)).toBe("complete"); + expect(update).toHaveBeenCalledOnce(); // A pause keeps the messages it deferred (work-intent outbox). expect(stop).toHaveBeenCalledWith(["target"], "Agent lifecycle stop requested", { preservePauseDeferred: true }); expect(db.select).not.toHaveBeenCalled(); diff --git a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts index e55f627582a..ea0b6ba6da7 100644 --- a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts @@ -371,6 +371,19 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu expect(await db.select().from(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId))).toHaveLength(0); }); + // H1 bug B: a pause's stop is told apart from termination by the cause the + // pause records on each run it stops. + it("records what a pause stopped each active run for", async () => { + const agent = await hire(); + await db.update(companies).set({ status: "paused", pauseReason: "budget" }).where(eq(companies.id, companyId)); + await db.update(agents).set({ lifecycleState: "pausing", lifecycleHolds: ["company_paused"] }).where(eq(agents.id, agent.id)); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "queued" }).returning(); + await createLifecycleDriver(db, {} as never).runHost(await current(agent.id) as never); + const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + expect(stopped.status).toBe("cancelled"); + expect(stopped.resultJson).toMatchObject({ pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } }); + }); + it("runs the saved harness test and keeps a failed configuration out of ready", async () => { const driver = createLifecycleDriver(db, {} as never); const work = worker(driver.runHost); diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 831d8228d3d..761624744fa 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -3,7 +3,7 @@ import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.j import { instanceSettingsService } from "../../../services/instance-settings.js"; import { randomBytes, randomUUID } from "node:crypto"; import { and, eq } from "drizzle-orm"; -import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { activityLog, @@ -754,6 +754,94 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("deferred_issue_execution"); }); + // H1 bug B: a budget pause stopped the agent's run, and the release blocked + // the task behind a reconciliation incident with no control. A pause is an + // intentional stop; resuming continues the task. + it.each(["Agent lifecycle stop requested", "Cancelled due to budget pause", "Cancelled due to agent pause"])( + "keeps a native task's status and opens no incident when a pause stopped its run (%s)", async (reason) => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" }, + recordedAt: new Date().toISOString() }, + ...(reason === "Agent lifecycle stop requested" ? { pauseStop: { holds: ["budget"], companyPauseReason: null } } : {}) } }).where(eq(heartbeatRuns.id, runId)); + await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "in_progress" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0); + }); + + // Review: the full release (createReleaseIssueExecution), not only the + // native incident step, must leave a paused agent's task unblocked. + it.each(["native", "legacy"] as const)("keeps a %s task unblocked through the full release when its agent's pause stopped the run", async (runtimeMode) => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + await db.update(agents).set({ status: "paused", lifecycleState: "pausing", lifecycleHolds: ["manual"] }).where(eq(agents.id, agentId)); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode, ...(runtimeMode === "native" ? { nativeIssueId: issueId } : {}), + errorCode: "cancelled", resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", + expected: true, initiator: { type: "system" } }, pauseStop: { holds: ["manual"], companyPauseReason: null } } }) + .where(eq(heartbeatRuns.id, runId)); + await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId)); + const escalateStrandedAssignedIssue = vi.fn(async () => undefined); + const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps), + recovery: { escalateStrandedAssignedIssue, escalateStrandedRecoveryIssueInPlace: async () => undefined }, + }); + await release({ companyId, runId, now: new Date() }); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "in_progress" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0); + expect(escalateStrandedAssignedIssue).not.toHaveBeenCalled(); + }); + + it("still opens a reconciliation incident when termination replaced the pause that stamped the run", async () => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + await db.update(agents).set({ lifecycleState: "terminating" }).where(eq(agents.id, agentId)); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true, + initiator: { type: "system" } }, pauseStop: { holds: ["budget"], companyPauseReason: null } } }) + .where(eq(heartbeatRuns.id, runId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0]) + .toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + }); + + it("still opens a reconciliation incident when agent termination stopped the run", async () => { + // Termination and cleanup use the lifecycle stop reason too, with no pauseStop. + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", + expected: true, initiator: { type: "system" } } } }).where(eq(heartbeatRuns.id, runId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0]) + .toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + }); + + it("still opens a reconciliation incident for a native run cancelled by another system stop", async () => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason: "Cancelled because the company was archived", source: "control_plane", + expected: true, initiator: { type: "system" } } } }).where(eq(heartbeatRuns.id, runId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0]) + .toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + }); + it.each(["in_progress", "blocked"])("preserves recovery ownership and queued messages when a native task fails from %s", async (status) => { const companyId = await seedCompany(); const agentId = await seedAgent({ companyId }); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index c3bbf5219a8..d59c9340a0a 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -17,6 +17,7 @@ import { getExecutionBlocker, hasPendingRunCleanup } from "../../../services/exe import { nativeManualAdmissionHoldsExecution } from "../../../services/native-runtime/native-runner-ownership.js"; import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm"; import { readReleaseRun, type ReleaseRun } from "./release-run-read.js"; +import { isGovernancePauseStop } from "../../../services/governance-pause-stop.js"; import type { Db } from "@paperclipai/db"; import { deliveredQueuedCommentIds } from "../../../services/queued-comment-delivery.js"; import { extractIssueReferenceIdentifiers, isUuidLike } from "@paperclipai/shared"; @@ -124,6 +125,9 @@ export function isOperatorStoppedRunRow(row: Pick = new Set([ + AGENT_LIFECYCLE_STOP_REASON, BUDGET_PAUSE_STOP_REASON, AGENT_PAUSE_STOP_REASON]); + +const record = (value: unknown): Record => + value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + +export function isGovernancePauseStop(run: { status: string; resultJson?: Record | null }): boolean { + if (run.status !== "cancelled") return false; + const cancellation = record(run.resultJson?.cancellation); + if (record(cancellation.initiator).type !== "system" || cancellation.expected !== true + || typeof cancellation.reason !== "string" || !GOVERNANCE_PAUSE_STOP_REASONS.has(cancellation.reason)) return false; + // Termination and cleanup issue the same lifecycle stop reason. Only the + // pausing path records pauseStop, so require it for that reason. + return cancellation.reason !== AGENT_LIFECYCLE_STOP_REASON + || Object.keys(record(run.resultJson?.pauseStop)).length > 0; +} From 510746c682912ef390e931f9cf0e8fcbd3899f42 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 21:06:21 -0600 Subject: [PATCH 04/12] fix(native): restart a remote run whose controller never sent its runner a command (#20153) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A follow-up run on a remote sandbox inherits the previous run's session checkpoint at admission, rebound to the new run's identity > - A remote runner receives work only through the controller's Postgres checkpoint, which the controller writes before every command. A run with no checkpoint rows, no runner events and no recorded runner process never sent its runner anything > - Restart recovery still read the rebound admission checkpoint's provider session id as provider evidence, and classified such a run `reattach_remote_runner`. The executor then found the root still owned by the previous run and failed with `native_remote_recovery_lease_mismatch: stage=runner_verification field=identity_run_id` > - This pull request classifies that run as an incomplete bootstrap, and lets the executor start it from the previous owner's settled root > - The benefit is that a follow-up interrupted before its runner ever got a command restarts on its own ## Linked Issues or Issue Description Refs #20137 **What happened?** Daytona per-turn restart-chaos M1 rerun w-20261109T013733-s9501 (master 8e3b64a257, image 34bc12bf), t02 (codex M1 parent): 1. Wake run `a867cb18` was admitted at 01:48:31 with a rebound checkpoint inherited from run `dfeddafa`: identity of the new run, `cursor: null`, no terminal turns, no active turn. 2. It recorded `native.process_start_requested` at 01:48:36, then a K4 hold froze the control plane. 3. At 01:53:11 recovery classified it `reattach_remote_runner` (`hasCheckpoint: true`, `hasProviderEvidence: true`, from the rebound checkpoint), although the run had no `native_runner_checkpoints` rows, no runner-originated events and no recorded process identity. 4. The executor found the session root still owned by `dfeddafa` and failed with `native_remote_recovery_lease_mismatch: stage=runner_verification field=identity_run_id`. The task needed a Retry. **Expected behavior** A remote run whose runner provably never received a command restarts its bootstrap. The executor treats the previous owner's root as it would for any follow-up start. **Steps to reproduce** 1. Claim a remote sandbox follow-up that recorded `native.process_start_requested` but has no controller checkpoint rows, no runner events and no `native.process_identity_recorded`, with only the admission-rebound session checkpoint. 2. Restart. Before this change, recovery claims `reattach_remote_runner`, and the executor fails with `identity_run_id`. **Paperclip version or commit** 8e3b64a257 (wave). Branch rebased on current master. ## What Changed - `native-restart-recovery.ts`: the new `remoteRunnerNeverReceivedCommands` condition. - It applies to a remote sandbox run (not a plugin task) with a prepared remote workspace when all of these hold: - no durable controller record, and no staged attachment; - only the admission-rebound checkpoint (or none), and no provider process events; - no runner-originated events, and no `native.process_identity_recorded`; - no run process fields; - the same claimed and dead previous-controller conditions as `remoteRunnerNeverLaunched`. - It yields `bootstrap_incomplete` with the reason `remote_runner_never_received_commands`, and does not count the rebound checkpoint as this run's checkpoint. - `native-session-executor.ts`: - a `bootstrap_incomplete` claim requires the incomplete-bootstrap proof only when the root carries this run's own identity; - a root still owned by the previous run takes the normal previous-owner path, and may retire a finished previous owner, as any follow-up start does. - Policy change, called out for review: a claimed remote run that recorded only `native.process_start_requested` was held as `ambiguous_remote_recovery_evidence`. Under the conditions above it now bootstraps again. - The controller writes every command to the Postgres checkpoint before sending it, so with no checkpoint rows no command reached any runner, and no provider work can exist. - A runner process that was spawned but never commanded is stopped before the new bootstrap by the existing stale-runner stop. - A recorded process identity, an adapter invocation, a runner event, process fields or a provider session still hold the run. - Tests: - `native-runner-restart-recovery.integration.test.ts`: the classification for a follow-up with an inherited checkpoint and for a first run without one, plus a held case when a runner process identity was recorded. The old held expectation for "a runner process start was requested" moved here; - `native-recovered-owner-followup.test.ts`: the executor admitting a `bootstrap_incomplete` follow-up from the previous owner's settled root. ## Verification - Red/green: - On master, the classification test claims `reattach_remote_runner`, as in production. With the change it claims `bootstrap_incomplete` with `remote_runner_never_received_commands`. - A recorded runner process identity keeps the run off this path. - The executor test is admitted with the change. On master it fails. - `vitest run` over the restart-recovery, bootstrap, recovered-owner and failed-workspace-recovery test files and `native-session-executor.test.ts`: 9 files, all passing after the policy test moved; `native-runner-restart-recovery.integration.test.ts` alone: 144 passed (14 skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk: - The new path requires positive proof that nothing reached a runner: no controller record, no runner events and no recorded process. Any of these keeps the existing classification. - In the executor, a root that carries this run's identity still needs the full bootstrap proof. - A root that carries the previous run's identity goes through the same previous-owner verification as any follow-up start. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../native-recovered-owner-followup.test.ts | 14 +++++++- .../native-runtime/native-restart-recovery.ts | 35 +++++++++++++++++-- ...unner-restart-recovery.integration.test.ts | 30 ++++++++++++++-- .../native-runtime/native-session-executor.ts | 5 +-- 4 files changed, 76 insertions(+), 8 deletions(-) diff --git a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts index ae9be61febe..e1887754d71 100644 --- a/server/src/services/native-runtime/native-recovered-owner-followup.test.ts +++ b/server/src/services/native-runtime/native-recovered-owner-followup.test.ts @@ -202,8 +202,10 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered const remoteExecute = vi.fn(async (_request: unknown): Promise<{ exitCode: number; timedOut: boolean; stdout: string; stderr: string }> => { throw new Error("admission must not reach the sandbox"); }); - const start = (outcome: "created" | "replacement" | "resumed" = "created") => createRunnerdBackend({ + const start = (outcome: "created" | "replacement" | "resumed" = "created", + restartRecovery?: Parameters[0]["restartRecovery"]) => createRunnerdBackend({ db, execution, runnerInstanceId: next.runnerInstanceId, onLog: async (_stream, chunk) => { logs.push(chunk); }, + ...(restartRecovery ? { restartRecovery } : {}), runnerExecutionTarget: { kind: "remote", transport: "sandbox", providerKey: "daytona", environmentId, leaseId, remoteCwd: "/workspace", reusableLeaseConfigured: true, runner: { execute: remoteExecute }, @@ -658,6 +660,16 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered expect(await followUp.start()).toBe(outcome); }, 60_000); + it("starts a follow-up whose own bootstrap never reached its root from the previous owner's settled root", async () => { + const prior = await seedRecoveredPriorRun(); + const followUp = await admitFollowUp(prior); + const root = await restoreHeldRoot(prior, followUp); + expect(await followUp.start("created", { kind: "bootstrap_incomplete", runId: followUp.runId, leaseOwner: "successor", + controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null } as never)).toBe(ADMITTED); + expect(await quarantined()).toEqual([]); + expect(await archivedOwners(root)).toEqual([prior.runId]); + }, 60_000); + it("admits again when a retry rebuilds the same projection from the database", async () => { const prior = await seedRecoveredPriorRun(); const followUp = await admitFollowUp(prior); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index bf8033add40..23f9ba7c2e6 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -678,7 +678,7 @@ export function classifyNativeRunnerRecoveryEvidence(input: { /** Durable evidence proves a claimed remote bootstrap reached no provider work. */ remoteBootstrapProvenIncomplete?: boolean; remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn" - | "remote_runner_exited_before_provider_work"; + | "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands"; processStartMatches: boolean; knownProviderProcessAlive?: boolean; knownProviderProcessIdentityAmbiguous?: boolean; @@ -793,6 +793,16 @@ async function hasNativeRunnerOriginatedEvidence( return event !== undefined; } +async function hasNativeRunEvent( + tx: Pick, run: { companyId: string; id: string }, eventType: string, +): Promise { + const [event] = await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents) + .where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), + eq(heartbeatRunEvents.eventType, eventType))) + .limit(1); + return event !== undefined; +} + export async function hasNativeExecutionStartEvidence( tx: Pick, run: { companyId: string; id: string }, ): Promise { @@ -1408,6 +1418,24 @@ export async function claimNativeRestartRecoveries(input: { const remoteBootstrapProvenIncomplete = remoteBootstrapProof !== null && (!hasProviderEvidence || remoteBootstrapProof === "session_opened_without_turn"); + const admissionCheckpointOnly = !hasCheckpoint || (checkpointIdentityMatches && checkpointRecord.cursor == null + && (!Array.isArray(checkpointRecord.terminalTurns) || checkpointRecord.terminalTurns.length === 0) + && checkpointRecord.activeTurnId == null && checkpointRecord.semanticResult == null && checkpointRecord.terminal == null); + const remoteRunnerNeverReceivedCommands = !unclaimedBootstrap && !remoteRunnerNeverLaunched + && !remoteRunnerExitedBeforeProviderWork && !remoteBootstrapProvenIncomplete + && remoteSandbox && remoteWorkspace !== null && !pluginTask && !durableAuthority && stagedAttachment === null + && admissionCheckpointOnly && providerEvents.length === 0 + && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION + && row.coordinator.phase === "observed" && Boolean(row.coordinator.leaseOwner) + && row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1 + && Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid && row.coordinator.controllerProcessStartedAt) + && !row.coordinator.failureDetail?.successorRunId + && !row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt + && !row.run.sessionIdBefore && !row.run.sessionIdAfter + && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId) + && !(await hasNativeRunnerOriginatedEvidence(tx, row.run)) + && !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED)); + // Read before this claim appends its own history entry. const cleanDrainSuspension = suspendedByCleanDrain(row.coordinator); const resumeKeepsAttempt = cleanDrainSuspension || await restartResumeKeepsAttempt(tx, { @@ -1417,9 +1445,10 @@ export async function claimNativeRestartRecoveries(input: { remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority), unclaimedBootstrap, remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched - || remoteRunnerExitedBeforeProviderWork, + || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands, remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched" : remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work" + : remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands" : remoteBootstrapProof === "session_opened_without_turn" ? "remote_session_opened_without_turn" : undefined, runnerPidAlive, runnerGroupAlive, @@ -1427,7 +1456,7 @@ export async function claimNativeRestartRecoveries(input: { knownProviderProcessAlive: providerProcesses.livePids.length > 0, knownProviderProcessIdentityAmbiguous: providerProcesses.ambiguousLivePids.length > 0, - hasCheckpoint, + hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands, stagedAttachmentVerified: stagedAttachment !== null, checkpointIdentityMatches: checkpointIdentityMatches && diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 5dad4687c4e..26b9a82da58 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -2690,6 +2690,34 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); return fixture; } + it.each([ + { label: "restarts a remote follow-up", identityRecorded: false, inherited: true }, + { label: "restarts a first remote run", identityRecorded: false, inherited: false }, + { label: "keeps a recorded runner process held instead of restarting", identityRecorded: true, inherited: true }, + ])("$label whose controller never sent its runner a command", async ({ identityRecorded, inherited }) => { + const fixture = await seedUnlaunchedRemoteClaim(`NEVER-COMMANDED-${identityRecorded}-${inherited}`, + { event: { eventType: "native.process_start_requested" } }); + const [seeded] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + if (inherited) await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...seeded.runnerProfileJson, sessionCheckpoint: { + driverKind: "codex_app_server", backendKind: "runner", sessionId: "inherited-thread", cursor: null, + identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId }, + providerSessionId: "inherited-thread", activeTurnId: null, terminal: null, terminalTurns: [], semanticResult: null, + pendingRuntimeRequests: [], lineage: [], + } } }).where(eq(heartbeatRuns.id, fixture.runId)); + if (identityRecorded) await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, seq: 901, + stream: "system", level: "info", eventType: "native.process_identity_recorded" }); + const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] }); + if (identityRecorded) { + expect(claim).not.toMatchObject({ kind: "bootstrap_incomplete" }); + return; + } + expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 }); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" }); + }); + it("restarts a remote bootstrap retry that shutdown cut before any runner was requested", async () => { const fixture = await seedUnlaunchedRemoteClaim("BOOTSTRAP-RETRY", { bootstrapRetry: true }); expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, @@ -2865,8 +2893,6 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); }); it.each<[string, Parameters[1], { kind: string; reason: string }]>([ - ["a runner process start was requested", { event: { eventType: "native.process_start_requested" } }, - { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], ["a runner process identity was recorded", { event: { eventType: "native.process_identity_recorded" } }, { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], ["the adapter was invoked", { event: { eventType: "adapter.invoke" } }, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 4782abec0a4..8e006540a3f 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -5808,7 +5808,7 @@ async function migrateRunnerdStateRootForExecution(input: { if (result.kind !== "state") throw new Error("Plugin Runner checkpoint unavailable"); savePluginRunnerCheckpoint(scoped, result.state, identity); } - if (input.restartRecovery?.kind === "bootstrap_incomplete") { + if (input.restartRecovery?.kind === "bootstrap_incomplete" && identity.runId === input.execution.binding.runId) { // A local runner's process ids belong to this host, so a fresh provider // session it opened before any turn can be proven gone here. const localProcesses = !input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local"; @@ -5883,7 +5883,8 @@ async function migrateRunnerdStateRootForExecution(input: { // Name the failed retirement proof: the shared error otherwise hides // which of several independent fences held the follow-up. let retirementRefusal = ""; - if (verification === "terminal_state_indeterminate" && !input.restartRecovery && !input.retainedWarmOwnerRunId) { + if (verification === "terminal_state_indeterminate" && !input.retainedWarmOwnerRunId + && (!input.restartRecovery || input.restartRecovery.kind === "bootstrap_incomplete")) { const retirement = await retireFinishedPerTurnRunnerdOwner({ ...input, root: scoped, identity }); if (retirement.retired) return; if (retirement.refused !== "lifecycle_not_per_turn") retirementRefusal = `; retirement_refused=${retirement.refused}`; From fa7950121dbce5de9e84c63d8aa8f72a64479198 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 21:06:26 -0600 Subject: [PATCH 05/12] fix(native): move an interrupted bootstrap's session aside on the run's own created sandbox lease (#20154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When restart recovery proves a remote bootstrap reached no provider work, it bootstraps the same run again in the same sandbox. Before relaunching, it stops the interrupted runner (#20103) > - The relaunch starts a fresh controller record, whose command ids restart at `command_prp_00000001`. The sandbox still held the interrupted runner's journal, which had already recorded that command id with different data > - The relaunched runnerd rejected it ("commandId was reused with different command data") and exited while waiting for `run.prepare`. Every same-run retry collided the same way until the retry budget ran out and the task blocked > - The proven interrupted-bootstrap session move (#20103) ran only for a resumed lease. A same-run retry on a per-turn sandbox reuses the run's own created lease, so nothing moved the interrupted session aside > - This pull request applies that same proof-gated session move on the run's own created lease > - The benefit is that a same-run bootstrap retry starts its runner on a clean journal, and the old journal is kept as evidence ## Linked Issues or Issue Description Refs #20103 **What happened?** Daytona per-turn restart-chaos M1 rerun w-20261109T013733-s9501 (master 8e3b64a257, image 34bc12bf): - **t10 (codex M1):** the codex subtask's run `f8a8f483` went through eleven same-run bootstrap retries between 01:56 and 02:07. - Each relaunched runnerd logged `paperclip-runnerd: commandId was reused with different command data` and exited about 12 s later while waiting for `run.prepare`. - The run then failed `native_session_retry_exhausted` ("Provider startup failed on 3 attempts before any provider work began"). The subtask blocked, and so did the parent. - **t02:** the same "runnerd exited while waiting for run.prepare" pattern appeared on retry run `2e13568f`. **Expected behavior** A same-run bootstrap retry moves the interrupted session aside, keeping it as a `.bootstrap-restart-