From 80c308f872313e110be25629940affd6c07fce51 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 3 Nov 2026 13:24:16 -0800 Subject: [PATCH] fix(runner): admit follow-ups after recovered per-turn owners (#18846) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one conversation across task turns. > - After a per-turn sandbox turn, the controller saves a Runner suspension receipt or a failover backup. > - A full sandbox disk can stop that save after the turn is complete. > - Recovery then commits the result, but the controller has no proof that the Runner suspended. > - Each follow-up rebuilds the old controller journal from the database and rejects it again. > - This pull request lets admission retire that old authority when the prior owner is provably finished. > - The benefit is that the task continues with its saved workspace and does not stay blocked. ## Linked Issues or Issue Description Refs #18705 (checkpoint save on a full sandbox disk). Refs #18829, #18718 and #18343 (related admission and controller recovery). Refs #18811 (a related warm continuation wait). **What happened?** A per-turn native run in a remote sandbox finished its turn. The post-turn `harness_state.backup.persist` and `session.checkpoint.persist` steps failed because the sandbox disk was full. The controller got no Runner suspension receipt and no failover backup. Workspace recovery later saved the files. The run committed its result and ended as `succeeded`. A board comment then started a follow-up in a new sandbox. Admission failed after about two minutes with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Execution recovery then blocked the task. The cause is in durable-root admission. `restorePerTurnRunnerdController` rebuilds the prior controller journal from the database checkpoint. `verifyPriorRunnerdStateForSessionScope` finds a terminal owner in the same scope with no suspended lifecycle and no retained warm owner. It returns `terminal_state_indeterminate`. Admission quarantines the root and fails. The database checkpoint does not change, so every later admission rebuilds the same journal and fails again. A board Retry after that failed follow-up does not repeat the same error. Runtime selection picks the failed follow-up as the resume source, because it carries an inherited provider checkpoint. Controller recovery then fails with `native_controller_recovery_unsupported:metadata_format`. The task stays blocked. This pull request prevents that failed follow-up. It does not change source selection (see Risks). **Expected behavior** When the prior owner is provably finished, the follow-up must start. It must keep the saved workspace. It must not start while the prior owner can still be alive, while terminal delivery or provider cleanup is unsettled, or while quarantine holds evidence from a different owner. **Steps to reproduce** 1. Complete a per-turn remote sandbox run. Commit its result through recovery and release its controller lease. 2. Keep its database controller checkpoint and recovery pointer. Do not save a suspension receipt or a failover backup. 3. Admit a follow-up in a new sandbox with no local controller projection. 4. Admission rebuilds the old journal, quarantines it, and fails with the error above. 5. Admit the same run again. It rebuilds the journal and fails the same way. The new regression test runs these steps against PostgreSQL. **Paperclip version or commit** Reproduced on `e0276e0166`. **Deployment mode** Native Runner with a per-turn lifecycle, the built-in remote sandbox transport, and the ACPX provider. The admission path is the same for other per-turn sandbox providers. ## What Changed - Add `retireFinishedPerTurnRunnerdOwner` to durable-root admission in `native-session-executor.ts`. It runs only when the existing verifier returns `terminal_state_indeterminate`, and only without a restart claim or a retained warm owner. - The function requires all of these. Otherwise the existing quarantine and rejection apply. - Per-turn lifecycle on the built-in remote sandbox transport, with a `created` or `replacement` sandbox. A resumed sandbox, a warm session, and a plugin task keep their existing paths. - No local Runner lifecycle evidence for the old owner. The session root contains only known authority entries. - A journal for that owner with its terminal delivered and only idle goal snapshots after it. All commands are completed. Each tool input has a completed result. No warm transition or staged attachment exists. - A terminal prior run in the same task. Its coordinator is `committed` with a result, has no lease owner, and has no recovery state. - A stopped prior controller, from `evaluateNativeControllerTakeover` with `requireStoppedController`. A durable controller is fenced by its released database lease, as in remote warm recovery. - Each quarantined root for the scope passes the same journal and local Runner evidence checks for the same owner. A retry rebuilds only the database journal, so quarantined Runner evidence must stay decisive. - When the proof holds, the function reads the journal again, moves the old authority into `continuity-breaks/