Reconcile dormant Codex predecessors through ordinary recovery (#19132)

Browse Source

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native Runner keeps provider and controller journals between
turns.
> - A process can stop after a completed turn while those journals
remain valid.
> - The next owner must reconcile that source before it starts new work.
> - This pull request reuses ordinary recovery and authenticated run
attachment.
> - Existing journals and owner admission keep retained work under its
source.
> - The benefit is conversation recovery without another snapshot format
or native mode.

## Linked Issues or Issue Description

Refs #19057. Built on merged #19129. This is the Codex recovery
transport foundation. Process selection and controller scheduling remain
separate changes.

On normal startup, `paperclip-runnerd` selects durable transport from
its connection arguments. With `--connect-url`, the Runner dials the
controller. With `--listen-address 0.0.0.0 --listen-path
/api/runner/v1/connect/<run-id>` and optional `--listen-port` (default
`43127`), the controller dials the Runner. Both use the same durable
entry point for fresh launches and recovery. The bootstrap ticket
authenticates startup, and `recoveryIntegrityVersion` advertises a
recovery capability.

## What Changed

- Observe the existing recovered command journal before provider
restoration. A saved dormant Codex session with no uncertain effectful
command must still be dormant after restore. Ordinary active same-run
recovery keeps its existing behavior.
- Preserve prepared providers during snapshots, drain and cleanup. New
completed-predecessor admission excludes prepared state. Existing
cleanup and ordinary continuation keep their established behavior.
- Advertise `recoveryIntegrityVersion: 1`. Reject the removed
`--source-reconcile` flag. No separate native source mode or command
whitelist remains.
- Derive controller admission from the authenticated `authorityRunId`
and actual Runner identity. Reuse retain-only settlement through
attachment, authentication and checkpoint persistence. Preserve dynamic
ownership across home configuration and later warm reuse.
- Require the selected execution driver's trusted
`sourceRecoveryVersion: 1` prelaunch attestation for a stopped
predecessor. Binary metadata alone cannot authorize that launch.
Ordinary live adoption keeps its existing path.
- Support fresh authorized preparation from `recoveryLaunch: {
requirements }` without replacing the source's ordinary cache. Require
an authenticated owner throughout that transport's lifetime. Preserve
immutable recorded policy through target preparation and later warm
attachment; compare explicit settings using the provider factory's
effective defaults.
- Complete interrupted attachment at the existing fresh-snapshot
boundary: persist the checkpoint, recheck current ownership, complete
the external recovery callback, then release a real predecessor dispatch
hold. Keep callback/checkpoint failures retryable and cancellation
cleanup retained.
- Preserve completed-session event delivery after target attachment,
exact-thread history projections, source registration identity, exact
command replay and nullable usage redaction.
- Recheck current ownership and the queued run when a slow notification
consumer presents a permission request. Suppress stale presentation
while preserving the committed event and pending request for
source-owner settlement.

## Verification

Current head `e90bb379178db75f78e440c92e108f0bbe01e073` merges master
`499712c68b3c` and is fully verified. [PR
CI](https://github.com/paperclipai/paperclip/actions/runs/42058217937),
[Runner
Windows](https://github.com/paperclipai/paperclip/actions/runs/42058217814),
and [Docker
Runner](https://github.com/paperclipai/paperclip/actions/runs/42058217365)
all passed. [Standard Greptile review
18](https://github.com/paperclipai/paperclip/pull/19132#issuecomment-6645302809)
scores this exact head 5/5 with no unresolved threads. The earlier Apex
review 16 applies to `c09ebb4a2a6c`, as recorded below; no new Apex
review was requested after the documentation or master-forward changes.

The conflict integration preserves upstream external cold-state
restoration before completed-handoff inspection, predecessor ownership
and pending-command checks, and the lost-source early-read exclusion.
Live adoption still does not restore a backup. All 43 focused process
cases passed across cold restore (including denied/foreign state),
staged handoff, ordinary adoption and source recovery. Runner `tsc
--noEmit`, parent-relative diff checks and added-line
secret/personal-data scan passed.

[Read the rewritten
guide](https://github.com/paperclipai/paperclip/blob/e90bb379178db75f78e440c92e108f0bbe01e073/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md#recovering-codex-between-turns):
it explains dial and listener startup, then follows completed run A
through ordinary recovery and authenticated attachment to run B. The
guide was rewritten in documentation commit `75db893fb9ab` and is
unchanged by the master-forward. Its listener CLI test and both
goal-update process tests passed.

Earlier feature head `c09ebb4a2a` includes a process regression for a
goal notification arriving between the restored goal response and the
final thread response. Before the fix the Runner admitted the provider.
The new tests prove that an active goal refuses admission and reaps the
provider, and that a paused goal is delivered normally. Both process
tests and five adjacent ordinary recovery cases pass locally. GitHub CI
passed on the first attempt, including the main, Windows, and Docker
workflows. Standard review 15 and Apex review 16 each scored this commit
5/5, with zero unresolved threads. [GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/41991764330) ·
[Greptile
review](https://github.com/paperclipai/paperclip/pull/19132#issuecomment-6645302809)

- The final ownership-race checks pass 7 focused cases, including owner
changes during checkpoint persistence, handler publication and saved
receipt recovery.
- Native `codex_provider ordinary_recovery`: 6 passed, including
saved/live mismatch, pending and indeterminate same-run goals, settled
idle restoration and prepared reads/cleanup without a provider restart.
- Native `paperclip-runnerd tests::`: 9 passed, including metadata and
legacy flag rejection.
- Native
`interrupted_result_delivery_reconciles_exact_receipts_without_reexecuting_operations`:
passed.
- Final focused compatibility run: 24 process cases passed, including
all 19 retained-settlement rows with queued snapshots, both home
variants of A-to-B-to-C, ordinary attachment/suspension and
environment-prepared close.
- Existing `recovers an active second warm business run after deleting
its controller context assets`: passed.
- Runner `tsc --noEmit`, `git diff --check`, and added-line secret and
personal-data scans passed.
- Real native processes with the maintained fake Codex provider prove
abrupt A exit, ordinary recover A, same-process attach B, a second
successful turn, ordinary warm attach C and a third successful turn on
the same thread. Both isolated and preconfigured homes pass. The
isolated case also exercises environment option copying.
- Additional process cases cover cacheless preparation, same-PID
controller restart adoption, revoked preparation, retained tool-input
admission, direct attachment, ordinary suspension and unexpected live
goals.
- A prior revised-head CI run exposed 8 retained-cleanup failures caused
by prepared auto-restoration. That new behavior and its journal-binding
code are removed. All 19 affected settlement variants now pass locally.
The same CI run also had one environment-prepared close failure (focused
check passes) and two unchanged Cloud sign-in E2E heading timeouts.
Original logs remain recorded. Full suites and platform checks run in
CI.

Previous head `16772ad3e1abfdae9abcc64f8221fa3644ecee58`: [PR workflow
41981127140](https://github.com/paperclipai/paperclip/actions/runs/41981127140),
[Runner Windows
41981126926](https://github.com/paperclipai/paperclip/actions/runs/41981126926),
and [Docker Runner
41981126975](https://github.com/paperclipai/paperclip/actions/runs/41981126975)
passed. Greptile review 14 scores this exact head 5/5 with no unresolved
findings. The PR remains draft.

CI evidence: an unchanged `idle-completed-history` local HTTP readiness
case failed in the initial server 11/12 job (127092367616) and its first
targeted retry (127096952128). The isolated case and the full 75-case
file passed locally. The same-head debug retry (127103567635) passed,
and the full workflow is green. Original failures remain recorded. No
fixture diagnostic, assertion change, or timeout change was pushed.

Focused review-fix validation:

- 57 process cases passed across recovery, attachment and all 19
retained-cleanup variants.
- Final staged matrix: 5 cases, each exercising
queued/prepared/activated/completed boundaries, across owned
connect/listen, legacy no-owner connect/listen, and retained
cancellation cleanup.
- Four completion-failure cases pass: synchronous/asynchronous callback
failure, checkpoint failure before any callback, and owner rebind during
persistence before any callback. Fresh-snapshot retry succeeds without
duplicate provider work.
- Saved-policy positive and explicit-conflict negative pass, including
an authenticated compatible target grant and B-to-C continuity with
omitted live settings.
- Factory/Core checks: 34 passed, including effective ACPX defaults,
real policy changes, already-dispatching target work and predecessor
admission.
- Ordinary adoption compatibility checks: 11 passed. Cacheless owner
construction/revocation and policy checks: 6 passed.
- Deferred permission ordering/forwarding: 30 passed. The new
regressions reproduce owner changes before queueing and before
slow-consumer delivery; matching-owner presentation and later
source-owner settlement remain intact.
- Runner `tsc --noEmit`, `git diff --check`, and full-PR added-line
secret/personal-data scan passed at the corrected head.

New tests reproduced both review findings and premature completion on
failed persistence/changed ownership before the fixes. Full suites
remain assigned to CI.

## Risks

- Fake-provider tests do not prove vendor restore inertness or rollout
fsync.
- The process owner must prove stopped ownership, no successor and
strict dormant provider admission before a predecessor launch. This PR
does not select or stop host tasks.
- Ordinary suspended active-goal continuation remains distinct from
completed-predecessor admission.
- Live dormant checks use bounded goal and thread reads. Unsupported or
conflicting responses fail closed.
- Prepared provider state is outside new predecessor admission. Reads
and cleanup retain their baseline behavior. The feature targets
completed between-turn session-open or exited sessions.
- No tenant changes or image publication are included.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with
local code and command tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this does not duplicate
planned work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked the related public issue or PR
- [x] I have not referenced internal issues or instance links
- [x] My branch name describes the change
- [x] Focused local tests pass
- [x] I have added tests and updated documentation
- [x] I have documented risks
- [x] All Paperclip CI gates are green at the revised head
- [x] Greptile is 5/5 with no unresolved comments at the revised head
- [x] I will address review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

...

This commit is contained in:

Nicky LeachandPaperclip authored and GitHub committed 2026-11-05 08:16:39 -08:00

1 parent b1d00eeca0

commit e80d745060

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

Diff Content Not Available