mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
Nicky LeachandClaude Opus 5.5
a7d3d4b78c
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner moves a native session from one run to the next
with an authority rotation, for Codex and for every ACPX provider
> - A rotation archives the prior run's controller state under
`authority-epochs/` and archives the Runner state, locally or on an
external Runner host
> - If the run fails after the rotation but before the new Runner
starts, retry preparation restores the prior run's controller checkpoint
into the session root
> - The retry then finds an active `control-plane` next to the archive
marker and fails with `native_runner_authority_archive_conflict`. Every
later run for the session fails the same way
> - The local and external rotations were two copies of the same steps,
with different retry rules, and the sandbox archive script had a third
set of rules
> - This pull request gives every provider and Runner placement one
resumable rotation path, with the same retry rules for controller state
and Runner state
> - The benefit is that one failed startup no longer makes a native
session unusable
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A warm Codex session on an external (plugin) Runner failed its first
attempt after the authority rotation finished. The two automatic retries
both failed with `native_session_recovery_failed: Error:
native_runner_authority_archive_conflict`, and the run stopped. On disk,
the session root had
`authority-epochs/epoch-<key>/control-plane/control-plane-state.json`
and an active `control-plane/control-plane-state.json` with the same
identity (in this case, also the same sha256). The remote Launcher had
already archived the Runner state under the same key.
**Expected behavior**
A retry resumes the interrupted rotation. The rotation already supports
this when only the archive marker exists.
**Steps to reproduce**
1. Run a native Codex or ACPX session on an external Runner.
2. Start a new run for that session. Make the new Runner fail to start
after the rotation finishes.
3. Let the run retry. Retry preparation (`restoreWarmRunnerdController`
/ `restorePerTurnRunnerdController` →
`restoreNativeControllerProjections`) restores the prior run's
controller checkpoint into the session root.
4. The retry fails with `native_runner_authority_archive_conflict`. All
later runs for the session also fail.
The new `authority rotation after the prior controller state is
restored` tests reproduce steps 2–4 directly.
**Paperclip version or commit**
`master` at 35280516c0.
**Deployment mode**
Hosted, with a remote (plugin) Runner environment.
**Agent adapter(s) involved**
Codex (native Paperclip Runner). ACPX providers use the same rotation
code.
## What Changed
- **One rotation path.** `rotateAuthorityEpoch` replaces
`rotateLocalAuthorityEpoch` and `rotateExternalAuthorityEpoch`. The only
difference between Runner placements is a `RunnerStateArchive` (`read`,
`archive`):
- `localRunnerStateArchive(root)` for Runner state under `<root>/runner`
- `#externalRunnerStateArchive()` for a plugin task or a sandbox host
- **One caller.** The transport chooses the rotation source (the live
control plane, or the latest archive), then makes one call. Locally
owned state is still quarantined on failure.
- **Committed rotation, normalized once.** Retry preparation restores
the prior run's controller checkpoint, but an earlier attempt may
already have archived that authority and its Runner state.
`takeCommittedRotation` runs right after the transport reads the control
plane. When the epoch archive for the active copy's exact identity
exists, it removes the restored copy and continues on the archive-only
path. Every later check (the warm-handoff evidence read, the
source-attachment settlement read, and the rotation) then sees the state
the rotation committed, instead of reading retired live Runner state. A
copy that still holds an unfinished warm transition, staged attachment,
or pending `run.attach` fails closed.
- **Restored controller state.** When the epoch archive exists, an
active control plane with the same schema and the exact archived
identity is a copy of authority that the rotation already retired. The
rotation removes it and resumes from the archived snapshot. This
compares identity, not bytes, because a checkpoint restore serializes
the state again and drops fields such as `ackedSourceSeq`. Any other
active state still fails closed.
- **Duplicate Runner state.** Local Runner state now follows the
computer Launcher's rules: move it, accept it already archived, drop a
byte-identical live copy beside the archive, or fail closed. The local
archive also checks that the live state is suspended before it moves it.
- **Sandbox archive script.** The script is now
`REMOTE_RUNNER_AUTHORITY_ARCHIVE_SCRIPT` and follows the same rules. It
uses `cmp` for the duplicate check and fails closed when `cmp` is not
available.
- **Sandbox host preparation after a resumed rotation.** A resumed
rotation archives the remote Runner state without preparing the host
first. When `archiveExternalRunnerState` succeeds, the server now
records the archived snapshot path. The launch's host preparation
(`inspectRemoteHarnessState`) validates that snapshot instead of the
missing live `runner-state.json`, with the same provider-state and
identity checks. Only these host preparation checks use the archived
snapshot; close checkpointing and backup-restore verification always
inspect the live `runner-state.json`. The SSH staging check treats it as
present, so retired state is not staged back.
- **Host preparation.** The external archive prepares its host only when
it reads live Runner state. Before, only one of the two caller branches
prepared the host. A resumed rotation finds the state already archived,
and preparing again could stage it a second time.
## Verification
- `npx vitest run src/live/runnerd-codex-transport.test.ts -t
"rotation"` in `packages/paperclip-runner`: 16 passed.
- External and local rotations with a byte-identical, a re-serialized,
and an other-authority restored control plane. The first two resume from
the archived snapshot. The third fails closed and keeps the copy.
- Local rotation with an identical duplicate Runner state (resumes) and
a different duplicate (fails closed and keeps it).
- The existing crash-before and crash-after remote archive retry cases.
- Before the source change, the byte-identical external case failed with
`native_runner_authority_archive_conflict`.
- `npx vitest run src/live/runnerd-codex-transport.test.ts -t "transport
retry after a committed rotation"` in `packages/paperclip-runner`: 4
passed. These go through the transport's real start path. A completed
warm handoff and a source attachment hook now resume from the archive;
before the normalization, both read the archived live Runner state and
failed. An unfinished warm transition and a pending attachment fail
closed; before, the pending attachment was silently dropped.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
authority archiv"` in `server`: 6 passed. Five new cases run the real
archive script with `sh` against temporary files: a new archive, an
existing archive, an identical duplicate, a different duplicate, and no
state. The existing mocked archive test also passes.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "resumed
rotation archived"` in `server`: 2 passed. A resumed sandbox with a
retained backup goes through the real archive script and host
preparation. It failed with `runner_harness_state_mismatch:
resumed_unavailable` before the host preparation fix. A settled close
checkpoint with no live Runner state then fails closed with
`runner_remote_checkpoint_incomplete`, rather than checking the archived
snapshot. Without a completed rotation, a missing live state still fails
closed.
- `pnpm run typecheck:typescript` in `packages/paperclip-runner` and
`node scripts/typecheck-layers.mjs` in `server`: pass.
- The full `runnerd-codex-transport*.test.ts`,
`cursor-cloud-runner-launch.test.ts`, and adoption test files fail on
the same 110 tests with and without this change. These tests need the
built `paperclip-runnerd` binary, which needs `cargo`. `cargo` is not
installed locally, so CI covers them.
## Risks
- The change is only on the rotation path. A rotation that has no
archive marker behaves as before: it reads and checks live Runner state,
then archives the control plane.
- A rotation removes an active control plane only when an archive for
that exact identity exists and the active copy has the same schema and
identity. A new run's authority has a different identity, so it cannot
match.
- On a resume, the rotation now uses the archived controller snapshot,
not the caller's copy. The archive is the exact pre-rotation state. A
checkpoint restore can omit fields from it.
- An external Runner whose options define `readRunnerState` and
`archiveExternalRunnerState` but not `prepareExternalRunnerState` can
now rotate a live control plane. Before, that branch required the
prepare hook. The plugin and sandbox drivers define all three.
- The sandbox duplicate check needs `cmp`. A host without `cmp` fails
closed as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). It diagnosed the
failure from the run events, the server logs, and the on-disk controller
and Launcher state, then wrote the fix and the tests.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>