Files
PaperClipAI/packages
DottaandClaude Opus 5.5 d9d5958d09 fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path

- Paperclip's controller and the sandbox Runner exchange numbered
commands; the Runner refuses a command whose number is not exactly one
past the last it processed.
- Staging's most common "Runner gone" failure was a follow-up controller
whose journal was behind the Runner. #20928 removed the cause on the
server (a warm follow-up now launches from the journal its retired
predecessor saved) and added a defence that reconciles a journal behind
only by the predecessor's settled close.
- The Runner still treated an out-of-order command as fatal:
`durable/state.rs` returned an error and the process exited. A
controller with a stale journal from any other cause (state saved by an
older release, a mid-run deploy) therefore lost the Runner process and
the run could only fail as "Runner gone"
(`runner_remote_process_identity_unavailable`).
- This pull request makes the Runner refuse such a command with a
precise result instead of exiting, and makes the server rest that run
with a precise code and a Retry whose next run reconciles the journal
(the #20928 defence) before it opens the session.

## Linked Issues or Issue Description

- Refs #20928 (the server-side root-cause fix and the journal
reconciliation this relies on)
- Refs #20907 (makes runnerd exit text visible in the run log)
- Duplicate search: searched PRs for "controller sequence contiguous",
"controller_sequence_mismatch" and "out-of-order command"; none changes
the Runner's handling.

## What Changed

- `runner-core/src/durable/state.rs`: a new command whose
`controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap)
now yields `CommandDisposition::Reject` with `status: "rejected"` and `{
code: "controller_sequence_mismatch", message, expectedControllerSeq,
receivedControllerSeq }`. The command is not executed or journaled and
the last sequence is unchanged; the process keeps running. Replay of an
already-processed command and the compacted-history rejection are
unchanged.
- `native-session-executor.ts`: a failure carrying
`controller_sequence_mismatch` is classified as
`native_controller_journal_behind_runner`, a permanent (terminal)
failure with a Retry action that says no provider work began and that
Retry reconciles the journal from the saved checkpoint or stops with a
precise error.
- Tests:
- Rust `tests/durable_recovery.rs`: out-of-order (behind and gap)
commands are rejected with the precise code, the journal is unchanged,
and the next contiguous command executes. The existing
`command_gaps_and_identifier_reuse_fail_closed` unit test now asserts
the rejection instead of an error.
- Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a
follow-up controller with a journal missing the predecessor's close now
gets `"code":"controller_sequence_mismatch"`, the Runner process stays
alive and its durable sequence is unchanged; with the current journal it
resumes.
  - Server: the classification and terminal disposition of the new code.

## Verification

- Red (master Runner): the Rust test panics on `expect("an out-of-order
command is not fatal")`; the real-runnerd test receives `runnerd exited
unexpectedly with code 1: paperclip-runnerd: controller sequence must be
contiguous: expected 6, received 4`.
- Green: `cargo test -p paperclip-runner-core --test durable_recovery`
(4 passed); `cargo test -p paperclip-runner-core` lib suite (the only
other failure, `task_memory::tests::monitor_bounds_notices`, is a timing
test that passes alone); `npx vitest run
src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built
from this branch (2 passed); server `native-session-executor.test.ts`
focused tests (71 passed).
- `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`:
profiles match the release manifest. No ACPX profile, attestation, pin
or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the
change is Runner source behaviour with an existing command-result shape,
so no profile version or `commandDigest` changes.

## Risks

- A controller that is *ahead* of the Runner (a gap) is now rejected
instead of fatal. Neither case executes the command; the Runner journal
is unchanged, so this is strictly safer than exiting.
- The rested run needs a Retry. Before this change the same run failed
after its retries as "Runner gone" with no precise cause.
- Runner source changes the Daytona image content ID; a new image is
needed for chaos waves on this revision.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool
use (shell, file edits, tests) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:14:31 -06:00
..