Files
DottaandClaude Opus 5.5 c0c26188db fix(runner): stamp event generations and record an earlier generation's events as history (#19960)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through the Paperclip Runner, which turns
provider output (Codex, OpenCode, Claude through ACPX) into durable
runner events
> - In the restart chaos campaign, events from an earlier turn or runner
generation failed the current turn about ten times (RC-15, 20, 20b, 20c,
33, 33b, 59 and 61)
> - #19718, #19722, #19772 and #19783 each special-cased one variant,
and other variants still fail the run
> - The durable-ownership plan (section G, invariant 6) asks for one
rule: every event carries its generation and turn, and an earlier
generation's events are recorded as history and never fail the current
turn
> - This pull request stamps the runner generation on every event and
applies one stale-event rule in the ACPX provider state, the Codex
driver (which Runnerd also uses for OpenCode) and the pending
`turn.start` check
> - The benefit is that a late report about a settled or restored turn
no longer stops a healthy turn, while events for a turn the runner never
saw still fail closed

## Linked Issues or Issue Description

Refs #19718, #19722, #19772, #19783

**What happened?**
A runner or provider can report a turn after that turn settled: a
resumed provider reports the lost turn's usage or terminal, an ACPX
sidecar emits a last delta after the turn terminal, or a held failure of
the previous turn sits in front of a new `turn.start`. Several of these
paths fail the current turn: ACPX `AcpxEventScope` rejects any event for
a non-active turn ("named a stale turn"), the Codex driver fails
`turn_binding_mismatch` for a settled turn's late item between turns,
and `runnerdPendingTurnStartFailure` reports any held non-recoverable
failure, whatever turn it names. Runner events carried no generation.

**Expected behavior**
An event for a turn that is not active, and that an earlier generation
accepted or this generation already settled, is recorded as history and
never fails the current turn. An event for a turn the runner never saw
still fails closed.

**Steps to reproduce**
ACPX: settle turn 1, start turn 2, and deliver a late `runtime.event`
for turn 1. Before this change, the provider state returns an error and
the run fails.

## What Changed

- Runner (`durable/state.rs`, `durable/runner.rs`): every event payload
carries `generation`, the runner process generation that emitted it. A
runner process that recovers state starts the next generation
(`begin_recovered_generation`); it is saved with the next normal write,
so recovery adds no write before the first turn. Loading stays pure. Old
saved state loads as generation 0. Outbox events keep their original
generation.
- PRP event schemas v1–v3 declare the optional `generation` property;
generated validators and the manifest are regenerated.
- ACPX (`acpx_event_scope.rs`, `acpx_provider_state.rs`): an event that
names a settled turn of this run becomes a bounded
`stale_generation_event` diagnostic (32 per state) and changes no state.
Unknown turns still fail.
- Codex driver (`codex-session-state.ts`,
`codex-session-notifications.ts`): one predicate, `isHistoryTurn()`: the
turn is not active, and it was restored from the checkpoint (an earlier
generation) or settled in this generation. The classifier's stale-turn
branch, the resume-usage branch, restored-terminal replay and the main
fail point (`notificationNamesActiveTurn`) all use it. A terminal for an
earlier generation's turn is history even when it disagrees with the
checkpoint. A terminal that contradicts a turn this same generation
settled is still a `conflicting_turn_terminal` integrity failure, and a
provider that starts a settled turn again is still rejected.
- Transport (`runnerd-codex-transport.ts`): a pending `turn.start`
ignores a held failure that names another provider turn.
- `DURABLE_TRANSPORT.md` documents the stamp and the rule.

## Verification

- `cargo test -p paperclip-runner-core`: new
`records_late_events_for_a_settled_turn_as_history` (red before),
`stamps_every_event_with_the_runner_process_generation`; two existing
ACPX tests now assert "recorded as history, no state change" instead of
"rejected". Durable tests: 112 passed. `cargo fmt` clean.
- `codex-app-server-driver.stale-generation.test.ts` (new, 9 tests):
late item, delta, terminal and workspace change for a settled turn
during and before the next turn, an earlier generation's conflicting
terminal after recovery, and an unknown turn that still fails. Red on
master: 8 of 9 (the during-turn cases on master already dropped the
event, with a different diagnostic; the between-turn and recovery cases
failed the session).
- `runnerd-pending-turn-start-failure.test.ts` (new): an earlier turn's
held failure no longer fails the pending start; the pending turn's own
failure and a failed run terminal still do.
- `vitest run src/drivers/codex src/live/runnerd-codex-transport`: 787
passed; 2 Codex executable-resolution tests timed out under host load
and pass when run alone.
- `tsc -p tsconfig.json` for the runner package, `check:protocol-types`,
`check:replay-goldens`: clean.
- Not run: chaos waves (the coordinator schedules them on this host).

## Risks

- A late report about a settled turn is now dropped with a diagnostic
instead of failing. A provider that keeps working on a settled turn
could have its output ignored silently; a provider that starts a settled
turn again, or names a turn the runner never saw, still fails closed.
- `generation` is a new optional field. Consumers that do not know it
ignore it, as the PRP contract requires.
- The generation counter is written with the next normal state save, so
there is no new write before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:16:00 -06:00
..