fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner keeps one ACPX provider conversation across the
per-turn runs of a task. Each run cold-attaches the session that the
previous run settled.
> - The saved ACPX state binds the digest of the launch profile that
wrote it. Every deploy ships a new provider pack, so that digest
changes.
> - After a deploy, the next run of such a task fails `run.attach` with
`ACPX durable launch profile digest does not match runner startup`.
Every same-run retry fails the same way.
> - On one instance this happened seven times. Six tasks used their
retry budget and ended blocked, or in progress with no runner. In the
seventh, the scheduled same-run resume never ran at all: the task stayed
`in_progress` with nothing running, and its monitor did not fire.
> - This pull request re-binds a settled session to the upgraded launch
profile, types the remaining mismatches so the server stops retrying
them, and makes a scheduled remote resume survive the stale-lock sweep.
> - The benefit is that a task continues its conversation after a
deploy. When it cannot, it stops with a clear next action instead of
stalling.

## Linked Issues or Issue Description

Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes).
Refs #15795, an open change to the same stale-lock sweep function. It
changes run teardown, not lock cleanup, so a second merge needs at most
a small rebase.

**What happened?**

A per-turn native task used provider `acpx` with Claude, driver
`acpx_runtime`, on a remote sandbox.

1. Run A succeeded and settled its provider session. The next run with
the same build cold-attached it.
2. The instance deployed a new build.
3. A board approval woke run B in a fresh sandbox. Run B failed after
about five minutes: `native_session_recovery_failed: Error: PRP command
run.attach failed: {...
"result":{"code":"command_execution_failed","message":"ACPX durable
launch profile digest does not match runner startup"}}`.
4. The coordinator recorded `retryable_failure`, attempt 1,
`native_session_interrupted`, recovery mode `exact_checkpoint_resume`,
recovery state `resuming_session`, and a retry 30 seconds later.
5. Two seconds after the failure, the stale-lock sweep cleared the
task's execution and checkout locks, because run B was `failed`.
6. Forty minutes later the attempt was still 1. The scheduled monitor
had not fired. The task stayed `in_progress` with nothing running.

**Expected behavior**

A session that the previous run settled continues after a deploy. If it
cannot continue, the run stops at once with a clear next action. A
scheduled resume either runs or fails visibly.

**Steps to reproduce**

1. Save a suspended ACPX session under one launch profile.
2. Change the sidecar bytes and the provider-pack authority at the same
paths.
3. Cold-attach the next run in a new runner. Restore fails with the
digest mismatch.
4. For the stall: leave a remote native run `failed` with a due
`retryable_failure` coordinator, run the stale-lock sweep, then run
restart recovery for that run. It returns `blocked:
issue_execution_lock_changed` and writes nothing.

The new Rust and PostgreSQL tests run these steps.

**Paperclip version or commit**

`master` at `b0c4e345267`.

**Deployment mode**

Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies
to every ACPX agent with an ordinary startup.

## What Changed

### Root causes

- **Upgrade failure.** `AcpxLaunchProfile::canonical_digest` hashes the
provider-pack authority, the sidecar command and arguments, each
artifact path and SHA-256, and any candidate admission. For a remote
sandbox the authority is the provider-pack manifest digest, which covers
the Runner source revision and dist digest. So every deploy changes it.
`AcpxDurableState` stores only the digest, and `validate()` rejects any
difference. Only a Kimi candidate run binding had a carve-out.
- **Silent stall.** The failure write keeps the task locks for
`retryable_failure` and clears them only for `terminal_failure`. The
stale-lock sweep treats any `failed` run as cleanable, so it cleared
them. A remote retry goes through `claimNativeRestartRecoveries`. That
claim requires `issues.execution_run_id` to name the run, so it returned
`blocked: issue_execution_lock_changed` with no coordinator write and no
deadline. The live recovery queue then dropped the entry. The local
claim (`claimNativeSessionResumptions`) already reclaims a free lock, so
only remote runs stalled.
- **Monitor.** `noActiveNativeMonitorRun` holds a task's monitor while
any native run for it has a coordinator that is not `committed` or
`terminal_failure`. Run B's `retryable_failure` held it. That is by
design; it ends when the resume runs.

### Runner (Rust)

- `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore
re-binds a saved session to the current launch profile only when all of
these are true:
- The lifecycle is `suspended` and an identity exists. There is no
active turn, pending runtime request, undelivered provider event, or
unconfirmed provider exit.
  - The startup has no candidate admission.
- The saved descriptor's sidecar command and arguments equal this
startup's.
- Any other digest mismatch keeps failing with the same message, plus
`classification: "provider_state_incompatible"` in the durable command
result. `DurableRunnerError` carries an optional closed classification
for this.
- A saved descriptor that names an agent release this build does not
qualify (ACPX version, agent server package or version, or command
digest) also gets the class. `validate_session()` and its message are
unchanged; restore only adds the class.

### Runner (TypeScript)

- `runnerRecoveryFailureClass` reads the classification from the
authenticated `runnerCommandResult`. The Codex app-server driver (which
also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on
as `failureClass`.
- If replacement after a failed resume is not allowed,
`executeNativeSession` throws `NativeProviderStateIncompatibleError`
(message unchanged: `native_session_recovery_failed: ...`). If
replacement is allowed, the existing path opens a fresh session in the
same run, as before.

### Server

- `nativeSessionFailureSourceCode` maps that error class to
`native_provider_state_incompatible` with `instanceof`, not text. The
disposition is permanent: `terminal_failure` at once, the task blocks,
and the board owns the recovery action. Its next action says that a
Retry continues in a fresh provider session with the same workspace.
- `findNativeSessionResumeRun` also returns the barrier's `errorCode`.
`rebindNativeSessionCheckpoint` does not resume a source with
`native_provider_state_incompatible`. The existing fallback in
`buildNativeExecutionWithCheckpoint` then starts a fresh normalized
session with the full task bootstrap.
- `sweepStaleIssueLocks` keeps both locks while a referenced run's
coordinator owns a result-less `retryable_failure`, unless the task is
done or cancelled. Its update also requires that no referenced run has a
live status at the write, because a resume claim can return the run to
running after the status read.
- `claimNativeRestartRecoveries` reclaims a free execution lock for a
`retryable_failure` resume, like the local claim does, when the checkout
lock is free or held by this run and no other run for the task is
queued, running or scheduled. It records
`native.recovery.execution_reclaimed`. A lock held by another run still
blocks.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` and
`doc/execution-semantics.md`.

### Safety argument for the re-bind

What the binding protects, and what this change keeps:

- **Candidate admission** carries company, model, platform and expiry
authority. A candidate startup never re-binds. The Kimi run-binding
carve-out and all candidate rejections are unchanged.
- **The agent that loads the conversation** stays exact. `validate()`
runs on the saved descriptor before the digest check and requires this
build's qualified agent profile: ACPX version, agent server package and
version, and `commandDigest`. The saved identity must still match that
profile and the provider mode.
- **In-flight provider work** is never re-bound. A turn, runtime
request, undelivered event or unconfirmed exit keeps the strict binding,
so no old provider process or request continues under a new build. The
next launch only loads a settled conversation.
- **Launch shape.** A changed sidecar command path or arguments still
rejects. Only the content at the same paths may change.
- **Artifacts** are not trusted from the saved state. The current
profile comes from the trusted controller at startup, and session start
verifies every artifact's bytes and the restored provider identity, as
before.

What changes: a new provider-pack authority, or new sidecar or Node
bytes, at the same paths no longer invalidates a settled conversation.
The saved digest cannot show which part changed, so the rule does not
depend on it. I did not add state fields to tell authority changes from
artifact changes, because the rule would treat both the same.

One existing assertion changes.
`released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile`
expected a settled ordinary Kimi session to reject a changed authority,
artifact digest, or saved digest. Those three cases moved to a new
positive test. Command, arguments, every candidate grant field, a
missing profile and run binding still reject.

### How a task in the reported state recovers after this deploys

The task has old-digest saved state, a `retryable_failure` coordinator
at attempt 1, and cleared locks.

1. Startup recovery, and the live recovery queue, run restart recovery
for the due run. It reclaims the free execution lock and claims the run.
The resume runs now.
2. Run B's failed `run.attach` is a durable outcome of B's exact run
authority. If the resume replays that outcome instead of executing it
again, it fails with the old, untyped text. The remaining attempts then
fail the same way, and the run ends `terminal_failure` with
`native_session_retry_exhausted`. The task blocks with a board recovery
action. This takes about a minute.
3. If the task blocks, press **Retry** once. Retry selects run B as the
source (it has process identity and controller evidence, so the #18860
skip does not apply) and rebinds B's checkpoint, which holds A's
provider session. The new run cold-attaches under the new Runner, and
restore re-binds the old digest because the session is suspended and
quiescent. The conversation continues. Run B's failed attach never wrote
the state file, so the state is still A's settled state.
4. Tasks that already exhausted their retries (blocked, or in progress
after `native_session_retry_exhausted`) need the same single Retry.
Their earlier attempts ran under the old Runner.

Wait for the task to show blocked before pressing Retry. A Retry pressed
while the resume is running joins that run.

## Verification

- Rust, failing first:
`settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed
on `master` with the production message.
`unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed
on the missing classification. Both pass now.
- `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test
binaries pass. The library has 471 passed and 1 ignored.
- `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner
typecheck` (TypeScript, generated contracts, `cargo check`): pass.
- Server, failing first: `native-scheduled-resume-lock.test.ts`
(PostgreSQL). On `master` the sweep cleared the locks and restart
recovery returned `issue_execution_lock_changed`. Both negative controls
(another run's lock, a queued run) already passed. All four tests pass
now.
- `pnpm --filter @paperclipai/server typecheck`: pass.
- Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`,
`src/drivers/codex`, `src/drivers/provider-failure.test.ts`,
`src/live/runnerd-codex-transport*.test.ts`,
`src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011
tests pass. New: the typed error when same-session recovery is required,
and the recovery class reader and pass-through.
- Server vitest: the native executor, status corpus and provider-pack
admission suites (1,268 tests) pass. New: the classifier makes the typed
error permanent, and the same text stays retryable. Checkpoint selection
returns `errorCode`, and an incompatible source starts a fresh session
with the same workspace (unit and PostgreSQL).
- Server vitest over the recovery area: stale-lock sweep, native
resumption, process recovery, failed-workspace recovery, restart
recovery (integration, diskless, preparation), controller recovery,
startup retry, startup cleanup, recovered-owner follow-up, session
resume, runtime selection and recovery deadlines. 926 passed. Four
`heartbeat-process-recovery` tests failed in that loaded run. Three pass
when I run them alone. `redacts opaque environment-bound credentials
from Sentry diagnostics: result` fails the same way on unmodified
`master` on this host (a local environment value is redacted).
- Review round 1: a new sweep test returns the run to running before the
clear. It cleared the locks without the write recheck and keeps them
with it. The strict Rust restore test covers an older agent server
release and an older ACPX release. The ACPX backend and durable runner
unit tests, the lock, sweep, resume projection and controller recovery
suites, `cargo fmt --check` and the server typecheck pass. In the second
full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s
for a child exit on this loaded host. They do not use the changed code
and passed in the first full run.
- After the rebase onto the current `master`: the new lock test,
failed-workspace recovery, stale-lock sweep and session resume suites
pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and
`git diff --check` pass.
- CI on `929ef4508d2`: in the first run, `isolated-repository-rehome`
hit its 15 s test timeout, `idle-completed-history` could not reach its
fixture runtime service, and e2e shard 3 was cancelled by a runner
shutdown. A rerun of the failed jobs passed. All 55 checks pass.
- Not verified: a live remote sandbox deploy, and whether run B's
same-run resume replays its recorded `run.attach` failure or executes it
again. The Retry path is tested at source selection and checkpoint
rebind (server) and at cold attach (Runner). It is not tested end to end
through a sandbox.

## Risks

- A settled session now continues under new sidecar bytes. If a sidecar
change made an old ACPX record unreadable, session start fails as any
failed resume does. It does not corrupt state, and the run follows its
normal recovery policy.
- The sweep now keeps locks for a failed native run with a scheduled
resume. If that resume can never be claimed (for example the agent stays
paused), the task keeps the lock until the coordinator settles or the
task closes. Before, the sweep cleared it, and the resume still never
ran.
- If another run holds the task's lock, a scheduled remote resume still
stays blocked and the coordinator stays `retryable_failure`. This change
does not settle that case. It needs a separate decision about
superseding the resume.
- `native_provider_state_incompatible` is a new failure code. No UI or
migration change is needed. Older servers never see it.
- No schema migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, test runs). The runtime does not expose the context
window or reasoning configuration.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/execution-semantics.md

@@ -1258,6 +1258,13 @@ checks coordinator ownership in its terminal update, so a result recorded after
 the process check is also protected. Terminal task status remains authoritative;
 exhausted finalization retries do not keep an otherwise orphaned run alive.
 
+A native failure that schedules a same-run resume marks the run `failed` but
+keeps the task's execution and checkout locks for that resume. The sweep leaves
+those locks in place and clears them only when the task is done or cancelled.
+If a scheduled resume finds its execution lock free, restart recovery reclaims
+it, as the local resume claim does, unless another run for the task is queued,
+running, or scheduled. A lock held by another run still blocks the resume.
+
 Automatic productivity reviews are retired. Run counts, missing comments, and elapsed task time do not create review tasks or impose continuation holds. Bounded continuation, provider recovery, budget limits, explicit blockers, and normal review/approval stages remain in force. Existing productivity-review tasks, comments, assignments, and dependencies remain unchanged and readable; their historical origins still identify them as recovery work for recursion suppression.
 
 ### Issue-thread interaction resolution