fix(native): accept late provider notices in recovery (#18890)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner sessions keep a provider conversation across task
turns.
> - When a later run gets a new task lease, recovery must prove that the
prior run settled.
> - Recovery required `run.terminal` to be the last committed event.
> - A provider can send a late notification for a settled turn. The
Runner then records a `provider.notice.recorded` warning after
`run.terminal`.
> - This false denial stranded a clean session. The notice stays in the
journal, so each later run on that session meets the same denial.
> - This pull request lets recovery accept a recoverable notice after
the terminal event. All other new events still block recovery.
> - The benefit is that a late warning no longer ends session
continuity. A denial now also names the check that failed.

## Linked Issues or Issue Description

No issue exists. I searched open and closed pull requests and issues for
the error code, the event name and the function name. I found no
duplicate.

**What happened?**

A run failed in about nine seconds with `runner_state_identity_mismatch:
plugin_checkpoint_recovery_unproven`. No provider work started. The
prior run had finished and was committed as succeeded. Its journal had
one `provider.notice.recorded` warning, "ignored a notification for a
settled Codex turn", 1 ms after `run.terminal`. A second session had the
same warning about 61 seconds after a normally completed turn.

**Expected behavior**

Recovery accepts the session and continues the provider conversation. A
recoverable notice does not start work, so it must not block recovery.

**Steps to reproduce**

1. Start a warm native session in a plugin sandbox environment.
2. Finish a run so that a provider notice follows `run.terminal` in the
journal.
3. Wait until the idle timeout stops the Runner task and releases the
lease.
4. Start the next run on that session. It gets a new task lease.
5. The run fails with the error above. The notice stays in the journal,
so a retry on the same session meets the same denial.

The `local-receipt-late-notice` test mode in
`native-session-executor.test.ts` reproduces this failure on the unfixed
code.

**Paperclip version or commit**

Observed on `d8a9b3eb0c`. The same check is on `master` before this
change.

**Deployment mode**

Cloud deployment, with a native Runner in a plugin sandbox environment.

## What Changed

- `isSettledRemoteWarmControlState` accepts a `provider.notice.recorded`
event after the terminal event. The notice must be recoverable and have
`info` or `warning` severity. It must come from the same runner, run,
session and lease.
- Error notices, unrecoverable notices, notices from another source and
new work still block recovery.
- Checkpoint recovery and local quiescent recovery now use this
function. They no longer have separate checks that require
`run.terminal` to be the last event.
- Each checkpoint recovery denial adds a specific reason after the
existing code, for example `plugin_checkpoint_recovery_unproven:
journal_not_settled`. The `runner_state_identity_mismatch: ` prefix does
not change.
- I reworded the existing comment on `isSettledRemoteWarmControlState`
so that it stays correct. I added no new code comments.

## Verification

- **Failure before the fix:** I added the tests first and ran them on
the unfixed code. 38 tests failed. In the 4 `local-receipt-late-notice`
cases, checkpoint recovery rejected with
`runner_state_identity_mismatch`. In the `late provider notice` case,
quiescent recovery rejected with `quarantine_owner_unverified`. The new
unit test for notices also failed. The other 32 failures were assertions
for the new denial reasons.
- **Targeted tests after the fix:** 134 tests passed, with 987 skipped
by the filter.
- `cd server && pnpm exec vitest run
src/services/native-runtime/remote-warm-session-recovery.test.ts
src/services/native-runtime/native-session-executor.test.ts -t "settled
remote warm control evidence|recovers a blocked|automatically recovers
only a proven settled local session"`
- **Type error found in review:** Vitest does not check types, and my
first version passed these tests. Review of the diff showed an
incompatible `identity` parameter type. I changed that type to
`Record<string, unknown>`.
- **After the type change:** `remote-warm-session-recovery.test.ts`
passed (23 tests) and `pnpm run typecheck` in `server` passed. I did not
run the 134 tests again, because this change affected only types.
- **Real data, read-only:** I ran the new function on 285 session
journals from a test deployment. The result changed for 2 journals.
These are the 2 stranded sessions, and each has only provider notices
after `run.terminal`.
- **Not run locally:** I did not run the full test suite. I stopped a
run of the wider native-runtime test files before it reported a result.
CI provides the full regression coverage.
- **Not verified:** no live run has reached `read_state` on a stranded
session after this fix. The tests use a fixture for that call.

## Risks

- **Stricter identity check:** checkpoint and quiescent recovery now
also require the journal's saved identity to match. Recovery reads this
identity from the same file, so only damaged state should fail this
check.
- **Wider goal snapshot rule:** these two paths now accept idle goal
snapshots after the terminal event. Warm handoff already used this rule.
- **Wider notice rule:** warm handoff and the settled projection check
also accept recoverable notices now. A notice does not start work, and
the checks for pending commands and unanswered tool input stay in place.
- **Longer error text:** code that compares the full
`plugin_checkpoint_recovery_unproven` message for equality would stop
matching. I found no other reference to this code in the repository.
- **Stranded sessions:** these sessions need no data repair. They can
still fail with `checkpoint_not_settled` if the sandbox checkpoint is
not clean.
- **Migrations:** this change has no migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code,
with tool use and code execution. A human directed the diagnosis and
approved the change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>

server/src/services/native-runtime/native-session-executor.ts

@@ -5049,25 +5049,25 @@ async function recoverPluginRunnerCheckpoint(input: {
       target.sandboxLeaseAcquisition?.outcome === "resumed") return;
   let root = scoped;
   const quarantined = !existsSync(scoped);
-  const deny = (): never => { throw new Error("runner_state_identity_mismatch: plugin_checkpoint_recovery_unproven"); };
+  const deny = (reason: string): never => { throw new Error(`runner_state_identity_mismatch: plugin_checkpoint_recovery_unproven: ${reason}`); };
   if (quarantined) {
     const quarantine = resolve(runnerdStateBase(), "quarantine");
     if (!existsSync(quarantine)) return;
-    if (!isSafeNativeStateDirectory(quarantine)) return deny();
+    if (!isSafeNativeStateDirectory(quarantine)) return deny("quarantine_unsafe");
     const candidates = readdirSync(quarantine).filter(name => name.startsWith(`${basename(scoped)}.identity_indeterminate.`));
     if (candidates.length === 0) return;
-    if (candidates.length !== 1) return deny();
+    if (candidates.length !== 1) return deny("quarantine_ambiguous");
     root = resolve(quarantine, candidates[0]!);
   }
   if (!existsSync(root)) return;
-  if (!isSafeNativeStateDirectory(root)) return deny();
+  if (!isSafeNativeStateDirectory(root)) return deny("root_unsafe");
   if (hasRetainedWarmTransitionEvidence(root)) {
-    if (quarantined) return deny();
+    if (quarantined) return deny("quarantined_warm_transition");
     return;
   }
   const identity = readRunnerdDurableIdentity(root);
   if (!durableIdentityMatchesSession(identity, input.execution) || identity.runId === input.execution.binding.runId) {
-    if (quarantined) return deny();
+    if (quarantined) return deny("quarantined_identity_mismatch");
     return;
   }
   const direct = runnerdAuthorityLifecycle(root, identity);
@@ -5075,32 +5075,29 @@ async function recoverPluginRunnerCheckpoint(input: {
     !existsSync(resolve(root, "runner", "runner-state.json")) && pluginRunnerdAuthorityLifecycle(root, identity) === "suspended";
   // Never hide contradictory direct authority behind a remote checkpoint.
   if (direct !== "absent" && !(quarantined && direct === "suspended") && !verifiedPluginReceipt) {
-    if (quarantined) return deny();
+    if (quarantined) return deny("quarantined_authority_conflict");
     return;
   }
   await assertCleanupActivationCommitted(input.db, root, input.execution);
   const prior = await verifyPriorRunnerdStateForSessionScope({ ...input, root, identity, allowVerifiedBackup: true });
   if (prior !== "terminal_state_indeterminate" && !((quarantined || verifiedPluginReceipt) && prior === "verified")) {
-    if (quarantined) return deny();
+    if (quarantined) return deny("quarantined_prior_owner_unverified");
     return;
   }
   const control = record(JSON.parse(readBoundedNativeFile(resolve(root, "control-plane", "control-plane-state.json"),
     NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, "runner_durable_identity_too_large").toString("utf8")));
-  const terminal = record(Array.isArray(control.committedEvents) ? control.committedEvents.at(-1) : null);
-  if (!Array.isArray(control.commands) || control.commands.some(command => record(command).status !== "completed") ||
-      terminal.eventType !== "run.terminal" || record(terminal.envelope).runId !== identity.runId ||
-      record(terminal.envelope).normalizedSessionId !== identity.normalizedSessionId) return deny();
+  if (!isSettledRemoteWarmControlState(control, identity)) return deny("journal_not_settled");
   const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
     leaseId: target.leaseId, priorIdentity: identity, inspectCheckpoint: true });
   const checkpoint = environmentTaskResultSchema.parse(await target.runnerTask({ kind: "read_state", identity, resumeFrom }));
   if (checkpoint.kind !== "state" || checkpoint.state.lifecycle !== "suspended" ||
       !Array.isArray(checkpoint.state.outbox) || checkpoint.state.outbox.length !== 0 ||
       checkpoint.state.pendingTerminalDelivery != null || checkpoint.state.pendingProviderCleanup != null ||
-      checkpoint.state.warmTransition != null) return deny();
+      checkpoint.state.warmTransition != null) return deny("checkpoint_not_settled");
   savePluginRunnerCheckpoint(root, checkpoint.state, identity);
-  if (await verifyPriorRunnerdStateForSessionScope({ ...input, root, identity, allowVerifiedBackup: true }) !== "verified") return deny();
+  if (await verifyPriorRunnerdStateForSessionScope({ ...input, root, identity, allowVerifiedBackup: true }) !== "verified") return deny("checkpoint_unverified");
   if (quarantined) {
-    if (existsSync(scoped)) return deny();
+    if (existsSync(scoped)) return deny("quarantine_destination_exists");
     renameSync(root, scoped);
   }
 }
@@ -5517,11 +5514,7 @@ async function recoverQuiescentRunnerdState(input: {
       const runner = record(JSON.parse(runnerBytes));
       const control = record(JSON.parse(controlBytes));
       const provider = record(JSON.parse(providerBytes));
-      const commands = control.commands;
       const events = control.committedEvents;
-      const terminalEnvelope = record(
-        record(Array.isArray(events) ? events.at(-1) : null).envelope,
-      );
       if (
         runner.schema !== RUNNERD_STATE_SCHEMA ||
         !["ready", "suspended"].includes(String(runner.lifecycle)) ||
@@ -5537,12 +5530,8 @@ async function recoverQuiescentRunnerdState(input: {
         !Array.isArray(runner.outbox) ||
         runner.outbox.length !== 0 ||
         runner.pendingTerminalDelivery !== null ||
-        !Array.isArray(commands) ||
-        commands.some((command) => record(command).status !== "completed") ||
         !Array.isArray(events) ||
-        record(events.at(-1)).eventType !== "run.terminal" ||
-        terminalEnvelope.runId !== identity.runId ||
-        terminalEnvelope.normalizedSessionId !== identity.normalizedSessionId ||
+        !isSettledRemoteWarmControlState(control, identity) ||
         provider.completedTurnAuthoritative !== true
       )
         continue;