fix(native): start a fresh session when an accepted run's provider state was not saved (#20811)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run in a per-turn remote sandbox saves a provider state
backup after its result is accepted. The sandbox is then released, so
the backup is the only copy of the provider session.
> - When that backup fails to save (for example, #20772: storage
throttling), the run keeps its accepted result. Nothing records that its
checkpoint can no longer be resumed.
> - The next run on the session rebinds that checkpoint and asks the
Runner to resume. The Runner has no state and refuses (#20757). With the
normal policy, the run opens a replacement session after that failed
attempt. With a governed-wait or unfinished-goal checkpoint, the run
fails with `native_provider_state_incompatible`, and only Retry or the
next message starts fresh.
> - The server already knew the state was gone when the backup failed.
The failed resume is wasted work.
> - This pull request marks the accepted run's checkpoint as not
resumable at the moment the backup fails. The next run then starts a
fresh provider session directly.
> - The benefit is that a lost backup no longer costs a failed run or a
failed resume attempt before the task continues.

## Linked Issues or Issue Description

Refs #20757 (the missing-state refusal that the next run reaches today).
Refs #20772 (the throttled backup upload that loses the backup). Refs
#20809 (a related fix for governed-wait follow-ups that reach the
missing state; this PR stops most of them from reaching it).

**What happened?**

1. A native run in a per-turn sandbox finished, and its result was
accepted.
2. `harness_state.backup.persist` failed. The run recorded
`native.post_result_failure`. Its run row kept a normal
`sessionCheckpoint`.
3. The sandbox was released.
4. The next run on the same session selected that run as its resume
source, and `rebindNativeSessionCheckpoint` returned the checkpoint.
5. The Runner found no saved state and refused with `PRP provider resume
state is unavailable`. The run then opened a replacement session after
the failed attempt, or (governed-wait or unfinished-goal checkpoint)
failed with `native_provider_state_incompatible`. Retry or the next
message then started fresh.

**Expected behavior**

The next run starts a fresh provider session at once, because the server
already knows the saved state is not there. The accepted result and the
task outcome stay as they are.

**Steps to reproduce**

1. Give a run row an accepted result and a normal `sessionCheckpoint`.
2. Make its close checkpoint fail in a per-turn sandbox
(`reusableLeaseConfigured: false`).
3. Select the next run's resume source with `findNativeSessionResumeRun`
and call `rebindNativeSessionCheckpoint`.
4. On `master` it returns the checkpoint, so the next run tries to
resume state that does not exist.

**Paperclip version or commit**

`master` at `556b249897`.

## What Changed

- `server/src/services/native-runtime/native-provider-state-unsaved.ts`
(new): `markAcceptedRunProviderStateUnsaved` adds
`nativeProviderStateUnsaved: { code:
"native_provider_state_incompatible", reason: "harness_backup_not_saved"
}` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it
only when the run's coordinator holds a result with `schema_status =
'accepted'`. `clearRunProviderStateUnsaved` removes it.
- `server/src/services/native-runtime/native-session-executor.ts`: the
remote close checkpoint (`checkpointRemoteRunner`) now wraps the
existing body. When the checkpoint throws in a sandbox without a
reusable lease, it calls the marker, writes one run-log line, and throws
the original error. A marker or log failure cannot replace that error.
If a later checkpoint of the same run saves, it clears the mark.
- `server/src/services/native-runtime/native-session-resume.ts`:
`rebindNativeSessionCheckpoint` returns `null` for a source with the
mark, as it already does for `errorCode =
native_provider_state_incompatible`. `findNativeSessionResumeRun`
projects the new key.
- `doc/execution-semantics.md`: one additive paragraph.
- Tests in `native-session-resume.test.ts` and
`native-session-executor.test.ts`.

Guard conditions. The mark is written only when all of these are true:

- The close checkpoint threw before the durable backup was stored (after
the upload's own bounded retries). A later local publication, stamp or
cleanup failure does not mark the run.
- The target is a remote `sandbox` with `reusableLeaseConfigured !==
true`. A retained sandbox or an SSH host keeps the live state, so its
resume path is unchanged.
- The run holds an accepted result. A failure before acceptance changes
nothing.

Saved state from the previous release: run rows without the key keep
today's resume path. An older server that reads a row with the key
ignores it, and so it behaves as it does today. No schema change.

User-visible: the next run's transcript shows "Paperclip session
started" instead of "resumed", with a new session id. The source run's
log shows `provider state was not saved after the accepted result; the
next run starts a fresh provider session`. The fresh session gets the
full task context, as for other fresh starts.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-resume.test.ts -t "not
resumable only after|provider state was not saved"`: 9 pass (6 pure, 3
embedded Postgres).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "marks
the provider state unsaved|uses the bundled Linux image authority for
remote opencode-new-durable"`: 8 pass. They include a durable store
failure (marked) and a local publication failure after the store (not
marked).
- Red on `master` (source files reverted, tests kept): the 3 "starts the
next run in a fresh provider session directly" cases fail (`expected {…}
to be null`), the DB "accepted" case fails the same way, and the
executor per-turn case fails (`mark` called 0 times).
- Guard removal checks:
- Sandbox guard replaced with `true`: the "retained sandbox" case fails
(`mark` called once).
  - `!checkpointBackupStored` removed: the "publish-failed" case fails.
- `schema_status = 'accepted'` removed from the marker: the "rejected"
case fails (`expected true to be false`).
- Negative cases that pass: no result, rejected result, retained
sandbox, records without the key or with another code (all three
recovery policies), and a later save that clears the mark.
- `cd server && npx tsc --noEmit -p .`: clean.
- Not related to this change and failing on `master` in this worktree
too: `runtime-selection.test.ts` connector skill cases (missing
`.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts`
"archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required
CI is the authority for those suites.

## Risks

- Low. The change only skips a resume that cannot succeed. The fresh
session uses the same workspace and full task context, like a Retry
after `native_provider_state_incompatible`.
- If the backup failed but the provider state were still somewhere
recoverable, the next run would lose provider context. For a per-turn
sandbox without a reusable lease, the restore path uses only the backup,
so this does not apply. Retained sandboxes and SSH hosts are excluded.
- One extra conditional `UPDATE` runs only on the failure path. The
normal path before `turn.start` has no new waits or work.
- `doc/execution-semantics.md` is also edited by #20809. The paragraphs
are at different places; whichever merges second rebases.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

server/src/services/native-runtime/native-session-executor.ts

@@ -271,6 +271,7 @@ import {
   stageNativeRunnerWakeAttachments,
 } from "./native-runner-file-handoff.js";
 import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
+import { clearRunProviderStateUnsaved, markAcceptedRunProviderStateUnsaved } from "./native-provider-state-unsaved.js";
 import { verifyRetainedMaintenanceNoLaunch } from "./native-maintenance-no-launch.js";
 import { issueRecoveryActionService } from "../issue-recovery-actions.js";
 import { reportRunFailure } from "../run-failure-report.js";
@@ -16655,15 +16656,50 @@ async function createRunnerdBackendWithinSessionClaim(
     );
   };
 
+  // A per-turn sandbox is released after the run, so the backup is the only
+  // copy of the provider session. When it is not saved after the result was
+  // accepted, mark the checkpoint not resumable: the next run starts a fresh
+  // provider session instead of failing once on the missing state. A retained
+  // sandbox or SSH host keeps the live state, so it keeps the resume path.
+  let providerStateUnsavedMarked = false;
+  // Set once the durable backup is stored. A later local publication, stamp,
+  // or cleanup failure leaves that backup restorable, so it is not marked.
+  let checkpointBackupStored = false;
   const checkpointRemoteRunner = async (
     settlement: "settled" | "unsettled",
   ) => {
+    let saved: boolean;
+    checkpointBackupStored = false;
+    try {
+      saved = await persistRemoteRunnerCheckpoint(settlement);
+    } catch (error) {
+      if (!checkpointBackupStored && remoteTarget?.transport === "sandbox" && remoteTarget.reusableLeaseConfigured !== true) {
+        providerStateUnsavedMarked = await markAcceptedRunProviderStateUnsaved(input.db, input.execution.binding)
+          .catch(() => false) || providerStateUnsavedMarked;
+        // A log failure cannot replace the checkpoint error.
+        if (providerStateUnsavedMarked) await Promise.resolve(input.onLog?.(
+          "stderr",
+          "[paperclip-runner] provider state was not saved after the accepted result; the next run starts a fresh provider session\n",
+        )).catch(() => undefined);
+      }
+      throw error;
+    }
+    // A later save of this run's state restores the resume path. If clearing
+    // fails, the next run still starts fresh, which is safe.
+    if (saved && providerStateUnsavedMarked) {
+      await clearRunProviderStateUnsaved(input.db, input.execution.binding).catch(() => undefined);
+      providerStateUnsavedMarked = false;
+    }
+  };
+  const persistRemoteRunnerCheckpoint = async (
+    settlement: "settled" | "unsettled",
+  ): Promise<boolean> => {
     if (
       !remoteCommandRunner ||
       !remoteStateDirectory ||
       !remoteRunnerFilesystemRoot
     )
-      return;
+      return false;
     const durableStorage = durableNativeHarnessStorage();
     const currentAuthority = sessionToolAuthorityEpochs.get(sessionScopeId);
     const backupOwner = durableStorage && currentAuthority ? sessionHarnessBackupOwners.get(currentAuthority) : null;
@@ -16704,7 +16740,7 @@ async function createRunnerdBackendWithinSessionClaim(
         `[paperclip-runner] remote checkpoint ${incompleteFailure ? "failed" : "skipped"}: exact suspended harness state unavailable (process=${settlement} reason=${checkpointable.incompleteReason})\n`,
       );
       if (incompleteFailure) throw incompleteFailure;
-      return;
+      return false;
     }
     const backupSpanAttributes = {
       provider: input.execution.provider.kind,
@@ -16802,6 +16838,10 @@ async function createRunnerdBackendWithinSessionClaim(
                     scopeDigest: createHash("sha256").update(sessionScopeId).digest("hex"), root: pendingRoot,
                     signal: AbortSignal.timeout(300_000) });
                 }
+                // Without durable storage the local publication below is the
+                // save; its result is uncertain once it starts, so keep the
+                // resume path rather than discard a backup that may exist.
+                checkpointBackupStored = true;
                 await publishNativeHarnessBackup({
                   db: input.db, companyId: input.execution.binding.companyId,
                   scopeDigest: createHash("sha256").update(sessionScopeId).digest("hex"), backupRoot, pendingRoot,
@@ -16833,6 +16873,7 @@ async function createRunnerdBackendWithinSessionClaim(
       );
       throw error;
     }
+    return true;
   };
 
   const prepareExternalRunnerState =