fix(recovery): re-prove an exhausted save's agent stop and free the workspace (#20295)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run ends, Paperclip saves the files that the run
changed in its remote workspace. Until that save settles, the run holds
the workspace, so no other task can write it
> - When the save fails, a bounded system recovery proves that the agent
stopped (a provider stop receipt) and then saves the files again
> - If the stop cannot be proven before the 30-minute recovery window
ends, the recovery is exhausted with `agent_stop_unconfirmed`.
Exhaustion was terminal: Paperclip never proved the stop again, even
after the run ended and its sandbox had a stop receipt
> - The hold then stayed forever. On staging, a child task's save was
exhausted this way on Nov 5. Its parent and its two sibling tasks share
that workspace, and all of them stayed blocked for four days
> - This pull request re-arms such an exhausted save automatically, on a
bounded backoff, once the run is provably over. After the bounded
re-arms, the hold continues from the last save with the unsaved files
attached
> - The benefit is that a dirty workspace hold always settles without a
person, and no unsaved work is discarded silently

## Linked Issues or Issue Description

Refs #20273 (the stall sweep stops failing wakes against this kind of
hold)
Refs #20278 (a settled restore hold returns its warm sandbox)
Refs #19860 (subtasks get their own workspace)

**What happened?**
A child task's run wrote its remote workspace and failed to save it
(`native_workspace_sync_out_retry_exhausted`). The save recovery made
one attempt. That attempt failed with `agent_stop_unconfirmed`, and the
recovery window ended before a second attempt. The recovery became
`exhausted`. Ten minutes after the first attempt, the sandbox lease got
a provider stop receipt, so the agent was provably gone. But nothing
looked at the exhausted recovery again. The run kept the workspace's
writer hold. The parent task and every other task on that workspace
could not start ("The workspace still has an unfinished writer or save
operation."). The blocker told a person to press Retry.

**Expected behavior**
An unproven stop is a wait, not a refusal of the saved files. When the
run is over, recovery must prove the stop again and finish the save, so
the hold settles automatically. If the stop still cannot be proven,
Paperclip must keep the unsaved files and then release the hold. A
person must not need to act.

**Steps to reproduce**
1. Run a task in a remote (sandbox) workspace. Make its final save fail.
2. Keep the workspace recovery from proving the agent stop until its
30-minute window ends (for example, keep the finalization ownership
busy).
3. See that the recovery action is `exhausted` with
`agent_stop_unconfirmed`, and that it stays so after the lease gets a
stop receipt.
4. Wake another task bound to the same workspace. See that its setup
fails because the workspace is held.

**Root cause**
- `server/src/services/workspace-recovery.ts`
(`recoverRetainedWorkspace`): an `exhausted` state only goes to the
sibling settlement and to the retained-source fallback. The fallback
accepts only permanent source refusals (`isUnverifiableSourceFailure`).
Every other exhausted state is handed to the board and never attempted
again.
- `recheckStoppedWorkspaceRecovery` skips `exhausted` states, and it
only accepts acknowledged cancellations. A failed run whose lease got a
stop receipt later is never re-examined.
- `retainedWorkspaceCandidates(…, obligationsOnly)` and
`nextWorkspaceRecoveryAt` do not select exhausted
`agent_stop_unconfirmed` holds, so the recovery worker never wakes for
them.

## What Changed

- New `server/src/services/workspace-recovery-stop-proof.ts`
(`rearmExhaustedStopProof`). For an exhausted save whose last failure is
`agent_stop_unconfirmed` (`recheck_evidence`), it starts a new bounded
system recovery generation. It does this only when the source run is
provably over: the run is terminal, no local execution remains, and no
controller lease or finalizer lease is live. The new generation repeats
the strict stop proof (a provider stop receipt) and the save. It records
an `issue.workspace_recovery_rearmed` system activity.
- Re-arms are bounded: at most 4. The first is due at once. The next
ones are due 30 minutes, 2 hours and 8 hours after the previous re-arm.
The `stopProofRecovery` marker in the recovery action evidence records
the count and the next due time.
- `workspace-recovery-fallback-discovery.ts`: new discovery conditions
(`stopProofRearmPendingCondition`, `stopProofRearmDueAt`,
`stopProofRearmDueCondition`) and `retainedHoldFallbackEligible`.
`retainedWorkspaceCandidates` (worker pass) and
`nextWorkspaceRecoveryAt` use them, so the recovery worker wakes when a
re-arm is due.
- `workspace-recovery-fallback.ts`: when every re-arm is spent and the
stop is still unproven, the hold takes the existing retained-source
fallback. The unsaved files are attached to the task (unapplied, secrets
omitted), the unverified epoch is abandoned, and the task continues from
the last save.
- `system-recovery-state.ts`: the exhausted `agent_stop_unconfirmed`
text now says that Paperclip tries again automatically. It no longer
asks a person to press Retry. Retry stays available.
- Review follow-ups: the re-arm applies to every workspace recovery the
worker owns (restore and historical holds too). A due re-arm is
scheduled no earlier than the source run's controller or finalizer lease
expiry, so the worker is not woken every 15 seconds while a lease is
live. A fallback whose bounded checks are spent is reconsidered once
when an allocation of the source run changes, and
`settleStopOnlyCleanup` wakes the workspace recovery worker when it
records a stop receipt. The automatic-retry text is shown only while a
re-arm remains.
- `doc/execution-semantics.md`: documents the re-arm, its bounds, the
fallback, and the upgrade behavior.

### Previous-release story (`rules/upgrade-safe-run-state.mdc`)

- **Carried forward, no migration.** A hold exhausted by an older
release has no `stopProofRecovery` marker. The recovery worker treats it
as due at once and re-arms it on its first pass after deploy. The new
regression test saves the exact Nov 5 evidence shape (`attempts: 1`,
`retriesUsed: 0`, `state: exhausted`, no marker) and resumes it with the
new code.
- **Mid-run deploy.** A run that is still running, or whose controller
lease or finalizer lease is live, is not re-armed. The test covers a
live controller lease. The re-arm uses the same CAS (generation and
revision) and the same row locks as the other recovery writers, so an
old worker's late write cannot overwrite the new generation.
- **No new refusal codes.** No saved run identity changes. The re-arm
writes only recovery-action evidence.

## Verification

- Red (before the fix), `npx vitest run
src/services/workspace-recovery-stop-proof.test.ts`: 3 failed, 1 passed.
The staging-shape test failed because the worker did not wake
(`nextWorkspaceRecoveryAt` returned `null`) and the hold stayed. The
backoff and fallback tests failed for the same discovery gap.
- Green (after the fix): 4 of 4 pass.
- Staging shape: a parent and 3 children that share one remote
workspace. One child's save is exhausted with `agent_stop_unconfirmed`,
and its run ended days ago with a stop receipt. Before recovery, the
hold refuses the parent (`remoteWorkspaceHoldRefusesStart` is `true`).
The recovery worker discovers the hold, re-arms it, proves the stop,
saves the files and completes. The workspace is not dirty, it has a new
snapshot, and the hold is `free` for the parent. The action stays
system-owned and its text asks no person to act.
- A live controller lease keeps the hold. No re-arm happens and no lease
is resumed.
- A re-arm that is not yet due is not discovered. The worker's next wake
is in the future, and the text says that Paperclip tries again
automatically.
- After 4 re-arms, the hold continues from the last save. The unsaved
files are captured as a task attachment, and the hold is `free`.
- A live controller lease defers the worker's next wake to the lease
expiry, and the hold is not selected before it.
- A spent fallback is not selected, and a later lease change (a late
stop receipt) makes it selected again; the worker then settles it.
- `system-recovery-state.test.ts`: the automatic-retry text appears only
while a re-arm remains.
- Related suites, all pass: `src/services/workspace-recovery*`,
`src/services/remote-workspace*`,
`src/services/system-recovery-state.test.ts`,
`src/services/task-retry.test.ts`,
`src/__tests__/heartbeat-observed-workspace-preservation.test.ts` (34
files, 1510 tests), plus the stop-cleanup suites
(`environment-retained-orphan-sandbox`, `live-recovery-work`,
`retained-native-session-cleanup-work`,
`legacy-workspace-restore-recovery`, native workspace warm-reuse,
cancelled preservation, reconstruction and failed recovery; 11 files,
622 tests).
- `pnpm verify`: exit 2 ("broader verification required", because
`doc/execution-semantics.md` is a global CI input). The focused suites
above cover the change, and required CI supplies the broad coverage.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.

## Risks

- A re-armed generation resumes the stopped sandbox to save its files.
This is the same physical path as a manual Retry. It runs only when the
source run is provably over.
- Re-arms are bounded (4) and spaced, so a stop that can never be proven
does not loop. After that, the existing retained-source fallback settles
the hold. If the retained files cannot be read, the fallback records
why. This is the same behavior as for a refused source.
- The parent's workspace receives the child's saved files, because the
child (bound by an older release) wrote that workspace. This is the
existing #19860 behavior: the files that a legacy child left in the
shared workspace stay there. #20314 moves the stranded legacy children
to their own workspace.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser read-only API checks) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -2536,6 +2536,30 @@ recovery worker discovers these holds itself, so holds saved by earlier releases
 take the same path after deploy. An exhausted hold whose run's files a sibling
 recovery action already saved settles as recovered.
 
+An exhausted save whose last failure was `agent_stop_unconfirmed` is not a
+refusal of the saved files: its recovery window ran out before the agent's stop
+could be proven. This applies to every workspace recovery the worker owns,
+export holds and restore holds alike. Once the source run is provably over (its
+run is terminal, no local execution remains, and neither its controller lease
+nor its finalizer lease is live), the recovery worker re-arms it with a new
+bounded system generation. A live lease defers the re-arm to that lease's
+expiry; the worker is not woken earlier. The new generation repeats the strict
+stop proof (a provider stop receipt for every allocation of the run) and the
+save, then settles the hold exactly like any completed recovery, so other tasks
+on the workspace can start. Re-arms are bounded: at most four, the first at once
+and the next ones 30 minutes, 2 hours and 8 hours after the previous re-arm,
+each recorded in the action's `stopProofRecovery` marker and as
+`issue.workspace_recovery_rearmed` system activity. When every re-arm of an
+export hold is spent and the stop still cannot be proven, the hold takes the
+retained-source fallback above: the unsaved files are attached to the task,
+unapplied, and the task continues from the last save. A fallback whose bounded
+checks are spent is reconsidered once whenever an allocation of the source run
+changes afterwards, for example when a late provider stop receipt arrives.
+While a re-arm remains, the hold names the wait and says that Paperclip tries
+again automatically; after that it names Retry. Holds exhausted by earlier
+releases carry no marker and are re-armed on the worker's first pass after
+deploy, with no migration.
+
 An exhausted workspace export records the exact Blocked status version it
 creates, bound to its source run, result, contract and identity. Recovery may
 restore the accepted passive wait only while that receipt still owns the task
@@ -3397,7 +3421,8 @@ Recorded system recovery failures retain their original attempt and intent.
 When system workspace recovery exhausts its bounded attempts or deadline, the
 recovery action stays active and its owner changes from system to board. Its
 next action names the attempt count and last failure code and tells the board to
-use **Retry recovery**. The task blocker shows that same text. Retry starts one
+use **Retry recovery**. An unproven agent stop (`agent_stop_unconfirmed`) instead
+names the automatic re-arm described above; Retry remains available. The task blocker shows that same text. Retry starts one
 new bounded system generation and returns ownership to the system. A sweep that
 finds an action already exhausted and still system-owned applies the same handoff
 as bookkeeping only; it grants no attempt. Exhaustion never submits a provider

server/src/services/issue-recovery-actions.ts

@@ -93,7 +93,7 @@ export type ResolveIssueRecoveryActionInput = {
 function readModelNextAction(row: IssueRecoveryActionRow): string {
   const system = readSystemRecovery(row.evidence);
   return system?.state === "exhausted" && row.evidence?.workspaceRecovery
-    ? exhaustedSystemRecoveryAction(system) : row.nextAction;
+    ? exhaustedSystemRecoveryAction(system, row.evidence) : row.nextAction;
 }
 
 function toReadModel(row: IssueRecoveryActionRow): IssueRecoveryAction {

server/src/services/sandbox-stop-and-retain.ts

@@ -1,5 +1,6 @@
 import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
 import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
 import { randomUUID } from "node:crypto";
 import { and, eq, sql, type SQL } from "drizzle-orm";
 import { environmentLeases, heartbeatRuns, type Db } from "@paperclipai/db";
@@ -198,7 +199,11 @@ export async function settleStopOnlyCleanup(db: Db, lease: Lease, options: { att
     ? { [ORPHANED_SANDBOX_RECOVERY_KEY]: { recoveredOrphanedRunLease: true, ownerProof: recordedIntent.ownerProof ?? null,
       runId: lease.heartbeatRunId, recoveredAt: now.toISOString(), fromStopIntent: true } } : {};
   const [updated] = await writeEnvironmentCleanup(db, async tx => {
-    if (stopped) await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! });
+    if (stopped) {
+      await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! });
+      // A stop receipt is new evidence for a save hold waiting on it.
+      await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
+    }
     return tx.update(environmentLeases).set({
       status: stopped ? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed",
       failureReason: stopped ? null : "sandbox_stop_pending",

server/src/services/system-recovery-state.test.ts

@@ -82,6 +82,17 @@ describe("durable system recovery budget", () => {
     expect(exhaustedSystemRecoveryAction({ ...once, attempts: 6 })).toContain("stopped after 6 attempts");
   });
 
+  it("promises an automatic retry for an unproven stop only while a re-arm remains", () => {
+    const stop = { ...failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!,
+      new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence"), { now: 0 }), state: "exhausted" as const };
+    const marker = (rearms: number) => ({ stopProofRecovery: { schema: "paperclip.stop-proof-recovery.v1", rearms, nextRearmAt: null } });
+    expect(exhaustedSystemRecoveryAction(stop, {})).toMatch(/tries again automatically/);
+    expect(exhaustedSystemRecoveryAction(stop, marker(3))).toMatch(/tries again automatically/);
+    // Spent re-arms, or no evidence to judge them, keep the Retry text.
+    expect(exhaustedSystemRecoveryAction(stop, marker(4))).toContain("Retry to try again");
+    expect(exhaustedSystemRecoveryAction(stop)).toContain("Retry to try again");
+  });
+
   it("names the last failure and its cause in the board handoff", () => {
     const state = reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!;
     const withCause = failSystemRecovery(state, new SystemRecoveryFailure("provider_unavailable", "retry_transient"),

server/src/services/system-recovery-state.ts

@@ -127,13 +127,46 @@ export function failSystemRecovery(
   };
 }
 
+/**
+ * An exhausted save whose last failure was an unproven agent stop is not a
+ * permanent refusal: once the source run has ended, a later attempt can prove
+ * the stop. Recovery re-arms it a bounded number of times on a backoff
+ * (workspace-recovery-stop-proof), then an export hold continues from the last
+ * save with the unsaved files attached, like a refused source.
+ */
+export const STOP_PROOF_RECOVERY_KEY = "stopProofRecovery";
+export const STOP_PROOF_RECOVERY_SCHEMA = "paperclip.stop-proof-recovery.v1";
+export const STOP_PROOF_MAX_REARMS = 4;
+export const STOP_PROOF_WAIT_CODE = "agent_stop_unconfirmed";
+
+export function readStopProofRecovery(evidence: Record<string, unknown> | null | undefined): { rearms: number; nextRearmAt: number | null } | null {
+  const marker = evidence?.[STOP_PROOF_RECOVERY_KEY] as Record<string, unknown> | undefined;
+  if (!marker || marker.schema !== STOP_PROOF_RECOVERY_SCHEMA || typeof marker.rearms !== "number") return null;
+  return { rearms: marker.rearms, nextRearmAt: typeof marker.nextRearmAt === "number" ? marker.nextRearmAt : null };
+}
+
+export function isExhaustedStopProofWait(state: Pick<SystemRecoveryState, "state" | "failure"> | null | undefined) {
+  return state?.state === "exhausted" && state.failure?.code === STOP_PROOF_WAIT_CODE
+    && state.failure.retryClass === "recheck_evidence";
+}
+
+/** Every bounded re-arm of an unproven stop is spent. */
+export function stopProofRearmsSpent(evidence: Record<string, unknown> | null | undefined) {
+  return (readStopProofRecovery(evidence)?.rearms ?? 0) >= STOP_PROOF_MAX_REARMS;
+}
+
 /** User-facing text for a spent recovery budget. It names what happens next and
  * never asks a person to inspect a sandbox or environment. The code stays for support. */
-export function exhaustedSystemRecoveryAction(state: SystemRecoveryState): string {
+export function exhaustedSystemRecoveryAction(state: SystemRecoveryState, evidence?: Record<string, unknown> | null): string {
   // Another Retry cannot release a copyback owner that stopped without reporting.
   if (state.failure?.code === "copyback_stop_unconfirmed")
     return "Recovery of saved work is waiting for the previous server's file copy to stop. "
       + "Recovery resumes once Paperclip can confirm that it stopped. No provider turn is submitted.";
+  // An unproven agent stop is re-proven automatically once the run is over
+  // (workspace-recovery-stop-proof), while a bounded re-arm remains.
+  if (evidence !== undefined && isExhaustedStopProofWait(state) && !stopProofRearmsSpent(evidence))
+    return "Recovery of saved work is waiting to confirm that the previous run stopped. "
+      + "Paperclip tries again automatically, and the agent will not repeat its work.";
   const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : "";
   if (isDeterministicRecoveryRefusal(state)) {
     const code = state.failure!.code;

server/src/services/system-recovery.ts

@@ -5,7 +5,8 @@ import { writeExecutionControl } from "./execution-control-notifications.js";
 import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
 import { createSystemRecovery, exhaustedSystemRecoveryAction, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
 import { workspaceRecoverySourceEligible, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
-import { retainedSourceFallbackDueAt, retainedSourceFallbackPendingCondition } from "./workspace-recovery-fallback-discovery.js";
+import { retainedSourceFallbackDueAt, retainedSourceFallbackPendingCondition, stopProofRearmDueAt,
+  stopProofRearmPendingCondition } from "./workspace-recovery-fallback-discovery.js";
 
 export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) {
   return db.transaction(async tx => {
@@ -59,7 +60,7 @@ export async function updateSystemRecovery(db: Db, source: WorkspaceRecoverySour
   // A spent budget is not system work any more. Hand it to the board with the
   // one action that can progress it, instead of leaving an inert system owner.
   const exhausted = next.state === "exhausted";
-  const handoff = exhausted ? exhaustedSystemRecoveryAction(next) : null;
+  const handoff = exhausted ? exhaustedSystemRecoveryAction(next, await loadActionEvidence(db, source)) : null;
   const compatibility = {
     schema: "paperclip.workspace-recovery.v1", runId: source.runId, status,
     stage: next.phase === "verify_execution" ? "verifying_stop" : next.phase === "preserve_files" ? "restoring_files" : "saving_receipts",
@@ -90,7 +91,7 @@ export async function updateSystemRecovery(db: Db, source: WorkspaceRecoverySour
 export async function handOffExhaustedSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity,
   state: SystemRecoveryState) {
   if (state.state !== "exhausted") return false;
-  const handoff = exhaustedSystemRecoveryAction(state);
+  const handoff = exhaustedSystemRecoveryAction(state, await loadActionEvidence(db, source));
   const [updated] = await db.update(issueRecoveryActions).set({
     ownerType: "board", ownerAgentId: null, ownerUserId: null, nextAction: handoff,
     evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{workspaceRecovery,message}', ${JSON.stringify(handoff)}::jsonb)`,
@@ -105,6 +106,12 @@ export async function handOffExhaustedSystemRecovery(db: Db, source: WorkspaceRe
   return Boolean(updated);
 }
 
+async function loadActionEvidence(db: Db, source: WorkspaceRecoverySourceIdentity) {
+  const [row] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
+    eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId)));
+  return row?.evidence ?? null;
+}
+
 export async function loadSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) {
   const [row] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
     eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.sourceIssueId, source.issueId),
@@ -115,6 +122,7 @@ export async function loadSystemRecovery(db: Db, source: WorkspaceRecoverySource
 export async function nextWorkspaceRecoveryAt(db: Db): Promise<number | null> {
   const rows = await db.select({ at: sql<number>`case
     when ${retainedSourceFallbackPendingCondition()} then ${retainedSourceFallbackDueAt()}
+    when ${stopProofRearmPendingCondition()} then ${stopProofRearmDueAt()}
     when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}') = 'number'
     then (${issueRecoveryActions.evidence}#>>'{systemRecovery,nextAttemptAt}')::bigint else 0 end` })
     .from(issueRecoveryActions).innerJoin(heartbeatRuns, and(
@@ -126,6 +134,7 @@ export async function nextWorkspaceRecoveryAt(db: Db): Promise<number | null> {
       workspaceRecoverySourceEligible(true),
       sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
         or ${retainedSourceFallbackPendingCondition()}
+        or ${stopProofRearmPendingCondition()}
         or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'
           and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
           and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'

server/src/services/workspace-recovery-fallback-discovery.ts

@@ -1,13 +1,72 @@
 import { sql } from "drizzle-orm";
 import { issueRecoveryActions } from "@paperclipai/db";
+import { isExhaustedStopProofWait, STOP_PROOF_MAX_REARMS, STOP_PROOF_RECOVERY_KEY, STOP_PROOF_RECOVERY_SCHEMA, STOP_PROOF_WAIT_CODE,
+  stopProofRearmsSpent, type SystemRecoveryState } from "./system-recovery-state.js";
 
 /** Source refusals that repeat on every retry (see workspace-recovery-fallback). */
 export const UNVERIFIABLE_SOURCE_CODES = ["allocation_ownership_ambiguous", "source_binding_invalid", "source_unavailable", "historical_source_missing"] as const;
 export const PERMANENT_SOURCE_RETRY_CLASSES = ["service_required", "use_alternative_strategy"] as const;
 export const RETAINED_SOURCE_FALLBACK_CHECK_KEY = "retainedSourceFallbackCheck";
 
+/** Whether an exhausted hold may continue from the last save: its source is
+ * refused permanently, or its stop stayed unproven through every re-arm. */
+export function retainedHoldFallbackEligible(evidence: Record<string, unknown> | null | undefined, state: SystemRecoveryState | null) {
+  if (!state || state.state !== "exhausted") return false;
+  const failure = state.failure;
+  if (failure && (UNVERIFIABLE_SOURCE_CODES as readonly string[]).includes(failure.code)
+    && (PERMANENT_SOURCE_RETRY_CLASSES as readonly string[]).includes(failure.retryClass)) return true;
+  return isExhaustedStopProofWait(state) && stopProofRearmsSpent(evidence);
+}
+
+const stopProofMarker = sql`${issueRecoveryActions.evidence}->${STOP_PROOF_RECOVERY_KEY}`;
+const stopProofRearms = sql`case when ${stopProofMarker}->>'schema' = ${STOP_PROOF_RECOVERY_SCHEMA}
+  and jsonb_typeof(${stopProofMarker}->'rearms') = 'number' then (${stopProofMarker}->>'rearms')::int else 0 end`;
+const exhaustedStopProofWait = sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'exhausted'
+  and ${issueRecoveryActions.evidence}#>>'{systemRecovery,failure,code}' = ${STOP_PROOF_WAIT_CODE}
+  and ${issueRecoveryActions.evidence}#>>'{systemRecovery,failure,retryClass}' = 'recheck_evidence')`;
+
+/** An exhausted unproven-stop hold whose next bounded re-arm is pending. Holds
+ * saved by older releases carry no marker and are due immediately. A finalizer
+ * owner whose lease is not live waits for the abandoned-owner settlement (the
+ * attempt would refuse it, and the passive sweep revisits the hold after it);
+ * a live one makes the re-arm due at its expiry (stopProofRearmDueAt). */
+export function stopProofRearmPendingCondition() {
+  return sql`(${issueRecoveryActions.status} in ('active', 'escalated')
+    and ${exhaustedStopProofWait}
+    and not exists (select 1 from native_run_finalizations finalizer
+      where finalizer.company_id = ${issueRecoveryActions.companyId} and finalizer.run_id::text = ${issueRecoveryActions.evidence}->>'runId'
+        and finalizer.lease_owner is not null and coalesce(finalizer.lease_expires_at <= now(), true))
+    and ${issueRecoveryActions.evidence}->'retainedSourceFallback' is null
+    and ${issueRecoveryActions.evidence}->'executionReconciliation' is null
+    and ${stopProofRearms} < ${STOP_PROOF_MAX_REARMS})`;
+}
+
+/** When the next re-arm is due, in epoch milliseconds (0 = now): its backoff,
+ * and no earlier than the expiry of a controller or finalizer lease that the
+ * source run still holds, so a held run is checked at that deadline only. */
+export function stopProofRearmDueAt() {
+  return sql`greatest(case when ${stopProofMarker}->>'schema' = ${STOP_PROOF_RECOVERY_SCHEMA}
+      and jsonb_typeof(${stopProofMarker}->'nextRearmAt') = 'number' then (${stopProofMarker}->>'nextRearmAt')::bigint else 0 end,
+    coalesce((select (extract(epoch from source.controller_lease_expires_at) * 1000)::bigint from heartbeat_runs source
+      where source.company_id = ${issueRecoveryActions.companyId} and source.id::text = ${issueRecoveryActions.evidence}->>'runId'), 0),
+    coalesce((select max((extract(epoch from finalizer.lease_expires_at) * 1000)::bigint) from native_run_finalizations finalizer
+      where finalizer.company_id = ${issueRecoveryActions.companyId} and finalizer.run_id::text = ${issueRecoveryActions.evidence}->>'runId'
+        and finalizer.lease_owner is not null), 0))`;
+}
+
+export function stopProofRearmDueCondition(now: number) {
+  return sql`(${stopProofRearmPendingCondition()} and ${stopProofRearmDueAt()} <= ${now}::bigint)`;
+}
+
 const list = (values: readonly string[]) => sql.join(values.map(value => sql`${value}`), sql`, `);
 const check = sql`${issueRecoveryActions.evidence}->${RETAINED_SOURCE_FALLBACK_CHECK_KEY}`;
+// New stop evidence (any change to an allocation of the source run, such as a
+// late provider stop receipt) after the last deferred check reconsiders the
+// hold once, even after its bounded checks are spent.
+const leaseChangedSinceCheck = sql`(jsonb_typeof(${check}->'checkedAt') = 'number' and exists (
+  select 1 from environment_leases lease where lease.company_id = ${issueRecoveryActions.companyId}
+    and lease.heartbeat_run_id::text = ${issueRecoveryActions.evidence}->>'runId'
+    and lease.updated_at > to_timestamp((${check}->>'checkedAt')::bigint / 1000.0)))`;
 
 /**
  * An exhausted hold that the fallback may still settle: its source is refused
@@ -27,6 +86,11 @@ export function retainedSourceFallbackPendingCondition() {
         and exists (select 1 from heartbeat_runs source where source.company_id = ${issueRecoveryActions.companyId}
           and source.id::text = ${issueRecoveryActions.evidence}->>'runId'
           and source.runner_profile_json#>>'{remoteWorkspace,state}' = 'prepared'))
+      or (${issueRecoveryActions.cause} in ('native_workspace_sync_out_unrecoverable', 'native_workspace_sync_out_retry_exhausted')
+        and ${exhaustedStopProofWait} and ${stopProofRearms} >= ${STOP_PROOF_MAX_REARMS}
+        and exists (select 1 from heartbeat_runs source where source.company_id = ${issueRecoveryActions.companyId}
+          and source.id::text = ${issueRecoveryActions.evidence}->>'runId'
+          and source.runner_profile_json#>>'{remoteWorkspace,state}' = 'prepared'))
       or exists (select 1 from issue_recovery_actions sibling
         where sibling.company_id = ${issueRecoveryActions.companyId} and sibling.source_issue_id = ${issueRecoveryActions.sourceIssueId}
           and sibling.id <> ${issueRecoveryActions.id} and sibling.evidence->>'runId' = ${issueRecoveryActions.evidence}->>'runId'
@@ -34,6 +98,7 @@ export function retainedSourceFallbackPendingCondition() {
           and sibling.evidence#>>'{workspaceRecovery,status}' = 'recovered'))
     and (${check} is null
       or ${check}->>'generationId' is distinct from ${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}'
+      or ${leaseChangedSinceCheck}
       or jsonb_typeof(${check}->'nextCheckAt') = 'number'))`;
 }
 

server/src/services/workspace-recovery-fallback.ts

@@ -26,7 +26,7 @@ import { abandonUnverifiedRemoteWorkspaceEpoch } from "./remote-workspace-epoch-
 import { hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, restoreSandboxStopAfterFailedRestart, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
 import { nativeWorkspaceFinalizationBlockSchema } from "./native-runtime/preserved-native-result-finalization.js";
 import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
-import { PERMANENT_SOURCE_RETRY_CLASSES, RETAINED_SOURCE_FALLBACK_CHECK_KEY, UNVERIFIABLE_SOURCE_CODES } from "./workspace-recovery-fallback-discovery.js";
+import { PERMANENT_SOURCE_RETRY_CLASSES, RETAINED_SOURCE_FALLBACK_CHECK_KEY, UNVERIFIABLE_SOURCE_CODES, retainedHoldFallbackEligible } from "./workspace-recovery-fallback-discovery.js";
 import { captureReadOnlyWorkspaceArchive, sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
 import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js";
 import { readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
@@ -115,7 +115,7 @@ async function inspectFallback(db: Db, input: Source, lock: boolean, requireStop
   const system = readSystemRecovery(action.evidence);
   if (!exportCauses.includes(action.cause) || !["active", "escalated"].includes(action.status) || action.evidence.runId !== run.id
     || action.evidence.executionReconciliation || system?.state !== "exhausted" || system.sourceRunId !== run.id
-    || !isUnverifiableSourceFailure(system.failure)) return refuse("not_unverifiable_source_hold");
+    || !retainedHoldFallbackEligible(action.evidence, system)) return refuse("not_unverifiable_source_hold");
   if (run.runtimeMode !== "native" || !terminal.includes(run.status) || !run.finishedAt
     || (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issue.id) return refuse("source_run_not_terminal");
   // No second executor: the source run, its controller and its finalizer have stopped.
@@ -417,7 +417,7 @@ export async function recordRetainedSourceFallbackDeferral(db: Db, input: Source
 /**
  * The recovery worker's entry for an exhausted hold: settle it when a sibling
  * saved its files, or continue from the last save when its source is refused
- * permanently. Runs under the workspace ownership lock, like any physical
+ * permanently or its stop stayed unproven through every bounded re-arm. Runs under the workspace ownership lock, like any physical
  * recovery attempt. Returns whether the hold is settled.
  */
 export async function settleExhaustedRetainedHold(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService },
@@ -425,7 +425,10 @@ export async function settleExhaustedRetainedHold(input: Source & { db: Db; envi
   const { db } = input;
   const sibling = await settleSiblingRecoveredHold(db, input);
   if (sibling === "settled") return true;
-  if (!options.fallback || !isUnverifiableSourceFailure(state.failure)) {
+  const [row] = options.fallback ? await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
+    eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
+    eq(issueRecoveryActions.id, input.actionId))) : [];
+  if (!options.fallback || !retainedHoldFallbackEligible(row?.evidence, state)) {
     // A sibling marked recovered without a receipt that still verifies cannot
     // settle this hold. Record it so discovery does not select it again.
     if (sibling === "unverified") await recordRetainedSourceFallbackDeferral(db, input, "sibling_receipt_unverified");

server/src/services/workspace-recovery-receipt.ts

@@ -67,7 +67,7 @@ export function workspaceRecoveryDisplay(evidence: Record<string, unknown>, runI
  * described from its current state instead of the stored text. */
 export function blockedWorkspaceRecoveryMessage(evidence: Record<string, unknown> | null | undefined): string | null {
   const system = readSystemRecovery(evidence ?? undefined);
-  return system?.state === "exhausted" ? exhaustedSystemRecoveryAction(system) : null;
+  return system?.state === "exhausted" ? exhaustedSystemRecoveryAction(system, evidence ?? null) : null;
 }
 
 async function verifiedNativeCopybackReceipt(db: Db, input: {

server/src/services/workspace-recovery-stop-proof.test.ts

@@ -0,0 +1,250 @@
+import { randomUUID } from "node:crypto";
+import { readFile } from "node:fs/promises";
+import path from "node:path";
+import { eq } from "drizzle-orm";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { activityLog, environmentLeases, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
+import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
+import { readRemoteWorkspaceAuthority } from "@paperclipai/shared";
+import { readRemoteWorkspaceWriterHold, remoteWorkspaceHoldRefusesStart } from "./remote-workspace-writer-hold.js";
+import { readSystemRecovery } from "./system-recovery-state.js";
+import { adoptSystemRecovery } from "./system-recovery.js";
+import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
+import { prepareSandboxStopAndRetain, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
+import { nextWorkspaceRecoveryAt } from "./system-recovery.js";
+import { RETAINED_SOURCE_FALLBACK_OUTCOME } from "./workspace-recovery-fallback.js";
+import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
+import { reconcileRetainedWorkspaces, recoverRetainedWorkspace, retainedWorkspaceCandidates } from "./workspace-recovery.js";
+import { acceptedRemoteExport, action, connectionString, db, run, seed, useWorkspaceRecoveryFixture } from "./__tests__/workspace-recovery-fixture.js";
+
+const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
+vi.mock("./environment-execution-target.js", async importOriginal => {
+  const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
+  return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
+    (input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
+});
+
+// Recovery text never asks a person to inspect or fix a sandbox.
+const adminText = /sandbox|environment|administrator|board member|owner must|inspect the run/i;
+const workspace = async (id: string) => (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id)))[0]!;
+
+describe("exhausted stop-proof holds", () => {
+  useWorkspaceRecoveryFixture(targets);
+  beforeEach(() => {
+    vi.stubEnv("DATABASE_URL", connectionString);
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "63".repeat(32));
+  });
+  afterEach(() => { vi.unstubAllEnvs(); });
+
+  /**
+   * The staging shape (Nov 5): a parent task and three children that an older
+   * release bound to the parent's remote workspace. One child's run wrote the
+   * workspace, failed to save it, and its save recovery was exhausted after one
+   * attempt with `agent_stop_unconfirmed`: the recovery window ran out while
+   * the stop could not be proven yet. The run has since ended and its sandbox
+   * has a provider stop receipt, so the agent is provably gone.
+   */
+  async function exhaustedStopProofHold() {
+    const f = await acceptedRemoteExport("retained");
+    const source = await run(f.runId);
+    const [parent] = await db.insert(issues).values({ companyId: f.companyId, title: "Landing page", status: "blocked",
+      assigneeAgentId: source.agentId, executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" }).returning();
+    await db.update(executionWorkspaces).set({ sourceIssueId: parent.id }).where(eq(executionWorkspaces.id, f.workspaceId));
+    await db.update(issues).set({ parentId: parent.id, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, f.issueId));
+    const siblings = await db.insert(issues).values(["Hero copy", "Pricing table"].map(title => ({ companyId: f.companyId, title,
+      parentId: parent.id, status: "blocked" as const, assigneeAgentId: source.agentId, executionWorkspaceId: f.workspaceId,
+      executionWorkspacePreference: "reuse_existing" }))).returning();
+    const firstFailureAt = Date.now() - 5 * 86_400_000;
+    // The writer went silent days ago, past the workspace holder cutoff.
+    await db.update(heartbeatRuns).set({ finishedAt: new Date(firstFailureAt - 60_000) }).where(eq(heartbeatRuns.id, f.runId));
+    const nextAction = "Automatic recovery of saved work stopped after 1 attempts (last failure: agent_stop_unconfirmed). "
+      + "Retry to try again; the agent will not repeat its work.";
+    const prior = await action(f.actionId);
+    const current = readSystemRecovery(prior.evidence)!;
+    // The exact evidence the Nov 5 release saved; no field this change adds.
+    await db.update(issueRecoveryActions).set({ ownerType: "system", nextAction, evidence: { ...prior.evidence,
+      systemRecovery: { ...current, revision: 3, phase: "verify_execution", state: "exhausted", attempts: 1, retriesUsed: 0,
+        attemptId: randomUUID(), firstFailureAt, deadlineAt: firstFailureAt + 30 * 60_000, nextAttemptAt: null,
+        failure: { code: "agent_stop_unconfirmed", retryClass: "recheck_evidence", evidenceFingerprint: "d".repeat(64) } },
+      workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked", stage: "verifying_stop",
+        attempt: 1, attemptLimit: 6, retryable: true, requestIds: [] },
+      coordinatorAttempt: 1, originalFailureCode: "native_workspace_sync_out_failed", workspaceFinalizeAttempt: 3 } })
+      .where(eq(issueRecoveryActions.id, f.actionId));
+    return { ...f, parentIssueId: parent.id, siblingIssueIds: siblings.map(row => row.id) };
+  }
+
+  async function parentHold(f: { companyId: string; workspaceId: string }) {
+    const authority = readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)!;
+    return readRemoteWorkspaceWriterHold(db as never, { companyId: f.companyId, authority, runId: randomUUID() });
+  }
+
+  it("re-proves the stop, saves the files and frees the shared workspace without a person", async () => {
+    const f = await exhaustedStopProofHold();
+    // The hold refuses every other task on the shared workspace, the parent included.
+    expect(remoteWorkspaceHoldRefusesStart(await parentHold(f))).toBe(true);
+
+    // The recovery worker discovers it and wakes for it.
+    expect(await nextWorkspaceRecoveryAt(db)).toBeLessThanOrEqual(Date.now());
+    const ids = (await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id);
+    expect(ids).toContain(f.actionId);
+    const onRecovered = vi.fn(async () => undefined);
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, onRecovered, true) ?? []);
+
+    const settled = await action(f.actionId);
+    expect(readSystemRecovery(settled.evidence)).toMatchObject({ state: "complete" });
+    expect(readSystemRecovery(settled.evidence)!.preservationOperationIds.length).toBeGreaterThan(0);
+    expect(readWorkspaceRecovery(settled.evidence)?.status).toBe("recovered");
+    expect(settled.evidence.stopProofRecovery).toMatchObject({ schema: "paperclip.stop-proof-recovery.v1", rearms: 1 });
+    expect(settled.ownerType).toBe("system");
+    expect(String(settled.nextAction)).not.toMatch(adminText);
+    expect(onRecovered).toHaveBeenCalledTimes(1);
+
+    // The child's files are saved into the workspace it wrote; the hold is gone.
+    const authority = readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)!;
+    expect(authority).toMatchObject({ dirty: false });
+    expect(authority.snapshotId).not.toBeNull();
+    expect(await parentHold(f)).toEqual({ kind: "free" });
+    expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
+
+    const [rearmed] = await db.select().from(activityLog).where(eq(activityLog.action, "issue.workspace_recovery_rearmed"));
+    expect(rearmed).toMatchObject({ actorType: "system", entityId: f.issueId, runId: f.runId });
+  }, 60_000);
+
+  it("keeps the hold while the source run's controller still holds its lease", async () => {
+    const f = await exhaustedStopProofHold();
+    const expiresAt = Date.now() + 60 * 60_000;
+    await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(expiresAt) }).where(eq(heartbeatRuns.id, f.runId));
+
+    // The worker is not woken until the lease can expire, and does not select the hold before that.
+    expect(await nextWorkspaceRecoveryAt(db)).toBeGreaterThanOrEqual(expiresAt - 1_000);
+    expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).not.toContain(f.actionId);
+    await recoverRetainedWorkspace(f.input);
+
+    const held = await action(f.actionId);
+    expect(readSystemRecovery(held.evidence)).toMatchObject({ state: "exhausted", failure: { code: "agent_stop_unconfirmed" } });
+    expect(held.evidence.stopProofRecovery).toBeUndefined();
+    expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
+  }, 60_000);
+
+  it("waits for the next re-arm on its backoff", async () => {
+    const f = await exhaustedStopProofHold();
+    const prior = await action(f.actionId);
+    await db.update(issueRecoveryActions).set({ evidence: { ...prior.evidence, stopProofRecovery: {
+      schema: "paperclip.stop-proof-recovery.v1", rearms: 1, generationId: randomUUID(), previousGenerationId: randomUUID(),
+      rearmedAt: Date.now() - 60_000, nextRearmAt: Date.now() + 60 * 60_000 } } }).where(eq(issueRecoveryActions.id, f.actionId));
+
+    expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).not.toContain(f.actionId);
+    expect(await nextWorkspaceRecoveryAt(db)).toBeGreaterThan(Date.now());
+    await recoverRetainedWorkspace(f.input);
+    const waiting = await action(f.actionId);
+    expect(readSystemRecovery(waiting.evidence)?.state).toBe("exhausted");
+    // The wait names the automatic retry; it asks no person to act.
+    expect(waiting.nextAction).toMatch(/tries again automatically/);
+    expect(String(waiting.nextAction)).not.toMatch(adminText);
+  }, 60_000);
+
+  it("after its bounded re-arms, attaches the unsaved files and continues from the last save", async () => {
+    const f = await exhaustedStopProofHold();
+    const prior = await action(f.actionId);
+    await db.update(issueRecoveryActions).set({ evidence: { ...prior.evidence, stopProofRecovery: {
+      schema: "paperclip.stop-proof-recovery.v1", rearms: 4, generationId: randomUUID(), previousGenerationId: randomUUID(),
+      rearmedAt: Date.now() - 86_400_000, nextRearmAt: null } } }).where(eq(issueRecoveryActions.id, f.actionId));
+
+    expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).toContain(f.actionId);
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []);
+
+    const settled = await action(f.actionId);
+    expect(settled).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+    const decision = settled.evidence.executionReconciliation as { retainedSourceFallback: { quarantine: { state: string } } };
+    // No unsaved work is discarded silently: it is attached to the child task.
+    expect(decision.retainedSourceFallback.quarantine.state).toBe("captured");
+    expect(await parentHold(f)).toEqual({ kind: "free" });
+  }, 60_000);
+
+  it("reconsiders a spent fallback when a late stop receipt arrives", async () => {
+    const f = await exhaustedStopProofHold();
+    const prior = await action(f.actionId);
+    const generationId = readSystemRecovery(prior.evidence)!.generationId;
+    const checkedAt = Date.now() - 60_000;
+    await db.update(issueRecoveryActions).set({ evidence: { ...prior.evidence,
+      stopProofRecovery: { schema: "paperclip.stop-proof-recovery.v1", rearms: 4, generationId, previousGenerationId: randomUUID(),
+        rearmedAt: Date.now() - 86_400_000, nextRearmAt: null },
+      retainedSourceFallbackCheck: { generationId, reason: "execution_not_stopped", checks: 12, checkedAt, nextCheckAt: null } } })
+      .where(eq(issueRecoveryActions.id, f.actionId));
+    await db.update(environmentLeases).set({ updatedAt: new Date(checkedAt - 60_000) }).where(eq(environmentLeases.heartbeatRunId, f.runId));
+    const ids = async () => (await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id);
+    // Every bounded check is spent: the worker stops selecting the hold.
+    expect(await ids()).not.toContain(f.actionId);
+
+    // A provider stop receipt lands on the source allocation afterwards.
+    await db.update(environmentLeases).set({ updatedAt: new Date() }).where(eq(environmentLeases.heartbeatRunId, f.runId));
+    expect(await ids()).toContain(f.actionId);
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []);
+    expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+    expect(await parentHold(f)).toEqual({ kind: "free" });
+  }, 60_000);
+
+  /** Exhausts a saved-work recovery with the Nov 5 evidence shape: one attempt, `agent_stop_unconfirmed`, window spent. */
+  async function exhaustStopWait(actionId: string, runId: string) {
+    const prior = await action(actionId);
+    const current = readSystemRecovery(prior.evidence)!;
+    const firstFailureAt = Date.now() - 5 * 86_400_000;
+    await db.update(issueRecoveryActions).set({ evidence: { ...prior.evidence,
+      systemRecovery: { ...current, revision: current.revision + 3, state: "exhausted", attempts: 1, retriesUsed: 0, attemptId: randomUUID(),
+        firstFailureAt, deadlineAt: firstFailureAt + 30 * 60_000, nextAttemptAt: null,
+        failure: { code: "agent_stop_unconfirmed", retryClass: "recheck_evidence", evidenceFingerprint: "e".repeat(64) } },
+      workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId, status: "blocked", stage: "verifying_stop",
+        attempt: 1, attemptLimit: 6, retryable: true, requestIds: [] } } }).where(eq(issueRecoveryActions.id, actionId));
+  }
+
+  it("re-arms an exhausted restore hold too, through the worker's own discovery", async () => {
+    // A restore hold (native continuation reconciliation), not a workspace export.
+    const f = await seed("native", true);
+    await adoptSystemRecovery(db, f);
+    expect((await action(f.actionId)).cause).toBe("native_continuation_requires_reconciliation");
+    await exhaustStopWait(f.actionId, f.runId);
+
+    expect(await nextWorkspaceRecoveryAt(db)).toBeLessThanOrEqual(Date.now());
+    expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).toContain(f.actionId);
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []);
+
+    const rearmed = await action(f.actionId);
+    expect(rearmed.evidence.stopProofRecovery).toMatchObject({ rearms: 1 });
+    expect(readSystemRecovery(rearmed.evidence)).toMatchObject({ state: "complete" });
+    expect(rearmed.ownerType).toBe("system");
+  }, 60_000);
+
+  it("leaves a stale finalizer owner to the abandoned-owner settlement instead of spending re-arms", async () => {
+    const f = await exhaustedStopProofHold();
+    await db.update(nativeRunFinalizations).set({ leaseOwner: `native-finalizer:${randomUUID()}`, leaseExpiresAt: new Date(Date.now() - 60_000) })
+      .where(eq(nativeRunFinalizations.runId, f.runId));
+
+    expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).not.toContain(f.actionId);
+    const next = await nextWorkspaceRecoveryAt(db);
+    expect(next === null || next > Date.now()).toBe(true);
+    await recoverRetainedWorkspace(f.input);
+    const held = await action(f.actionId);
+    expect(held.evidence.stopProofRecovery).toBeUndefined();
+    expect(readSystemRecovery(held.evidence)?.state).toBe("exhausted");
+
+    // Once that settlement releases the owner, the hold is due again.
+    await db.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, f.runId));
+    expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).toContain(f.actionId);
+  }, 60_000);
+
+  it("wakes the workspace recovery worker when a stop receipt is recorded", async () => {
+    const f = await acceptedRemoteExport("retained");
+    const woken: string[] = [];
+    const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.workspaceRecovery, () => woken.push("workspace-recovery"));
+    try {
+      const [active] = await db.update(environmentLeases).set({ status: "active", releasedAt: null, cleanupStatus: null,
+        metadata: { ...f.lease.metadata, remoteExecutionTermination: undefined } }).where(eq(environmentLeases.id, f.lease.id)).returning();
+      const stopping = await prepareSandboxStopAndRetain(db, active);
+      expect(stopping).not.toBeNull();
+      woken.length = 0;
+      expect(await settleStopOnlyCleanup(db, stopping!, { attemptId: String(stopping!.metadata?.pendingCleanupAttemptId ?? randomUUID()),
+        receipt: { providerLeaseId: f.providerLeaseId, state: "stopped" } })).not.toBeNull();
+      await vi.waitFor(() => expect(woken.length).toBeGreaterThan(0), { timeout: 5_000 });
+    } finally { stop(); }
+  }, 60_000);
+});

server/src/services/workspace-recovery-stop-proof.ts

@@ -0,0 +1,89 @@
+import { and, eq } from "drizzle-orm";
+import { heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
+import { logger } from "../middleware/logger.js";
+import { adapterExecutionControls } from "./adapter-execution-control.js";
+import { persistActivity, publishActivity } from "./activity-log.js";
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
+import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
+import { createSystemRecovery, isExhaustedStopProofWait, readStopProofRecovery, readSystemRecovery, STOP_PROOF_MAX_REARMS,
+  STOP_PROOF_RECOVERY_KEY, STOP_PROOF_RECOVERY_SCHEMA, type SystemRecoveryState } from "./system-recovery-state.js";
+import { terminalWorkspaceRunStates, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
+
+/** Delay before re-arm n+1, measured from re-arm n. The first re-arm is due at once. */
+const STOP_PROOF_REARM_DELAYS_MS = [30 * 60_000, 2 * 60 * 60_000, 8 * 60 * 60_000];
+
+/** User-facing text while a re-armed save runs. It names no person who must act. */
+export const STOP_PROOF_REARM_MESSAGE = "Paperclip is saving the previous run's files again now that the run has ended. "
+  + "No provider turn is submitted.";
+
+/**
+ * Re-arms an exhausted save whose last failure was an unproven agent stop.
+ * Exhaustion there only means the recovery window ran out before the stop
+ * could be proven; it is not a refusal of the saved files. Once the source run
+ * is provably over (terminal, no local execution, no live controller or
+ * finalizer lease), a new recovery generation repeats the strict stop proof
+ * (a provider stop receipt) and the save. Bounded: at most
+ * STOP_PROOF_MAX_REARMS re-arms on a growing backoff, after which the hold
+ * continues from the last save with the unsaved files attached
+ * (workspace-recovery-fallback). Holds exhausted by older releases carry no
+ * marker and are re-armed on the first pass, with no migration.
+ * Returns whether a new generation was written.
+ */
+export async function rearmExhaustedStopProof(db: Db, input: WorkspaceRecoverySourceIdentity, observed: SystemRecoveryState,
+  now = Date.now()): Promise<boolean> {
+  if (!isExhaustedStopProofWait(observed)) return false;
+  const outcome = await db.transaction(async tx => {
+    // Lock order shared with the fallback: task, coordinator, run, recovery action.
+    const [issue] = await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, input.companyId),
+      eq(issues.id, input.issueId))).for("update");
+    const [coordinator] = await tx.select({ leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt })
+      .from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId),
+        eq(nativeRunFinalizations.runId, input.runId))).for("update");
+    const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
+      eq(heartbeatRuns.id, input.runId))).for("update");
+    const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
+      eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update");
+    if (!issue || !run || !action) return null;
+    const system = readSystemRecovery(action.evidence);
+    if (!system || system.generationId !== observed.generationId || system.revision !== observed.revision
+      || !isExhaustedStopProofWait(system) || system.sourceRunId !== run.id) return null;
+    if (!["active", "escalated"].includes(action.status)
+      || action.evidence.runId !== run.id || action.evidence.executionReconciliation || action.evidence.retainedSourceFallback) return null;
+    // The agent is provably gone: its run ended and nothing still executes or
+    // finalizes it. The new attempt still proves the sandbox stop itself.
+    const at = new Date(now);
+    if (!terminalWorkspaceRunStates.includes(run.status) || !run.finishedAt || adapterExecutionControls.has(run.id)
+      || (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > at)
+      // The attempt refuses any recorded finalizer owner (inspectWorkspaceRecoverySource);
+      // a stale one is released by the abandoned-owner settlement first.
+      || coordinator?.leaseOwner) return null;
+    const marker = readStopProofRecovery(action.evidence);
+    const rearms = marker?.rearms ?? 0;
+    if (rearms >= STOP_PROOF_MAX_REARMS || (marker?.nextRearmAt ?? 0) > now) return null;
+    // Any workspace recovery the worker owns: export holds and restore holds alike.
+    const next: SystemRecoveryState = { ...createSystemRecovery(run.id), revision: system.revision + 1,
+      preservationOperationIds: system.preservationOperationIds, providerCooldownAt: system.providerCooldownAt,
+      nextAttemptAt: system.providerCooldownAt };
+    const count = rearms + 1;
+    const delay = STOP_PROOF_REARM_DELAYS_MS[Math.min(count - 1, STOP_PROOF_REARM_DELAYS_MS.length - 1)];
+    const stopProofRecovery = { schema: STOP_PROOF_RECOVERY_SCHEMA, rearms: count, generationId: next.generationId,
+      previousGenerationId: system.generationId, rearmedAt: now, nextRearmAt: count < STOP_PROOF_MAX_REARMS ? now + delay : null };
+    const prior = action.evidence.workspaceRecovery as { requestIds?: unknown } | undefined;
+    await notifyIssueRecovery(tx, { companyId: input.companyId });
+    await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
+    await tx.update(issueRecoveryActions).set({ ownerType: "system", ownerAgentId: null, ownerUserId: null,
+      nextAction: STOP_PROOF_REARM_MESSAGE, updatedAt: at,
+      evidence: { ...action.evidence, systemRecovery: next, [STOP_PROOF_RECOVERY_KEY]: stopProofRecovery,
+        workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: run.id, status: "queued", stage: "verifying_stop",
+          attempt: 0, attemptLimit: 6, retryable: true, message: STOP_PROOF_REARM_MESSAGE,
+          requestIds: Array.isArray(prior?.requestIds) ? prior.requestIds : [] } },
+    }).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, action.id)));
+    return persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
+      action: "issue.workspace_recovery_rearmed", entityType: "issue", entityId: input.issueId, runId: run.id,
+      details: { recoveryActionId: action.id, failureCode: system.failure?.code ?? null, rearm: count,
+        maxRearms: STOP_PROOF_MAX_REARMS, previousGenerationId: system.generationId, generationId: next.generationId } });
+  });
+  if (!outcome) return false;
+  try { publishActivity(outcome.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
+  return true;
+}

server/src/services/workspace-recovery.ts

@@ -36,7 +36,8 @@ import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfil
 import { canAdoptNativeSourceRetention, retainNativeRecoverySource, settleRecoveredNativeSourceRetention } from "./native-runtime/native-source-retention.js";
 import { saveRemoteWorkspace } from "./remote-workspaces.js";
 import { continueFromLastSavedWorkspace, isUnverifiableSourceFailure, recordRetainedSourceFallbackDeferral, settleExhaustedRetainedHold } from "./workspace-recovery-fallback.js";
-import { retainedSourceFallbackDueCondition } from "./workspace-recovery-fallback-discovery.js";
+import { retainedSourceFallbackDueCondition, stopProofRearmDueCondition } from "./workspace-recovery-fallback-discovery.js";
+import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
 import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
 
 const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
@@ -198,13 +199,16 @@ export async function requestWorkspaceRecovery(input: Source & { db: Db; idempot
 export async function recoverRetainedWorkspace(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService }) {
   const { db } = input;
   await adoptSystemRecovery(db, input);
-  const initial = await loadSystemRecovery(db, input);
+  let initial = await loadSystemRecovery(db, input);
   // A hold whose files a sibling saved, or whose source is refused permanently
   // (including holds saved by older releases), settles without a person.
   // An adoptable native source keeps its retention path; only the sibling
   // settlement applies to it.
   if (initial?.state === "exhausted"
     && await settleExhaustedRetainedHold(input, initial, { fallback: !canAdoptNativeSourceRetention(initial) })) return;
+  // An exhausted wait for stop proof is re-armed, bounded, once the run is over.
+  if (initial?.state === "exhausted" && await rearmExhaustedStopProof(db, input, initial))
+    initial = await loadSystemRecovery(db, input);
   if (initial?.state === "exhausted" && !canAdoptNativeSourceRetention(initial)) {
     // A budget spent before this hold existed still names its receipt, so the
     // board confirmation in its handoff can release it (RC-50 wave databases).
@@ -624,7 +628,7 @@ export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedO
           and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'))` : undefined,
       obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
         or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and (${cleanup} or ${interruptedPreparation})) or ${waitingInput}
-        or ${retainedSourceFallbackDueCondition(Date.now())})` : undefined,
+        or ${retainedSourceFallbackDueCondition(Date.now())} or ${stopProofRearmDueCondition(Date.now())})` : undefined,
       sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted')
         and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb
           or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${Date.now()}::bigint)))`,