fix(native): let a Stop before the provider session opens continue once (#19206)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. **What happened?** A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). **Expected behavior** The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. **Steps to reproduce** 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. **Paperclip version or commit** Base: `87dfba6818`. **Deployment mode** Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 01:57:31 +01:00 · 2026-11-05 03:13:44 -06:00
1 parent 833d068741
commit beb73381ca
@@ -1,10 +1,10 @@
 import { isPreDispatchReviewWait, neverStartedDispatchCancellationPredicate } from "./pre-dispatch-review-wait.js";
 import { canonicalExecutionUuid } from "./execution-identity.js";
 import { preDispatchExecutionWaitRecoveryActionPredicate } from "./pre-dispatch-execution-wait.js";
-import { and, desc, eq, gt, inArray, not, or, sql } from "drizzle-orm";
+import { and, desc, eq, gt, inArray, not, or, sql, type SQL } from "drizzle-orm";
 import { conversationRecoveryActionPredicate, getConversationOwnershipBlocker } from "./conversation-continuation.js";
 import { z } from "zod";
-import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, toolConnections, workspaceOperations, type Db } from "@paperclipai/db";
+import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, toolConnections, workspaceOperations, type Db } from "@paperclipai/db";
 import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation.js";
 import { canRetryStoppedRun } from "./cancelled-native-startup.js";
 import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
@@ -126,13 +126,58 @@ export function nativeExecutionRecoveryActionPredicate() {
      and ${heartbeatRuns.runtimeMode} = 'native')`;
 }

+// Runner startup lifecycle families: they open no turn. Every other event the
+// runner sourced counts as provider work, so a new or unlisted family keeps
+// the hold (fail closed). Server-side records without a source event (spans,
+// process identity, lifecycle) count only when they name turn or provider work.
+export const PROVIDER_STARTUP_EVENT_PREFIXES = ["harness.", "session.", "workspace.", "runtime.phase.",
+  "runtime.activity.", "provider.notice.", "auth.credentials.", "mcp_app.discovered", "mcp_app.initializing", "mcp_app.ready",
+  "mcp_app.resource.", "mcp_app.host_context.", "mcp_app.teardown", "mcp_app.failed"] as const;
+export const PROVIDER_STARTUP_EVENT_TYPES = ["run.attached", "run.detached"] as const;
+const SERVER_TURN_WORK_EVENT_PREFIXES = ["turn.", "item.", "tool.", "semantic_tool.", "run.result."] as const;
+const SERVER_TURN_WORK_EVENT_TYPES = ["provider.event", "provider.rpc_result"] as const;
+
+const anyPrefix = (eventType: SQL, prefixes: readonly string[]) =>
+  sql.join(prefixes.map(prefix => sql`starts_with(${eventType}, ${prefix})`), sql` or `);
+const anyType = (eventType: SQL, types: readonly string[]) =>
+  sql`${eventType} in (${sql.join(types.map(type => sql`${type}`), sql`, `)})`;
+
+/** SQL truth for an event row (alias columns) that may prove provider turn work. */
+export function providerTurnWorkEventCondition(eventType: SQL, sourceEventId: SQL) {
+  return sql`((${sourceEventId} is not null and not (${anyType(eventType, PROVIDER_STARTUP_EVENT_TYPES)}
+      or ${anyPrefix(eventType, PROVIDER_STARTUP_EVENT_PREFIXES)}))
+    or ${anyType(eventType, SERVER_TURN_WORK_EVENT_TYPES)} or ${anyPrefix(eventType, SERVER_TURN_WORK_EVENT_PREFIXES)})`;
+}
+
+/**
+ * A Stop that the native startup owner fenced before it published a provider
+ * handle proves that no provider turn ran. The run's own durable record must
+ * agree: no turn, tool or provider event and no native result. `stopped` is
+ * the SQL alias of the stopped run.
+ */
+export function stoppedBeforeProviderTurnCondition() {
+  return sql`(stopped.status = 'cancelled' and stopped.runtime_mode = 'native'
+    and stopped.result_json->'nativeCancellation'->>'schema' = 'paperclip.native-cancellation.v1'
+    and stopped.result_json->'nativeCancellation'->>'runId' = stopped.id::text
+    and stopped.result_json->'nativeCancellation'->>'scope' = 'run'
+    and stopped.result_json->'nativeCancellation'->>'dispatchState' = 'acknowledged'
+    and stopped.result_json->'nativeCancellation'->'beforeProviderTurn' = 'true'::jsonb
+    and not exists (select 1 from ${heartbeatRunEvents} work
+      where work.company_id = stopped.company_id and work.run_id = stopped.id
+        and ${providerTurnWorkEventCondition(sql`work.event_type`, sql`work.source_event_id`)})
+    and not exists (select 1 from ${nativeRunResults} result
+      where result.company_id = stopped.company_id and result.run_id = stopped.id))`;
+}
+
 /**
 * A file-preservation hold binds a newer task owner only until automatic
 * recovery saved the stopped run's files. That owner's turn is a fresh
 * assignment, not a replay: the stopped run's unknown outcomes stay with its
- * own agent, which keeps the hold. The proof is the durable completed-recovery
- * receipt bound to this action, so later reuse of the workspace by the new
- * owner (new epoch, writer or snapshot) cannot re-arm the hold.
+ * own agent, which keeps the hold. A Stop fenced before any provider turn has
+ * no unknown outcome, so its hold releases for the same agent too. The proof is
+ * the durable completed-recovery receipt bound to this action, so later reuse
+ * of the workspace by the current owner (new epoch, writer or snapshot) cannot
+ * re-arm the hold.
 */
 export function restoreHoldReleasedForCurrentOwnerPredicate() {
  const sourceRunId = sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')`;
@@ -150,11 +195,12 @@ export function restoreHoldReleasedForCurrentOwnerPredicate() {
        and stopped.status in ('failed', 'timed_out', 'interrupted', 'cancelled')
        and stopped.finished_at is not null
        and task.assignee_agent_id is not null
-        and task.assignee_agent_id <> stopped.agent_id
-        and not coalesce(task.status = 'in_review'
-          and task.execution_state->>'status' = 'pending'
-          and task.execution_state->'currentParticipant'->>'type' = 'agent'
-          and task.execution_state->'currentParticipant'->>'agentId' = stopped.agent_id::text, false))`,
+        and ((task.assignee_agent_id <> stopped.agent_id
+          and not coalesce(task.status = 'in_review'
+            and task.execution_state->>'status' = 'pending'
+            and task.execution_state->'currentParticipant'->>'type' = 'agent'
+            and task.execution_state->'currentParticipant'->>'agentId' = stopped.agent_id::text, false))
+          or ${stoppedBeforeProviderTurnCondition()}))`,
    completedWorkspaceRecoveryCondition(sourceRunId),
    // Copyback put the files back into the task workspace that the new owner
    // reuses. A snapshot-only save needs an explicit fresh-workspace request
@@ -8421,6 +8421,107 @@ describe("native session cancellation", () => {
    }
  });

+  it.each(["in-time", "late"] as const)("records that a Stop during startup (%s) preceded any provider turn", async timing => {
+    const root = await mkdtemp(join(tmpdir(), "native-startup-stop-proof-"));
+    const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
+    process.env.PAPERCLIP_RUNNER_STATE_DIR = root;
+    let open!: () => void, started!: () => void;
+    const opening = new Promise<void>(resolve => { open = resolve; });
+    const admitted = new Promise<void>(resolve => { started = resolve; });
+    const submitTurn = vi.fn();
+    state.cancel.mockClear();
+    state.execute.mockReset().mockImplementationOnce(async (options) => {
+      started(); await opening;
+      await options.onSession({ cancel: state.cancel });
+      submitTurn();
+      throw new Error("a stopped startup must not reach prompt submission");
+    });
+    const runResultJson: Record<string, unknown> = {};
+    const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
+    const outcome = executePaperclipNativeSession({ db: leaseDb(execution, {}, runResultJson, updates), execution, runnerInstanceId: "runner" })
+      .catch(error => error);
+    const persistence = cancellationDb();
+    try {
+      await admitted;
+      if (timing === "in-time") {
+        const stopping = cancelNativeSession(execution.binding.runId, "operator Stop", { db: persistence.db, scope: "run" });
+        await new Promise(resolve => setImmediate(resolve));
+        open();
+        expect(await stopping).toMatchObject({ dispatched: true });
+        await outcome;
+        expect(persistence.getResultJson().nativeCancellation).toMatchObject({ dispatchState: "acknowledged", beforeProviderTurn: true });
+      } else {
+        vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
+        const stopping = cancelNativeSession(execution.binding.runId, "operator Stop", { db: persistence.db, scope: "run" }).catch(error => error);
+        await new Promise(resolve => setImmediate(resolve));
+        await vi.advanceTimersByTimeAsync(30_000);
+        expect(await stopping).toMatchObject({ name: "NativeCancellationPendingRecoveryError" });
+        vi.useRealTimers();
+        Object.assign(runResultJson, persistence.getResultJson());
+        open(); await outcome;
+        const acknowledgement = updates.find(update => update.table === heartbeatRuns &&
+          (update.values.resultJson as Record<string, any> | undefined)?.nativeCancellation?.dispatchState === "acknowledged");
+        expect(acknowledgement?.values.resultJson).toMatchObject({ nativeCancellation: { beforeProviderTurn: true } });
+      }
+      expect(state.cancel).toHaveBeenCalledOnce();
+      expect(submitTurn).not.toHaveBeenCalled();
+    } finally {
+      vi.useRealTimers(); open(); await outcome;
+      if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
+      else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
+      await rm(root, { recursive: true, force: true });
+    }
+  });
+
+  it("records no pre-turn proof for a Stop after the provider handle was published", async () => {
+    const running = executePaperclipNativeSession({ db: leaseDb(), execution, runnerInstanceId: "runner" });
+    await vi.waitFor(() => expect(state.release).toBeTypeOf("function"));
+    const persistence = cancellationDb();
+    try {
+      expect(await cancelNativeSession(execution.binding.runId, "operator Stop", { db: persistence.db, scope: "run" }))
+        .toMatchObject({ dispatched: true });
+      expect(persistence.getResultJson().nativeCancellation).toMatchObject({ dispatchState: "acknowledged" });
+      expect(persistence.getResultJson().nativeCancellation).not.toHaveProperty("beforeProviderTurn");
+    } finally { state.release?.(); await running; }
+  });
+
+  it("records no pre-turn proof for a startup Stop under restart recovery", async () => {
+    // A recovered controller may reattach a provider that already worked.
+    const current = structuredClone(execution);
+    current.binding.runId = "restart-recovery-startup-stop";
+    const { currentNativeControllerIdentity } = await import("./native-restart-recovery.js");
+    const controller = await currentNativeControllerIdentity();
+    const db = leaseDb(current, { leaseOwner: "recovered-controller", controllerBootId: controller.bootId,
+      controllerPid: controller.pid, controllerGeneration: 2, attempt: 1 });
+    let open!: () => void, started!: () => void;
+    const opening = new Promise<void>(resolve => { open = resolve; });
+    const admitted = new Promise<void>(resolve => { started = resolve; });
+    const submitTurn = vi.fn();
+    state.cancel.mockClear();
+    state.execute.mockReset().mockImplementationOnce(async (options) => {
+      started(); await opening;
+      await options.onSession({ cancel: state.cancel });
+      submitTurn();
+      throw new Error("a stopped startup must not reach prompt submission");
+    });
+    const outcome = executePaperclipNativeSession({ db, execution: current, runnerInstanceId: "runner",
+      leaseOwner: "recovered-controller", restartRecovery: { kind: "bootstrap_incomplete" as const, runId: current.binding.runId,
+        leaseOwner: "recovered-controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "hard" as const,
+        recoveryRequestId: null } }).catch(error => error);
+    const persistence = cancellationDb({ runId: current.binding.runId });
+    try {
+      await admitted;
+      const stopping = cancelNativeSession(current.binding.runId, "operator Stop", { db: persistence.db, scope: "run" });
+      await new Promise(resolve => setImmediate(resolve));
+      open();
+      expect(await stopping).toMatchObject({ dispatched: true });
+      await outcome;
+      expect(persistence.getResultJson().nativeCancellation).toMatchObject({ dispatchState: "acknowledged" });
+      expect(persistence.getResultJson().nativeCancellation).not.toHaveProperty("beforeProviderTurn");
+      expect(submitTurn).not.toHaveBeenCalled();
+    } finally { open(); await outcome; }
+  });
+
  it("routes control-plane cancellation to the active normalized session and removes the handle", async () => {
    const running = executePaperclipNativeSession({
      db: leaseDb(),
@@ -353,6 +353,10 @@ type NativeSessionStartup = {
  executionLeaseRenewal?: NativeExecutionLeaseRenewal;
  /** A fresh claim whose preparation failed before any provider turn. */
  unstartedClaim?: { leaseOwner: string; attempt: number };
+  /** A provider session handle was published; a turn may have followed it. */
+  sessionPublished?: boolean;
+  /** Restart recovery reattaches a provider that may already have worked. */
+  providerWorkMayPrecede?: boolean;
  executionContext?: NativeExecutionContext;
  settled: Promise<void>;
  settle: () => void;
@@ -7739,6 +7743,8 @@ export function cancelNativeSession(
    scope?: "turn" | "run" | "issue";
    replacementAccepted?: boolean;
    cancellationRequestId?: string;
+    /** Internal: the startup owner fenced this Stop before any provider turn. */
+    beforeProviderTurn?: boolean;
  },
 ): Promise<{
  dispatched: boolean;
@@ -7754,6 +7760,8 @@ export async function cancelNativeSession(
    scope?: "turn" | "run" | "issue";
    replacementAccepted?: boolean;
    cancellationRequestId?: string;
+    /** Internal: the startup owner fenced this Stop before any provider turn. */
+    beforeProviderTurn?: boolean;
  },
 ): Promise<
  | boolean
@@ -7779,7 +7787,8 @@ export async function cancelNativeSession(
 async function performNativeCancellation(
  runId: string,
  reason: string,
-  options?: { db: Db; scope?: "turn" | "run" | "issue"; replacementAccepted?: boolean; cancellationRequestId?: string },
+  options?: { db: Db; scope?: "turn" | "run" | "issue"; replacementAccepted?: boolean; cancellationRequestId?: string;
+    beforeProviderTurn?: boolean },
 ) {
  const runStop = (options?.scope ?? "run") === "run";
  let decision: NativeStatusDecision | null = null;
@@ -8098,6 +8107,11 @@ async function performNativeCancellation(
  const startup = runStop && !activeNativeSessions.has(runId) ? nativeSessionStartups.get(runId) : undefined;
  let settleStartupCancellation: (() => void) | undefined;
  let settleActiveCancellation: (() => void) | undefined;
+  // Proof for the acknowledgement: this Stop reached startup before it
+  // published a provider handle. Publication then fences that handle, so no
+  // provider turn can follow. A reattached provider may already have worked.
+  const beforeProviderTurn = runStop && (options?.beforeProviderTurn === true || Boolean(startup
+    && startup.sessionPublished !== true && startup.providerWorkMayPrecede !== true));
  if (startup) {
    startup.stopRequested = true;
    startup.executionContext?.abort(new Error("native_session_cancelled"));
@@ -8298,6 +8312,7 @@ async function performNativeCancellation(
                decisionId,
                acknowledgementAuditId,
                acknowledgedAt: new Date().toISOString(),
+                ...(beforeProviderTurn ? { beforeProviderTurn: true } : {}),
              },
            },
            updatedAt: new Date(),
@@ -8479,6 +8494,7 @@ export async function executePaperclipNativeSession(input: {
  let settle!: () => void;
  const startup: NativeSessionStartup = {
    priorRestartDetach: nativeRunsDetachingForRestart.get(runId),
+    providerWorkMayPrecede: Boolean(input.restartRecovery),
    settled: new Promise<void>(resolve => { settle = resolve; }),
    settle: () => settle(),
    promise: new Promise<ActiveNativeSession | null>(resolve => { resolveStartup = resolve; }),
@@ -10340,6 +10356,9 @@ async function executePaperclipNativeSessionWithinScope(
                const active: ActiveNativeSession = { session, cancelRequested: false };
                activeNativeSessions.set(input.execution.binding.runId, active);
                const startup = nativeSessionStartups.get(input.execution.binding.runId);
+                const stopBeforeProviderTurn = startup?.stopRequested === true && startup.sessionPublished !== true
+                  && startup.providerWorkMayPrecede !== true;
+                if (startup) startup.sessionPublished = true;
                startup?.resolve(active);
                if (startup?.stopRequested) {
                  // Publication wakes the Stop caller; wait for its durable ACK
@@ -10358,7 +10377,8 @@ async function executePaperclipNativeSessionWithinScope(
                    )).limit(1);
                    const pendingStop = record(record(stopRun?.resultJson).nativeCancellation);
                    if (pendingStop.scope === "run" && pendingStop.dispatchState === "pending") {
-                      await cancelNativeSession(input.execution.binding.runId, "Stop requested during native startup", { db: input.db, scope: "run" });
+                      await cancelNativeSession(input.execution.binding.runId, "Stop requested during native startup",
+                        { db: input.db, scope: "run", beforeProviderTurn: stopBeforeProviderTurn });
                    }
                  } catch (error) {
                    // Best effort: the in-memory Stop below still fences this
@@ -7,7 +7,8 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp
 import { SystemRecoveryFailure } from "./system-recovery-state.js";
 import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
 import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
-import { ENVIRONMENT_LEASE_POLICIES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLeasePolicy,
+import { LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
+import { ENVIRONMENT_LEASE_POLICIES, EXECUTION_RECONCILIATION_CAUSES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLeasePolicy,
  type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";

 export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
@@ -81,6 +82,14 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
  if (!run) throw invalid("source_run_missing");
  if (!action) throw invalid("recovery_action_missing");
  if (action.evidence.runId !== run.id) throw invalid("recovery_action_run_mismatch");
+  // This file-preservation hold blocks every dispatch for the task until its
+  // recovery receipt completes (see getExecutionBlocker).
+  const restoreRecovery = action.evidence.workspaceRestoreRecovery as { schema?: unknown } | undefined;
+  const automaticRecovery = action.evidence.automaticRecovery as { replay?: unknown } | undefined;
+  const holdGatesTaskDispatch = restoreRecovery?.schema === LEGACY_WORKSPACE_RECOVERY_SCHEMA
+    && action.evidence.workspaceRestoreFailure !== "restore_unsafe_archive"
+    && (EXECUTION_RECONCILIATION_CAUSES as readonly string[]).includes(action.cause)
+    && (["active", "escalated"].includes(action.status) || automaticRecovery?.replay === "blocked");
  if (!terminalWorkspaceRunStates.includes(run.status)) throw invalid("source_run_not_terminal");
  if ((run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== input.issueId) throw invalid("source_issue_mismatch");
  const rawReference = run.runnerProfileJson?.nativeWorkspaceSync;
@@ -208,6 +217,12 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
          and a.evidence->>'runId' = ${run.id}
          and a.evidence#>>'{explicitUserContinuation,runId}' = ${heartbeatRuns.id}::text
          and a.evidence#>>'{explicitUserContinuation,readiness,mode}' = 'fresh_isolated_workspace')`,
+        // A never-started queued turn for this task cannot write: this hold
+        // gates its dispatch until recovery completes. Counting it as a writer
+        // would leave both waiting for each other.
+        holdGatesTaskDispatch ? sql`not coalesce(${heartbeatRuns.status} = 'queued' and ${heartbeatRuns.startedAt} is null
+          and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null
+          and ${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}, false)` : undefined,
        or(eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.issueId, input.issueId),
          sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`,
          sql`${heartbeatRuns.contextSnapshot}->>'executionWorkspaceId' = ${descriptor.workspaceId}`))).limit(1);
@@ -3619,6 +3619,116 @@ describe("automatic retained workspace recovery", () => {
    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: false });
  }, 30_000);

+  // Shape of a Stop or reassignment that reached a remote native run while its
+  // provider session was still opening: the startup owner fenced the handle, the
+  // run settled cancelled with a file-preservation hold, and the follow-up turn
+  // (the redirect comment, or the new owner's wake) was already queued.
+  async function stopBeforeSessionOpen(f: Awaited<ReturnType<typeof seed>>, options: { errorCode: string; proof?: boolean }) {
+    const nativeCancellation = { schema: "paperclip.native-cancellation.v1", runId: f.runId, companyId: f.companyId,
+      issueId: f.issueId, scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true,
+      intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(), acknowledgedAt: new Date().toISOString(),
+      ...(options.proof === false ? {} : { beforeProviderTurn: true }) };
+    const stopped = await run(f.runId);
+    await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: options.errorCode, finishedAt: new Date(),
+      resultJson: { ...stopped.resultJson, nativeCancellation, startupCancellation: { errorCode: options.errorCode, beforeNativeSelection: false } },
+    }).where(eq(heartbeatRuns.id, f.runId));
+    const held = await action(f.actionId);
+    await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked", resolvedAt: new Date(),
+      fingerprint: `native-workspace:${f.runId}`, evidence: { ...held.evidence, continuationDelivery: "invalidated",
+        automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked", actionOutcome: "unknown" } },
+    }).where(eq(issueRecoveryActions.id, f.actionId));
+    await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issueId));
+  }
+  const queueFollowUp = async (f: Awaited<ReturnType<typeof seed>>, agentId: string) =>
+    (await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, runtimeMode: "legacy", status: "queued",
+      invocationSource: "on_demand", contextSnapshot: { issueId: f.issueId, wakeReason: "issue_commented" } }).returning())[0]!;
+  const pendingTurns = (f: Awaited<ReturnType<typeof seed>>) => db.select().from(heartbeatRuns).where(and(
+    eq(heartbeatRuns.companyId, f.companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry", "running"])));
+
+  it("continues the same agent's redirect once after a Stop that landed before the provider session opened", async () => {
+    const f = await seed();
+    const agentId = (await run(f.runId)).agentId;
+    await stopBeforeSessionOpen(f, { errorCode: "cancelled" });
+    // Startup lifecycle from the runner opens no turn and is not provider work.
+    for (const [seq, eventType] of ["harness.starting", "harness.ready", "workspace.ready", "runtime.phase.changed",
+      "session.starting", "session.started", "session.updated", "auth.credentials.updated"].entries()) await db.insert(heartbeatRunEvents).values({
+      companyId: f.companyId, runId: f.runId, agentId, seq: 9_000 + seq, eventType, stream: "system", level: "info",
+      message: eventType, sourceEventId: randomUUID() });
+    const followUp = await queueFollowUp(f, agentId);
+    // No executor may touch the workspace while its files are unrecovered.
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true });
+    // The queued follow-up never started and is gated by this hold, so it is not
+    // a workspace writer. Recovery must not wait for it.
+    await recoverRetainedWorkspace(f.input);
+    expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "complete" });
+    expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
+    // No provider turn ran, so no outcome is unknown: the same agent continues.
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
+    // Exactly once: the queued follow-up is the only turn, and a sweep adds none.
+    await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: f.issueId });
+    expect((await pendingTurns(f)).map(item => item.id)).toEqual([followUp.id]);
+    // The audit row keeps its no-replay evidence.
+    expect(await action(f.actionId)).toMatchObject({ status: "resolved", outcome: "blocked",
+      evidence: { automaticRecovery: { replay: "blocked" } } });
+  }, 30_000);
+
+  it("hands a task reassigned before the provider session opened to its new owner once", async () => {
+    const f = await seed();
+    await stopBeforeSessionOpen(f, { errorCode: "issue_reassigned" });
+    const newOwnerId = randomUUID();
+    await db.insert(agents).values({ id: newOwnerId, companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner",
+      status: "idle", runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
+    await db.update(issues).set({ assigneeAgentId: newOwnerId }).where(eq(issues.id, f.issueId));
+    const handoff = await queueFollowUp(f, newOwnerId);
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true });
+    await recoverRetainedWorkspace(f.input);
+    expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "complete" });
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
+    await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: f.issueId });
+    expect((await pendingTurns(f)).map(item => item.id)).toEqual([handoff.id]);
+  }, 30_000);
+
+  it.each(["no-proof", "turn-event", "provider-event", "unlisted-provider-event", "running-writer", "other-task-writer",
+    "unscoped-queued-writer", "non-gating-hold"] as const)
+  ("keeps the stopped agent held when provider work cannot be excluded (%s)", async kind => {
+    const f = await seed();
+    const agentId = (await run(f.runId)).agentId;
+    await stopBeforeSessionOpen(f, { errorCode: "cancelled", proof: kind !== "no-proof" });
+    if (kind === "turn-event") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId,
+      eventType: "turn.started", stream: "system", level: "info", message: "turn started" });
+    if (kind === "provider-event") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.runId, agentId,
+      seq: 9_999, eventType: "item.completed", stream: "system", level: "info", message: "provider item", sourceEventId: randomUUID() });
+    // A runner family outside the startup lifecycle counts as work even when unlisted.
+    if (kind === "unlisted-provider-event") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.runId,
+      agentId, seq: 9_999, eventType: "delegation.completed", stream: "system", level: "info", message: "delegated",
+      sourceEventId: randomUUID() });
+    if (kind === "running-writer" || kind === "other-task-writer" || kind === "unscoped-queued-writer") {
+      // A started turn, or a queued one that this task's dispatch gate does not cover, is a real writer.
+      await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, runtimeMode: "legacy",
+        status: kind === "running-writer" ? "running" : "queued",
+        ...(kind === "unscoped-queued-writer" ? { issueId: f.issueId } : {}),
+        contextSnapshot: kind === "running-writer" ? { issueId: f.issueId }
+          : kind === "other-task-writer" ? { issueId: randomUUID(), executionWorkspaceId: f.issueId } : {} });
+      await expect(recoverySources.inspectWorkspaceRecoverySource(db, f)).rejects.toMatchObject({ code: "workspace_writer_active" });
+      return;
+    }
+    if (kind === "non-gating-hold") {
+      // A resolved action without a no-replay hold does not gate dispatch, so a
+      // queued turn for the task can start and must still count as a writer.
+      const held = await action(f.actionId);
+      await db.update(issueRecoveryActions).set({ evidence: { ...held.evidence,
+        automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "allowed" } } })
+        .where(eq(issueRecoveryActions.id, f.actionId));
+      await queueFollowUp(f, agentId);
+      await expect(recoverySources.inspectWorkspaceRecoverySource(db, f)).rejects.toMatchObject({ code: "workspace_writer_active" });
+      return;
+    }
+    await queueFollowUp(f, agentId);
+    await recoverRetainedWorkspace(f.input);
+    expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId, runId: f.runId });
+  }, 30_000);
+
  it("keeps a snapshot-only recovery held for a new assignee because the task workspace lacks the saved files", async () => {
    const f = await seed("native", false);
    await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned" }).where(eq(heartbeatRuns.id, f.runId));