fix(tasks): hold a Retry while the task's own native session close is retained (#19609)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native provider session close fails, the runner keeps a
cleanup owner or quarantine in process memory. Its admission gate
refuses new sessions for that task (scoped to the task since #19532)
> - The server could not see that state. A task's Retry was accepted
(202), its run was dispatched, and the runner refused it at once with
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`
> - The user saw a Retry that failed instantly, and nothing said what to
do
> - This pull request lets the server read the runner's per-task cleanup
gate and hold the Retry with a clear reason. The Retry starts
automatically when the close is released
> - The benefit is that a Retry is never dispatched only to fail on a
block that is already known

## Linked Issues or Issue Description

**What happened?**
After two hard restarts, a Codex run's close stayed incomplete. The
board Retry for that task was accepted, and its run failed in about 1 s
with `native_session_cleanup_quarantined: prior session cleanup remains
incomplete`. The next Retry did the same.

**Steps to reproduce**
1. Let a native session close fail or stay pending for a task.
2. Press Retry on that task.

**Expected behavior**
The Retry waits with a clear reason, and starts when the cleanup
finishes. If the cleanup needs an operator, Retry is not offered and the
one working action is shown.

**Actual behavior**
The Retry is accepted and its run fails at once.

## What Changed

- `native-session-runtime.ts`:
- New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a
read-only view of the in-process admission gate for one task: `pending`
(a retained close or a bounded automatic close recovery still owns the
task's provider resources), `operator_required`, or `null`. It grants no
authority.
- New `onNativeSessionCleanupSettled(listener)`. It is called whenever a
cleanup owner or quarantine entry is released.
- `explicit-native-continuation.ts` (the continuation gate for a Retry
or a new message):
- `pending`: the input is held with `local_cleanup` ("Retry is waiting
for the previous execution to finish cleanup. Your message is saved.").
It starts automatically when the close is released.
- `operator_required`: handled exactly like a recorded
`native_session_cleanup_quarantined` run. Retry is not offered. A new
message runs the existing stopped-process proof
(`verifyStoppedNativeSessionForContinuation`), which retires the runner
quarantine when the old processes are proven stopped.
- `native-cleanup-release-wake.ts` (registered in
`server/src/index.ts`): each cleanup release wakes deferred execution
for the company in a transaction, so a held Retry re-evaluates.
- `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime
functions.

## Verification

- Red before the gate (server change stashed): `pnpm --filter
@paperclipai/server exec vitest run
src/services/quarantined-task-retry.test.ts -t "holds a Retry"`
dispatched a successor run while the task's close was retained
(`expected [ { … } ] to have a length of +0 but got 1`).
- Green after the gate:
- The Retry is held (`deferred_issue_execution`, `executionWait.reason =
local_cleanup`), and no successor is dispatched.
  - A different task is not blocked.
- When the close settles, the release is announced, and the next pass
dispatches exactly one successor.
- The whole `quarantined-task-retry.test.ts` file passes (28 tests).
`src/native-session-runtime.test.ts` passes. Server `tsc --noEmit`
passes after the runner package build.
- Live chaos replays: results are added in a comment.

## Risks

- The gate reads process memory, so it applies only where the server and
the runner session share a process (local runner). Other deployments
keep today's behavior.
- A held Retry re-evaluates on the release wake (sent in its own
transaction) or on the existing 30 s deferred-execution pass. When the
bounded automatic close budget is spent with nothing else scheduled, the
view reports `operator_required`. Retry is then no longer offered, and a
new message reaches the stopped-process recovery instead of waiting
forever.
- No schema or API shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #19532.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/explicit-native-continuation.ts

@@ -17,7 +17,7 @@ import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation.
 import { readQueuedInteractionResponse } from "./queued-interaction-response.js";
 import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js";
 import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "./native-local-process-stop.js";
-import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/paperclip-runner/index.js";
+import { completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
 import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
 import { z } from "zod";
 import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm";
@@ -490,7 +490,17 @@ export async function admitExplicitNativeContinuation(input: {
         if (run.processGroupId && !processStopped(-run.processGroupId)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
       }
     }
-    if (!remote && run.runtimeMode === "native" && run.errorCode === "native_session_cleanup_quarantined") {
+    // The runner keeps a closing or quarantined provider session in process
+    // memory, keyed by this task. Its admission gate would refuse a successor
+    // at once, so hold the input here with the matching reason instead of
+    // dispatching a run that can only fail. A pending close settles by itself
+    // and its release wakes deferred execution; the held input then starts.
+    const retainedCleanup = !remote && run.runtimeMode === "native"
+      ? nativeSessionCleanupBlock({ companyId, issueId }) : null;
+    if (retainedCleanup === "pending" && run.errorCode !== "native_session_cleanup_quarantined")
+      return blocked("local_cleanup", "Waiting for the previous provider session to finish cleanup. Your message will start automatically.");
+    if (!remote && run.runtimeMode === "native" &&
+        (run.errorCode === "native_session_cleanup_quarantined" || retainedCleanup === "operator_required")) {
       // This is new user input, never permission to retry the interrupted turn.
       if (retry) return blocked("cleanup_quarantined", "Send a new message after the previous provider has stopped.");
       const { verifyStoppedNativeSessionForContinuation } = await import("./native-runtime/native-session-executor.js");

server/src/services/native-runtime/native-cleanup-release-wake.ts

@@ -0,0 +1,17 @@
+import type { Db } from "@paperclipai/db";
+import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
+import { onNativeSessionCleanupSettled } from "../../vendor/paperclip-runner/index.js";
+
+/**
+ * A retained native session close holds task input in the runner's process
+ * memory. When the runner releases it, wake deferred execution for that company
+ * so a held Retry or message re-evaluates its gate. The wake is a hint only:
+ * dispatch repeats every admission check under its own locks.
+ */
+export function registerNativeCleanupReleaseWake(db: Db, onError: (error: unknown, companyId: string) => void): () => void {
+  return onNativeSessionCleanupSettled(({ companyId }) => {
+    if (!companyId) return;
+    void db.transaction(async tx => notifyDeferredExecutionWork(tx, { companyId }))
+      .catch((error: unknown) => onError(error, companyId));
+  });
+}

server/src/services/quarantined-task-retry.test.ts

@@ -3,13 +3,14 @@ import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
 import { hostname, tmpdir } from "node:os";
 import { join } from "node:path";
 import { and, eq } from "drizzle-orm";
-import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
 import {
   agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces, heartbeatRunEvents,
   heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations,
 } from "@paperclipai/db";
 import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
-import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
+import { executeNativeSession, nativeSessionCleanupBlock, onNativeSessionCleanupSettled, parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
+import type { NativeSession } from "../vendor/paperclip-runner/index.js";
 import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
 import { heartbeatService } from "./heartbeat.js";
 import { getExecutionBlocker } from "./execution-blocker.js";
@@ -19,6 +20,8 @@ import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-loc
 import { nativeSha256 } from "./native-runtime/canonical.js";
 import { verifyStoppedNativeSessionForContinuation } from "./native-runtime/native-session-executor.js";
 import { requestTaskRetry } from "./task-retry.js";
+import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
+import { registerNativeCleanupReleaseWake } from "./native-runtime/native-cleanup-release-wake.js";
 
 describe("task Retry after inherited native cleanup quarantine", () => {
   let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
@@ -240,4 +243,83 @@ describe("task Retry after inherited native cleanup quarantine", () => {
       await rm(base, { recursive: true, force: true });
     }
   });
+
+  it("holds a Retry while the task's own provider session close is still retained, then starts it", async () => {
+    const base = await mkdtemp(join(tmpdir(), "retained-close-retry-"));
+    const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
+    process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
+    let releaseClose = () => {};
+    try {
+      const f = await seed(base);
+      // The source failed for another reason; only the runner's in-process
+      // admission gate still holds this task's provider session close.
+      await db.update(heartbeatRuns).set({ errorCode: "provider_transport_failed" }).where(eq(heartbeatRuns.id, f.sourceId));
+      const settled = vi.fn();
+      const unsubscribe = onNativeSessionCleanupSettled(settled);
+      // The production release wake, observed on the delivery queue it signals.
+      const wakeFailure = vi.fn();
+      const unregisterWake = registerNativeCleanupReleaseWake(db, wakeFailure);
+      const deferredWake = vi.fn();
+      const unsubscribeDeferred = subscribeDeliveryWork(db, DELIVERY_QUEUES.deferredExecutionReceipts, deferredWake);
+      const pendingClose = new Promise<void>(resolve => { releaseClose = resolve; });
+      const identity = { runId: randomUUID(), sessionId: "retained-close-session", companyId: f.companyId, issueId: f.issueId, agentId: f.agentId };
+      const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
+      const session: NativeSession = {
+        identity: () => identity, capabilities: async () => capabilities,
+        async *events() { throw new Error("retained close transport stopped"); },
+        startTurn: async () => ({ turnId: "retained-close-turn" }), result: async () => null,
+        async snapshot() { return { backendKind: "mock", sessionId: "retained-close-driver", identity,
+          providerSessionId: "retained-close-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },
+        close: () => pendingClose,
+      };
+      await expect(executeNativeSession({
+        input: parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
+          binding: { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runId: identity.runId, executionWorkspaceId: f.workspaceId },
+          task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" },
+          workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null },
+          session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
+            lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
+          completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
+            contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } },
+          interactionResponses: [], credentialBindings: [] }),
+        backend: { descriptor: async () => ({ kind: "runner", name: "retained-close-backend", version: "1", capabilities }),
+          openSession: async () => session },
+        controlPlane: { openRun: async () => {}, checkpointSession: async () => {},
+          appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
+          replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
+        runnerInstanceId: f.runnerInstanceId, controlPlaneInstanceId: "control",
+      })).rejects.toThrow("retained close transport stopped");
+      expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("pending");
+      expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: randomUUID() })).toBeNull();
+
+      await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
+      const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
+        actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
+      const heartbeat = heartbeatService(db);
+      await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
+      const queued = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
+      // No successor is dispatched that the runner would refuse at once.
+      expect(await queued()).toHaveLength(0);
+      const [held] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
+      expect(held).toMatchObject({ status: "deferred_issue_execution", runId: null });
+      expect(held.payload).toMatchObject({ executionWait: { reason: "local_cleanup" } });
+
+      // The close settles by itself; its release wakes the held Retry, which then starts.
+      releaseClose();
+      await vi.waitFor(() => expect(settled).toHaveBeenCalledWith({ companyId: f.companyId, issueIds: [f.issueId] }));
+      await vi.waitFor(() => expect(deferredWake).toHaveBeenCalled());
+      expect(wakeFailure).not.toHaveBeenCalled();
+      expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
+      unsubscribe();
+      unregisterWake();
+      unsubscribeDeferred();
+      // The deferred-execution worker runs this pass on that wake.
+      await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
+      expect(await queued()).toHaveLength(1);
+    } finally {
+      releaseClose();
+      if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
+      await rm(base, { recursive: true, force: true });
+    }
+  });
 });