fix(tasks): hold a Retry while the task's own native session close is retained (#19609)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native provider session close fails, the runner keeps a
cleanup owner or quarantine in process memory. Its admission gate
refuses new sessions for that task (scoped to the task since #19532)
> - The server could not see that state. A task's Retry was accepted
(202), its run was dispatched, and the runner refused it at once with
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`
> - The user saw a Retry that failed instantly, and nothing said what to
do
> - This pull request lets the server read the runner's per-task cleanup
gate and hold the Retry with a clear reason. The Retry starts
automatically when the close is released
> - The benefit is that a Retry is never dispatched only to fail on a
block that is already known

## Linked Issues or Issue Description

**What happened?**
After two hard restarts, a Codex run's close stayed incomplete. The
board Retry for that task was accepted, and its run failed in about 1 s
with `native_session_cleanup_quarantined: prior session cleanup remains
incomplete`. The next Retry did the same.

**Steps to reproduce**
1. Let a native session close fail or stay pending for a task.
2. Press Retry on that task.

**Expected behavior**
The Retry waits with a clear reason, and starts when the cleanup
finishes. If the cleanup needs an operator, Retry is not offered and the
one working action is shown.

**Actual behavior**
The Retry is accepted and its run fails at once.

## What Changed

- `native-session-runtime.ts`:
- New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a
read-only view of the in-process admission gate for one task: `pending`
(a retained close or a bounded automatic close recovery still owns the
task's provider resources), `operator_required`, or `null`. It grants no
authority.
- New `onNativeSessionCleanupSettled(listener)`. It is called whenever a
cleanup owner or quarantine entry is released.
- `explicit-native-continuation.ts` (the continuation gate for a Retry
or a new message):
- `pending`: the input is held with `local_cleanup` ("Retry is waiting
for the previous execution to finish cleanup. Your message is saved.").
It starts automatically when the close is released.
- `operator_required`: handled exactly like a recorded
`native_session_cleanup_quarantined` run. Retry is not offered. A new
message runs the existing stopped-process proof
(`verifyStoppedNativeSessionForContinuation`), which retires the runner
quarantine when the old processes are proven stopped.
- `native-cleanup-release-wake.ts` (registered in
`server/src/index.ts`): each cleanup release wakes deferred execution
for the company in a transaction, so a held Retry re-evaluates.
- `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime
functions.

## Verification

- Red before the gate (server change stashed): `pnpm --filter
@paperclipai/server exec vitest run
src/services/quarantined-task-retry.test.ts -t "holds a Retry"`
dispatched a successor run while the task's close was retained
(`expected [ { … } ] to have a length of +0 but got 1`).
- Green after the gate:
- The Retry is held (`deferred_issue_execution`, `executionWait.reason =
local_cleanup`), and no successor is dispatched.
  - A different task is not blocked.
- When the close settles, the release is announced, and the next pass
dispatches exactly one successor.
- The whole `quarantined-task-retry.test.ts` file passes (28 tests).
`src/native-session-runtime.test.ts` passes. Server `tsc --noEmit`
passes after the runner package build.
- Live chaos replays: results are added in a comment.

## Risks

- The gate reads process memory, so it applies only where the server and
the runner session share a process (local runner). Other deployments
keep today's behavior.
- A held Retry re-evaluates on the release wake (sent in its own
transaction) or on the existing 30 s deferred-execution pass. When the
bounded automatic close budget is spent with nothing else scheduled, the
view reports `operator_required`. Retry is then no longer offered, and a
new message reaches the stopped-process recovery instead of waiting
forever.
- No schema or API shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #19532.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-06 18:52:46 -06:00
1 parent 29e05f7e06
commit 785ead5c7a
7 files changed
+213 -6

No files matched your search

@@ -41,6 +41,7 @@ import {
executeNativeSession,
completeTerminatedRemoteNativeSessionCleanup,
completeTerminatedLocalNativeSessionCleanup,
nativeSessionCleanupBlock,
type ExecuteNativeSessionOptions,
} from "./native-session-runtime.js";
@@ -2525,11 +2526,17 @@ describe("executeNativeSession recovery", () => {
expect(admissionAttempt).toBe(3);
expect(scheduledAttempt).toBe(0);
const block = { companyId: lifetimeIdentity.companyId, issueId: input.binding.issueId };
// Automatic recovery is still scheduled: the server may hold input.
expect(nativeSessionCleanupBlock(block)).toBe("pending");
await vi.advanceTimersByTimeAsync(180_000);
expect(scheduledAttempt).toBe(3);
await vi.advanceTimersByTimeAsync(300_000);
expect(scheduledAttempt).toBe(3);
expect(openSession).toHaveBeenCalledOnce();
// The automatic budget is spent and nothing else is scheduled: waiting
// would never end, so the task needs stopped-process recovery.
expect(nativeSessionCleanupBlock(block)).toBe("operator_required");
} finally {
vi.useRealTimers();
}
@@ -85,6 +85,78 @@ const failedSessionCleanupOwners = new Map<
Promise<void>,
NativeSessionCleanupDomain
>();
type NativeSessionCleanupSettledListener = (scope: {
companyId: string;
issueIds: string[];
}) => void;
const cleanupSettledListeners = new Set<NativeSessionCleanupSettledListener>();
function cleanupDomainScope(domain: NativeSessionCleanupDomain): {
companyId: string;
issueIds: string[];
} {
const [companyId] = JSON.parse(domain.key) as string[];
const issueIds = domain.resources.flatMap((resource) => {
const [kind, id] = JSON.parse(resource) as string[];
return kind === "issue" && typeof id === "string" ? [id] : [];
});
return { companyId: companyId ?? "", issueIds };
}
/** A retained cleanup owner or quarantine entry was released. Observers can
* re-evaluate work that waited on it; this grants no execution authority. */
function announceCleanupSettled(domain: NativeSessionCleanupDomain): void {
const scope = cleanupDomainScope(domain);
for (const listener of cleanupSettledListeners) {
try {
listener(structuredClone(scope));
} catch {
// Observers cannot change cleanup ownership.
}
}
}
/** Subscribe to cleanup releases. Returns the unsubscribe function. */
export function onNativeSessionCleanupSettled(
listener: NativeSessionCleanupSettledListener,
): () => void {
cleanupSettledListeners.add(listener);
return () => cleanupSettledListeners.delete(listener);
}
/**
* Read-only view of the in-process admission gate for one task. "pending":
* a retained close or bounded automatic close recovery still owns the task's
* provider resources, and admission would be refused now. "operator_required":
* the close cannot settle automatically. Never grants or retires authority.
*/
export function nativeSessionCleanupBlock(input: {
companyId: string;
issueId: string;
}): "operator_required" | "pending" | null {
const issueResource = JSON.stringify(["issue", input.issueId]);
const matches = (domain: NativeSessionCleanupDomain) =>
domain.resources.includes(issueResource) &&
cleanupDomainScope(domain).companyId === input.companyId;
let block: "operator_required" | "pending" | null = null;
for (const entry of quarantinedSessionCleanups) {
if (!matches(entry.domain)) continue;
// An entry whose bounded automatic close budget is spent has no further
// scheduled attempt. Waiting would never end, so it needs the same
// stopped-process recovery as an unrecoverable close.
const automaticRecoveryExhausted =
!entry.attempt && !entry.recovery && !entry.timer &&
entry.automaticAttempts >= MAX_QUARANTINED_SESSION_AUTOMATIC_CLOSE_ATTEMPTS;
if (entry.operatorRecoveryRequired || automaticRecoveryExhausted)
return "operator_required";
block = "pending";
}
for (const domain of failedSessionCleanupOwners.values()) {
if (matches(domain)) block = "pending";
}
return block;
}
const FAILED_SESSION_CLOSE_RETRY_MS = 1_000;
const MAX_FAILED_SESSION_CLOSE_RETRIES = 3;
const MAX_QUARANTINED_SESSION_CLOSE_RETRIES = 3;
@@ -154,7 +226,10 @@ export function completeRetainedNativeSessionCleanup(
) {
throw new NativeSessionCleanupQuarantinedError();
}
for (const entry of matches) quarantinedSessionCleanups.delete(entry);
for (const entry of matches) {
quarantinedSessionCleanups.delete(entry);
announceCleanupSettled(entry.domain);
}
return matches.length;
}
@@ -182,6 +257,7 @@ export function completeTerminatedRemoteNativeSessionCleanup(binding: {
if (entry.timer) clearTimeout(entry.timer);
entry.timer = null;
quarantinedSessionCleanups.delete(entry);
announceCleanupSettled(entry.domain);
}
return true;
}
@@ -205,6 +281,7 @@ export function completeTerminatedLocalNativeSessionCleanup(binding: {
for (const entry of matches) {
if (entry.timer) clearTimeout(entry.timer);
quarantinedSessionCleanups.delete(entry);
announceCleanupSettled(entry.domain);
}
return true;
}
@@ -679,7 +756,10 @@ function retainFailedSessionCleanupOwner(
): void {
failedSessionCleanupOwners.set(cleanup, cleanupDomain);
void cleanup
.finally(() => failedSessionCleanupOwners.delete(cleanup))
.finally(() => {
failedSessionCleanupOwners.delete(cleanup);
announceCleanupSettled(cleanupDomain);
})
.catch(() => undefined);
}
@@ -752,6 +832,7 @@ function startQuarantinedSessionCleanupRecovery(
try {
await attempt;
quarantinedSessionCleanups.delete(cleanup);
announceCleanupSettled(cleanup.domain);
} catch (error) {
if (error instanceof NativeSessionCloseUnrecoverableError) {
cleanup.operatorRecoveryRequired = true;
@@ -768,6 +849,7 @@ function startQuarantinedSessionCleanupRecovery(
void recovery
.finally(() => {
failedSessionCleanupOwners.delete(recovery);
announceCleanupSettled(cleanup.domain);
if (cleanup.recovery === recovery) {
cleanup.recovery = null;
cleanup.recoveryMaxAttempts = null;
+7
View File
@@ -82,6 +82,7 @@ import { setSentryRunLinkResolver } from "./sentry-links.js";
import { createSentryRunLinkResolver } from "./services/sentry-run-links.js";
import { waitForPendingRunFailureReports } from "./services/run-failure-report.js";
import { verifyStoppedNativeSessionForReplacement } from "./services/native-runtime/native-session-executor.js";
import { registerNativeCleanupReleaseWake } from "./services/native-runtime/native-cleanup-release-wake.js";
import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js";
import { deliverExecutionStatuses } from "./services/execution-status-delivery.js";
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js";
@@ -1357,6 +1358,12 @@ async function startServerWithDatabaseTeardown(
], () => !heartbeatSchedulerStopped);
const unsubscribeLocalLoginCleanup = localLoginCleanup.subscribeCleanupReady(localLoginCleanupWork.wake);
server.on("close", unsubscribeLocalLoginCleanup);
// A retained native session close holds task input in process memory. When
// it is released, wake deferred execution so held Retries re-evaluate.
const unsubscribeNativeCleanupSettled = registerNativeCleanupReleaseWake(db, (error, companyId) => {
logger.warn({ err: error, companyId }, "native cleanup release could not wake deferred execution");
});
server.on("close", unsubscribeNativeCleanupSettled);
registerExecutionControlWork(deliveryWork, [
{ queue: DELIVERY_QUEUES.executionFinalization, run: () => reconcileAbandonedExecutionControl(db, new Date(), {
startupCleanup: async scope => {
@@ -17,7 +17,7 @@ import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation.
import { readQueuedInteractionResponse } from "./queued-interaction-response.js";
import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js";
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "./native-local-process-stop.js";
import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/paperclip-runner/index.js";
import { completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
import { z } from "zod";
import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm";
@@ -490,7 +490,17 @@ export async function admitExplicitNativeContinuation(input: {
if (run.processGroupId && !processStopped(-run.processGroupId)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
}
}
if (!remote && run.runtimeMode === "native" && run.errorCode === "native_session_cleanup_quarantined") {
// The runner keeps a closing or quarantined provider session in process
// memory, keyed by this task. Its admission gate would refuse a successor
// at once, so hold the input here with the matching reason instead of
// dispatching a run that can only fail. A pending close settles by itself
// and its release wakes deferred execution; the held input then starts.
const retainedCleanup = !remote && run.runtimeMode === "native"
? nativeSessionCleanupBlock({ companyId, issueId }) : null;
if (retainedCleanup === "pending" && run.errorCode !== "native_session_cleanup_quarantined")
return blocked("local_cleanup", "Waiting for the previous provider session to finish cleanup. Your message will start automatically.");
if (!remote && run.runtimeMode === "native" &&
(run.errorCode === "native_session_cleanup_quarantined" || retainedCleanup === "operator_required")) {
// This is new user input, never permission to retry the interrupted turn.
if (retry) return blocked("cleanup_quarantined", "Send a new message after the previous provider has stopped.");
const { verifyStoppedNativeSessionForContinuation } = await import("./native-runtime/native-session-executor.js");
@@ -0,0 +1,17 @@
import type { Db } from "@paperclipai/db";
import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
import { onNativeSessionCleanupSettled } from "../../vendor/paperclip-runner/index.js";
/**
* A retained native session close holds task input in the runner's process
* memory. When the runner releases it, wake deferred execution for that company
* so a held Retry or message re-evaluates its gate. The wake is a hint only:
* dispatch repeats every admission check under its own locks.
*/
export function registerNativeCleanupReleaseWake(db: Db, onError: (error: unknown, companyId: string) => void): () => void {
return onNativeSessionCleanupSettled(({ companyId }) => {
if (!companyId) return;
void db.transaction(async tx => notifyDeferredExecutionWork(tx, { companyId }))
.catch((error: unknown) => onError(error, companyId));
});
}
@@ -3,13 +3,14 @@ import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { hostname, tmpdir } from "node:os";
import { join } from "node:path";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import {
agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces, heartbeatRunEvents,
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
import { executeNativeSession, nativeSessionCleanupBlock, onNativeSessionCleanupSettled, parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
import type { NativeSession } from "../vendor/paperclip-runner/index.js";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { heartbeatService } from "./heartbeat.js";
import { getExecutionBlocker } from "./execution-blocker.js";
@@ -19,6 +20,8 @@ import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-loc
import { nativeSha256 } from "./native-runtime/canonical.js";
import { verifyStoppedNativeSessionForContinuation } from "./native-runtime/native-session-executor.js";
import { requestTaskRetry } from "./task-retry.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
import { registerNativeCleanupReleaseWake } from "./native-runtime/native-cleanup-release-wake.js";
describe("task Retry after inherited native cleanup quarantine", () => {
let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
@@ -240,4 +243,83 @@ describe("task Retry after inherited native cleanup quarantine", () => {
await rm(base, { recursive: true, force: true });
}
});
it("holds a Retry while the task's own provider session close is still retained, then starts it", async () => {
const base = await mkdtemp(join(tmpdir(), "retained-close-retry-"));
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
let releaseClose = () => {};
try {
const f = await seed(base);
// The source failed for another reason; only the runner's in-process
// admission gate still holds this task's provider session close.
await db.update(heartbeatRuns).set({ errorCode: "provider_transport_failed" }).where(eq(heartbeatRuns.id, f.sourceId));
const settled = vi.fn();
const unsubscribe = onNativeSessionCleanupSettled(settled);
// The production release wake, observed on the delivery queue it signals.
const wakeFailure = vi.fn();
const unregisterWake = registerNativeCleanupReleaseWake(db, wakeFailure);
const deferredWake = vi.fn();
const unsubscribeDeferred = subscribeDeliveryWork(db, DELIVERY_QUEUES.deferredExecutionReceipts, deferredWake);
const pendingClose = new Promise<void>(resolve => { releaseClose = resolve; });
const identity = { runId: randomUUID(), sessionId: "retained-close-session", companyId: f.companyId, issueId: f.issueId, agentId: f.agentId };
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
const session: NativeSession = {
identity: () => identity, capabilities: async () => capabilities,
async *events() { throw new Error("retained close transport stopped"); },
startTurn: async () => ({ turnId: "retained-close-turn" }), result: async () => null,
async snapshot() { return { backendKind: "mock", sessionId: "retained-close-driver", identity,
providerSessionId: "retained-close-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },
close: () => pendingClose,
};
await expect(executeNativeSession({
input: parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
binding: { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runId: identity.runId, executionWorkspaceId: f.workspaceId },
task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" },
workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } },
interactionResponses: [], credentialBindings: [] }),
backend: { descriptor: async () => ({ kind: "runner", name: "retained-close-backend", version: "1", capabilities }),
openSession: async () => session },
controlPlane: { openRun: async () => {}, checkpointSession: async () => {},
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
runnerInstanceId: f.runnerInstanceId, controlPlaneInstanceId: "control",
})).rejects.toThrow("retained close transport stopped");
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("pending");
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: randomUUID() })).toBeNull();
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
const heartbeat = heartbeatService(db);
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
const queued = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
// No successor is dispatched that the runner would refuse at once.
expect(await queued()).toHaveLength(0);
const [held] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
expect(held).toMatchObject({ status: "deferred_issue_execution", runId: null });
expect(held.payload).toMatchObject({ executionWait: { reason: "local_cleanup" } });
// The close settles by itself; its release wakes the held Retry, which then starts.
releaseClose();
await vi.waitFor(() => expect(settled).toHaveBeenCalledWith({ companyId: f.companyId, issueIds: [f.issueId] }));
await vi.waitFor(() => expect(deferredWake).toHaveBeenCalled());
expect(wakeFailure).not.toHaveBeenCalled();
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
unsubscribe();
unregisterWake();
unsubscribeDeferred();
// The deferred-execution worker runs this pass on that wake.
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
expect(await queued()).toHaveLength(1);
} finally {
releaseClose();
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
await rm(base, { recursive: true, force: true });
}
});
});
+2
View File
@@ -191,6 +191,8 @@ export const nativeRestartInterruptedTurnId = runner.nativeRestartInterruptedTur
export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup;
export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup;
export const nativeSessionCleanupBlock = runner.nativeSessionCleanupBlock;
export const onNativeSessionCleanupSettled = runner.onNativeSessionCleanupSettled;
export const probeAcpxClaudeInstallation = runner.probeAcpxClaudeInstallation;
export const probeAcpxGrokInstallation = runner.probeAcpxGrokInstallation;