From 785ead5c7ab8fcaedd3ab791c2ec0486764eb0e1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 18:52:46 -0600 Subject: [PATCH] fix(tasks): hold a Retry while the task's own native session close is retained (#19609) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native provider session close fails, the runner keeps a cleanup owner or quarantine in process memory. Its admission gate refuses new sessions for that task (scoped to the task since #19532) > - The server could not see that state. A task's Retry was accepted (202), its run was dispatched, and the runner refused it at once with `native_session_cleanup_quarantined: prior session cleanup remains incomplete` > - The user saw a Retry that failed instantly, and nothing said what to do > - This pull request lets the server read the runner's per-task cleanup gate and hold the Retry with a clear reason. The Retry starts automatically when the close is released > - The benefit is that a Retry is never dispatched only to fail on a block that is already known ## Linked Issues or Issue Description **What happened?** After two hard restarts, a Codex run's close stayed incomplete. The board Retry for that task was accepted, and its run failed in about 1 s with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. The next Retry did the same. **Steps to reproduce** 1. Let a native session close fail or stay pending for a task. 2. Press Retry on that task. **Expected behavior** The Retry waits with a clear reason, and starts when the cleanup finishes. If the cleanup needs an operator, Retry is not offered and the one working action is shown. **Actual behavior** The Retry is accepted and its run fails at once. ## What Changed - `native-session-runtime.ts`: - New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a read-only view of the in-process admission gate for one task: `pending` (a retained close or a bounded automatic close recovery still owns the task's provider resources), `operator_required`, or `null`. It grants no authority. - New `onNativeSessionCleanupSettled(listener)`. It is called whenever a cleanup owner or quarantine entry is released. - `explicit-native-continuation.ts` (the continuation gate for a Retry or a new message): - `pending`: the input is held with `local_cleanup` ("Retry is waiting for the previous execution to finish cleanup. Your message is saved."). It starts automatically when the close is released. - `operator_required`: handled exactly like a recorded `native_session_cleanup_quarantined` run. Retry is not offered. A new message runs the existing stopped-process proof (`verifyStoppedNativeSessionForContinuation`), which retires the runner quarantine when the old processes are proven stopped. - `native-cleanup-release-wake.ts` (registered in `server/src/index.ts`): each cleanup release wakes deferred execution for the company in a transaction, so a held Retry re-evaluates. - `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime functions. ## Verification - Red before the gate (server change stashed): `pnpm --filter @paperclipai/server exec vitest run src/services/quarantined-task-retry.test.ts -t "holds a Retry"` dispatched a successor run while the task's close was retained (`expected [ { … } ] to have a length of +0 but got 1`). - Green after the gate: - The Retry is held (`deferred_issue_execution`, `executionWait.reason = local_cleanup`), and no successor is dispatched. - A different task is not blocked. - When the close settles, the release is announced, and the next pass dispatches exactly one successor. - The whole `quarantined-task-retry.test.ts` file passes (28 tests). `src/native-session-runtime.test.ts` passes. Server `tsc --noEmit` passes after the runner package build. - Live chaos replays: results are added in a comment. ## Risks - The gate reads process memory, so it applies only where the server and the runner session share a process (local runner). Other deployments keep today's behavior. - A held Retry re-evaluates on the release wake (sent in its own transaction) or on the existing 30 s deferred-execution pass. When the bounded automatic close budget is spent with nothing else scheduled, the view reports `operator_required`. Retry is then no longer offered, and a new message reaches the stopped-process recovery instead of waiting forever. - No schema or API shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/native-session-runtime.test.ts | 7 ++ .../src/native-session-runtime.ts | 86 ++++++++++++++++++- server/src/index.ts | 7 ++ .../services/explicit-native-continuation.ts | 14 ++- .../native-cleanup-release-wake.ts | 17 ++++ .../services/quarantined-task-retry.test.ts | 86 ++++++++++++++++++- server/src/vendor/paperclip-runner/index.ts | 2 + 7 files changed, 213 insertions(+), 6 deletions(-) create mode 100644 server/src/services/native-runtime/native-cleanup-release-wake.ts diff --git a/packages/paperclip-runner/src/native-session-runtime.test.ts b/packages/paperclip-runner/src/native-session-runtime.test.ts index 897dd4290a9..575bb67bb7a 100644 --- a/packages/paperclip-runner/src/native-session-runtime.test.ts +++ b/packages/paperclip-runner/src/native-session-runtime.test.ts @@ -41,6 +41,7 @@ import { executeNativeSession, completeTerminatedRemoteNativeSessionCleanup, completeTerminatedLocalNativeSessionCleanup, + nativeSessionCleanupBlock, type ExecuteNativeSessionOptions, } from "./native-session-runtime.js"; @@ -2525,11 +2526,17 @@ describe("executeNativeSession recovery", () => { expect(admissionAttempt).toBe(3); expect(scheduledAttempt).toBe(0); + const block = { companyId: lifetimeIdentity.companyId, issueId: input.binding.issueId }; + // Automatic recovery is still scheduled: the server may hold input. + expect(nativeSessionCleanupBlock(block)).toBe("pending"); await vi.advanceTimersByTimeAsync(180_000); expect(scheduledAttempt).toBe(3); await vi.advanceTimersByTimeAsync(300_000); expect(scheduledAttempt).toBe(3); expect(openSession).toHaveBeenCalledOnce(); + // The automatic budget is spent and nothing else is scheduled: waiting + // would never end, so the task needs stopped-process recovery. + expect(nativeSessionCleanupBlock(block)).toBe("operator_required"); } finally { vi.useRealTimers(); } diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.ts index 6b21866e901..5d6743e53fa 100644 --- a/packages/paperclip-runner/src/native-session-runtime.ts +++ b/packages/paperclip-runner/src/native-session-runtime.ts @@ -85,6 +85,78 @@ const failedSessionCleanupOwners = new Map< Promise, NativeSessionCleanupDomain >(); + +type NativeSessionCleanupSettledListener = (scope: { + companyId: string; + issueIds: string[]; +}) => void; +const cleanupSettledListeners = new Set(); + +function cleanupDomainScope(domain: NativeSessionCleanupDomain): { + companyId: string; + issueIds: string[]; +} { + const [companyId] = JSON.parse(domain.key) as string[]; + const issueIds = domain.resources.flatMap((resource) => { + const [kind, id] = JSON.parse(resource) as string[]; + return kind === "issue" && typeof id === "string" ? [id] : []; + }); + return { companyId: companyId ?? "", issueIds }; +} + +/** A retained cleanup owner or quarantine entry was released. Observers can + * re-evaluate work that waited on it; this grants no execution authority. */ +function announceCleanupSettled(domain: NativeSessionCleanupDomain): void { + const scope = cleanupDomainScope(domain); + for (const listener of cleanupSettledListeners) { + try { + listener(structuredClone(scope)); + } catch { + // Observers cannot change cleanup ownership. + } + } +} + +/** Subscribe to cleanup releases. Returns the unsubscribe function. */ +export function onNativeSessionCleanupSettled( + listener: NativeSessionCleanupSettledListener, +): () => void { + cleanupSettledListeners.add(listener); + return () => cleanupSettledListeners.delete(listener); +} + +/** + * Read-only view of the in-process admission gate for one task. "pending": + * a retained close or bounded automatic close recovery still owns the task's + * provider resources, and admission would be refused now. "operator_required": + * the close cannot settle automatically. Never grants or retires authority. + */ +export function nativeSessionCleanupBlock(input: { + companyId: string; + issueId: string; +}): "operator_required" | "pending" | null { + const issueResource = JSON.stringify(["issue", input.issueId]); + const matches = (domain: NativeSessionCleanupDomain) => + domain.resources.includes(issueResource) && + cleanupDomainScope(domain).companyId === input.companyId; + let block: "operator_required" | "pending" | null = null; + for (const entry of quarantinedSessionCleanups) { + if (!matches(entry.domain)) continue; + // An entry whose bounded automatic close budget is spent has no further + // scheduled attempt. Waiting would never end, so it needs the same + // stopped-process recovery as an unrecoverable close. + const automaticRecoveryExhausted = + !entry.attempt && !entry.recovery && !entry.timer && + entry.automaticAttempts >= MAX_QUARANTINED_SESSION_AUTOMATIC_CLOSE_ATTEMPTS; + if (entry.operatorRecoveryRequired || automaticRecoveryExhausted) + return "operator_required"; + block = "pending"; + } + for (const domain of failedSessionCleanupOwners.values()) { + if (matches(domain)) block = "pending"; + } + return block; +} const FAILED_SESSION_CLOSE_RETRY_MS = 1_000; const MAX_FAILED_SESSION_CLOSE_RETRIES = 3; const MAX_QUARANTINED_SESSION_CLOSE_RETRIES = 3; @@ -154,7 +226,10 @@ export function completeRetainedNativeSessionCleanup( ) { throw new NativeSessionCleanupQuarantinedError(); } - for (const entry of matches) quarantinedSessionCleanups.delete(entry); + for (const entry of matches) { + quarantinedSessionCleanups.delete(entry); + announceCleanupSettled(entry.domain); + } return matches.length; } @@ -182,6 +257,7 @@ export function completeTerminatedRemoteNativeSessionCleanup(binding: { if (entry.timer) clearTimeout(entry.timer); entry.timer = null; quarantinedSessionCleanups.delete(entry); + announceCleanupSettled(entry.domain); } return true; } @@ -205,6 +281,7 @@ export function completeTerminatedLocalNativeSessionCleanup(binding: { for (const entry of matches) { if (entry.timer) clearTimeout(entry.timer); quarantinedSessionCleanups.delete(entry); + announceCleanupSettled(entry.domain); } return true; } @@ -679,7 +756,10 @@ function retainFailedSessionCleanupOwner( ): void { failedSessionCleanupOwners.set(cleanup, cleanupDomain); void cleanup - .finally(() => failedSessionCleanupOwners.delete(cleanup)) + .finally(() => { + failedSessionCleanupOwners.delete(cleanup); + announceCleanupSettled(cleanupDomain); + }) .catch(() => undefined); } @@ -752,6 +832,7 @@ function startQuarantinedSessionCleanupRecovery( try { await attempt; quarantinedSessionCleanups.delete(cleanup); + announceCleanupSettled(cleanup.domain); } catch (error) { if (error instanceof NativeSessionCloseUnrecoverableError) { cleanup.operatorRecoveryRequired = true; @@ -768,6 +849,7 @@ function startQuarantinedSessionCleanupRecovery( void recovery .finally(() => { failedSessionCleanupOwners.delete(recovery); + announceCleanupSettled(cleanup.domain); if (cleanup.recovery === recovery) { cleanup.recovery = null; cleanup.recoveryMaxAttempts = null; diff --git a/server/src/index.ts b/server/src/index.ts index c3c1e172f18..77710bcc477 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -82,6 +82,7 @@ import { setSentryRunLinkResolver } from "./sentry-links.js"; import { createSentryRunLinkResolver } from "./services/sentry-run-links.js"; import { waitForPendingRunFailureReports } from "./services/run-failure-report.js"; import { verifyStoppedNativeSessionForReplacement } from "./services/native-runtime/native-session-executor.js"; +import { registerNativeCleanupReleaseWake } from "./services/native-runtime/native-cleanup-release-wake.js"; import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js"; import { deliverExecutionStatuses } from "./services/execution-status-delivery.js"; import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js"; @@ -1357,6 +1358,12 @@ async function startServerWithDatabaseTeardown( ], () => !heartbeatSchedulerStopped); const unsubscribeLocalLoginCleanup = localLoginCleanup.subscribeCleanupReady(localLoginCleanupWork.wake); server.on("close", unsubscribeLocalLoginCleanup); + // A retained native session close holds task input in process memory. When + // it is released, wake deferred execution so held Retries re-evaluate. + const unsubscribeNativeCleanupSettled = registerNativeCleanupReleaseWake(db, (error, companyId) => { + logger.warn({ err: error, companyId }, "native cleanup release could not wake deferred execution"); + }); + server.on("close", unsubscribeNativeCleanupSettled); registerExecutionControlWork(deliveryWork, [ { queue: DELIVERY_QUEUES.executionFinalization, run: () => reconcileAbandonedExecutionControl(db, new Date(), { startupCleanup: async scope => { diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index d3207e5b97b..e35448a0bb6 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -17,7 +17,7 @@ import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation. import { readQueuedInteractionResponse } from "./queued-interaction-response.js"; import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js"; import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "./native-local-process-stop.js"; -import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/paperclip-runner/index.js"; +import { completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js"; import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js"; import { z } from "zod"; import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm"; @@ -490,7 +490,17 @@ export async function admitExplicitNativeContinuation(input: { if (run.processGroupId && !processStopped(-run.processGroupId)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically."); } } - if (!remote && run.runtimeMode === "native" && run.errorCode === "native_session_cleanup_quarantined") { + // The runner keeps a closing or quarantined provider session in process + // memory, keyed by this task. Its admission gate would refuse a successor + // at once, so hold the input here with the matching reason instead of + // dispatching a run that can only fail. A pending close settles by itself + // and its release wakes deferred execution; the held input then starts. + const retainedCleanup = !remote && run.runtimeMode === "native" + ? nativeSessionCleanupBlock({ companyId, issueId }) : null; + if (retainedCleanup === "pending" && run.errorCode !== "native_session_cleanup_quarantined") + return blocked("local_cleanup", "Waiting for the previous provider session to finish cleanup. Your message will start automatically."); + if (!remote && run.runtimeMode === "native" && + (run.errorCode === "native_session_cleanup_quarantined" || retainedCleanup === "operator_required")) { // This is new user input, never permission to retry the interrupted turn. if (retry) return blocked("cleanup_quarantined", "Send a new message after the previous provider has stopped."); const { verifyStoppedNativeSessionForContinuation } = await import("./native-runtime/native-session-executor.js"); diff --git a/server/src/services/native-runtime/native-cleanup-release-wake.ts b/server/src/services/native-runtime/native-cleanup-release-wake.ts new file mode 100644 index 00000000000..8d23f85b240 --- /dev/null +++ b/server/src/services/native-runtime/native-cleanup-release-wake.ts @@ -0,0 +1,17 @@ +import type { Db } from "@paperclipai/db"; +import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js"; +import { onNativeSessionCleanupSettled } from "../../vendor/paperclip-runner/index.js"; + +/** + * A retained native session close holds task input in the runner's process + * memory. When the runner releases it, wake deferred execution for that company + * so a held Retry or message re-evaluates its gate. The wake is a hint only: + * dispatch repeats every admission check under its own locks. + */ +export function registerNativeCleanupReleaseWake(db: Db, onError: (error: unknown, companyId: string) => void): () => void { + return onNativeSessionCleanupSettled(({ companyId }) => { + if (!companyId) return; + void db.transaction(async tx => notifyDeferredExecutionWork(tx, { companyId })) + .catch((error: unknown) => onError(error, companyId)); + }); +} diff --git a/server/src/services/quarantined-task-retry.test.ts b/server/src/services/quarantined-task-retry.test.ts index 56006603a50..8a02a628709 100644 --- a/server/src/services/quarantined-task-retry.test.ts +++ b/server/src/services/quarantined-task-retry.test.ts @@ -3,13 +3,14 @@ import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { hostname, tmpdir } from "node:os"; import { join } from "node:path"; import { and, eq } from "drizzle-orm"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations, } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; -import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js"; +import { executeNativeSession, nativeSessionCleanupBlock, onNativeSessionCleanupSettled, parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js"; +import type { NativeSession } from "../vendor/paperclip-runner/index.js"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { heartbeatService } from "./heartbeat.js"; import { getExecutionBlocker } from "./execution-blocker.js"; @@ -19,6 +20,8 @@ import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-loc import { nativeSha256 } from "./native-runtime/canonical.js"; import { verifyStoppedNativeSessionForContinuation } from "./native-runtime/native-session-executor.js"; import { requestTaskRetry } from "./task-retry.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js"; +import { registerNativeCleanupReleaseWake } from "./native-runtime/native-cleanup-release-wake.js"; describe("task Retry after inherited native cleanup quarantine", () => { let postgres: Awaited>; @@ -240,4 +243,83 @@ describe("task Retry after inherited native cleanup quarantine", () => { await rm(base, { recursive: true, force: true }); } }); + + it("holds a Retry while the task's own provider session close is still retained, then starts it", async () => { + const base = await mkdtemp(join(tmpdir(), "retained-close-retry-")); + const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR; + process.env.PAPERCLIP_RUNNER_STATE_DIR = base; + let releaseClose = () => {}; + try { + const f = await seed(base); + // The source failed for another reason; only the runner's in-process + // admission gate still holds this task's provider session close. + await db.update(heartbeatRuns).set({ errorCode: "provider_transport_failed" }).where(eq(heartbeatRuns.id, f.sourceId)); + const settled = vi.fn(); + const unsubscribe = onNativeSessionCleanupSettled(settled); + // The production release wake, observed on the delivery queue it signals. + const wakeFailure = vi.fn(); + const unregisterWake = registerNativeCleanupReleaseWake(db, wakeFailure); + const deferredWake = vi.fn(); + const unsubscribeDeferred = subscribeDeliveryWork(db, DELIVERY_QUEUES.deferredExecutionReceipts, deferredWake); + const pendingClose = new Promise(resolve => { releaseClose = resolve; }); + const identity = { runId: randomUUID(), sessionId: "retained-close-session", companyId: f.companyId, issueId: f.issueId, agentId: f.agentId }; + const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true }; + const session: NativeSession = { + identity: () => identity, capabilities: async () => capabilities, + async *events() { throw new Error("retained close transport stopped"); }, + startTurn: async () => ({ turnId: "retained-close-turn" }), result: async () => null, + async snapshot() { return { backendKind: "mock", sessionId: "retained-close-driver", identity, + providerSessionId: "retained-close-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; }, + close: () => pendingClose, + }; + await expect(executeNativeSession({ + input: parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null }, + binding: { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runId: identity.runId, executionWorkspaceId: f.workspaceId }, + task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" }, + workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1", + contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } }, + interactionResponses: [], credentialBindings: [] }), + backend: { descriptor: async () => ({ kind: "runner", name: "retained-close-backend", version: "1", capabilities }), + openSession: async () => session }, + controlPlane: { openRun: async () => {}, checkpointSession: async () => {}, + appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }), + replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} }, + runnerInstanceId: f.runnerInstanceId, controlPlaneInstanceId: "control", + })).rejects.toThrow("retained close transport stopped"); + expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("pending"); + expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: randomUUID() })).toBeNull(); + + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board", + actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() }); + const heartbeat = heartbeatService(db); + await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!); + const queued = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))); + // No successor is dispatched that the runner would refuse at once. + expect(await queued()).toHaveLength(0); + const [held] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!)); + expect(held).toMatchObject({ status: "deferred_issue_execution", runId: null }); + expect(held.payload).toMatchObject({ executionWait: { reason: "local_cleanup" } }); + + // The close settles by itself; its release wakes the held Retry, which then starts. + releaseClose(); + await vi.waitFor(() => expect(settled).toHaveBeenCalledWith({ companyId: f.companyId, issueIds: [f.issueId] })); + await vi.waitFor(() => expect(deferredWake).toHaveBeenCalled()); + expect(wakeFailure).not.toHaveBeenCalled(); + expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull(); + unsubscribe(); + unregisterWake(); + unsubscribeDeferred(); + // The deferred-execution worker runs this pass on that wake. + await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!); + expect(await queued()).toHaveLength(1); + } finally { + releaseClose(); + if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous; + await rm(base, { recursive: true, force: true }); + } + }); }); diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index f72c65e333d..e74290588c0 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -191,6 +191,8 @@ export const nativeRestartInterruptedTurnId = runner.nativeRestartInterruptedTur export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup; export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup; +export const nativeSessionCleanupBlock = runner.nativeSessionCleanupBlock; +export const onNativeSessionCleanupSettled = runner.onNativeSessionCleanupSettled; export const probeAcpxClaudeInstallation = runner.probeAcpxClaudeInstallation; export const probeAcpxGrokInstallation = runner.probeAcpxGrokInstallation;