mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
fix(tasks): hold a Retry while the task's own native session close is retained (#19609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native provider session close fails, the runner keeps a cleanup owner or quarantine in process memory. Its admission gate refuses new sessions for that task (scoped to the task since #19532) > - The server could not see that state. A task's Retry was accepted (202), its run was dispatched, and the runner refused it at once with `native_session_cleanup_quarantined: prior session cleanup remains incomplete` > - The user saw a Retry that failed instantly, and nothing said what to do > - This pull request lets the server read the runner's per-task cleanup gate and hold the Retry with a clear reason. The Retry starts automatically when the close is released > - The benefit is that a Retry is never dispatched only to fail on a block that is already known ## Linked Issues or Issue Description **What happened?** After two hard restarts, a Codex run's close stayed incomplete. The board Retry for that task was accepted, and its run failed in about 1 s with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. The next Retry did the same. **Steps to reproduce** 1. Let a native session close fail or stay pending for a task. 2. Press Retry on that task. **Expected behavior** The Retry waits with a clear reason, and starts when the cleanup finishes. If the cleanup needs an operator, Retry is not offered and the one working action is shown. **Actual behavior** The Retry is accepted and its run fails at once. ## What Changed - `native-session-runtime.ts`: - New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a read-only view of the in-process admission gate for one task: `pending` (a retained close or a bounded automatic close recovery still owns the task's provider resources), `operator_required`, or `null`. It grants no authority. - New `onNativeSessionCleanupSettled(listener)`. It is called whenever a cleanup owner or quarantine entry is released. - `explicit-native-continuation.ts` (the continuation gate for a Retry or a new message): - `pending`: the input is held with `local_cleanup` ("Retry is waiting for the previous execution to finish cleanup. Your message is saved."). It starts automatically when the close is released. - `operator_required`: handled exactly like a recorded `native_session_cleanup_quarantined` run. Retry is not offered. A new message runs the existing stopped-process proof (`verifyStoppedNativeSessionForContinuation`), which retires the runner quarantine when the old processes are proven stopped. - `native-cleanup-release-wake.ts` (registered in `server/src/index.ts`): each cleanup release wakes deferred execution for the company in a transaction, so a held Retry re-evaluates. - `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime functions. ## Verification - Red before the gate (server change stashed): `pnpm --filter @paperclipai/server exec vitest run src/services/quarantined-task-retry.test.ts -t "holds a Retry"` dispatched a successor run while the task's close was retained (`expected [ { … } ] to have a length of +0 but got 1`). - Green after the gate: - The Retry is held (`deferred_issue_execution`, `executionWait.reason = local_cleanup`), and no successor is dispatched. - A different task is not blocked. - When the close settles, the release is announced, and the next pass dispatches exactly one successor. - The whole `quarantined-task-retry.test.ts` file passes (28 tests). `src/native-session-runtime.test.ts` passes. Server `tsc --noEmit` passes after the runner package build. - Live chaos replays: results are added in a comment. ## Risks - The gate reads process memory, so it applies only where the server and the runner session share a process (local runner). Other deployments keep today's behavior. - A held Retry re-evaluates on the release wake (sent in its own transaction) or on the existing 30 s deferred-execution pass. When the bounded automatic close budget is spent with nothing else scheduled, the view reports `operator_required`. Retry is then no longer offered, and a new message reaches the stopped-process recovery instead of waiting forever. - No schema or API shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
29e05f7e06
commit
785ead5c7a
7 files changed
+213
-6
No files matched your search
@@ -41,6 +41,7 @@ import {
|
||||
executeNativeSession,
|
||||
completeTerminatedRemoteNativeSessionCleanup,
|
||||
completeTerminatedLocalNativeSessionCleanup,
|
||||
nativeSessionCleanupBlock,
|
||||
type ExecuteNativeSessionOptions,
|
||||
} from "./native-session-runtime.js";
|
||||
|
||||
@@ -2525,11 +2526,17 @@ describe("executeNativeSession recovery", () => {
|
||||
|
||||
expect(admissionAttempt).toBe(3);
|
||||
expect(scheduledAttempt).toBe(0);
|
||||
const block = { companyId: lifetimeIdentity.companyId, issueId: input.binding.issueId };
|
||||
// Automatic recovery is still scheduled: the server may hold input.
|
||||
expect(nativeSessionCleanupBlock(block)).toBe("pending");
|
||||
await vi.advanceTimersByTimeAsync(180_000);
|
||||
expect(scheduledAttempt).toBe(3);
|
||||
await vi.advanceTimersByTimeAsync(300_000);
|
||||
expect(scheduledAttempt).toBe(3);
|
||||
expect(openSession).toHaveBeenCalledOnce();
|
||||
// The automatic budget is spent and nothing else is scheduled: waiting
|
||||
// would never end, so the task needs stopped-process recovery.
|
||||
expect(nativeSessionCleanupBlock(block)).toBe("operator_required");
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
|
||||
@@ -85,6 +85,78 @@ const failedSessionCleanupOwners = new Map<
|
||||
Promise<void>,
|
||||
NativeSessionCleanupDomain
|
||||
>();
|
||||
|
||||
type NativeSessionCleanupSettledListener = (scope: {
|
||||
companyId: string;
|
||||
issueIds: string[];
|
||||
}) => void;
|
||||
const cleanupSettledListeners = new Set<NativeSessionCleanupSettledListener>();
|
||||
|
||||
function cleanupDomainScope(domain: NativeSessionCleanupDomain): {
|
||||
companyId: string;
|
||||
issueIds: string[];
|
||||
} {
|
||||
const [companyId] = JSON.parse(domain.key) as string[];
|
||||
const issueIds = domain.resources.flatMap((resource) => {
|
||||
const [kind, id] = JSON.parse(resource) as string[];
|
||||
return kind === "issue" && typeof id === "string" ? [id] : [];
|
||||
});
|
||||
return { companyId: companyId ?? "", issueIds };
|
||||
}
|
||||
|
||||
/** A retained cleanup owner or quarantine entry was released. Observers can
|
||||
* re-evaluate work that waited on it; this grants no execution authority. */
|
||||
function announceCleanupSettled(domain: NativeSessionCleanupDomain): void {
|
||||
const scope = cleanupDomainScope(domain);
|
||||
for (const listener of cleanupSettledListeners) {
|
||||
try {
|
||||
listener(structuredClone(scope));
|
||||
} catch {
|
||||
// Observers cannot change cleanup ownership.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Subscribe to cleanup releases. Returns the unsubscribe function. */
|
||||
export function onNativeSessionCleanupSettled(
|
||||
listener: NativeSessionCleanupSettledListener,
|
||||
): () => void {
|
||||
cleanupSettledListeners.add(listener);
|
||||
return () => cleanupSettledListeners.delete(listener);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only view of the in-process admission gate for one task. "pending":
|
||||
* a retained close or bounded automatic close recovery still owns the task's
|
||||
* provider resources, and admission would be refused now. "operator_required":
|
||||
* the close cannot settle automatically. Never grants or retires authority.
|
||||
*/
|
||||
export function nativeSessionCleanupBlock(input: {
|
||||
companyId: string;
|
||||
issueId: string;
|
||||
}): "operator_required" | "pending" | null {
|
||||
const issueResource = JSON.stringify(["issue", input.issueId]);
|
||||
const matches = (domain: NativeSessionCleanupDomain) =>
|
||||
domain.resources.includes(issueResource) &&
|
||||
cleanupDomainScope(domain).companyId === input.companyId;
|
||||
let block: "operator_required" | "pending" | null = null;
|
||||
for (const entry of quarantinedSessionCleanups) {
|
||||
if (!matches(entry.domain)) continue;
|
||||
// An entry whose bounded automatic close budget is spent has no further
|
||||
// scheduled attempt. Waiting would never end, so it needs the same
|
||||
// stopped-process recovery as an unrecoverable close.
|
||||
const automaticRecoveryExhausted =
|
||||
!entry.attempt && !entry.recovery && !entry.timer &&
|
||||
entry.automaticAttempts >= MAX_QUARANTINED_SESSION_AUTOMATIC_CLOSE_ATTEMPTS;
|
||||
if (entry.operatorRecoveryRequired || automaticRecoveryExhausted)
|
||||
return "operator_required";
|
||||
block = "pending";
|
||||
}
|
||||
for (const domain of failedSessionCleanupOwners.values()) {
|
||||
if (matches(domain)) block = "pending";
|
||||
}
|
||||
return block;
|
||||
}
|
||||
const FAILED_SESSION_CLOSE_RETRY_MS = 1_000;
|
||||
const MAX_FAILED_SESSION_CLOSE_RETRIES = 3;
|
||||
const MAX_QUARANTINED_SESSION_CLOSE_RETRIES = 3;
|
||||
@@ -154,7 +226,10 @@ export function completeRetainedNativeSessionCleanup(
|
||||
) {
|
||||
throw new NativeSessionCleanupQuarantinedError();
|
||||
}
|
||||
for (const entry of matches) quarantinedSessionCleanups.delete(entry);
|
||||
for (const entry of matches) {
|
||||
quarantinedSessionCleanups.delete(entry);
|
||||
announceCleanupSettled(entry.domain);
|
||||
}
|
||||
return matches.length;
|
||||
}
|
||||
|
||||
@@ -182,6 +257,7 @@ export function completeTerminatedRemoteNativeSessionCleanup(binding: {
|
||||
if (entry.timer) clearTimeout(entry.timer);
|
||||
entry.timer = null;
|
||||
quarantinedSessionCleanups.delete(entry);
|
||||
announceCleanupSettled(entry.domain);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
@@ -205,6 +281,7 @@ export function completeTerminatedLocalNativeSessionCleanup(binding: {
|
||||
for (const entry of matches) {
|
||||
if (entry.timer) clearTimeout(entry.timer);
|
||||
quarantinedSessionCleanups.delete(entry);
|
||||
announceCleanupSettled(entry.domain);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
@@ -679,7 +756,10 @@ function retainFailedSessionCleanupOwner(
|
||||
): void {
|
||||
failedSessionCleanupOwners.set(cleanup, cleanupDomain);
|
||||
void cleanup
|
||||
.finally(() => failedSessionCleanupOwners.delete(cleanup))
|
||||
.finally(() => {
|
||||
failedSessionCleanupOwners.delete(cleanup);
|
||||
announceCleanupSettled(cleanupDomain);
|
||||
})
|
||||
.catch(() => undefined);
|
||||
}
|
||||
|
||||
@@ -752,6 +832,7 @@ function startQuarantinedSessionCleanupRecovery(
|
||||
try {
|
||||
await attempt;
|
||||
quarantinedSessionCleanups.delete(cleanup);
|
||||
announceCleanupSettled(cleanup.domain);
|
||||
} catch (error) {
|
||||
if (error instanceof NativeSessionCloseUnrecoverableError) {
|
||||
cleanup.operatorRecoveryRequired = true;
|
||||
@@ -768,6 +849,7 @@ function startQuarantinedSessionCleanupRecovery(
|
||||
void recovery
|
||||
.finally(() => {
|
||||
failedSessionCleanupOwners.delete(recovery);
|
||||
announceCleanupSettled(cleanup.domain);
|
||||
if (cleanup.recovery === recovery) {
|
||||
cleanup.recovery = null;
|
||||
cleanup.recoveryMaxAttempts = null;
|
||||
|
||||
@@ -82,6 +82,7 @@ import { setSentryRunLinkResolver } from "./sentry-links.js";
|
||||
import { createSentryRunLinkResolver } from "./services/sentry-run-links.js";
|
||||
import { waitForPendingRunFailureReports } from "./services/run-failure-report.js";
|
||||
import { verifyStoppedNativeSessionForReplacement } from "./services/native-runtime/native-session-executor.js";
|
||||
import { registerNativeCleanupReleaseWake } from "./services/native-runtime/native-cleanup-release-wake.js";
|
||||
import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js";
|
||||
import { deliverExecutionStatuses } from "./services/execution-status-delivery.js";
|
||||
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js";
|
||||
@@ -1357,6 +1358,12 @@ async function startServerWithDatabaseTeardown(
|
||||
], () => !heartbeatSchedulerStopped);
|
||||
const unsubscribeLocalLoginCleanup = localLoginCleanup.subscribeCleanupReady(localLoginCleanupWork.wake);
|
||||
server.on("close", unsubscribeLocalLoginCleanup);
|
||||
// A retained native session close holds task input in process memory. When
|
||||
// it is released, wake deferred execution so held Retries re-evaluate.
|
||||
const unsubscribeNativeCleanupSettled = registerNativeCleanupReleaseWake(db, (error, companyId) => {
|
||||
logger.warn({ err: error, companyId }, "native cleanup release could not wake deferred execution");
|
||||
});
|
||||
server.on("close", unsubscribeNativeCleanupSettled);
|
||||
registerExecutionControlWork(deliveryWork, [
|
||||
{ queue: DELIVERY_QUEUES.executionFinalization, run: () => reconcileAbandonedExecutionControl(db, new Date(), {
|
||||
startupCleanup: async scope => {
|
||||
|
||||
@@ -17,7 +17,7 @@ import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation.
|
||||
import { readQueuedInteractionResponse } from "./queued-interaction-response.js";
|
||||
import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js";
|
||||
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "./native-local-process-stop.js";
|
||||
import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/paperclip-runner/index.js";
|
||||
import { completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
||||
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
|
||||
import { z } from "zod";
|
||||
import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm";
|
||||
@@ -490,7 +490,17 @@ export async function admitExplicitNativeContinuation(input: {
|
||||
if (run.processGroupId && !processStopped(-run.processGroupId)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
|
||||
}
|
||||
}
|
||||
if (!remote && run.runtimeMode === "native" && run.errorCode === "native_session_cleanup_quarantined") {
|
||||
// The runner keeps a closing or quarantined provider session in process
|
||||
// memory, keyed by this task. Its admission gate would refuse a successor
|
||||
// at once, so hold the input here with the matching reason instead of
|
||||
// dispatching a run that can only fail. A pending close settles by itself
|
||||
// and its release wakes deferred execution; the held input then starts.
|
||||
const retainedCleanup = !remote && run.runtimeMode === "native"
|
||||
? nativeSessionCleanupBlock({ companyId, issueId }) : null;
|
||||
if (retainedCleanup === "pending" && run.errorCode !== "native_session_cleanup_quarantined")
|
||||
return blocked("local_cleanup", "Waiting for the previous provider session to finish cleanup. Your message will start automatically.");
|
||||
if (!remote && run.runtimeMode === "native" &&
|
||||
(run.errorCode === "native_session_cleanup_quarantined" || retainedCleanup === "operator_required")) {
|
||||
// This is new user input, never permission to retry the interrupted turn.
|
||||
if (retry) return blocked("cleanup_quarantined", "Send a new message after the previous provider has stopped.");
|
||||
const { verifyStoppedNativeSessionForContinuation } = await import("./native-runtime/native-session-executor.js");
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
|
||||
import { onNativeSessionCleanupSettled } from "../../vendor/paperclip-runner/index.js";
|
||||
|
||||
/**
|
||||
* A retained native session close holds task input in the runner's process
|
||||
* memory. When the runner releases it, wake deferred execution for that company
|
||||
* so a held Retry or message re-evaluates its gate. The wake is a hint only:
|
||||
* dispatch repeats every admission check under its own locks.
|
||||
*/
|
||||
export function registerNativeCleanupReleaseWake(db: Db, onError: (error: unknown, companyId: string) => void): () => void {
|
||||
return onNativeSessionCleanupSettled(({ companyId }) => {
|
||||
if (!companyId) return;
|
||||
void db.transaction(async tx => notifyDeferredExecutionWork(tx, { companyId }))
|
||||
.catch((error: unknown) => onError(error, companyId));
|
||||
});
|
||||
}
|
||||
@@ -3,13 +3,14 @@ import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { hostname, tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces, heartbeatRunEvents,
|
||||
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations,
|
||||
} from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
|
||||
import { executeNativeSession, nativeSessionCleanupBlock, onNativeSessionCleanupSettled, parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
|
||||
import type { NativeSession } from "../vendor/paperclip-runner/index.js";
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { getExecutionBlocker } from "./execution-blocker.js";
|
||||
@@ -19,6 +20,8 @@ import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-loc
|
||||
import { nativeSha256 } from "./native-runtime/canonical.js";
|
||||
import { verifyStoppedNativeSessionForContinuation } from "./native-runtime/native-session-executor.js";
|
||||
import { requestTaskRetry } from "./task-retry.js";
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { registerNativeCleanupReleaseWake } from "./native-runtime/native-cleanup-release-wake.js";
|
||||
|
||||
describe("task Retry after inherited native cleanup quarantine", () => {
|
||||
let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
@@ -240,4 +243,83 @@ describe("task Retry after inherited native cleanup quarantine", () => {
|
||||
await rm(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("holds a Retry while the task's own provider session close is still retained, then starts it", async () => {
|
||||
const base = await mkdtemp(join(tmpdir(), "retained-close-retry-"));
|
||||
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
||||
let releaseClose = () => {};
|
||||
try {
|
||||
const f = await seed(base);
|
||||
// The source failed for another reason; only the runner's in-process
|
||||
// admission gate still holds this task's provider session close.
|
||||
await db.update(heartbeatRuns).set({ errorCode: "provider_transport_failed" }).where(eq(heartbeatRuns.id, f.sourceId));
|
||||
const settled = vi.fn();
|
||||
const unsubscribe = onNativeSessionCleanupSettled(settled);
|
||||
// The production release wake, observed on the delivery queue it signals.
|
||||
const wakeFailure = vi.fn();
|
||||
const unregisterWake = registerNativeCleanupReleaseWake(db, wakeFailure);
|
||||
const deferredWake = vi.fn();
|
||||
const unsubscribeDeferred = subscribeDeliveryWork(db, DELIVERY_QUEUES.deferredExecutionReceipts, deferredWake);
|
||||
const pendingClose = new Promise<void>(resolve => { releaseClose = resolve; });
|
||||
const identity = { runId: randomUUID(), sessionId: "retained-close-session", companyId: f.companyId, issueId: f.issueId, agentId: f.agentId };
|
||||
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
|
||||
const session: NativeSession = {
|
||||
identity: () => identity, capabilities: async () => capabilities,
|
||||
async *events() { throw new Error("retained close transport stopped"); },
|
||||
startTurn: async () => ({ turnId: "retained-close-turn" }), result: async () => null,
|
||||
async snapshot() { return { backendKind: "mock", sessionId: "retained-close-driver", identity,
|
||||
providerSessionId: "retained-close-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },
|
||||
close: () => pendingClose,
|
||||
};
|
||||
await expect(executeNativeSession({
|
||||
input: parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
||||
binding: { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runId: identity.runId, executionWorkspaceId: f.workspaceId },
|
||||
task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" },
|
||||
workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } },
|
||||
interactionResponses: [], credentialBindings: [] }),
|
||||
backend: { descriptor: async () => ({ kind: "runner", name: "retained-close-backend", version: "1", capabilities }),
|
||||
openSession: async () => session },
|
||||
controlPlane: { openRun: async () => {}, checkpointSession: async () => {},
|
||||
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
|
||||
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
|
||||
runnerInstanceId: f.runnerInstanceId, controlPlaneInstanceId: "control",
|
||||
})).rejects.toThrow("retained close transport stopped");
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("pending");
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: randomUUID() })).toBeNull();
|
||||
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
||||
actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
const queued = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
// No successor is dispatched that the runner would refuse at once.
|
||||
expect(await queued()).toHaveLength(0);
|
||||
const [held] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
expect(held).toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
expect(held.payload).toMatchObject({ executionWait: { reason: "local_cleanup" } });
|
||||
|
||||
// The close settles by itself; its release wakes the held Retry, which then starts.
|
||||
releaseClose();
|
||||
await vi.waitFor(() => expect(settled).toHaveBeenCalledWith({ companyId: f.companyId, issueIds: [f.issueId] }));
|
||||
await vi.waitFor(() => expect(deferredWake).toHaveBeenCalled());
|
||||
expect(wakeFailure).not.toHaveBeenCalled();
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
|
||||
unsubscribe();
|
||||
unregisterWake();
|
||||
unsubscribeDeferred();
|
||||
// The deferred-execution worker runs this pass on that wake.
|
||||
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
expect(await queued()).toHaveLength(1);
|
||||
} finally {
|
||||
releaseClose();
|
||||
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
||||
await rm(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
+2
@@ -191,6 +191,8 @@ export const nativeRestartInterruptedTurnId = runner.nativeRestartInterruptedTur
|
||||
|
||||
export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup;
|
||||
export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup;
|
||||
export const nativeSessionCleanupBlock = runner.nativeSessionCleanupBlock;
|
||||
export const onNativeSessionCleanupSettled = runner.onNativeSessionCleanupSettled;
|
||||
|
||||
export const probeAcpxClaudeInstallation = runner.probeAcpxClaudeInstallation;
|
||||
export const probeAcpxGrokInstallation = runner.probeAcpxGrokInstallation;
|
||||
|
||||
Reference in new issue
Block a user