diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index fc5dedd5249..0716abc09d3 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1697,7 +1697,7 @@ or the exact provenance inside its verified committed saved snapshot. The latter uses one temporary read-only archive verification per preparation when needed. Task relationships alone do not authorize importing or retaining files. -When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority. +When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. A fresh-workspace Retry (selected at admission, or by its preparation when the saved files could not be restored) is renewable only on this shutdown path: the person asked for that fresh attempt and it never ran, so the successor's receipt records the same workspace selection and the successor prepares the same attempt (a remote fresh workspace is keyed by the Retry, so it is reused). The stranded-task sweep offers every such stopped Retry or message turn to the scheduler before its generic checks, because the person's Retry started a fresh session: state the replaced session left behind (an unfinished session goal, or a native finalization that the Retry already replaced) does not own the task. While a terminal run's own executor still settles it (a live local execution or an unexpired controller lease), the sweep leaves the task alone, so it cannot clear the claim that run's bounded retry needs. When admission refuses, the sweep escalates the task once to `blocked` with a plain notice that asks for Retry, instead of refusing again on every pass. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority. Explicit retry admission checks the local parent adapter controller and verified environment cleanup. The existing queue-first release policy may retain only its own terminal execution claim while that cleanup is pending; finalization and the periodic stale-lock sweep reconsider the same policy after cleanup. Generic terminal-lock cleanup and stale checkout adoption leave these claims for that settlement. New user input, reassignment, and a successor's claim take precedence. Exhausted retries and revoked or missing authorization release the terminal claim without requesting another retry. Pending or failed environment cleanup cannot mint a successor receipt. diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index 34c3444c00e..c02780a1b6c 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -2,7 +2,7 @@ import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-n import { agentDirectoryPreparationWaitPredicate, isAgentDirectoryPreparationWait } from "./pre-dispatch-execution-evidence.js"; import { assertDurableChatWakeupRequest, assertDurableChatWakeupReceipt, type DurableChatWakeupRequest } from "./durable-chat-wakeup.js"; import { notifyExecutionRunWork } from "./execution-control-notifications.js"; -import { canSelectFreshWorkspace, retainedIssueWorkspaceForRetry, type ContinuationReadiness } from "./fresh-workspace-continuation.js"; +import { canSelectFreshWorkspace, freshReadinessSchema, retainedIssueWorkspaceForRetry, type ContinuationReadiness } from "./fresh-workspace-continuation.js"; import { verifiedWorkspaceSnapshotReceipt } from "./workspace-recovery-snapshot.js"; import { readTaskRetryIntent } from "./task-retry.js"; import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js"; @@ -721,8 +721,12 @@ export async function admitExplicitNativeContinuation(input: { type ContinuationRetryAuthority = { continuation: { commentId: string } | { commentId: null; taskRetryRequestId: string }; actorId: string; sourceAuthorizationId: string; evidence: Record; + /** The workspace a fresh-workspace Retry selected, carried to the successor. */ + readiness?: CarriedRetryReadiness; }; +type CarriedRetryReadiness = z.infer; + async function readCommentRetryAuthority(db: Db, input: { companyId: string; issueId: string; parentRunId: string; explicit: Record; }): Promise { @@ -753,13 +757,24 @@ async function readCommentRetryAuthority(db: Db, input: { /** A task Retry admits one plain continuation in the saved workspace. When that * run fails transiently, its bounded retry keeps the same actor and receipt; - * chat, queued-message and fresh-workspace Retries keep their own contracts. */ + * chat and queued-message Retries keep their own contracts. + * + * A fresh-workspace Retry (selected at admission, or when its preparation found + * the saved files unrestorable) is renewable only for a run that a graceful + * shutdown stopped before provider work: the person asked for that fresh + * attempt and it never ran. The successor carries the same workspace selection, + * so it prepares the same attempt rather than the workspace the Retry left. */ async function readTaskRetryRetryAuthority(db: Db, input: { companyId: string; issueId: string; agentId: string; parentRunId: string; explicit: Record; + workspaceSelectionRenewable: boolean; }): Promise { const { companyId, issueId, explicit } = input; const requestId = z.string().guid().safeParse(explicit.taskRetryRequestId); - if (explicit.commentId !== null || !requestId.success || explicit.queuedCommentInterruptId || explicit.readiness) return null; + if (explicit.commentId !== null || !requestId.success || explicit.queuedCommentInterruptId) return null; + const admittedReadiness = explicit.readiness === undefined ? null : freshReadinessSchema.safeParse(explicit.readiness); + if (admittedReadiness && (!input.workspaceSelectionRenewable || !admittedReadiness.success + || admittedReadiness.data.retryIntentId !== requestId.data + || admittedReadiness.data.sourceRunId !== explicit.previousRunId)) return null; const [wake] = await db.select().from(agentWakeupRequests).where(and( eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.id, requestId.data), eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.reason, "task_retry"), @@ -767,20 +782,44 @@ async function readTaskRetryRetryAuthority(db: Db, input: { sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`, )); const intent = readTaskRetryIntent(wake?.payload); + const freshAttempt = intent?.strategy === "fresh_isolated_workspace"; if (!wake?.requestedByActorId || !wake.runId || !intent || intent.generationId !== wake.id || - intent.actorId !== wake.requestedByActorId || intent.strategy !== "continue" || + intent.actorId !== wake.requestedByActorId || + !(intent.strategy === "continue" || (freshAttempt && input.workspaceSelectionRenewable)) || intent.chatActionId || intent.queuedChatRequestId || intent.canonicalIntentId) return null; // The admitted run's authority may be its action ledger, an earlier // automatic-retry receipt, or the Retry receipt itself. Dispatch validation // of the parent below proves which one applies. - const [receipt] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( - eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId), - eq(issueRecoveryActions.status, "resolved"), - sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId' = ${input.parentRunId}`, - sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'taskRetryRequestId' = ${wake.id}`, - )).limit(1); + const receipts = await db.select({ id: issueRecoveryActions.id, evidence: issueRecoveryActions.evidence }) + .from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + eq(issueRecoveryActions.status, "resolved"), + sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId' = ${input.parentRunId}`, + sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'taskRetryRequestId' = ${wake.id}`, + )); + let readiness: CarriedRetryReadiness | undefined = admittedReadiness?.success ? admittedReadiness.data : undefined; + if (freshAttempt && readiness?.mode !== "fresh_isolated_workspace") { + if (readiness) return null; + // An automatic retry of the fresh attempt carries the selection on its receipt. + const inherited = receipts.map(row => freshReadinessSchema.safeParse( + ((row.evidence.explicitUserContinuation as Record | undefined)?.automaticRetry as + Record | undefined)?.readiness)).find(parsed => parsed.success)?.data; + if (inherited) { + if (inherited.mode !== "fresh_isolated_workspace" || inherited.retryIntentId !== wake.id + || inherited.sourceRunId !== intent.sourceRunId) return null; + readiness = inherited; + } else { + // Preparation selected the fresh attempt after admission. Rebuild the + // selection it recorded: this Retry, its source and that source's leases. + if (intent.sourceRunId !== explicit.previousRunId) return null; + const leases = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, intent.sourceRunId))); + readiness = { mode: "fresh_isolated_workspace", retryIntentId: wake.id, sourceRunId: intent.sourceRunId, + retainedSourceIds: leases.map(lease => lease.id) }; + } + } return { continuation: { commentId: null, taskRetryRequestId: wake.id }, actorId: wake.requestedByActorId, - sourceAuthorizationId: receipt?.id ?? wake.id, evidence: {} }; + sourceAuthorizationId: receipts[0]?.id ?? wake.id, evidence: {}, ...(readiness ? { readiness } : {}) }; } /** A graceful server shutdown (an upgrade or restart) stopped this legacy run @@ -835,7 +874,8 @@ export async function admitExplicitContinuationRetry(input: { const explicit = context.explicitUserContinuation as Record | undefined; if (!explicit || !z.string().guid().safeParse(explicit.previousRunId).success || explicit.failedRunId) return null; const authority = explicit.commentId === null - ? await readTaskRetryRetryAuthority(db, { companyId, issueId, agentId, parentRunId: parent.id, explicit }) + ? await readTaskRetryRetryAuthority(db, { companyId, issueId, agentId, parentRunId: parent.id, explicit, + workspaceSelectionRenewable: shutdownInterrupted }) : await readCommentRetryAuthority(db, { companyId, issueId, parentRunId: parent.id, explicit }); if (!authority) return null; const [superseding] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( @@ -883,6 +923,7 @@ export async function admitExplicitContinuationRetry(input: { ...continuation, actorId: authority.actorId, runId: input.successorRunId, ...authority.evidence, recordedAt: input.now.toISOString(), automaticRetry: { sourceRunId: parent.id, sourceAuthorizationId: authority.sourceAuthorizationId, + ...(authority.readiness ? { readiness: authority.readiness } : {}), }, } }, }); diff --git a/server/src/services/fresh-workspace-continuation.ts b/server/src/services/fresh-workspace-continuation.ts index 9888ae7ebf3..c05d2b3c233 100644 --- a/server/src/services/fresh-workspace-continuation.ts +++ b/server/src/services/fresh-workspace-continuation.ts @@ -27,7 +27,8 @@ export type ContinuationReadiness = | { mode: "resume_stopped_native_workspace"; retryIntentId: string; sourceRunId: string; workspaceId: string; cwd: string } | { mode: "reuse_issue_workspace"; retryIntentId: string; sourceRunId: string; workspaceId: string; cwd: string | null } | { mode: "fresh_isolated_workspace"; retryIntentId: string; sourceRunId: string; retainedSourceIds: string[] }; -const freshReadinessSchema = z.discriminatedUnion("mode", [z.object({ +/** Workspace selections a Retry receipt records and its continuation can carry. */ +export const freshReadinessSchema = z.discriminatedUnion("mode", [z.object({ mode: z.literal("fresh_isolated_workspace"), retryIntentId: z.string().uuid(), sourceRunId: z.string().uuid(), retainedSourceIds: z.array(z.string().uuid()), }), z.object({ @@ -312,6 +313,31 @@ async function stoppedNativeContinuationWorkspace(db: Db, source: typeof heartbe sourceRunId: source.id, workspaceId: bound.id, cwd: parsed.data.workspaceCwd }; } +/** An automatic continuation of a fresh-workspace Retry attempt that a shutdown + * stopped before provider work prepares the same attempt. Its own admission + * receipt (`admitExplicitContinuationRetry`) records the selection; the run's + * context must still name the same parent and Retry. */ +async function readCarriedRetryReadiness(db: Db, run: Pick, issueId: string): Promise { + const explicit = run.contextSnapshot?.explicitUserContinuation as Record | undefined; + if (!explicit || typeof explicit.previousRunId !== "string" || typeof explicit.taskRetryRequestId !== "string") return null; + const [receipt] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + eq(issueRecoveryActions.cause, "explicit_user_continuation_retry"), eq(issueRecoveryActions.fingerprint, run.id), + eq(issueRecoveryActions.status, "resolved"), + sql`${issueRecoveryActions.evidence}#>>'{explicitUserContinuation,runId}' = ${run.id}`, + sql`${issueRecoveryActions.evidence}#>'{explicitUserContinuation,automaticRetry,readiness}' is not null`, + )).limit(1); + const authorization = receipt?.evidence.explicitUserContinuation as Record | undefined; + const automaticRetry = authorization?.automaticRetry as Record | undefined; + const parsed = freshReadinessSchema.safeParse(automaticRetry?.readiness); + if (!parsed.success || automaticRetry?.sourceRunId !== explicit.previousRunId + || authorization?.previousRunId !== explicit.previousRunId + || authorization?.taskRetryRequestId !== explicit.taskRetryRequestId + || parsed.data.retryIntentId !== explicit.taskRetryRequestId) return null; + return parsed.data; +} + /** Preparation trusts the committed admission receipt, not a raw wake payload. */ export async function readFreshWorkspaceReadiness(db: Db, run: Pick): Promise { @@ -345,6 +371,8 @@ export async function readFreshWorkspaceReadiness(db: Db, run: Pick>'runId' = ${issueRecoveryActions.evidence}#>>'{explicitUserContinuation,readiness,sourceRunId}'`, )).limit(1); if (!action) { + const carried = await readCarriedRetryReadiness(db, run, issueId); + if (carried) return carried; const [wake] = await db.select().from(agentWakeupRequests).where(and( eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId), eq(agentWakeupRequests.runId, run.id), eq(agentWakeupRequests.reason, "task_retry"), diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 9ce4709d5c4..8af7ec013d8 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1033,6 +1033,8 @@ export function heartbeatService( // when its lane (for example interrupted_preparation after cleanup) // differs from the synchronous budget check below. if (outcome && result.outcome === "retry_exhausted") outcome.retryExhausted = true; + if (outcome && result.outcome === "not_scheduled" && "errorCode" in result + && result.errorCode === "continuation_user_authorization_missing") outcome.continuationRefused = true; return result.outcome === "scheduled" ? result.run : null; }, // Mirrors scheduleBoundedRetryForRun's transient budget check: a failed diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 9b491b8d1eb..424f0efc47f 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -38,7 +38,7 @@ import { decideLegacyContinuation, legacyDispositionEpisode, legacyDispositionFingerprint, LEGACY_DISPOSITION_REPAIR_INSTRUCTION, type LegacyDispositionEpisode, } from "./legacy-continuation.js"; -import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController, liveLegacyControllerExpiresAt } from "../legacy-controller-lease.js"; +import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController, legacyControllerBootId, liveLegacyControllerExpiresAt } from "../legacy-controller-lease.js"; import { isCompletedOnboardingHandoffWake } from "../chat-completion-delivery.js"; import { instanceSettingsService } from "../instance-settings.js"; import { isWaitingConversation, settleConversationTurnForRun, deliverConversationComments } from "../agent-conversations.js"; @@ -205,6 +205,13 @@ const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [ "running", "scheduled_retry", ] as const; +const CONTINUATION_REFUSED_NOTICE: StrandedRecoveryNoticeSeed = { + title: "Retry needed", + body: "Paperclip could not continue this task on its own. The earlier Retry or message no longer matches the task, " + + "for example because the task or the message changed since. Nothing was lost. Press Retry to continue.", + tone: "warning", + nextAction: "Press Retry to continue the task.", +}; const UNSUCCESSFUL_HEARTBEAT_RUN_TERMINAL_STATUSES = [ "interrupted", "failed", @@ -926,13 +933,23 @@ type RecoveryIssue = Omit; const { description: _unusedIssueDescription, ...recoveryIssueFields } = getTableColumns(issues); +/** Why the stranded sweep could not queue a successor. A spent budget and a + * refused continuation both end automatic recovery for that run, so the sweep + * escalates the task once instead of trying again on every pass. */ +export type StrandedRecoveryOutcome = { + retryExhausted?: boolean; + /** The person's original Retry or message no longer authorizes this run's + * continuation (for example the task or its message changed since). */ + continuationRefused?: boolean; +}; + export function recoveryService( db: Db, deps: { enqueueWakeup: RecoveryWakeup; scheduleRecoveryRetry?: ( runId: string, - outcome?: { retryExhausted?: boolean }, + outcome?: StrandedRecoveryOutcome, ) => Promise; /** Settle retained explicit retry claims through the queue-first release policy. */ settleExplicitContinuationRetry?: (run: typeof heartbeatRuns.$inferSelect) => Promise; @@ -1099,6 +1116,21 @@ export function recoveryService( return { consecutive, latestFinishedAt }; } + /** When a terminal legacy run's executor is still finishing it, the time to + * look again; null when nothing settles it any more. In this process that is + * a live execution; another process proves it with its unexpired controller + * lease, which it clears once settlement (retry scheduling, claim release) + * is done. This process's own lease without a live execution is stale. */ + async function terminalRunSettlingUntil(companyId: string, runId: string): Promise { + if (deps.liveRunExecutions?.has(runId)) return new Date(Date.now() + 30_000); + const [run] = await db.select({ leaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.runtimeMode, "legacy"), sql`${heartbeatRuns.controllerBootId} is not null`, + sql`${heartbeatRuns.controllerBootId} <> ${legacyControllerBootId}`, + sql`${heartbeatRuns.controllerLeaseExpiresAt} > now()`)); + return run?.leaseExpiresAt ?? null; + } + async function hasActiveExecutionPath( companyId: string, issueId: string, @@ -1146,6 +1178,10 @@ export function recoveryService( eq(nativeRunFinalizations.issueId, issueId), isNull(nativeRunFinalizations.resultId), agentId ? eq(heartbeatRuns.agentId, agentId) : sql`true`, + // A person's Retry or message already replaced this stopped run; + // the run it admitted owns the task, not this leftover state. + sql`not (coalesce(${nativeRunFinalizations.failureDetail}->>'replacementDenied', '') = 'explicit_user_continuation' + and ${heartbeatRuns.status} in ('failed', 'cancelled', 'timed_out', 'interrupted', 'succeeded'))`, or( inArray(nativeRunFinalizations.recoveryState, [ "awaiting_evidence", @@ -2013,7 +2049,7 @@ export function recoveryService( * budget. A null return without it means another authority owns the * run (native runtime, reconciliation) and the caller must leave it. */ - outcome?: { retryExhausted?: boolean }; + outcome?: StrandedRecoveryOutcome; }) { if (input.retryOfRunId) { const [predecessor] = await db @@ -2043,9 +2079,10 @@ export function recoveryService( return null; } if (deps.scheduleRecoveryRetry) { - const scheduling: { retryExhausted?: boolean } = {}; + const scheduling: StrandedRecoveryOutcome = {}; const retry = await deps.scheduleRecoveryRetry(predecessor.id, scheduling); if (retry) return retry; + if (input.outcome && scheduling.continuationRefused) input.outcome.continuationRefused = true; // A spent budget is the one "no retry" the sweeper must not wait // out: nothing else will ever queue a successor for this run, so // it reports the exhaustion instead of leaving the issue with no @@ -4546,6 +4583,70 @@ export function recoveryService( ); } + /** + * A person's Retry or message admitted a legacy run that a graceful shutdown + * (an upgrade) then stopped before provider work. That person already + * authorized the work and chose a fresh session, so the task's earlier + * session state (an unfinished goal, a superseded native session) does not + * own it. Offer each such task to the bounded scheduler directly; its + * admission keeps every authorization check and creates one continuation. + * A refusal escalates the task once with a working Retry. + */ + async function resumeShutdownStoppedRetryPreparations( + candidates: RecoveryIssue[], + result: { continuationRequeued: number; escalated: number; issueIds: string[] }, + onDeferredDeadline?: (at: number) => void, + ) { + const handled = new Set(); + const waiting = candidates.filter(issue => issue.status === "in_progress" && issue.assigneeAgentId + && !issue.executionRunId && !issue.conversationAgentId && issue.originKind !== "chat_channel"); + if (!waiting.length) return handled; + const stopped = await db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.issueId, + contextIssueId: sql`${heartbeatRuns.contextSnapshot}->>'issueId'` }).from(heartbeatRuns).where(and( + inArray(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`, waiting.map(issue => issue.id)), + eq(heartbeatRuns.status, "interrupted"), eq(heartbeatRuns.errorCode, "server_shutdown_interrupted"), + eq(heartbeatRuns.runtimeMode, "legacy"), + sql`${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'`, + sql`${heartbeatRuns.resultJson}#>>'{executionRecovery,providerWorkStarted}' = 'false'`, + sql`jsonb_typeof(${heartbeatRuns.contextSnapshot}->'explicitUserContinuation') = 'object'`, + )); + for (const run of stopped) { + const issue = waiting.find(row => row.id === run.contextIssueId); + if (!issue?.assigneeAgentId || handled.has(issue.id)) continue; + const latestRun = await getLatestIssueRun(issue.companyId, issue.id); + if (latestRun?.id !== run.id || latestRun.agentId !== issue.assigneeAgentId) continue; + const agent = await getAgent(issue.assigneeAgentId); + if (!agent || agent.companyId !== issue.companyId || !(await isAgentInvokable(agent)) + || await hasActiveExecutionPath(issue.companyId, issue.id) + || (await recoveryActionsSvc.getActiveForIssue(issue.companyId, issue.id))?.ownerType === "board" + || await isAutomaticRecoverySuppressedByPauseHold(db, issue.companyId, issue.id, treeControlSvc)) continue; + // Another server is still settling the run: look again when its lease + // lapses, since later generic skips may never reach their own check. + const settlingUntil = await terminalRunSettlingUntil(issue.companyId, run.id); + if (settlingUntil) { + onDeferredDeadline?.(settlingUntil.getTime()); + handled.add(issue.id); + continue; + } + handled.add(issue.id); + const outcome: StrandedRecoveryOutcome = {}; + const queued = await enqueueStrandedIssueRecovery({ issueId: issue.id, agentId: issue.assigneeAgentId, + reason: "issue_continuation_needed", retryReason: "issue_continuation_needed", + source: "issue.shutdown_stopped_retry_preparation", retryOfRunId: run.id, outcome }); + if (queued) { + result.continuationRequeued += 1; + result.issueIds.push(issue.id); + } else if ((outcome.continuationRefused || outcome.retryExhausted) + && !(await hasActiveExecutionPath(issue.companyId, issue.id)) + && await escalateStrandedAssignedIssue({ issue, previousStatus: "in_progress", latestRun, + notice: outcome.continuationRefused ? CONTINUATION_REFUSED_NOTICE : null })) { + result.escalated += 1; + result.issueIds.push(issue.id); + } + } + return handled; + } + async function reconcileStrandedAssignedIssues(opts?: { companyId?: string; issueId?: string; @@ -4602,6 +4703,8 @@ export function recoveryService( issueIds: [] as string[], }; + const handledIssueIds = await resumeShutdownStoppedRetryPreparations(candidates, result, opts?.onDeferredDeadline); + const candidateIssueIds = candidates.map((issue) => issue.id); const unfinishedGoalBindings = new Set(); if (candidateIssueIds.length > 0) { @@ -4627,6 +4730,7 @@ export function recoveryService( } for (const issue of candidates) { + if (handledIssueIds.has(issue.id)) continue; if (issue.originKind === "chat_channel") { if (opts?.skipExternalConversationSettlement) { @@ -4692,6 +4796,16 @@ export function recoveryService( } let latestRun = await getLatestIssueRun(issue.companyId, issue.id); + // The run's own executor still settles it: it schedules the bounded retry + // and releases the claim itself. Escalating in that window clears the + // claim the retry needs. Re-check once its controller lease lapses. + const settlingUntil = latestRun && isTerminalIssueRun(latestRun) + ? await terminalRunSettlingUntil(issue.companyId, latestRun.id) : null; + if (settlingUntil) { + opts?.onDeferredDeadline?.(settlingUntil.getTime()); + result.skipped += 1; + continue; + } if (latestRun?.agentId === agentId && isInteractionDeliveryDeniedRun(latestRun)) { result.skipped += 1; continue; @@ -5795,7 +5909,7 @@ export function recoveryService( continue; } - const recoveryOutcome: { retryExhausted?: boolean } = {}; + const recoveryOutcome: StrandedRecoveryOutcome = {}; const queued = await enqueueStrandedIssueRecovery({ issueId: issue.id, agentId, @@ -5809,7 +5923,7 @@ export function recoveryService( result.continuationRequeued += 1; result.issueIds.push(issue.id); } else if ( - recoveryOutcome.retryExhausted && + (recoveryOutcome.retryExhausted || recoveryOutcome.continuationRefused) && !(await hasActiveExecutionPath(issue.companyId, issue.id, null)) ) { // The failed run's bounded transient retries are all spent, so no @@ -5822,14 +5936,14 @@ export function recoveryService( issue, previousStatus: "in_progress", latestRun, - notice: { + notice: recoveryOutcome.retryExhausted ? { body: "Paperclip retried this issue's run after it ended without finishing, but the bounded retry budget " + "is spent and it still has no live execution path. " + "Moving it to `blocked` so it is visible for intervention.", title: "No live execution path", tone: "danger", - }, + } : CONTINUATION_REFUSED_NOTICE, }); if (updated) { result.escalated += 1; diff --git a/server/src/services/task-retry.test.ts b/server/src/services/task-retry.test.ts index 41da5162c66..84b4232cbf1 100644 --- a/server/src/services/task-retry.test.ts +++ b/server/src/services/task-retry.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; -import { agents, agentWakeupRequests, chatActions, chatConversations, chatEndpoints, companies, budgetPolicies, costEvents, createDb, type Db, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, workspaceOperations, heartbeatRunEvents, heartbeatRuns, issueComments, issues, approvals, issueApprovals, issueRecoveryActions, nativeRunFinalizations, toolApplications, toolConnections, toolInvocations, applyPendingMigrations } from "@paperclipai/db"; +import { agents, agentTaskSessions, agentWakeupRequests, chatActions, chatConversations, chatEndpoints, companies, budgetPolicies, costEvents, createDb, type Db, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, workspaceOperations, heartbeatRunEvents, heartbeatRuns, issueComments, issues, approvals, issueApprovals, issueRecoveryActions, nativeRunFinalizations, toolApplications, toolConnections, toolInvocations, applyPendingMigrations } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; import { priorRecoveryApiReceipt, unresolvedRecoveryToolInvocation } from "./recovery-effect-receipts.js"; import { requestTaskRetry } from "./task-retry.js"; @@ -300,6 +300,142 @@ describe("durable task Retry admission", () => { expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]!.executionRunId) .toEqual(kind === "claimed_by_other_run" ? expect.any(String) : null); }); + // Staging: a Retry that selected a fresh isolated + // workspace (at admission, or when its preparation found the saved files + // unrestorable) was stopped by an upgrade before provider work. The person + // asked for that fresh attempt and it never ran, so it continues as the same + // attempt: the successor prepares the workspace this Retry selected. + async function seedFreshAttemptStoppedByShutdown(selectedAt: "admission" | "preparation") { + const f = await seedRefusedShutdownContinuation(); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.retryRequestId)); + await db.update(agentWakeupRequests).set({ payload: { ...wake.payload, + taskRetry: { ...(wake.payload!.taskRetry as Record), strategy: "fresh_isolated_workspace" } } }) + .where(eq(agentWakeupRequests.id, f.retryRequestId)); + const readiness = { mode: "fresh_isolated_workspace", retryIntentId: f.retryRequestId, sourceRunId: f.source.id, + retainedSourceIds: [] as string[] }; + if (selectedAt === "admission") await db.update(heartbeatRuns).set({ contextSnapshot: { ...f.parent.contextSnapshot, + explicitUserContinuation: { ...(f.parent.contextSnapshot!.explicitUserContinuation as Record), readiness } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + const [parent] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.parent.id)); + return { ...f, parent, readiness }; + } + it.each(["admission", "preparation"] as const)( + "continues a fresh-workspace Retry that a shutdown stopped before provider work (selected at %s)", async selectedAt => { + const f = await seedFreshAttemptStoppedByShutdown(selectedAt); + const scope = { companyId: f.input.companyId }; + await Promise.all([heartbeatService(db).reconcileStrandedAssignedIssues(scope), + heartbeatService(db).reconcileStrandedAssignedIssues(scope)]); + const successors = await successorsOf(f.parent.id); + expect(successors).toEqual([expect.objectContaining({ status: "scheduled_retry" })]); + const successor = successors[0]!; + expect(successor.contextSnapshot?.explicitUserContinuation).toEqual({ + previousRunId: f.parent.id, commentId: null, taskRetryRequestId: f.retryRequestId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]!.executionRunId).toBe(successor.id); + await expect(dispatchRun(f.input, successor)).resolves.toMatchObject({ interruptedRunId: f.parent.id }); + // Preparation of the successor selects the same fresh attempt. + expect(await readFreshWorkspaceReadiness(db, successor)).toEqual(f.readiness); + // A second upgrade stops the continuation too: it carries the same attempt on. + const finishedAt = new Date(Date.now() - 30_000); + const [stopped] = await db.update(heartbeatRuns).set({ status: "interrupted", errorCode: "server_shutdown_interrupted", + startedAt: finishedAt, finishedAt, runtimeMode: "legacy", executionStage: "preparing", controllerBootId: randomUUID(), + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).where(eq(heartbeatRuns.id, successor.id)).returning(); + await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.input.issueId)); + await heartbeatService(db).reconcileStrandedAssignedIssues(scope); + const next = await successorsOf(stopped.id); + expect(next).toHaveLength(1); + await expect(dispatchRun(f.input, next[0]!)).resolves.toMatchObject({ interruptedRunId: stopped.id }); + expect(await readFreshWorkspaceReadiness(db, next[0]!)).toEqual(f.readiness); + }); + it.each(["other_retry", "other_source", "failed_not_stopped"] as const)( + "keeps a fresh-workspace Retry's own contract when it does not match: %s", async kind => { + const f = await seedFreshAttemptStoppedByShutdown("admission"); + const explicit = f.parent.contextSnapshot!.explicitUserContinuation as Record; + if (kind !== "failed_not_stopped") await db.update(heartbeatRuns).set({ contextSnapshot: { ...f.parent.contextSnapshot, + explicitUserContinuation: { ...explicit, readiness: { ...f.readiness, + ...(kind === "other_retry" ? { retryIntentId: randomUUID() } : { sourceRunId: randomUUID() }) } } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + else await db.update(heartbeatRuns).set({ status: "failed", errorCode: "remote_workspace_restore_failed" }) + .where(eq(heartbeatRuns.id, f.parent.id)); + if (kind === "failed_not_stopped") await db.update(issues).set({ executionRunId: f.parent.id }).where(eq(issues.id, f.input.issueId)); + const result = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: new Date() }); + expect(result.outcome).not.toBe("scheduled"); + expect(await successorsOf(f.parent.id)).toHaveLength(0); + }); + // Staging: the Retry replaced a native session that had + // left session state behind. The sweep treated that state as the task's live + // path and never offered the stopped Retry run its continuation. + it.each(["unfinished_goal", "superseded_native_finalization"] as const)( + "continues a shutdown-stopped Retry whose replaced session left state behind: %s", async state => { + const f = await seedRefusedShutdownContinuation(); + if (state === "unfinished_goal") await db.insert(agentTaskSessions).values({ companyId: f.input.companyId, + agentId: f.parent.agentId, adapterType: "paperclip_runner", taskKey: f.input.issueId, goalStatus: "active" }); + else { + const [native] = await db.insert(heartbeatRuns).values({ companyId: f.input.companyId, agentId: f.parent.agentId, + status: "failed", runtimeMode: "native", errorCode: "native_continuation_requires_reconciliation", nativeIssueId: f.input.issueId, + nativePhase: "retryable_failure", createdAt: new Date(f.parent.createdAt.getTime() - 5000), + finishedAt: new Date(f.parent.createdAt.getTime() - 4000), contextSnapshot: { issueId: f.input.issueId } }).returning(); + await db.insert(nativeRunFinalizations).values({ runId: native.id, companyId: f.input.companyId, issueId: f.input.issueId, + phase: "retryable_failure", attempt: 2, recoveryState: "resuming_session", + failureDetail: { replacementDenied: "explicit_user_continuation" } }); + } + await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + const successors = await successorsOf(f.parent.id); + expect(successors).toEqual([expect.objectContaining({ status: "scheduled_retry" })]); + await expect(dispatchRun(f.input, successors[0]!)).resolves.toMatchObject({ interruptedRunId: f.parent.id }); + }); + it("looks again at a shutdown-stopped Retry run once another server's lease on it lapses", async () => { + const f = await seedRefusedShutdownContinuation(); + await db.insert(agentTaskSessions).values({ companyId: f.input.companyId, agentId: f.parent.agentId, + adapterType: "paperclip_runner", taskKey: f.input.issueId, goalStatus: "active" }); + await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp() + interval '1 minute'` }) + .where(eq(heartbeatRuns.id, f.parent.id)); + const deadlines: number[] = []; + await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId, + onDeferredDeadline: at => deadlines.push(at) }); + expect(await successorsOf(f.parent.id)).toHaveLength(0); + expect(deadlines).toEqual([expect.any(Number)]); + expect(deadlines[0]!).toBeGreaterThan(Date.now()); + await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: null }).where(eq(heartbeatRuns.id, f.parent.id)); + await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect(await successorsOf(f.parent.id)).toHaveLength(1); + }); + // A refusal that cannot succeed later ends automatic recovery once, with a + // plain notice and a Retry that works, instead of a refusal every sweep. + it("escalates a refused shutdown continuation once instead of refusing it on every sweep", async () => { + const f = await seedRefusedShutdownContinuation(); + await db.update(agentWakeupRequests).set({ requestedByActorId: "other-user" }).where(eq(agentWakeupRequests.id, f.retryRequestId)); + const scope = { companyId: f.input.companyId }; + await heartbeatService(db).reconcileStrandedAssignedIssues(scope); + await heartbeatService(db).reconcileStrandedAssignedIssues(scope); + expect(await successorsOf(f.parent.id)).toHaveLength(0); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "blocked" }); + const notices = await db.select().from(issueComments).where(eq(issueComments.issueId, f.input.issueId)); + expect(notices.filter(row => row.body.includes("Press Retry to continue"))).toHaveLength(1); + const refusals = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.parent.id), + sql`${heartbeatRunEvents.message} like 'The automatic retry could not revalidate%'`)); + expect(refusals).toHaveLength(1); + }); + // Staging: a Retry run failed before provider work with a transient + // sandbox resume failure. The sweep escalated the task while the run's own + // executor was still settling it, which cleared the claim its bounded retry + // needed. The sweep now leaves the task to that executor. + it("leaves a failed Retry run to its own executor while that executor still settles it", async () => { + const f = await seedFailedRetryAdmittedRun(); + // The server running the failed run is still settling it: its controller + // lease is live until it has scheduled the retry and released the claim. + await db.update(heartbeatRuns).set({ errorCode: "reusable_sandbox_resume_failed", controllerBootId: randomUUID(), + controllerLeaseExpiresAt: sql`clock_timestamp() + interval '1 minute'`, + }).where(eq(heartbeatRuns.id, f.parent.id)); + await db.update(issues).set({ status: "in_progress", executionRunId: f.parent.id }).where(eq(issues.id, f.input.issueId)); + await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]) + .toMatchObject({ status: "in_progress", executionRunId: f.parent.id }); + const retry = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: new Date() }); + expect(retry.outcome).toBe("scheduled"); + if (retry.outcome !== "scheduled") return; + await expect(dispatchRun(f.input, retry.run)).resolves.toMatchObject({ interruptedRunId: f.parent.id }); + }); async function seedRefusedRetry() { const input = await seed(); const now = Date.now();