diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 1bb81b0e687..19fed8d72d4 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1718,7 +1718,7 @@ or the exact provenance inside its verified committed saved snapshot. The latter uses one temporary read-only archive verification per preparation when needed. Task relationships alone do not authorize importing or retaining files. -When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. A fresh-workspace Retry (selected at admission, or by its preparation when the saved files could not be restored) is renewable only on this shutdown path: the person asked for that fresh attempt and it never ran, so the successor's receipt records the same workspace selection and the successor prepares the same attempt (a remote fresh workspace is keyed by the Retry, so it is reused). The stranded-task sweep offers every such stopped Retry or message turn to the scheduler before its generic checks, because the person's Retry started a fresh session: state the replaced session left behind (an unfinished session goal, or a native finalization that the Retry already replaced) does not own the task. While a terminal run's own executor still settles it (a live local execution or an unexpired controller lease), the sweep leaves the task alone, so it cannot clear the claim that run's bounded retry needs. When admission refuses, the sweep escalates the task once to `blocked` with a plain notice that asks for Retry, instead of refusing again on every pass. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority. +When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. A fresh-workspace Retry (selected at admission, or by its preparation when the saved files could not be restored) is renewable only on this shutdown path: the person asked for that fresh attempt and it never ran, so the successor's receipt records the same workspace selection and the successor prepares the same attempt (a remote fresh workspace is keyed by the Retry, so it is reused). The stranded-task sweep offers every such stopped Retry or message turn to the scheduler before its generic checks, because the person's Retry started a fresh session: state the replaced session left behind (an unfinished session goal, or a native finalization that the Retry already replaced) does not own the task. While a terminal run's own executor still settles it (a live local execution or an unexpired controller lease), the sweep leaves the task alone, so it cannot clear the claim that run's bounded retry needs. The controller lease stops renewing once the run is written terminal, so a slow settlement, or a server that stops before scheduling, can outlast it. A failed run that stopped before provider work with a transient preparation failure (for example a reusable sandbox that could not be resumed) is then offered to its own bounded retry scheduler, which reuses a successor that already exists; the task escalates only when that retry is exhausted or refused. When admission refuses, the sweep escalates the task once to `blocked` with a plain notice that asks for Retry, instead of refusing again on every pass. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority. Explicit retry admission checks the local parent adapter controller and verified environment cleanup. The existing queue-first release policy may retain only its own terminal execution claim while that cleanup is pending; finalization and the periodic stale-lock sweep reconsider the same policy after cleanup. Generic terminal-lock cleanup and stale checkout adoption leave these claims for that settlement. New user input, reassignment, and a successor's claim take precedence. Exhausted retries and revoked or missing authorization release the terminal claim without requesting another retry. Pending or failed environment cleanup cannot mint a successor receipt. diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 12de5b29fab..a860dae7cc9 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -121,6 +121,7 @@ import { createHeartbeatRecovery, } from "./heartbeat/recovery.js"; import { + failedPreparationOwnsBoundedRetry, failedRunRetryBudgetSpent, isWorkspaceBusyDeferral, createHeartbeatRetries, @@ -1045,6 +1046,7 @@ export function heartbeatService( // or interrupted run that has already consumed every bounded transient // attempt cannot be retried again through this lane. transientRetryBudgetSpent: failedRunRetryBudgetSpent, + ownsPreparationRetry: failedPreparationOwnsBoundedRetry, }); const runDispatch = createRunDispatch(db); const { diff --git a/server/src/services/heartbeat/retries.ts b/server/src/services/heartbeat/retries.ts index 806c866f2af..0dbe6a94803 100644 --- a/server/src/services/heartbeat/retries.ts +++ b/server/src/services/heartbeat/retries.ts @@ -160,6 +160,29 @@ export function isTransientWorkspacePreparationCode(code: string | null | undefi || code === STORAGE_UNAVAILABLE_CODE; } +/** + * The persisted form of the setup-failure decision in run completion: this + * failed legacy run stopped before provider work with a transient preparation + * code, so its own bounded retry (scheduleBoundedRetryForRun) continues it. + * The stranded-task sweep uses it to offer a run whose executor has not + * scheduled that retry (still settling, or its server stopped) to the same + * idempotent scheduler instead of escalating it. + */ +export function failedPreparationOwnsBoundedRetry(run: Pick): boolean { + if (run.status !== "failed" || run.runtimeMode !== "legacy" || run.nativeSessionId !== null || run.runnerInstanceId !== null + || !isTransientWorkspacePreparationCode(run.errorCode)) return false; + const recovery = parseObject(run.resultJson?.executionRecovery); + if (recovery.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false; + // A restore failure is retried only when its owner proved it retryable. + if (run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE) + return parseObject(run.resultJson?.remoteWorkspaceRestore).retryable === true; + // An accepted interaction keeps its own continuation retry. + return !((run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || run.errorCode === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE + || run.errorCode === REPOSITORY_CATALOG_UNAVAILABLE_CODE || run.errorCode === REVIEWED_CHAT_BINDING_NOT_READY) + && isResolvedInteractionContinuationWakeContext(run.contextSnapshot)); +} + export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS = BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length; diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index dc5eaffb7fb..56c6698ddc6 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -893,6 +893,17 @@ function isOperatorCancelledRun( ); } +function nonRetryableContinuationNotice(errorCode: string | null) { + return { + body: + "Paperclip detected a non-retryable failure on this issue's continuation run " + + `(\`${errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` + + "so it is visible for intervention.", + title: "Continuation failed", + tone: "danger" as const, + }; +} + function isUnsuccessfulTerminalIssueRun(latestRun: LatestIssueRun) { return Boolean( latestRun && @@ -965,6 +976,14 @@ export function recoveryService( transientRetryBudgetSpent?: ( run: typeof heartbeatRuns.$inferSelect, ) => boolean; + /** + * Whether a failed run stopped before provider work with a transient + * preparation failure that its own bounded retry continues. The sweep + * offers such a run to `scheduleRecoveryRetry` instead of escalating it. + */ + ownsPreparationRetry?: ( + run: typeof heartbeatRuns.$inferSelect, + ) => boolean; liveRunExecutions?: Readonly<{ has(id: string): boolean }>; /** * The one settle path for a `running` legacy run whose adapter child is @@ -1133,6 +1152,13 @@ export function recoveryService( return run?.leaseExpiresAt ?? null; } + async function failedRunOwnsPreparationRetry(companyId: string, runId: string): Promise { + if (!deps.ownsPreparationRetry || !deps.scheduleRecoveryRetry) return false; + const [run] = await db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId))); + return Boolean(run && deps.ownsPreparationRetry(run)); + } + async function hasActiveExecutionPath( companyId: string, issueId: string, @@ -5814,6 +5840,7 @@ export function recoveryService( } continue; } + let preparationRetry = false; if (isUnsuccessfulTerminalIssueRun(latestRun)) { const classification = classifyContinuationFailure(latestRun); @@ -5841,19 +5868,22 @@ export function recoveryService( continue; } - if (classification.kind === "non_retryable") { + // A failed run that stopped before provider work with a transient + // preparation failure is continued by its own bounded retry. Its + // executor schedules that retry after the failed write, and a slow + // settlement or a stopped server can leave it unscheduled once the + // settling check above lapses. Offer it to the same idempotent + // scheduler below (one successor, the same budget); only exhaustion + // or a refused continuation escalates it, and any other refusal + // escalates it as before. + preparationRetry = latestRun!.agentId === agentId + && await failedRunOwnsPreparationRetry(issue.companyId, latestRun!.id); + if (classification.kind === "non_retryable" && !preparationRetry) { const updated = await escalateStrandedAssignedIssue({ issue, previousStatus: "in_progress", latestRun, - notice: { - body: - "Paperclip detected a non-retryable failure on this issue's continuation run " + - `(\`${classification.errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` + - "so it is visible for intervention.", - title: "Continuation failed", - tone: "danger", - }, + notice: nonRetryableContinuationNotice(classification.errorCode), }); if (updated) { result.escalated += 1; @@ -5864,7 +5894,7 @@ export function recoveryService( continue; } - if (didAutomaticRecoveryFail(latestRun, "issue_continuation_needed")) { + if (!preparationRetry && didAutomaticRecoveryFail(latestRun, "issue_continuation_needed")) { const { consecutive, latestFinishedAt } = await summarizeRecentContinuationRetries( issue.companyId, @@ -5930,7 +5960,7 @@ export function recoveryService( result.continuationRequeued += 1; result.issueIds.push(issue.id); } else if ( - (recoveryOutcome.retryExhausted || recoveryOutcome.continuationRefused) && + (recoveryOutcome.retryExhausted || recoveryOutcome.continuationRefused || preparationRetry) && !(await hasActiveExecutionPath(issue.companyId, issue.id, null)) ) { // The failed run's bounded transient retries are all spent, so no @@ -5950,7 +5980,11 @@ export function recoveryService( "Moving it to `blocked` so it is visible for intervention.", title: "No live execution path", tone: "danger", - } : CONTINUATION_REFUSED_NOTICE, + } : recoveryOutcome.continuationRefused ? CONTINUATION_REFUSED_NOTICE + // The scheduler declined the preparation retry for another + // reason (for example a durable owner or a permanent refusal): + // escalate it as a non-retryable failure, as before. + : nonRetryableContinuationNotice(latestRun?.errorCode ?? null), }); if (updated) { result.escalated += 1; diff --git a/server/src/services/task-retry.test.ts b/server/src/services/task-retry.test.ts index 84b4232cbf1..c3145dc846d 100644 --- a/server/src/services/task-retry.test.ts +++ b/server/src/services/task-retry.test.ts @@ -436,6 +436,61 @@ describe("durable task Retry admission", () => { if (retry.outcome !== "scheduled") return; await expect(dispatchRun(f.input, retry.run)).resolves.toMatchObject({ interruptedRunId: f.parent.id }); }); + // Staging shape: a person's Retry failed before provider work because the + // saved sandbox resume timed out. The run is already `failed`, but its + // executor has not scheduled the bounded retry yet: settling took longer than + // the controller lease that stopped renewing at the failed write, or the + // server running it stopped. Another server's sweep then reads the task. + it.each(["slow_executor", "executor_gone"] as const)( + "retries a failed Retry run's transient preparation failure when the sweep reaches it first: %s", async executor => { + const f = await seedFailedRetryAdmittedRun(); + await db.update(heartbeatRuns).set({ errorCode: "reusable_sandbox_resume_failed", controllerBootId: randomUUID(), + controllerLeaseExpiresAt: sql`clock_timestamp() - interval '1 second'`, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + const heartbeat = heartbeatService(db); + await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "in_progress" }); + const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id)); + expect(successors).toHaveLength(1); + expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "transient_failure", + scheduledRetryAttempt: 1 }); + if (executor === "slow_executor") { + // The executor's own scheduling adopts the sweep's successor. + expect(await heartbeat.scheduleBoundedRetry(f.parent.id, { now: new Date() })).toMatchObject({ + outcome: "scheduled", run: { id: successors[0].id } }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(1); + } + await expect(dispatchRun(f.input, successors[0])).resolves.toMatchObject({ interruptedRunId: f.parent.id }); + }); + it.each(["reusable_sandbox_resume_failed", "repository_catalog_unavailable"])( + "escalates a failed Retry run's transient preparation failure once when the scheduler declines it permanently: %s", async errorCode => { + const f = await seedFailedRetryAdmittedRun(); + // An external Dot execution is never retried by the bounded scheduler. + await db.update(heartbeatRuns).set({ errorCode, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, + nativeExecutionInput: { provider: { kind: "openai_dot" } } }, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + const heartbeat = heartbeatService(db); + await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "blocked" }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0); + const notices = async () => (await db.select().from(issueComments).where(eq(issueComments.issueId, f.input.issueId))).length; + const escalated = await notices(); + await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect(await notices()).toBe(escalated); + }); + it("still escalates a failed Retry run's transient preparation failure once its bounded retries are spent", async () => { + const f = await seedFailedRetryAdmittedRun(); + await db.update(heartbeatRuns).set({ errorCode: "reusable_sandbox_resume_failed", + scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 2, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "blocked" }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0); + }); async function seedRefusedRetry() { const input = await seed(); const now = Date.now();