From 4dc5492c2d15f290702ba03753bb6155b91cd920 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 22:20:55 -0600 Subject: [PATCH] fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on its next run > - When the resume fails before provider work (`reusable_sandbox_resume_failed`), the run gets a bounded automatic retry > - On a hosted staging server, 8 of 14 such failures in 10 hours did not get that retry. Each task ended `blocked` and needed a person to press Retry > - In all 8, the stranded-task sweep blocked the task between the failed write and the run's own retry scheduling. The block cleared the claim that the retry needed > - This pull request makes the sweep give such a run to the same bounded retry scheduler, not escalate it > - The benefit is that a transient sandbox resume failure no longer costs the task, also when settlement is slow or the server stops during an upgrade ## Linked Issues or Issue Description Refs #21108, #21113, #21078 **What happened?** A person pressed Retry on a task. The run tried to resume the task's saved Daytona sandbox before provider work. The resume failed (`cause=transport_timeout`, or `provider_unavailable status=503`), and the run was written `failed` with `reusable_sandbox_resume_failed`. Then the stranded-task sweep moved the task to `blocked` and cleared its claim. This happened 1 to 43 seconds after the failed write. The run's executor scheduled its bounded retry 2 to 81 seconds after the failed write, and the gate refused it ("The automatic retry could not revalidate the original user continuation."). Read-only audit of 14 failures (all legacy runtime, `executionRecovery: bootstrap, providerWorkStarted: false`): 6 got their retry, and the 5 most recent ones recovered. 8 were refused this way and wait for a person. In one example the run was written failed at 14:23:31, the task was blocked at 14:24:14, and the retry was refused at 14:24:52. The server was being replaced at that time. **Expected behavior** A transient preparation failure before provider work continues with its bounded retry. The sweep must not take the claim away from that retry. When the retry budget is spent, or the continuation is refused, the task escalates once, as before. **Steps to reproduce** 1. Press Retry on a task whose reusable sandbox resume fails before provider work. 2. Let the run's executor be slow to settle the failure (more than the remaining controller lease, at most 60 seconds), or stop its server after the failed write. 3. Let the stranded-task sweep run on another server. The task becomes `blocked`, and no retry is scheduled. **Root cause** `classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and the other transient preparation codes in `NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the executor already scheduled the dedicated bounded retry, so it escalates at once and never asks the scheduler. #21108 added a skip while the executor still settles the run. That skip uses a live local execution or an unexpired controller lease. The lease stops renewing when the run is written terminal (renewal requires `status = 'running'`). So the skip covers at most about 60 seconds after the last renewal, and it covers nothing when the executor's server stops during settlement. **Paperclip version or commit** Staging at `a65644d1e1`. Current `master` has the same code path. **Deployment mode** Hosted (cloud), two servers alive during each upgrade. **Agent adapter(s) involved** `codex_local` on a Daytona sandbox environment. The change is adapter-independent. ## What Changed - `server/src/services/heartbeat/retries.ts`: add `failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of the run-completion decision: a failed legacy run with no native session, bootstrap evidence that provider work did not start, and a transient preparation code. A restore failure counts only when its owner marked it retryable. Codes that an accepted interaction retries itself are excluded. - `server/src/services/recovery/service.ts`: when the latest run is such a run, the sweep does not escalate it as non-retryable. It goes to `scheduleRecoveryRetry`, which is the run's own bounded scheduler. The scheduler reuses a successor that already exists, so the executor and the sweep create one successor. The task escalates when the scheduler reports exhaustion or a refused continuation. When the scheduler declines the retry for another reason (for example a permanent refusal), the task escalates once as a non-retryable failure, as before. The #21108 settling skip still runs first. - `server/src/services/heartbeat.ts`: pass the predicate to the recovery service. - `doc/execution-semantics.md`: document the rule. ## Verification - New tests in `server/src/services/task-retry.test.ts`, in the staging shape: a Retry-admitted run failed with `reusable_sandbox_resume_failed`, bootstrap evidence, and an expired controller lease from another server. - "retries a failed Retry run's transient preparation failure when the sweep reaches it first" (`slow_executor`, `executor_gone`). Red before the fix: the task became `blocked`, as on staging. Green after the fix: the task stays `in_progress` with one `scheduled_retry` successor (`transient_failure`, attempt 1), and dispatch accepts it. For `slow_executor`, the executor's own scheduling then reuses the same successor, and there is still one successor. - "still escalates a failed Retry run's transient preparation failure once its bounded retries are spent": the task becomes `blocked` with no successor. - "escalates a failed Retry run's transient preparation failure once when the scheduler declines it permanently" (review finding): red before the follow-up commit (the task stayed `in_progress` with no run), green after (the task is `blocked` once, and a second sweep adds no notice). - `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass. - `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631 of 633 passed. The 2 failures were 15-second timeouts under parallel load in native Board-wait tests that this change does not touch. They passed when run alone. - `tsc -p server/tsconfig.json --noEmit` passes. - Not run: a live staging reproduction. The staging evidence is read-only (run events, task activity, run records). ## Risks - The sweep can now schedule the bounded retry before the executor does. The scheduler locks the task and the run and reuses an existing successor, so there is never a second successor. All continuation gate checks still apply. - The sweep acts only after the #21108 settling skip lapses, so a healthy run is not affected. No wait is added before a normal turn start. - No schema change. Rows from older releases follow the same rule because the predicate reads only persisted fields. - Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume calls that time out). #21113 handles the in-run wait for idle transport timeouts. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, read-only API reads, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 2 +- server/src/services/heartbeat.ts | 2 + server/src/services/heartbeat/retries.ts | 23 ++++++++++ server/src/services/recovery/service.ts | 58 +++++++++++++++++++----- server/src/services/task-retry.test.ts | 55 ++++++++++++++++++++++ 5 files changed, 127 insertions(+), 13 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 1bb81b0e687..19fed8d72d4 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1718,7 +1718,7 @@ or the exact provenance inside its verified committed saved snapshot. The latter uses one temporary read-only archive verification per preparation when needed. Task relationships alone do not authorize importing or retaining files. -When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. A fresh-workspace Retry (selected at admission, or by its preparation when the saved files could not be restored) is renewable only on this shutdown path: the person asked for that fresh attempt and it never ran, so the successor's receipt records the same workspace selection and the successor prepares the same attempt (a remote fresh workspace is keyed by the Retry, so it is reused). The stranded-task sweep offers every such stopped Retry or message turn to the scheduler before its generic checks, because the person's Retry started a fresh session: state the replaced session left behind (an unfinished session goal, or a native finalization that the Retry already replaced) does not own the task. While a terminal run's own executor still settles it (a live local execution or an unexpired controller lease), the sweep leaves the task alone, so it cannot clear the claim that run's bounded retry needs. When admission refuses, the sweep escalates the task once to `blocked` with a plain notice that asks for Retry, instead of refusing again on every pass. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority. +When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. A fresh-workspace Retry (selected at admission, or by its preparation when the saved files could not be restored) is renewable only on this shutdown path: the person asked for that fresh attempt and it never ran, so the successor's receipt records the same workspace selection and the successor prepares the same attempt (a remote fresh workspace is keyed by the Retry, so it is reused). The stranded-task sweep offers every such stopped Retry or message turn to the scheduler before its generic checks, because the person's Retry started a fresh session: state the replaced session left behind (an unfinished session goal, or a native finalization that the Retry already replaced) does not own the task. While a terminal run's own executor still settles it (a live local execution or an unexpired controller lease), the sweep leaves the task alone, so it cannot clear the claim that run's bounded retry needs. The controller lease stops renewing once the run is written terminal, so a slow settlement, or a server that stops before scheduling, can outlast it. A failed run that stopped before provider work with a transient preparation failure (for example a reusable sandbox that could not be resumed) is then offered to its own bounded retry scheduler, which reuses a successor that already exists; the task escalates only when that retry is exhausted or refused. When admission refuses, the sweep escalates the task once to `blocked` with a plain notice that asks for Retry, instead of refusing again on every pass. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority. Explicit retry admission checks the local parent adapter controller and verified environment cleanup. The existing queue-first release policy may retain only its own terminal execution claim while that cleanup is pending; finalization and the periodic stale-lock sweep reconsider the same policy after cleanup. Generic terminal-lock cleanup and stale checkout adoption leave these claims for that settlement. New user input, reassignment, and a successor's claim take precedence. Exhausted retries and revoked or missing authorization release the terminal claim without requesting another retry. Pending or failed environment cleanup cannot mint a successor receipt. diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 12de5b29fab..a860dae7cc9 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -121,6 +121,7 @@ import { createHeartbeatRecovery, } from "./heartbeat/recovery.js"; import { + failedPreparationOwnsBoundedRetry, failedRunRetryBudgetSpent, isWorkspaceBusyDeferral, createHeartbeatRetries, @@ -1045,6 +1046,7 @@ export function heartbeatService( // or interrupted run that has already consumed every bounded transient // attempt cannot be retried again through this lane. transientRetryBudgetSpent: failedRunRetryBudgetSpent, + ownsPreparationRetry: failedPreparationOwnsBoundedRetry, }); const runDispatch = createRunDispatch(db); const { diff --git a/server/src/services/heartbeat/retries.ts b/server/src/services/heartbeat/retries.ts index 806c866f2af..0dbe6a94803 100644 --- a/server/src/services/heartbeat/retries.ts +++ b/server/src/services/heartbeat/retries.ts @@ -160,6 +160,29 @@ export function isTransientWorkspacePreparationCode(code: string | null | undefi || code === STORAGE_UNAVAILABLE_CODE; } +/** + * The persisted form of the setup-failure decision in run completion: this + * failed legacy run stopped before provider work with a transient preparation + * code, so its own bounded retry (scheduleBoundedRetryForRun) continues it. + * The stranded-task sweep uses it to offer a run whose executor has not + * scheduled that retry (still settling, or its server stopped) to the same + * idempotent scheduler instead of escalating it. + */ +export function failedPreparationOwnsBoundedRetry(run: Pick): boolean { + if (run.status !== "failed" || run.runtimeMode !== "legacy" || run.nativeSessionId !== null || run.runnerInstanceId !== null + || !isTransientWorkspacePreparationCode(run.errorCode)) return false; + const recovery = parseObject(run.resultJson?.executionRecovery); + if (recovery.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false; + // A restore failure is retried only when its owner proved it retryable. + if (run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE) + return parseObject(run.resultJson?.remoteWorkspaceRestore).retryable === true; + // An accepted interaction keeps its own continuation retry. + return !((run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || run.errorCode === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE + || run.errorCode === REPOSITORY_CATALOG_UNAVAILABLE_CODE || run.errorCode === REVIEWED_CHAT_BINDING_NOT_READY) + && isResolvedInteractionContinuationWakeContext(run.contextSnapshot)); +} + export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS = BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length; diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index dc5eaffb7fb..56c6698ddc6 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -893,6 +893,17 @@ function isOperatorCancelledRun( ); } +function nonRetryableContinuationNotice(errorCode: string | null) { + return { + body: + "Paperclip detected a non-retryable failure on this issue's continuation run " + + `(\`${errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` + + "so it is visible for intervention.", + title: "Continuation failed", + tone: "danger" as const, + }; +} + function isUnsuccessfulTerminalIssueRun(latestRun: LatestIssueRun) { return Boolean( latestRun && @@ -965,6 +976,14 @@ export function recoveryService( transientRetryBudgetSpent?: ( run: typeof heartbeatRuns.$inferSelect, ) => boolean; + /** + * Whether a failed run stopped before provider work with a transient + * preparation failure that its own bounded retry continues. The sweep + * offers such a run to `scheduleRecoveryRetry` instead of escalating it. + */ + ownsPreparationRetry?: ( + run: typeof heartbeatRuns.$inferSelect, + ) => boolean; liveRunExecutions?: Readonly<{ has(id: string): boolean }>; /** * The one settle path for a `running` legacy run whose adapter child is @@ -1133,6 +1152,13 @@ export function recoveryService( return run?.leaseExpiresAt ?? null; } + async function failedRunOwnsPreparationRetry(companyId: string, runId: string): Promise { + if (!deps.ownsPreparationRetry || !deps.scheduleRecoveryRetry) return false; + const [run] = await db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId))); + return Boolean(run && deps.ownsPreparationRetry(run)); + } + async function hasActiveExecutionPath( companyId: string, issueId: string, @@ -5814,6 +5840,7 @@ export function recoveryService( } continue; } + let preparationRetry = false; if (isUnsuccessfulTerminalIssueRun(latestRun)) { const classification = classifyContinuationFailure(latestRun); @@ -5841,19 +5868,22 @@ export function recoveryService( continue; } - if (classification.kind === "non_retryable") { + // A failed run that stopped before provider work with a transient + // preparation failure is continued by its own bounded retry. Its + // executor schedules that retry after the failed write, and a slow + // settlement or a stopped server can leave it unscheduled once the + // settling check above lapses. Offer it to the same idempotent + // scheduler below (one successor, the same budget); only exhaustion + // or a refused continuation escalates it, and any other refusal + // escalates it as before. + preparationRetry = latestRun!.agentId === agentId + && await failedRunOwnsPreparationRetry(issue.companyId, latestRun!.id); + if (classification.kind === "non_retryable" && !preparationRetry) { const updated = await escalateStrandedAssignedIssue({ issue, previousStatus: "in_progress", latestRun, - notice: { - body: - "Paperclip detected a non-retryable failure on this issue's continuation run " + - `(\`${classification.errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` + - "so it is visible for intervention.", - title: "Continuation failed", - tone: "danger", - }, + notice: nonRetryableContinuationNotice(classification.errorCode), }); if (updated) { result.escalated += 1; @@ -5864,7 +5894,7 @@ export function recoveryService( continue; } - if (didAutomaticRecoveryFail(latestRun, "issue_continuation_needed")) { + if (!preparationRetry && didAutomaticRecoveryFail(latestRun, "issue_continuation_needed")) { const { consecutive, latestFinishedAt } = await summarizeRecentContinuationRetries( issue.companyId, @@ -5930,7 +5960,7 @@ export function recoveryService( result.continuationRequeued += 1; result.issueIds.push(issue.id); } else if ( - (recoveryOutcome.retryExhausted || recoveryOutcome.continuationRefused) && + (recoveryOutcome.retryExhausted || recoveryOutcome.continuationRefused || preparationRetry) && !(await hasActiveExecutionPath(issue.companyId, issue.id, null)) ) { // The failed run's bounded transient retries are all spent, so no @@ -5950,7 +5980,11 @@ export function recoveryService( "Moving it to `blocked` so it is visible for intervention.", title: "No live execution path", tone: "danger", - } : CONTINUATION_REFUSED_NOTICE, + } : recoveryOutcome.continuationRefused ? CONTINUATION_REFUSED_NOTICE + // The scheduler declined the preparation retry for another + // reason (for example a durable owner or a permanent refusal): + // escalate it as a non-retryable failure, as before. + : nonRetryableContinuationNotice(latestRun?.errorCode ?? null), }); if (updated) { result.escalated += 1; diff --git a/server/src/services/task-retry.test.ts b/server/src/services/task-retry.test.ts index 84b4232cbf1..c3145dc846d 100644 --- a/server/src/services/task-retry.test.ts +++ b/server/src/services/task-retry.test.ts @@ -436,6 +436,61 @@ describe("durable task Retry admission", () => { if (retry.outcome !== "scheduled") return; await expect(dispatchRun(f.input, retry.run)).resolves.toMatchObject({ interruptedRunId: f.parent.id }); }); + // Staging shape: a person's Retry failed before provider work because the + // saved sandbox resume timed out. The run is already `failed`, but its + // executor has not scheduled the bounded retry yet: settling took longer than + // the controller lease that stopped renewing at the failed write, or the + // server running it stopped. Another server's sweep then reads the task. + it.each(["slow_executor", "executor_gone"] as const)( + "retries a failed Retry run's transient preparation failure when the sweep reaches it first: %s", async executor => { + const f = await seedFailedRetryAdmittedRun(); + await db.update(heartbeatRuns).set({ errorCode: "reusable_sandbox_resume_failed", controllerBootId: randomUUID(), + controllerLeaseExpiresAt: sql`clock_timestamp() - interval '1 second'`, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + const heartbeat = heartbeatService(db); + await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "in_progress" }); + const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id)); + expect(successors).toHaveLength(1); + expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "transient_failure", + scheduledRetryAttempt: 1 }); + if (executor === "slow_executor") { + // The executor's own scheduling adopts the sweep's successor. + expect(await heartbeat.scheduleBoundedRetry(f.parent.id, { now: new Date() })).toMatchObject({ + outcome: "scheduled", run: { id: successors[0].id } }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(1); + } + await expect(dispatchRun(f.input, successors[0])).resolves.toMatchObject({ interruptedRunId: f.parent.id }); + }); + it.each(["reusable_sandbox_resume_failed", "repository_catalog_unavailable"])( + "escalates a failed Retry run's transient preparation failure once when the scheduler declines it permanently: %s", async errorCode => { + const f = await seedFailedRetryAdmittedRun(); + // An external Dot execution is never retried by the bounded scheduler. + await db.update(heartbeatRuns).set({ errorCode, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, + nativeExecutionInput: { provider: { kind: "openai_dot" } } }, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + const heartbeat = heartbeatService(db); + await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "blocked" }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0); + const notices = async () => (await db.select().from(issueComments).where(eq(issueComments.issueId, f.input.issueId))).length; + const escalated = await notices(); + await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect(await notices()).toBe(escalated); + }); + it("still escalates a failed Retry run's transient preparation failure once its bounded retries are spent", async () => { + const f = await seedFailedRetryAdmittedRun(); + await db.update(heartbeatRuns).set({ errorCode: "reusable_sandbox_resume_failed", + scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 2, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + }).where(eq(heartbeatRuns.id, f.parent.id)); + await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId }); + expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "blocked" }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0); + }); async function seedRefusedRetry() { const input = await seed(); const now = Date.now();