fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124)

<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on its next run
> - When the resume fails before provider work
(`reusable_sandbox_resume_failed`), the run gets a bounded automatic
retry
> - On a hosted staging server, 8 of 14 such failures in 10 hours did
not get that retry. Each task ended `blocked` and needed a person to
press Retry
> - In all 8, the stranded-task sweep blocked the task between the
failed write and the run's own retry scheduling. The block cleared the
claim that the retry needed
> - This pull request makes the sweep give such a run to the same
bounded retry scheduler, not escalate it
> - The benefit is that a transient sandbox resume failure no longer
costs the task, also when settlement is slow or the server stops during
an upgrade

## Linked Issues or Issue Description

Refs #21108, #21113, #21078

**What happened?**
A person pressed Retry on a task. The run tried to resume the task's
saved Daytona sandbox before provider work. The resume failed
(`cause=transport_timeout`, or `provider_unavailable status=503`), and
the run was written `failed` with `reusable_sandbox_resume_failed`. Then
the stranded-task sweep moved the task to `blocked` and cleared its
claim. This happened 1 to 43 seconds after the failed write. The run's
executor scheduled its bounded retry 2 to 81 seconds after the failed
write, and the gate refused it ("The automatic retry could not
revalidate the original user continuation."). Read-only audit of 14
failures (all legacy runtime, `executionRecovery: bootstrap,
providerWorkStarted: false`): 6 got their retry, and the 5 most recent
ones recovered. 8 were refused this way and wait for a person. In one
example the run was written failed at 14:23:31, the task was blocked at
14:24:14, and the retry was refused at 14:24:52. The server was being
replaced at that time.

**Expected behavior**
A transient preparation failure before provider work continues with its
bounded retry. The sweep must not take the claim away from that retry.
When the retry budget is spent, or the continuation is refused, the task
escalates once, as before.

**Steps to reproduce**
1. Press Retry on a task whose reusable sandbox resume fails before
provider work.
2. Let the run's executor be slow to settle the failure (more than the
remaining controller lease, at most 60 seconds), or stop its server
after the failed write.
3. Let the stranded-task sweep run on another server. The task becomes
`blocked`, and no retry is scheduled.

**Root cause**
`classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and
the other transient preparation codes in
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the
executor already scheduled the dedicated bounded retry, so it escalates
at once and never asks the scheduler. #21108 added a skip while the
executor still settles the run. That skip uses a live local execution or
an unexpired controller lease. The lease stops renewing when the run is
written terminal (renewal requires `status = 'running'`). So the skip
covers at most about 60 seconds after the last renewal, and it covers
nothing when the executor's server stops during settlement.

**Paperclip version or commit**
Staging at `a65644d1e1`. Current `master` has the same code path.

**Deployment mode**
Hosted (cloud), two servers alive during each upgrade.

**Agent adapter(s) involved**
`codex_local` on a Daytona sandbox environment. The change is
adapter-independent.

## What Changed

- `server/src/services/heartbeat/retries.ts`: add
`failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of
the run-completion decision: a failed legacy run with no native session,
bootstrap evidence that provider work did not start, and a transient
preparation code. A restore failure counts only when its owner marked it
retryable. Codes that an accepted interaction retries itself are
excluded.
- `server/src/services/recovery/service.ts`: when the latest run is such
a run, the sweep does not escalate it as non-retryable. It goes to
`scheduleRecoveryRetry`, which is the run's own bounded scheduler. The
scheduler reuses a successor that already exists, so the executor and
the sweep create one successor. The task escalates when the scheduler
reports exhaustion or a refused continuation. When the scheduler
declines the retry for another reason (for example a permanent refusal),
the task escalates once as a non-retryable failure, as before. The
#21108 settling skip still runs first.
- `server/src/services/heartbeat.ts`: pass the predicate to the recovery
service.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New tests in `server/src/services/task-retry.test.ts`, in the staging
shape: a Retry-admitted run failed with
`reusable_sandbox_resume_failed`, bootstrap evidence, and an expired
controller lease from another server.
- "retries a failed Retry run's transient preparation failure when the
sweep reaches it first" (`slow_executor`, `executor_gone`). Red before
the fix: the task became `blocked`, as on staging. Green after the fix:
the task stays `in_progress` with one `scheduled_retry` successor
(`transient_failure`, attempt 1), and dispatch accepts it. For
`slow_executor`, the executor's own scheduling then reuses the same
successor, and there is still one successor.
- "still escalates a failed Retry run's transient preparation failure
once its bounded retries are spent": the task becomes `blocked` with no
successor.
- "escalates a failed Retry run's transient preparation failure once
when the scheduler declines it permanently" (review finding): red before
the follow-up commit (the task stayed `in_progress` with no run), green
after (the task is `blocked` once, and a second sweep adds no notice).
- `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass.
- `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631
of 633 passed. The 2 failures were 15-second timeouts under parallel
load in native Board-wait tests that this change does not touch. They
passed when run alone.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live staging reproduction. The staging evidence is
read-only (run events, task activity, run records).

## Risks

- The sweep can now schedule the bounded retry before the executor does.
The scheduler locks the task and the run and reuses an existing
successor, so there is never a second successor. All continuation gate
checks still apply.
- The sweep acts only after the #21108 settling skip lapses, so a
healthy run is not affected. No wait is added before a normal turn
start.
- No schema change. Rows from older releases follow the same rule
because the predicate reads only persisted fields.
- Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume
calls that time out). #21113 handles the in-run wait for idle transport
timeouts.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 22:20:55 -06:00
1 parent afe86e9ded
commit 4dc5492c2d
5 files changed
+127 -13

No files matched your search

+1 -1
View File
@@ -1718,7 +1718,7 @@ or the exact provenance inside its verified committed saved snapshot. The latter
uses one temporary read-only archive verification per preparation when needed.
Task relationships alone do not authorize importing or retaining files.
When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. A fresh-workspace Retry (selected at admission, or by its preparation when the saved files could not be restored) is renewable only on this shutdown path: the person asked for that fresh attempt and it never ran, so the successor's receipt records the same workspace selection and the successor prepares the same attempt (a remote fresh workspace is keyed by the Retry, so it is reused). The stranded-task sweep offers every such stopped Retry or message turn to the scheduler before its generic checks, because the person's Retry started a fresh session: state the replaced session left behind (an unfinished session goal, or a native finalization that the Retry already replaced) does not own the task. While a terminal run's own executor still settles it (a live local execution or an unexpired controller lease), the sweep leaves the task alone, so it cannot clear the claim that run's bounded retry needs. When admission refuses, the sweep escalates the task once to `blocked` with a plain notice that asks for Retry, instead of refusing again on every pass. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority.
When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. A fresh-workspace Retry (selected at admission, or by its preparation when the saved files could not be restored) is renewable only on this shutdown path: the person asked for that fresh attempt and it never ran, so the successor's receipt records the same workspace selection and the successor prepares the same attempt (a remote fresh workspace is keyed by the Retry, so it is reused). The stranded-task sweep offers every such stopped Retry or message turn to the scheduler before its generic checks, because the person's Retry started a fresh session: state the replaced session left behind (an unfinished session goal, or a native finalization that the Retry already replaced) does not own the task. While a terminal run's own executor still settles it (a live local execution or an unexpired controller lease), the sweep leaves the task alone, so it cannot clear the claim that run's bounded retry needs. The controller lease stops renewing once the run is written terminal, so a slow settlement, or a server that stops before scheduling, can outlast it. A failed run that stopped before provider work with a transient preparation failure (for example a reusable sandbox that could not be resumed) is then offered to its own bounded retry scheduler, which reuses a successor that already exists; the task escalates only when that retry is exhausted or refused. When admission refuses, the sweep escalates the task once to `blocked` with a plain notice that asks for Retry, instead of refusing again on every pass. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority.
Explicit retry admission checks the local parent adapter controller and verified environment cleanup. The existing queue-first release policy may retain only its own terminal execution claim while that cleanup is pending; finalization and the periodic stale-lock sweep reconsider the same policy after cleanup. Generic terminal-lock cleanup and stale checkout adoption leave these claims for that settlement. New user input, reassignment, and a successor's claim take precedence. Exhausted retries and revoked or missing authorization release the terminal claim without requesting another retry. Pending or failed environment cleanup cannot mint a successor receipt.
+2
View File
@@ -121,6 +121,7 @@ import {
createHeartbeatRecovery,
} from "./heartbeat/recovery.js";
import {
failedPreparationOwnsBoundedRetry,
failedRunRetryBudgetSpent,
isWorkspaceBusyDeferral,
createHeartbeatRetries,
@@ -1045,6 +1046,7 @@ export function heartbeatService(
// or interrupted run that has already consumed every bounded transient
// attempt cannot be retried again through this lane.
transientRetryBudgetSpent: failedRunRetryBudgetSpent,
ownsPreparationRetry: failedPreparationOwnsBoundedRetry,
});
const runDispatch = createRunDispatch(db);
const {
+23
View File
@@ -160,6 +160,29 @@ export function isTransientWorkspacePreparationCode(code: string | null | undefi
|| code === STORAGE_UNAVAILABLE_CODE;
}
/**
* The persisted form of the setup-failure decision in run completion: this
* failed legacy run stopped before provider work with a transient preparation
* code, so its own bounded retry (scheduleBoundedRetryForRun) continues it.
* The stranded-task sweep uses it to offer a run whose executor has not
* scheduled that retry (still settling, or its server stopped) to the same
* idempotent scheduler instead of escalating it.
*/
export function failedPreparationOwnsBoundedRetry(run: Pick<HeartbeatRun,
"status" | "runtimeMode" | "nativeSessionId" | "runnerInstanceId" | "errorCode" | "resultJson" | "contextSnapshot">): boolean {
if (run.status !== "failed" || run.runtimeMode !== "legacy" || run.nativeSessionId !== null || run.runnerInstanceId !== null
|| !isTransientWorkspacePreparationCode(run.errorCode)) return false;
const recovery = parseObject(run.resultJson?.executionRecovery);
if (recovery.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false;
// A restore failure is retried only when its owner proved it retryable.
if (run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE)
return parseObject(run.resultJson?.remoteWorkspaceRestore).retryable === true;
// An accepted interaction keeps its own continuation retry.
return !((run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || run.errorCode === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|| run.errorCode === REPOSITORY_CATALOG_UNAVAILABLE_CODE || run.errorCode === REVIEWED_CHAT_BINDING_NOT_READY)
&& isResolvedInteractionContinuationWakeContext(run.contextSnapshot));
}
export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS =
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length;
+46 -12
View File
@@ -893,6 +893,17 @@ function isOperatorCancelledRun(
);
}
function nonRetryableContinuationNotice(errorCode: string | null) {
return {
body:
"Paperclip detected a non-retryable failure on this issue's continuation run " +
`(\`${errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` +
"so it is visible for intervention.",
title: "Continuation failed",
tone: "danger" as const,
};
}
function isUnsuccessfulTerminalIssueRun(latestRun: LatestIssueRun) {
return Boolean(
latestRun &&
@@ -965,6 +976,14 @@ export function recoveryService(
transientRetryBudgetSpent?: (
run: typeof heartbeatRuns.$inferSelect,
) => boolean;
/**
* Whether a failed run stopped before provider work with a transient
* preparation failure that its own bounded retry continues. The sweep
* offers such a run to `scheduleRecoveryRetry` instead of escalating it.
*/
ownsPreparationRetry?: (
run: typeof heartbeatRuns.$inferSelect,
) => boolean;
liveRunExecutions?: Readonly<{ has(id: string): boolean }>;
/**
* The one settle path for a `running` legacy run whose adapter child is
@@ -1133,6 +1152,13 @@ export function recoveryService(
return run?.leaseExpiresAt ?? null;
}
async function failedRunOwnsPreparationRetry(companyId: string, runId: string): Promise<boolean> {
if (!deps.ownsPreparationRetry || !deps.scheduleRecoveryRetry) return false;
const [run] = await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId)));
return Boolean(run && deps.ownsPreparationRetry(run));
}
async function hasActiveExecutionPath(
companyId: string,
issueId: string,
@@ -5814,6 +5840,7 @@ export function recoveryService(
}
continue;
}
let preparationRetry = false;
if (isUnsuccessfulTerminalIssueRun(latestRun)) {
const classification = classifyContinuationFailure(latestRun);
@@ -5841,19 +5868,22 @@ export function recoveryService(
continue;
}
if (classification.kind === "non_retryable") {
// A failed run that stopped before provider work with a transient
// preparation failure is continued by its own bounded retry. Its
// executor schedules that retry after the failed write, and a slow
// settlement or a stopped server can leave it unscheduled once the
// settling check above lapses. Offer it to the same idempotent
// scheduler below (one successor, the same budget); only exhaustion
// or a refused continuation escalates it, and any other refusal
// escalates it as before.
preparationRetry = latestRun!.agentId === agentId
&& await failedRunOwnsPreparationRetry(issue.companyId, latestRun!.id);
if (classification.kind === "non_retryable" && !preparationRetry) {
const updated = await escalateStrandedAssignedIssue({
issue,
previousStatus: "in_progress",
latestRun,
notice: {
body:
"Paperclip detected a non-retryable failure on this issue's continuation run " +
`(\`${classification.errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` +
"so it is visible for intervention.",
title: "Continuation failed",
tone: "danger",
},
notice: nonRetryableContinuationNotice(classification.errorCode),
});
if (updated) {
result.escalated += 1;
@@ -5864,7 +5894,7 @@ export function recoveryService(
continue;
}
if (didAutomaticRecoveryFail(latestRun, "issue_continuation_needed")) {
if (!preparationRetry && didAutomaticRecoveryFail(latestRun, "issue_continuation_needed")) {
const { consecutive, latestFinishedAt } =
await summarizeRecentContinuationRetries(
issue.companyId,
@@ -5930,7 +5960,7 @@ export function recoveryService(
result.continuationRequeued += 1;
result.issueIds.push(issue.id);
} else if (
(recoveryOutcome.retryExhausted || recoveryOutcome.continuationRefused) &&
(recoveryOutcome.retryExhausted || recoveryOutcome.continuationRefused || preparationRetry) &&
!(await hasActiveExecutionPath(issue.companyId, issue.id, null))
) {
// The failed run's bounded transient retries are all spent, so no
@@ -5950,7 +5980,11 @@ export function recoveryService(
"Moving it to `blocked` so it is visible for intervention.",
title: "No live execution path",
tone: "danger",
} : CONTINUATION_REFUSED_NOTICE,
} : recoveryOutcome.continuationRefused ? CONTINUATION_REFUSED_NOTICE
// The scheduler declined the preparation retry for another
// reason (for example a durable owner or a permanent refusal):
// escalate it as a non-retryable failure, as before.
: nonRetryableContinuationNotice(latestRun?.errorCode ?? null),
});
if (updated) {
result.escalated += 1;
+55
View File
@@ -436,6 +436,61 @@ describe("durable task Retry admission", () => {
if (retry.outcome !== "scheduled") return;
await expect(dispatchRun(f.input, retry.run)).resolves.toMatchObject({ interruptedRunId: f.parent.id });
});
// Staging shape: a person's Retry failed before provider work because the
// saved sandbox resume timed out. The run is already `failed`, but its
// executor has not scheduled the bounded retry yet: settling took longer than
// the controller lease that stopped renewing at the failed write, or the
// server running it stopped. Another server's sweep then reads the task.
it.each(["slow_executor", "executor_gone"] as const)(
"retries a failed Retry run's transient preparation failure when the sweep reaches it first: %s", async executor => {
const f = await seedFailedRetryAdmittedRun();
await db.update(heartbeatRuns).set({ errorCode: "reusable_sandbox_resume_failed", controllerBootId: randomUUID(),
controllerLeaseExpiresAt: sql`clock_timestamp() - interval '1 second'`,
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
}).where(eq(heartbeatRuns.id, f.parent.id));
const heartbeat = heartbeatService(db);
await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId });
expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "in_progress" });
const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id));
expect(successors).toHaveLength(1);
expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "transient_failure",
scheduledRetryAttempt: 1 });
if (executor === "slow_executor") {
// The executor's own scheduling adopts the sweep's successor.
expect(await heartbeat.scheduleBoundedRetry(f.parent.id, { now: new Date() })).toMatchObject({
outcome: "scheduled", run: { id: successors[0].id } });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(1);
}
await expect(dispatchRun(f.input, successors[0])).resolves.toMatchObject({ interruptedRunId: f.parent.id });
});
it.each(["reusable_sandbox_resume_failed", "repository_catalog_unavailable"])(
"escalates a failed Retry run's transient preparation failure once when the scheduler declines it permanently: %s", async errorCode => {
const f = await seedFailedRetryAdmittedRun();
// An external Dot execution is never retried by the bounded scheduler.
await db.update(heartbeatRuns).set({ errorCode,
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" },
nativeExecutionInput: { provider: { kind: "openai_dot" } } },
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
}).where(eq(heartbeatRuns.id, f.parent.id));
const heartbeat = heartbeatService(db);
await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId });
expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "blocked" });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0);
const notices = async () => (await db.select().from(issueComments).where(eq(issueComments.issueId, f.input.issueId))).length;
const escalated = await notices();
await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.input.companyId });
expect(await notices()).toBe(escalated);
});
it("still escalates a failed Retry run's transient preparation failure once its bounded retries are spent", async () => {
const f = await seedFailedRetryAdmittedRun();
await db.update(heartbeatRuns).set({ errorCode: "reusable_sandbox_resume_failed",
scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 2,
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
}).where(eq(heartbeatRuns.id, f.parent.id));
await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId });
expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]).toMatchObject({ status: "blocked" });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0);
});
async function seedRefusedRetry() {
const input = await seed();
const now = Date.now();