fix(native): keep crediting proven unstarted restarts past the generation limit (#20647)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run survives a control-plane restart through restart
recovery, which resumes the interrupted provider turn
> - Restart recovery charges a provider attempt only when the provider
did work that could be lost, and it stops after three charged attempts
> - From controller generation 12 on, every restart charged an attempt
before that work rule was checked
> - In a busy restart wave, one long task sees 12 to 15 restarts, most
caused by other tasks. A restart that hit before the provider accepted a
turn was charged, the budget ran out, and a person had to click Retry
> - This pull request keeps the work rule past generation 12 for
restarts that a durable record proves did not start the provider, and
bounds a crash loop with a count of consecutive restarts without
provider progress
> - The benefit is that a long task survives many unrelated restarts
without a manual Retry, while a run that never makes progress still
stops

## Linked Issues or Issue Description

Refs #20625 (same restart recovery path). No public issue exists for
this case.

**What happened?**

`restartResumeKeepsAttempt` returned "charge" for every restart once the
run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller
generations. It did this before it read the work-based outcome. In a
local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1)
the run reached generation 13. The last restart interrupted a resume
whose runner started but whose provider had not accepted a turn. That
restart was charged at the last attempt, so the claim failed with
`execution_recovery_budget_exhausted` and
`native_restart_recovery_blocked`. The task needed a manual Retry.

**Expected behavior**

A restart that a durable record proves did not start provider work does
not spend an attempt, also past generation 12. A crash loop that never
makes provider progress still stops automatically. When it stops, the
task shows a Retry that works.

**Steps to reproduce**

Run a native task past controller generation 12. Make the last restart
hit after `native.process_start_requested` but before `turn.accepted`,
with the provider attempt at 3. The new integration test `does not
charge a restart past the generation limit whose resume never started
the provider` reproduces the failure before the fix.

**Paperclip version or commit**

`master` at ed5e7493bb.

## What Changed

- `restartResumeKeepsAttempt` now returns `{ keepsAttempt,
noProgressRestarts }`. A new helper judges the interrupted attempt as
`provider_not_started`, `provider_progressed` or `charged`, with the
same proofs as before.
- `noProgressRestarts` is 0 when the interrupted attempt is proven to
have made provider progress. Otherwise it is the previous claim's count
plus one. A clean-drain claim carries the previous count.
- Before generation 12 the decision does not change.
- From generation 12 on, a restart keeps the attempt only when the
interrupted attempt is proven `provider_not_started` and the count is
below the new `NATIVE_NO_PROGRESS_RESTART_LIMIT` (12). Provider work
(progress, or an accepted turn without progress) and unprovable cases
still charge.
- Each restart claim writes `noProgressRestarts` into its
`recovery_history` entry. No schema change: `recovery_history` is JSONB.
- Claims saved by earlier releases have no count. They are read as zero.
- Integration tests: the three generation-cap remote tests now check the
new bound. New tests cover the unstarted restart past the limit, the
consecutive-limit stop with a working Retry, the progress reset, an
accepted turn without progress, an unrecorded previous claim, and an
old-format claim.
- The durable-ownership plan (section F) records the rule.

**Why 12.** It equals the existing generation limit
(`PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT`). The count includes
restarts before generation 12. Thus a run that never makes provider
progress is bounded exactly as before: the existing test `bounds a crash
loop that never starts the provider` passes unchanged. Only proven
progress resets the count. In the failing wave, the task had made
progress shortly before its last restart, so its count was 1.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`
- Red before the fix: with only the new constant added to the old code,
the 9 new and changed tests fail. With the fix, they pass. The existing
crash-loop, progress and unstarted-resume tests pass without change.
- The consecutive-limit test checks that the stopped run is
`terminal_failure` / `native_restart_recovery_blocked` with no lease
owner and nothing scheduled, that `taskInterruptionPresentation` shows
`failed` with `canRetry: true`, and that `requestTaskRetry` accepts the
Retry.
- `cd server && npx tsc --noEmit -p .`
- Full local run of the three suites above: 322 passed, 14 skipped.
- Live: local restart chaos wave L6 (seed 9106, concurrency 3, `--env
local --adapters codex,claude --workloads A1,B5,C1 --faults RR1,RR2,RR3
--tasks 6`) at 7f7803ab0c: 6 of 6 tasks passed, and no task needed
Retry. Task t03 (Claude B5 RR1) reached controller generation 13 again.
The claim at generation 13 resumed the run (`resume_dead_runner`), and
the server log has no `execution_recovery_budget_exhausted`. The same
task on master was blocked at that claim and needed Retry.

## Risks

- A run that makes provider progress between restarts past generation 12
still charges each of those restarts. This is the same as before, by
design.
- An upgrade or a mid-run deploy reads old claims as count zero. A run
that was already in a no-progress loop can get up to 12 more
proven-unstarted restarts before the bound applies. It cannot be charged
where the old rule did not charge.
- A rollback ignores the new field and restores the old rule.
- Past generation 12 the claim now reads the event log, as it already
does before generation 12. This is not on the turn-start path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 15:27:45 -06:00
1 parent 4f33bb3578
commit 30403781b9
3 files changed
+222 -34

No files matched your search

@@ -218,6 +218,14 @@ Charge an attempt only on a recorded outcome that proves provider work started.
The rules from #19720 and #19747 then become one rule over these outcome records, instead of watermark heuristics. The on-demand crash-point suite asserts this at the pre-spawn and pre-accept crash points.
**Crash-loop bound past the generation limit (2026-11-10, approved by Dotta).** Uncharged restarts used to stop at controller generation 12: from then on every restart charged, before the work rule was checked. In a concurrency-3 restart chaos wave one long task saw 12–15 restarts, most caused by other tasks. A restart that hit before the provider accepted any turn was charged, the budget ran out, and a person had to click Retry (wave `w-20261110T183431-s9106`, t03). The rule is now:
- Each restart claim records `noProgressRestarts` in its `recovery_history` entry: 0 when the interrupted attempt is proven to have made provider progress, otherwise one more than the previous claim's count. A clean-drain claim carries the previous count unchanged.
- Before generation 12 nothing changes.
- From generation 12 on, a restart is not charged only when a durable record proves the interrupted attempt never started provider work (the same `provider_not_started` proof as above, or a resume that never reached execution) and the count is still below `NATIVE_NO_PROGRESS_RESTART_LIMIT` (12). Provider work (progress, or an accepted turn without progress) and unprovable cases still charge, as before.
- When the count reaches the limit, every further restart charges, so the attempt budget runs out and recovery stops with `execution_recovery_budget_exhausted`, which offers Retry. The limit equals the generation limit, so a run that never makes progress is bounded exactly as before; only proven progress resets the count.
- Claims saved by earlier releases have no count and are read as zero. A mid-run deploy therefore gives at most 12 more proven-unstarted restarts before the bound applies, and never charges a restart that the old rule would have credited.
### G. Generation stamping
The runner stamps `generation` and `turnId` on every event. Driver and session code treat an older generation as history. #19718, #19722, #19772 and #19783 each special-cased one variant of this.
@@ -180,14 +180,29 @@ export function suspendedByCleanDrain(coordinator: {
* before the crash. Each claim records the run's event-log position, so the
* next claim judges only what happened after it. A resume the controller died
* before starting, a runner that never admitted its provider, and a turn.start
* the provider did not accept are not charged. Credit stops after
* NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT controller generations, so a crash
* loop that never starts the provider, or that makes progress and then crashes
* the controller every time, is still bounded.
* the provider did not accept are not charged. After
* NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT controller generations only a proven
* unstarted attempt keeps credit, and only until NATIVE_NO_PROGRESS_RESTART_LIMIT
* consecutive restarts made no provider progress, so a crash loop that never
* starts the provider, or that makes progress and then crashes the controller
* every time, is still bounded.
*/
export const NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT = PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT;
export const NATIVE_NO_PROGRESS_RESTART_LIMIT = PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT;
type RecoveryClaimProgressEntry = { disposition?: unknown; progressWatermarkSeq?: unknown; progressAttempt?: unknown };
type RecoveryClaimProgressEntry = {
disposition?: unknown; progressWatermarkSeq?: unknown; progressAttempt?: unknown; noProgressRestarts?: unknown;
};
function previousNoProgressRestarts(recoveryHistory: ReadonlyArray<RecoveryClaimProgressEntry>): number {
for (let index = recoveryHistory.length - 1; index >= 0; index -= 1) {
const entry = recoveryHistory[index];
if (!RECOVERY_CLAIM_DISPOSITIONS.has(entry?.disposition)) continue;
const count = entry?.noProgressRestarts;
return typeof count === "number" && Number.isSafeInteger(count) && count >= 0 ? count : 0;
}
return 0;
}
/**
* The most recent recovery claim's event-log position and the provider attempt
@@ -261,7 +276,9 @@ async function nativeRunEventWatermark(tx: Pick<Db, "select">, runId: string): P
* charge decision was never applied, so this claim carries it: the attempt
* ends at the attempt that resume was to run as, never above it.
* - A claim history written before claims recorded their position charges.
* At or after NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT every restart charges.
* At or after NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT a restart charges unless
* the interrupted attempt is proven unstarted and fewer than
* NATIVE_NO_PROGRESS_RESTART_LIMIT consecutive restarts made no progress.
*/
export async function restartResumeKeepsAttempt(tx: Pick<Db, "select">, input: {
runId: string;
@@ -270,14 +287,29 @@ export async function restartResumeKeepsAttempt(tx: Pick<Db, "select">, input: {
controllerGeneration: number;
recoveryHistory: ReadonlyArray<RecoveryClaimProgressEntry>;
};
}): Promise<boolean> {
if (input.coordinator.attempt <= 0) return false;
if (input.coordinator.controllerGeneration >= NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT) return false;
}): Promise<{ keepsAttempt: boolean; noProgressRestarts: number }> {
const outcome = await interruptedAttemptOutcome(tx, input);
const noProgressRestarts = outcome === "provider_progressed" ? 0
: previousNoProgressRestarts(input.coordinator.recoveryHistory) + 1;
const keepsAttempt = input.coordinator.controllerGeneration < NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT
? outcome !== "charged"
: outcome === "provider_not_started" && noProgressRestarts < NATIVE_NO_PROGRESS_RESTART_LIMIT;
return { keepsAttempt, noProgressRestarts };
}
async function interruptedAttemptOutcome(tx: Pick<Db, "select">, input: {
runId: string;
coordinator: {
attempt: number;
recoveryHistory: ReadonlyArray<RecoveryClaimProgressEntry>;
};
}): Promise<"provider_not_started" | "provider_progressed" | "charged"> {
if (input.coordinator.attempt <= 0) return "charged";
const previous = previousRecoveryClaim(input.coordinator.recoveryHistory);
if (previous === "unrecorded") return false;
if (previous === "unrecorded") return "charged";
if (previous !== "none" && previous.launchesRunner
&& !await nativeExecutionStartedSince(tx, input.runId, previous.watermark))
return input.coordinator.attempt >= previous.attempt;
return input.coordinator.attempt >= previous.attempt ? "provider_not_started" : "charged";
const claimedAt = previous === "none" ? 0 : previous.watermark;
// A reattachment adopted a provider that had already started, and it keeps
// the attempt; a normal retry after it starts a new attempt.
@@ -287,13 +319,14 @@ export async function restartResumeKeepsAttempt(tx: Pick<Db, "select">, input: {
: await providerAttemptWindowStart(tx, {
runId: input.runId, sinceSeq: claimedAt, providerAttempt: input.coordinator.attempt,
});
if (window === "unstarted") return true;
if (window === "unstarted") return "provider_not_started";
const outcome = await readProviderAttemptOutcome(tx, {
runId: input.runId,
sinceSeq: window.seq,
adoptedProviderWork,
});
return !providerAttemptChargesBudget(outcome);
if (providerAttemptChargesBudget(outcome)) return "charged";
return outcome === "provider_progressed" ? outcome : "provider_not_started";
}
@@ -1577,9 +1610,10 @@ export async function claimNativeRestartRecoveries(input: {
// Read before this claim appends its own history entry.
const cleanDrainSuspension = suspendedByCleanDrain(row.coordinator);
const resumeKeepsAttempt = cleanDrainSuspension || await restartResumeKeepsAttempt(tx, {
runId: row.run.id, coordinator: row.coordinator,
});
const restartCredit = cleanDrainSuspension
? { keepsAttempt: true, noProgressRestarts: previousNoProgressRestarts(row.coordinator.recoveryHistory) }
: await restartResumeKeepsAttempt(tx, { runId: row.run.id, coordinator: row.coordinator });
const resumeKeepsAttempt = restartCredit.keepsAttempt;
const classification = classifyNativeRunnerRecoveryEvidence({
remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority),
unclaimedBootstrap,
@@ -1812,7 +1846,8 @@ export async function claimNativeRestartRecoveries(input: {
: "bootstrap_incomplete",
recoveryRequestId: input.recoveryRequestId ?? null,
recoveryHistory: appendBoundedRecoveryHistory({
...event, progressWatermarkSeq, progressAttempt, ...(attemptKeptForProgress ? { attemptKeptForProgress } : {}),
...event, progressWatermarkSeq, progressAttempt, noProgressRestarts: restartCredit.noProgressRestarts,
...(attemptKeptForProgress ? { attemptKeptForProgress } : {}),
}),
failureCode: null,
nextAttemptAt: null,
@@ -93,7 +93,9 @@ vi.mock("../../sentry.js", async () => {
captureRunFailure: mockCaptureRunFailure,
};
});
import { claimNativeRestartRecoveries, currentNativeControllerIdentity, NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT, NATIVE_RESTART_RECOVERY_FAILED, nextNativeProviderAttempt, type NativeControllerIdentity } from "./native-restart-recovery.js";
import { claimNativeRestartRecoveries, currentNativeControllerIdentity, NATIVE_NO_PROGRESS_RESTART_LIMIT, NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT, NATIVE_RESTART_RECOVERY_FAILED, nextNativeProviderAttempt, type NativeControllerIdentity } from "./native-restart-recovery.js";
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
import { requestTaskRetry } from "../task-retry.js";
// Exact v5 wire prompt from before the task-monitor upgrade (#15446).
const historicalPrompt = {
@@ -3107,15 +3109,19 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
kind: "bootstrap_incomplete", runId: fixture.runId, providerAttempt: 3, chargesProviderAttempt: false })]);
});
it("blocks an exited remote runner at its last attempt once the controller generation cap is reached", async () => {
// Repeated exits before provider work stay bounded by the generation cap.
const fixture = await seedExitedRemoteRunner("EXITED-GEN-CAP", {}, 3);
await fixture.db.update(nativeRunFinalizations).set({ controllerGeneration: NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
kind: "blocked", runId: fixture.runId, reason: "execution_recovery_budget_exhausted" })]);
});
it.each([
{ recorded: null, expected: { kind: "bootstrap_incomplete", chargesProviderAttempt: false } },
{ recorded: NATIVE_NO_PROGRESS_RESTART_LIMIT - 2, expected: { kind: "bootstrap_incomplete", chargesProviderAttempt: false } },
{ recorded: NATIVE_NO_PROGRESS_RESTART_LIMIT - 1, expected: { kind: "blocked", reason: "execution_recovery_budget_exhausted" } },
])("bounds an exited remote runner at its last attempt past the generation cap by consecutive no-progress restarts ($recorded)",
async ({ recorded, expected }) => {
const fixture = await seedExitedRemoteRunner(`EXITED-GEN-CAP-${recorded}`, {}, 3);
await fixture.db.update(nativeRunFinalizations).set({ controllerGeneration: NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
if (recorded !== null) await recordUnstartedClaim(fixture, recorded);
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ runId: fixture.runId, ...expected })]);
});
it.each(["a provider event committed after the failure", "a session checkpoint saved after the failure"])(
"keeps an exited remote runner's retry held when %s contradicts the saved failure", async (label) => {
@@ -3174,13 +3180,20 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
it.each([
["claimed remote bootstrap", (label: string) => seedClaimedRemoteBootstrap(label, { attempt: 3 })],
["remote bootstrap retry", (label: string) => seedUnlaunchedRemoteClaim(label, { bootstrapRetry: true, attempt: 3 })],
] as const)("blocks a %s at its last attempt once the controller generation cap is reached", async (label, seed) => {
const fixture = await seed(`GEN-CAP-${label.replace(/\W+/g, "-")}`);
await fixture.db.update(nativeRunFinalizations).set({ controllerGeneration: NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
kind: "blocked", runId: fixture.runId, reason: "execution_recovery_budget_exhausted" })]);
] as const)("bounds a %s at its last attempt past the generation cap by consecutive no-progress restarts", async (label, seed) => {
const uncapped = await seed(`GEN-CAP-OPEN-${label.replace(/\W+/g, "-")}`);
await uncapped.db.update(nativeRunFinalizations).set({ controllerGeneration: NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT })
.where(eq(nativeRunFinalizations.runId, uncapped.runId));
expect(await claimNativeRestartRecoveries({ db: uncapped.db, controller: successor,
restartKind: "hard", runIds: [uncapped.runId] })).toEqual([expect.objectContaining({
kind: "bootstrap_incomplete", runId: uncapped.runId, providerAttempt: 3, chargesProviderAttempt: false })]);
const capped = await seed(`GEN-CAP-${label.replace(/\W+/g, "-")}`);
await capped.db.update(nativeRunFinalizations).set({ controllerGeneration: NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT })
.where(eq(nativeRunFinalizations.runId, capped.runId));
await recordUnstartedClaim(capped, NATIVE_NO_PROGRESS_RESTART_LIMIT - 1);
expect(await claimNativeRestartRecoveries({ db: capped.db, controller: successor,
restartKind: "hard", runIds: [capped.runId] })).toEqual([expect.objectContaining({
kind: "blocked", runId: capped.runId, reason: "execution_recovery_budget_exhausted" })]);
});
it("keeps a paused agent's remote bootstrap retry waiting instead of starting it", async () => {
@@ -3638,6 +3651,138 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
.toEqual([2, ...Array(NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT - 2).fill(2), 3]);
});
async function recordUnstartedClaim(fixture: Awaited<ReturnType<typeof seedRun>>, noProgressRestarts: number) {
const [last] = await fixture.db.select({ seq: sql<number>`coalesce(max(${heartbeatRunEvents.seq}), 0)::int` })
.from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, fixture.runId));
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ recoveryHistory: [...coordinator.recoveryHistory, {
disposition: "bootstrap_incomplete", progressWatermarkSeq: last?.seq ?? 0, progressAttempt: coordinator.attempt,
noProgressRestarts,
}] }).where(eq(nativeRunFinalizations.runId, fixture.runId));
}
async function seedPastGenerationLimit(label: string) {
const fixture = await seedInterruptedCheckpoint(label);
await fixture.db.update(nativeRunFinalizations).set({ controllerGeneration: NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
await appendRunEvents(fixture, [{ eventType: "tool.execution.completed", turnId: "interrupted-turn" }]);
return fixture;
}
async function lastClaimEntry(fixture: Awaited<ReturnType<typeof seedRun>>) {
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
return (coordinator.recoveryHistory as Array<Record<string, unknown>>).at(-1)!;
}
const providerNeverStarted = [{ eventType: "native.process_start_requested" },
{ eventType: "native.process_identity_recorded" }, { eventType: "session.resumed" }];
const providerProgressed = (round: number) => [{ eventType: "native.process_start_requested" },
{ eventType: "turn.accepted", turnId: `past-limit-${round}` },
{ eventType: "tool.execution.completed", turnId: `past-limit-${round}` }];
it("does not charge a restart past the generation limit whose resume never started the provider", async () => {
const fixture = await seedPastGenerationLimit("PAST-LIMIT-UNSTARTED");
expect(await resumeThenCrash(fixture)).toMatchObject({
disposition: { kind: "resume_dead_runner", chargesProviderAttempt: true }, attempt: 2 });
await appendRunEvents(fixture, providerProgressed(1));
expect(await resumeThenCrash(fixture)).toMatchObject({
disposition: { kind: "resume_dead_runner", chargesProviderAttempt: true }, attempt: 3 });
await appendRunEvents(fixture, providerNeverStarted);
expect(await resumeThenCrash(fixture)).toMatchObject({
disposition: { kind: "resume_dead_runner", chargesProviderAttempt: false }, attempt: 3 });
expect(await lastClaimEntry(fixture)).toMatchObject({ noProgressRestarts: 1 });
});
it("stops after consecutive restarts with no provider progress and leaves a working Retry", async () => {
const fixture = await seedPastGenerationLimit("PAST-LIMIT-NO-PROGRESS-LOOP");
expect(await resumeThenCrash(fixture)).toMatchObject({ attempt: 2 });
const outcomes: Array<{ kind?: string; attempt?: number }> = [];
for (let round = 1; round <= NATIVE_NO_PROGRESS_RESTART_LIMIT + 2; round += 1) {
await appendRunEvents(fixture, providerNeverStarted);
const { disposition, attempt } = await resumeThenCrash(fixture);
outcomes.push({ kind: disposition?.kind, attempt });
if (disposition?.kind !== "resume_dead_runner") break;
}
expect(outcomes).toEqual([
...Array(NATIVE_NO_PROGRESS_RESTART_LIMIT - 1).fill({ kind: "resume_dead_runner", attempt: 2 }),
{ kind: "resume_dead_runner", attempt: 3 },
{ kind: "blocked", attempt: undefined },
]);
const [blocked] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(blocked).toMatchObject({ phase: "terminal_failure", failureCode: "native_restart_recovery_blocked",
leaseOwner: null, nextAttemptAt: null });
const [issue] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId));
expect(await taskInterruptionPresentation(fixture.db, issue)).toMatchObject({
state: "failed", canRetry: true, observedRunId: fixture.runId });
const personId = randomUUID();
await fixture.db.insert(authUsers).values({ id: personId, name: "Retry person", email: `${personId}@fixture.invalid`,
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
await fixture.db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: personId,
status: "active", membershipRole: "member" });
expect(await requestTaskRetry({ db: fixture.db, companyId, issueId: fixture.issueId, actorId: personId,
actorSource: "session", idempotencyKey: `no-progress-loop-${fixture.runId}`, observedRunId: fixture.runId }))
.toMatchObject({ accepted: true });
});
it("resets the consecutive no-progress count when a restart proves provider progress", async () => {
const fixture = await seedPastGenerationLimit("PAST-LIMIT-PROGRESS-RESETS");
const counts: unknown[] = [];
const attempts: Array<number | undefined> = [];
const round = async (events: Parameters<typeof appendRunEvents>[1]) => {
await appendRunEvents(fixture, events);
attempts.push((await resumeThenCrash(fixture)).attempt);
counts.push((await lastClaimEntry(fixture)).noProgressRestarts);
};
await round([]);
for (let index = 0; index < 3; index += 1) await round(providerNeverStarted);
await round(providerProgressed(1));
await round(providerNeverStarted);
expect(counts).toEqual([0, 1, 2, 3, 0, 1]);
expect(attempts).toEqual([2, 2, 2, 2, 3, 3]);
});
it("still charges a restart past the generation limit whose provider accepted a turn without progress", async () => {
const fixture = await seedPastGenerationLimit("PAST-LIMIT-ACCEPTED-NO-PROGRESS");
expect(await resumeThenCrash(fixture)).toMatchObject({ attempt: 2 });
await appendRunEvents(fixture, [{ eventType: "native.process_start_requested" },
{ eventType: "turn.accepted", turnId: "accepted" }, { eventType: "item.completed", turnId: "accepted", kind: "usage" }]);
expect(await resumeThenCrash(fixture)).toMatchObject({
disposition: { kind: "resume_dead_runner", chargesProviderAttempt: true }, attempt: 3 });
expect(await lastClaimEntry(fixture)).toMatchObject({ noProgressRestarts: 1 });
});
it("still charges a restart past the generation limit when the previous claim did not record its position", async () => {
const fixture = await seedPastGenerationLimit("PAST-LIMIT-UNRECORDED");
expect(await resumeThenCrash(fixture)).toMatchObject({ attempt: 2 });
await fixture.db.update(nativeRunFinalizations).set({
recoveryHistory: sql`jsonb_set(${nativeRunFinalizations.recoveryHistory},
array[(jsonb_array_length(${nativeRunFinalizations.recoveryHistory}) - 1)::text],
(${nativeRunFinalizations.recoveryHistory} -> -1) - 'progressWatermarkSeq' - 'noProgressRestarts')`,
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
await appendRunEvents(fixture, providerNeverStarted);
expect(await resumeThenCrash(fixture)).toMatchObject({
disposition: { kind: "resume_dead_runner", chargesProviderAttempt: true }, attempt: 3 });
});
it("counts from zero for a claim saved before restarts recorded the count", async () => {
const fixture = await seedPastGenerationLimit("PAST-LIMIT-OLD-ROW");
expect(await resumeThenCrash(fixture)).toMatchObject({ attempt: 2 });
await appendRunEvents(fixture, providerNeverStarted);
await resumeThenCrash(fixture);
await fixture.db.update(nativeRunFinalizations).set({
recoveryHistory: sql`jsonb_set(${nativeRunFinalizations.recoveryHistory},
array[(jsonb_array_length(${nativeRunFinalizations.recoveryHistory}) - 1)::text],
(${nativeRunFinalizations.recoveryHistory} -> -1) - 'noProgressRestarts')`,
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
await appendRunEvents(fixture, providerNeverStarted);
expect(await resumeThenCrash(fixture)).toMatchObject({
disposition: { kind: "resume_dead_runner", chargesProviderAttempt: false }, attempt: 2 });
expect(await lastClaimEntry(fixture)).toMatchObject({ noProgressRestarts: 1 });
});
it("terminalizes the recorded failed-checkpoint incident atomically instead of resuming it on upgrade", async () => {
const fixture = await seedRun("FAILED-CHECKPOINT");
await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: {