fix(heartbeat): retry a legacy run that lost its lease before dispatch (#16358)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat runner executes legacy agent runs. A controller lease
watchdog keeps durable ownership of each legacy run, and it aborts the
run when it loses the lease.
> - #16084 records a lease loss as a `controller_lease` cancellation
when the adapter returns or throws. It does not cover a lease loss
during setup, before the adapter starts.
> - In setup, the abort reason goes to the setup catch. The catch
records an opaque `setup_failed` and sends the raw
`LegacyControllerLeaseLostError` to Sentry with no cause label.
> - Issue continuation recovery treats `setup_failed` as non-retryable.
The task moves to `blocked`, but no provider work started.
> - This pull request records a lease loss before dispatch as
`controller_lease_lost`, with the same `controller_lease` evidence.
Recovery retries it as transient infrastructure, and so does the
accepted-interaction (plan approval) continuation retry.
> - The benefit is that a host lease loss during setup no longer strands
the task, and the next Sentry event shows which renewal failure caused
the loss.

## Linked Issues or Issue Description

Refs #16084
Refs #16355
Refs #16359

**What happened?**
A staging server reported `LegacyControllerLeaseLostError: Legacy
controller lease lost` from the heartbeat. The run failed during setup
with `setup_failed`. Continuation recovery then said "non-retryable
failure on this issue's continuation run (`setup_failed`)" and moved the
task to `blocked`. A scanner had to move the task back to `in_progress`
by hand.

**Expected behavior**
A lease loss before adapter dispatch is a host infrastructure stop. The
run records the `controller_lease` cause, like the adapter-phase path.
The run keeps `providerWorkStarted: false`. Recovery retries the
continuation with its bounded transient budget, and it does not block
the task.

**Steps to reproduce**
1. Start a legacy run that has a controller lease.
2. Before the adapter dispatch, make the lease renewal fail: the update
matches no row, the query errors, or the lease deadline passes.
3. `controllerLease.assertOwned()` throws the abort reason in workspace
preparation or before dispatch.
4. Before this change: the run is `failed`/`setup_failed`, `resultJson`
has no `cancellation`, and continuation recovery blocks the task. After
this change: the run is `failed`/`controller_lease_lost` with
`cancellation.source: "controller_lease"` and a closed cause, and
continuation recovery classifies it as `transient_infra`.

**Additional context**
#16355 handles a different path: a restart orphan that the next boot
reaps as `process_lost`. This change handles a live controller that
loses its own lease during setup. This change does not find the root
cause of the lease loss. The closed cause label in the next Sentry event
gives that fact.

## What Changed

- `server/src/services/heartbeat/run-completion.ts`: `failRunSetup`
receives the execution abort signal. When the abort reason is a
`LegacyControllerLeaseLostError`, it writes error code
`controller_lease_lost`, adds the `controller_lease` cancellation
evidence, and keeps the existing bootstrap recovery facts. The agent
goes back to idle, not `error`, because the stop is a host fault. This
applies only before provider dispatch: `failRunSetup` also receives
`providerDispatchStarted`, and a setup-path error after a legacy adapter
or native dispatch began keeps `setup_failed`, so started work is never
retried.
- `server/src/services/legacy-controller-lease.ts`: add the
`CONTROLLER_LEASE_LOST_FAILURE_CODE` constant.
- Merged with #16359 (lease-loss timings): the setup failure keeps its
`resultJson.controllerLeaseLoss` timings and adds this PR's
`controller_lease` cancellation from the abort signal. #16359's setup
test now aborts the signal first, as `assertOwned()` does, and expects
`controller_lease_lost`.
- `server/src/services/heartbeat.ts`: pass the execution abort signal
and `providerDispatchStarted` (`legacyAdapterEntered ||
nativeDispatchStarted`) to `failRunSetup`.
- `server/src/services/recovery/service.ts`: add `controller_lease_lost`
to the transient-infrastructure continuation codes (3 bounded attempts
with backoff).
- `server/src/services/heartbeat/retries.ts`: add
`controller_lease_lost` to the accepted-interaction continuation retry
allowlist, next to `process_lost`. Without it, a plan-approval
continuation that lost its lease before dispatch was escalated and
blocked before generic recovery could see it.
- `server/src/services/heartbeat/run-lifecycle.ts`: add
`controller_lease_lost` to the pre-adapter setup codes, so the comment
policy does not queue a missing-comment retry.
- `doc/run-log-events.md`: document the new error code.
- Tests: setup lease loss when the abort reason is thrown, and when a
setup step throws a different error after the abort. A unit test shows a
lease loss after dispatch keeps `setup_failed`. A classifier test shows
`controller_lease_lost` is transient and `setup_failed` stays
non-retryable. Two assembled heartbeat tests
(`heartbeat-process-recovery.test.ts`) lose the real lease during
workspace preparation: one for a plain issue continuation (one
`transient_failure` retry) and one for an accepted plan-approval
interaction continuation (one `interaction_continuation_infra_retry`
retry). Both check the saved run, no adapter dispatch, and the issue
still `in_progress`.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: 25 passed (embedded
Postgres), including the new post-dispatch test.
- `cd server && npx vitest run src/services/run-cancellation.test.ts
src/services/recovery/provider-failure-classification.test.ts`: passed
on the first head (29 with run-completion; 25 classifier).
- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts`: 13 passed (needs `pnpm
--filter @paperclipai/plugin-sdk... build` first).
- After merging master (head `2148b27`): `run-completion`,
`legacy-controller-lease`, `run-cancellation`,
`provider-failure-classification` and `run-failure-report` suites, 310
passed; `heartbeat-process-recovery.test.ts -t "lease|dispatch"`, 31
passed.
- Head `c52825c`: `retries`, `run-completion`,
`heartbeat-retry-scheduling`, `legacy-controller-lease` and
`provider-failure-classification` suites, 134 passed;
`heartbeat-process-recovery.test.ts -t "lease|dispatch|interaction"`, 41
passed.
- Negative controls: with the lease-loss branch in `run-completion.ts`
disabled, the plain-continuation assembled test fails (`setup_failed`
instead of `controller_lease_lost`). With the `retries.ts` allowlist
change reverted, the accepted-interaction assembled test fails because
no retry is scheduled.
- Not run locally: full `tsc` and `pnpm build`. The sandbox has a 4 GB
memory limit and the OOM killer stops `tsc`. CI is the typecheck and
build gate.

## Risks

- Low risk. The change applies only when the execution abort reason is a
`LegacyControllerLeaseLostError` in the setup catch before dispatch.
Other setup failures keep their current codes.
- The retry uses the existing continuation recovery budgets and
admission checks (agent state, budget, pause). The
interaction-continuation retry keeps its existing 2-attempt budget. The
retry is a new run. The revoked controller cannot dispatch, because its
renewal compare-and-set fails.
- A run that the reaper settles first keeps the reaper outcome.
`setRunStatusIfRunning` makes the first terminal write win.
- Sentry still gets this unexpected stop, with the closed
`controllerLeaseLossCause` label. If the lease loss is frequent, the
event volume does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (targeted suites pass;
typecheck/build not possible in this sandbox, see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-10-14 21:31:42 -07:00
1 parent 3ea39f75c1
commit 76164b53cb
10 files changed
+213 -11

No files matched your search

@@ -2112,6 +2112,141 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
).resolves.toEqual({ status: "idle", errorReason: null });
});
it("retries a continuation whose controller lease is lost before dispatch instead of blocking the issue", async () => {
const { agentId, runId, issueId } = await seedQueuedIssueRunFixture();
const originalWatch = controllerLeases.watchLegacyControllerLease;
// Expire the claimed lease before workspace preparation renews it, so the
// real renewal is rejected and the watchdog stops setup before dispatch.
const watcher = vi.spyOn(controllerLeases, "watchLegacyControllerLease").mockImplementation((...args) => {
const lease = originalWatch(...args);
if (args[1].id !== runId) return lease;
return { ...lease, async assertOwned(stage?: "dispatching") {
if (stage === undefined) {
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() - 60_000) })
.where(eq(heartbeatRuns.id, runId));
}
return lease.assertOwned(stage);
} };
});
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
await waitForRunToSettle(heartbeat, runId);
await heartbeat.waitForRunExecutionDrain(runId);
expect(await heartbeat.getRun(runId)).toMatchObject({
status: "failed",
errorCode: "controller_lease_lost",
resultJson: {
cancellation: { source: "controller_lease", reason: "Legacy controller lease lost (renewal_rejected)" },
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
},
});
expect(
mockAdapterExecute.mock.calls.some(
([input]) => (input as { runId?: string } | undefined)?.runId === runId,
),
).toBe(false);
const retries = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId));
expect(retries).toHaveLength(1);
// One bounded transient retry, instead of escalating `setup_failed`.
expect(retries[0]).toMatchObject({ scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 1 });
expect(
await db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
.then((rows) => rows[0]?.status),
).toBe("in_progress");
expect(
await db.select({ status: agents.status }).from(agents).where(eq(agents.id, agentId))
.then((rows) => rows[0]?.status),
).not.toBe("error");
} finally {
await heartbeat.drainActiveRunExecutions().catch(() => undefined);
watcher.mockRestore();
}
});
it("retries an accepted interaction continuation whose controller lease is lost before dispatch", async () => {
const { companyId, agentId, runId, wakeupRequestId, issueId } = await seedQueuedIssueRunFixture();
const interactionId = randomUUID();
await db.insert(issueThreadInteractions).values({
id: interactionId,
companyId,
issueId,
kind: "request_confirmation",
status: "accepted",
continuationPolicy: "wake_assignee_on_accept",
createdByAgentId: agentId,
resolvedByUserId: "responsible-user",
resolvedAt: new Date("2026-03-19T00:00:00.000Z"),
payload: {
version: 1,
prompt: "Approve the plan?",
target: { type: "issue_document", issueId, key: "plan", revisionId: randomUUID() },
},
result: { version: 1, outcome: "accepted" },
});
const interactionContext = {
issueId,
interactionId,
interactionKind: "request_confirmation",
interactionStatus: "accepted",
mutation: "interaction",
};
await db.update(agentWakeupRequests)
.set({ source: "automation", reason: "issue_commented", payload: interactionContext })
.where(eq(agentWakeupRequests.id, wakeupRequestId));
await db.update(heartbeatRuns)
.set({
invocationSource: "automation",
contextSnapshot: { ...interactionContext, taskId: issueId, wakeReason: "issue_commented" },
})
.where(eq(heartbeatRuns.id, runId));
await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, issueId));
const originalWatch = controllerLeases.watchLegacyControllerLease;
const watcher = vi.spyOn(controllerLeases, "watchLegacyControllerLease").mockImplementation((...args) => {
const lease = originalWatch(...args);
if (args[1].id !== runId) return lease;
return { ...lease, async assertOwned(stage?: "dispatching") {
if (stage === undefined) {
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() - 60_000) })
.where(eq(heartbeatRuns.id, runId));
}
return lease.assertOwned(stage);
} };
});
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
await waitForRunToSettle(heartbeat, runId);
await heartbeat.waitForRunExecutionDrain(runId);
expect(await heartbeat.getRun(runId)).toMatchObject({
status: "failed",
errorCode: "controller_lease_lost",
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
});
expect(
mockAdapterExecute.mock.calls.some(
([input]) => (input as { runId?: string } | undefined)?.runId === runId,
),
).toBe(false);
const retries = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId));
expect(retries).toHaveLength(1);
expect(retries[0]).toMatchObject({
scheduledRetryReason: INTERACTION_CONTINUATION_INFRA_RETRY_REASON,
scheduledRetryAttempt: 1,
});
expect(
await db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
.then((rows) => rows[0]?.status),
).toBe("in_progress");
} finally {
await heartbeat.drainActiveRunExecutions().catch(() => undefined);
watcher.mockRestore();
}
});
it("does not queue immediate recovery when the failed run's issue is hidden", async () => {
mockAdapterExecute.mockResolvedValueOnce({
exitCode: 1,
+3 -1
View File
@@ -3645,7 +3645,9 @@ export function heartbeatService(
);
} else {
await runCompletion.failRunSetup({
run, runId, outerErr, identityRedactor, readFailureReportSecrets,
run, runId, outerErr, signal: executionControl.controller.signal,
providerDispatchStarted: legacyAdapterEntered || nativeDispatchStarted,
identityRedactor, readFailureReportSecrets,
});
}
} finally {
+5 -1
View File
@@ -16,6 +16,7 @@ import {
accountingForScheduledRetry,
} from "../execution-recovery-attempt.js";
import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-scheduler.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
import { randomUUID } from "node:crypto";
import {
and,
@@ -320,9 +321,12 @@ function isRetryableInteractionContinuationInfrastructureFailure(
"error" | "errorCode" | "resultJson"
>,
) {
// `controller_lease_lost` is only recorded before provider dispatch, so the
// retry cannot repeat started work.
if (
run.errorCode === WORKSPACE_VALIDATION_FAILURE_CODE ||
run.errorCode === "process_lost"
run.errorCode === "process_lost" ||
run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE
) {
return true;
}
@@ -108,7 +108,7 @@ describe.skipIf(!support.supported)("heartbeat run completion boundary", () => {
};
const completion = createHeartbeatRunCompletion(db, deps);
const failure = (err: unknown): FailHeartbeatRunInput => ({ ...input, runId: run.id, err, identityRedactor: createAgentIdentityRedactor("private-identity-material"), requiredWorkspaceRestoreEvidence: { workspaceRestoreFailure: { reason: "missing_base" } }, legacyAdapterEntered: true, goalCheckpointSession: { current: null }, previousSessionDisplayId: "previous-session", taskSession: null });
const setup = (outerErr: unknown) => ({ run, runId: run.id, outerErr, identityRedactor: createAgentIdentityRedactor(), readFailureReportSecrets: () => [] });
const setup = (outerErr: unknown, providerDispatchStarted = false) => ({ run, runId: run.id, outerErr, signal: controller.signal, providerDispatchStarted, identityRedactor: createAgentIdentityRedactor(), readFailureReportSecrets: () => [] });
return { run, agent, wake, deps, controller, input, completion, failure, setup };
}
const read = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!);
@@ -187,9 +187,10 @@ describe.skipIf(!support.supported)("heartbeat run completion boundary", () => {
it("records the lease-loss cause and timings when setup fails on the abort reason", async () => {
const f = await fixture();
await f.completion.failRunSetup(f.setup(new LegacyControllerLeaseLostError("renewal_deadline",
{ sinceRenewalMs: 60_004, renewalPendingMs: null, deadlineLateMs: 55_000, renewalCount: 3 })));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "setup_failed", resultJson: {
f.controller.abort(new LegacyControllerLeaseLostError("renewal_deadline",
{ sinceRenewalMs: 60_004, renewalPendingMs: null, deadlineLateMs: 55_000, renewalCount: 3 }));
await f.completion.failRunSetup(f.setup(f.controller.signal.reason));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "controller_lease_lost", resultJson: {
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
controllerLeaseLoss: { cause: "renewal_deadline", sinceRenewalMs: 60_004, renewalPendingMs: null,
deadlineLateMs: 55_000, renewalCount: 3 } } });
@@ -327,6 +328,34 @@ describe.skipIf(!support.supported)("heartbeat run completion boundary", () => {
} });
});
it("records a lease lost before dispatch as a retryable host stop, not an opaque setup failure", async () => {
const f = await fixture();
f.controller.abort(new LegacyControllerLeaseLostError("renewal_rejected"));
await f.completion.failRunSetup(f.setup(f.controller.signal.reason));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "controller_lease_lost", resultJson: {
cancellation: { source: "controller_lease", expected: false, reason: "Legacy controller lease lost (renewal_rejected)" },
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } });
expect(f.deps.releaseIssueExecutionAndPromote).toHaveBeenCalledOnce();
expect(f.deps.finalizeAgentStatus).toHaveBeenCalledWith(f.agent.id, "failed", expect.any(String), expect.objectContaining({ keepIdleOnFailure: true }));
});
it("attributes a lease loss even when a setup step throws its own abort error", async () => {
const f = await fixture();
f.controller.abort(new LegacyControllerLeaseLostError("renewal_deadline"));
await f.completion.failRunSetup(f.setup(new Error("environment acquisition aborted")));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "controller_lease_lost",
resultJson: { cancellation: { source: "controller_lease", reason: "Legacy controller lease lost (renewal_deadline)" } } });
});
it("does not make a lease loss retryable once provider work has started", async () => {
const f = await fixture();
f.controller.abort(new LegacyControllerLeaseLostError("renewal_error"));
await f.completion.failRunSetup(f.setup(new Error("usage capture write failed"), true));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "setup_failed",
resultJson: { cancellation: { source: "controller_lease", reason: "Legacy controller lease lost (renewal_error)" } } });
expect(f.deps.finalizeAgentStatus).toHaveBeenCalledWith(f.agent.id, "failed", expect.any(String), expect.objectContaining({ keepIdleOnFailure: false }));
});
it("leaves a winning terminal outcome and agent state untouched after a late setup error", async () => {
const f = await fixture({ status: "succeeded", resultJson: { winner: "completion" } });
await f.completion.failRunSetup(f.setup(new Error("late setup error")));
@@ -7,7 +7,7 @@ import { CONFIGURATION_INCOMPLETE_FAILURE_CODE, ConfigurationIncompleteFailure }
import { isWorkspaceValidationFailedRun, isWorkspaceValidationFailure, attachPaperclipSessionMetadataToSessionParams } from "./workspaces.js";
import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../legacy-workspace-restore-recovery.js";
import { cancellationResultJson, controllerLeaseLossCancellation, readRunCancellation } from "../run-cancellation.js";
import { LegacyControllerLeaseLostError } from "../legacy-controller-lease.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE, LegacyControllerLeaseLostError } from "../legacy-controller-lease.js";
import { accountRunCost } from "../run-cost-accounting.js";
import { isConversation, settleConversationTurn } from "../agent-conversations.js";
import { legacyExecutionNeedsReconciliationWithEvidence } from "../legacy-execution-recovery.js";
@@ -133,6 +133,9 @@ export interface FailHeartbeatRunSetupInput {
run: Run;
runId: string;
outerErr: unknown;
signal: AbortSignal;
/** True once a legacy adapter or native dispatch began provider work. */
providerDispatchStarted: boolean;
identityRedactor: Pick<ReturnType<typeof createAgentIdentityRedactor>, "redact">;
readFailureReportSecrets: () => string[];
}
@@ -1258,7 +1261,13 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
}
async function failRunSetup(input: FailHeartbeatRunSetupInput) {
const { run, runId, outerErr, identityRedactor, readFailureReportSecrets } = input;
const { run, runId, outerErr, signal, providerDispatchStarted, identityRedactor, readFailureReportSecrets } = input;
// The lease watchdog stopped preparation because this host no longer owns
// the run. That is host infrastructure, not a defect in the agent or setup.
// A post-dispatch error can also land here (for example a failed usage
// capture write), so only a stop before dispatch is a retryable lease loss.
const controllerLeaseLost = !providerDispatchStarted &&
signal.reason instanceof LegacyControllerLeaseLostError;
// Setup code before adapter.execute threw (e.g. ensureRuntimeState, resolveWorkspaceForRun).
// The inner catch did not fire, so we must record the failure here.
const message = redactCurrentUserText(
@@ -1297,6 +1306,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
nonRetryablePreflightFailureCode(outerErr);
const workspaceGitScanFailure = isWorkspaceGitScanError(outerErr) ? outerErr : null;
const setupFailureErrorCode =
(controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ??
workspaceGitScanFailure?.code ??
workspaceValidationSetupFailure?.code ??
configurationIncompleteSetupFailure?.code ??
@@ -1341,8 +1351,10 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
const setupFailureResultJson = {
...setupFailureDetails,
...(connectionFailure ? { connectionFailure } : {}),
// A lease lost during setup surfaces as the thrown abort reason.
// A lease lost during setup surfaces as the thrown abort reason, or as
// the signal's reason when a setup step threw a different error after it.
...controllerLeaseLossEvidence(outerErr),
...controllerLeaseLossResultJson(signal),
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
};
const setupFailureWrite = await setRunStatusIfRunning(runId, "failed", {
@@ -1475,7 +1487,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
// Keep the failed run and its safe provider refusal authoritative,
// but return the agent to idle so clients do not also announce a
// misleading agent-wide error for the same rejected chat turn.
keepIdleOnFailure: Boolean(nonRetryablePreflightCode),
keepIdleOnFailure: Boolean(nonRetryablePreflightCode) || controllerLeaseLost,
}).catch(() => undefined);
}
}
@@ -24,6 +24,7 @@ import {
rethrowNativeCancellationLockConflict,
} from "../native-runtime/native-cancellation-request.js";
import { cancellationResultJson } from "../run-cancellation.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
import { accountRunCost } from "../run-cost-accounting.js";
import { isWaitingConversation } from "../agent-conversations.js";
import { PROCESS_IDENTITY_RECORDED } from "../native-local-process-stop.js";
@@ -247,6 +248,7 @@ export const NON_RETRYABLE_PREFLIGHT_FAILURE_CODES = new Set<string>([
// of these codes when a failure happens before `adapter.execute`.
const PRE_ADAPTER_SETUP_FAILURE_CODES = new Set<string>([
"setup_failed",
CONTROLLER_LEASE_LOST_FAILURE_CODE,
CONFIGURATION_INCOMPLETE_FAILURE_CODE,
WORKSPACE_VALIDATION_FAILURE_CODE,
...NON_RETRYABLE_PREFLIGHT_FAILURE_CODES,
@@ -10,6 +10,10 @@ export const LEGACY_CONTROLLER_RENEW_MS = 10_000;
type Run = typeof heartbeatRuns.$inferSelect;
/** A lease lost before adapter dispatch: no provider work began, so the
* continuation is retryable rather than an opaque `setup_failed`. */
export const CONTROLLER_LEASE_LOST_FAILURE_CODE = "controller_lease_lost";
/** Abort reason for a lost lease, so completion can attribute the stop. */
export class LegacyControllerLeaseLostError extends Error {
constructor(
@@ -21,6 +21,12 @@ describe("classifyAdapterFailureForRecovery", () => {
expect(classifyContinuationFailure(unknown as never).kind).toBe("transient_infra");
expect(isConfigurationIncompleteFailedRun(unknown)).toBe(false);
});
it("retries a continuation whose host lost the controller lease before dispatch", () => {
expect(classifyContinuationFailure({ errorCode: "controller_lease_lost", error: "Legacy controller lease lost", resultJson: null } as never))
.toMatchObject({ kind: "transient_infra", maxAttempts: 3 });
expect(classifyContinuationFailure({ errorCode: "setup_failed", error: "Legacy controller lease lost", resultJson: null } as never).kind)
.toBe("non_retryable");
});
it.each(["acpx_auth_required", "claude_auth_required", "codex_auth_required", "adapter_auth_missing", "refresh_token_reused", "authentication_required"])("does not automatically retry provider authentication failure %s", (errorCode) => {
const classification = classifyRunLiveness({ runStatus: "failed", issue: null, errorCode, authenticationRepairRequested: true });
const run = { errorCode, ...classification };
+3 -1
View File
@@ -13,7 +13,7 @@ import {
decideLegacyContinuation, legacyDispositionEpisode, legacyDispositionFingerprint,
LEGACY_DISPOSITION_REPAIR_INSTRUCTION, type LegacyDispositionEpisode,
} from "./legacy-continuation.js";
import { hasLiveLegacyController } from "../legacy-controller-lease.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController } from "../legacy-controller-lease.js";
import { isCompletedOnboardingHandoffWake } from "../chat-completion-delivery.js";
import { instanceSettingsService } from "../instance-settings.js";
import { isWaitingConversation, settleConversationTurn, deliverConversationComments } from "../agent-conversations.js";
@@ -495,6 +495,8 @@ const TRANSIENT_INFRA_CONTINUATION_ERROR_CODES = new Set<string>([
"claude_transient_upstream",
"provider_quota",
"timeout",
// The host lost run ownership before dispatch; no provider work began.
CONTROLLER_LEASE_LOST_FAILURE_CODE,
]);
const NON_RETRYABLE_CONTINUATION_ERROR_CODES = new Set<string>([