mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade stops the old server gracefully. A legacy run that is still preparing ends `interrupted` with `server_shutdown_interrupted`. The new server must continue it automatically (#21019, #21046). > - A run that a person started with Retry (or a message) carries `explicitUserContinuation`. Its automatic retries need their own receipt from `admitExplicitContinuationRetry` (#14987, #19384). > - That admission refused every shutdown-interrupted Retry run. The task then stayed in progress with no run, and the stranded-task sweep refused it again about every 2.5 minutes, forever. A person had to press Retry. > - An upgrade is routine. It must never leave a task that needs a manual Retry. > - This pull request lets the person's original authorization cover a continuation after a shutdown that stopped the run before provider work started. All other checks of the gate stay. > - The benefit is that Retry-started tasks continue through an upgrade, and tasks that earlier releases left waiting continue on their own after deploy. ## Linked Issues or Issue Description Refs #21019, #21046, #14987, #19384 **What happened?** On a hosted staging server, each automatic upgrade stopped legacy runs that were still preparing (`executionRecovery {kind: bootstrap, providerWorkStarted: false}`). When the run came from a person's Retry (wake reason `task_retry`), the continuation was refused with "The automatic retry could not revalidate the original user continuation." (`continuation_user_authorization_missing`). The task stayed `in_progress` with no run and no next retry. Runs from older upgrades were refused again by each sweep. A run from the same upgrade that did not come from a Retry got its continuation in the `interrupted_preparation` lane. **Expected behavior** A Retry-started run that a graceful shutdown stopped before provider work continues automatically, once, under the person's original Retry. Tasks that an earlier release already refused continue after deploy with no manual step. **Root cause** Three conditions refused it (`server/src/services/heartbeat/retries.ts`, `server/src/services/explicit-native-continuation.ts`): 1. The scheduler admits an explicit continuation only for the `transient_failure`, `server_restart`, preparation-wait and agent-directory lanes. A shutdown-interrupted preparation with unspent failure budget uses `interrupted_preparation`, so it was refused before admission ran. 2. `admitExplicitContinuationRetry` accepted only a `failed` or `timed_out` parent. These parents are `interrupted`. 3. It required `task.executionRunId` to be the parent. The refusal released the claim (`releaseIssueExecutionAndPromote`), so every later sweep was refused by this check too. **Paperclip version or commit** master `f30005440b`. ## What Changed - `retries.ts`: an explicit continuation may also use the `interrupted_preparation` lane. Lane choice is unchanged (see below). - `explicit-native-continuation.ts`: `admitExplicitContinuationRetry` also accepts a parent that is `interrupted` by a graceful shutdown (`isServerReplacementInterruption`, so `server_shutdown_interrupted`) with bootstrap evidence `providerWorkStarted: false` (`isShutdownInterruptedPreparation`). - For such a parent only, a released claim (`executionRunId` null) is accepted. Admission restores the claim to the parent under the task lock, as the shutdown path leaves it, so the scheduler passes the claim to the successor and dispatch accepts it. If the parent validation then refuses, admission releases the claim again in the same transaction. A claim that another run holds still refuses. - `doc/execution-semantics.md`: describe the rule and the conditions that still refuse. Also apply the sentence that #19384 left pending (a plain task Retry is renewable). - No schema change, no backfill. Nothing changes on the path that starts a turn. **Which lane a graceful-shutdown legacy preparation uses (after #21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation` for an `interrupted` run with the bootstrap evidence. `usesServerRestartLane(run, true)` returns false while the failure budget is unspent, so the run uses `interrupted_preparation`: it does not spend the failure budget, the first continuation is due at once, and it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a spent failure budget it uses `server_restart`. This is the intended design (#21046 comment on `usesServerRestartLane`), so no lane change. The `server_restart` lane is for `process_lost` replaced preparations and for runs past dispatch. **Conditions that still refuse** (each has a test or an existing test): - The task has another assignee, or is `done` or `cancelled`. - Another run holds the task's claim. - Newer work on the task (a run created after the parent, a retry child, or a live run). - The Retry request changed: other actor, not admitted, other generation, bound to another run, a chat Retry, a fresh-workspace Retry. A message continuation with an edited or deleted message. - A missing or revoked receipt (dispatch validation of the parent fails). - An execution blocker on the task, or a live adapter controller for the parent. - An `interrupted` parent with another cause (for example an operator interrupt), or with no bootstrap evidence, or with `providerWorkStarted: true`. A parent that failed after provider work keeps the existing rules (`failed`/`timed_out` only). - Native runtime parents (unchanged, see below). **Native runtime.** A native `paperclip_runner` run is not interrupted by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and the next server reattaches it (`restart_suspended`). It does not reach this gate. Native replacement paths (`native-safe-replacement`, `native-provider-capacity-retry`) remove `explicitUserContinuation` from the successor and use their own contract. So the native path does not have this refusal, and this pull request does not change it. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` (no successor), green on this branch: - "continues a Retry-admitted run that a graceful shutdown stopped while it was preparing": production shape. A legacy run admitted by a task Retry, `preparing`, owned by this server, with bootstrap evidence. `drainRunningRunsForShutdown("SIGTERM")` interrupts it with `server_shutdown_interrupted` and schedules one `interrupted_preparation` successor (`failureRetries: 0`, `interruptedPreparations: 1`) with its own receipt. The claim passes to the successor and dispatch accepts it. A concurrent stranded sweep and a second schedule call reuse the same successor. - "continues a task an earlier release left waiting after refusing its shutdown continuation" (a direct Retry run, and an automatic retry of a Retry run): the row and task as an older release left them (claim released). Two concurrent stranded sweeps schedule exactly one successor; the claim passes to it; dispatch accepts it. - "does not continue a shutdown-interrupted Retry run without its original authority" for: provider work started, no bootstrap proof, another interruption cause, claim held by another run, reassigned, superseded, other Retry actor, cancelled task. No successor, no receipt, and the claim is left as it was. - I confirmed that each source change is needed: without the lane change both main tests fail; without the released-claim rule the older-release test fails. - Suites: `task-retry`, `explicit-native-continuation`, `legacy-explicit-continuation-recovery`, `quarantined-task-retry` (587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling` (520/520). - `tsc --noEmit` for `server`. - `pnpm verify` selects a broad run (exit 2) because `retries.ts` is central; required CI covers that. **After deploy, the next audit should show:** a Retry-started run that a shutdown stops while preparing gets a `scheduled_retry` successor with `scheduledRetryReason: interrupted_preparation` (or `server_restart` with a spent failure budget) and no `continuation_user_authorization_missing` event. The tasks that earlier upgrades left waiting get one such successor on the first stranded sweep of the new server, with the task claim on the successor. ## Risks - Behavior change: a shutdown-interrupted preparation of a Retry or message turn now continues automatically, within the existing uncharged and bounded lanes. The run did no provider work, so there is nothing to reconcile. - Rows saved by older releases: a refused row whose claim was released is continued once. If another run took the claim, or newer work exists, it stays refused. - Two servers at once: admission runs in the scheduler transaction under the task and run row locks, and the scheduler reuses an existing successor (`retryOfRunId`). The tests run two sweeps concurrently and get one successor. An old server that still refuses cannot clear the claim after the new server moved it to the successor (its release matches only the stopped run). - The claim restore writes `executionRunId` for a terminal run inside the scheduling transaction. If admission refuses after the restore, it releases the claim again before the transaction commits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
814fe5d20f
commit
c4927e1ec2
4 files changed
+152
-5
No files matched your search
@@ -1697,7 +1697,7 @@ or the exact provenance inside its verified committed saved snapshot. The latter
|
||||
uses one temporary read-only archive verification per preparation when needed.
|
||||
Task relationships alone do not authorize importing or retaining files.
|
||||
|
||||
When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. One-run Retry and Interrupt intents retain their separate admission contracts; they are not renewable user-message authority. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority.
|
||||
When a legacy turn admitted by an explicit user message fails or times out, a bounded transient retry needs its own durable continuation authorization. Initial admission binds the message's exact body digest and revision. Scheduling requires that original binding and revalidates the human author, source task, assignee, and execution ownership, then records the successor's receipt in the same transaction as its run. Historical receipts without a message binding cannot authorize automatic retries and are not backfilled. The parent receipt remains intact. Dispatch checks the exact successor, retry parent, and unchanged message binding again. Deleted or edited input, changed actors or ownership, superseded work, and duplicate successors cannot renew authority. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority. A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request stays bound to the run it admitted. A graceful server shutdown (an upgrade or restart, `server_shutdown_interrupted`) that stops such a turn while it is still preparing, with bootstrap evidence that provider work did not start, is not a failure: the original user authorization still covers the work, so the turn continues automatically in the uncharged `interrupted_preparation` lane (or `server_restart` once its failure budget is spent) with its own receipt. When a release without this rule refused that continuation and released the task's claim, the stranded-task sweep restores the claim to the stopped run and admits the continuation once; it refuses when another run holds the claim. All other checks still apply: a changed assignee or Retry actor, a cancelled or completed task, newer or live work on the task, a revoked or missing receipt, a different interruption cause, and a turn whose provider work started keep refusing. Other retry policies cannot borrow this receipt: their scheduling fails closed. Successful explicit turns do not create a prose-only missing-comment follow-up with spent authority.
|
||||
|
||||
Explicit retry admission checks the local parent adapter controller and verified environment cleanup. The existing queue-first release policy may retain only its own terminal execution claim while that cleanup is pending; finalization and the periodic stale-lock sweep reconsider the same policy after cleanup. Generic terminal-lock cleanup and stale checkout adoption leave these claims for that settlement. New user input, reassignment, and a successor's claim take precedence. Exhausted retries and revoked or missing authorization release the terminal claim without requesting another retry. Pending or failed environment cleanup cannot mint a successor receipt.
|
||||
|
||||
|
||||
@@ -37,6 +37,8 @@ import { historicalAdapterType, isConversationAdapter } from "./conversation-con
|
||||
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
|
||||
import { firstTurnDeliveredCommentIds } from "./queued-comment-delivery.js";
|
||||
import { canResumePreservedResultFinalization } from "./preserved-result-finalization.js";
|
||||
import { isServerReplacementInterruption } from "./server-replacement-interruption.js";
|
||||
import { normalizeAgentNameKey } from "./agent-name-key.js";
|
||||
|
||||
type Run = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "companyId" | "agentId" | "status" | "startedAt" | "finishedAt" | "createdAt" |
|
||||
@@ -781,11 +783,31 @@ async function readTaskRetryRetryAuthority(db: Db, input: {
|
||||
sourceAuthorizationId: receipt?.id ?? wake.id, evidence: {} };
|
||||
}
|
||||
|
||||
/** A graceful server shutdown (an upgrade or restart) stopped this legacy run
|
||||
* while it was still preparing: the bootstrap evidence proves that no provider
|
||||
* work started. The interruption is not a failed attempt, so the user's
|
||||
* original authorization still covers the work it admitted.
|
||||
*/
|
||||
function isShutdownInterruptedPreparation(run: typeof heartbeatRuns.$inferSelect): boolean {
|
||||
const recovery = run.resultJson?.executionRecovery;
|
||||
const evidence = recovery && typeof recovery === "object" && !Array.isArray(recovery)
|
||||
? recovery as Record<string, unknown> : null;
|
||||
return run.status === "interrupted" && isServerReplacementInterruption(run)
|
||||
&& evidence?.kind === "bootstrap" && evidence.providerWorkStarted === false;
|
||||
}
|
||||
|
||||
/** Re-admit a bounded automatic retry without lending it its parent's receipt.
|
||||
* The caller holds the task lock and creates the successor in this transaction.
|
||||
* Unchanged user messages and plain task Retries are renewable; Interrupt and
|
||||
* failed-run retry intents keep their separate one-run contracts. The
|
||||
* scheduler still owns retry and cleanup policy.
|
||||
*
|
||||
* A failed or timed-out parent must still hold the task's claim. A parent that
|
||||
* a graceful shutdown stopped before provider work may also find the claim
|
||||
* released: releases without this rule refused its continuation and released
|
||||
* the claim, leaving the task in progress with no run. Admission then restores
|
||||
* the claim to the parent, as the shutdown path leaves it, and every other
|
||||
* check below still applies.
|
||||
*/
|
||||
export async function admitExplicitContinuationRetry(input: {
|
||||
db: Db; companyId: string; issueId: string; agentId: string;
|
||||
@@ -799,9 +821,12 @@ export async function admitExplicitContinuationRetry(input: {
|
||||
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, input.parentRunId),
|
||||
eq(heartbeatRuns.agentId, agentId),
|
||||
)).for("update");
|
||||
if (!task || task.assigneeAgentId !== agentId || task.executionRunId !== input.parentRunId ||
|
||||
const shutdownInterrupted = Boolean(parent && isShutdownInterruptedPreparation(parent));
|
||||
const releasedClaim = Boolean(task && task.executionRunId === null && shutdownInterrupted);
|
||||
if (!task || task.assigneeAgentId !== agentId || (task.executionRunId !== input.parentRunId && !releasedClaim) ||
|
||||
["done", "cancelled"].includes(task.status) || !parent ||
|
||||
(!["failed", "timed_out"].includes(parent.status) && !isAgentDirectoryPreparationWait(parent)) || !parent.finishedAt ||
|
||||
(!["failed", "timed_out"].includes(parent.status) && !isAgentDirectoryPreparationWait(parent) && !shutdownInterrupted) ||
|
||||
!parent.finishedAt ||
|
||||
parent.runtimeMode !== "legacy" || parent.contextSnapshot?.issueId !== issueId ||
|
||||
(parent.nativeIssueId !== null && parent.nativeIssueId !== issueId) ||
|
||||
adapterExecutionControls.has(parent.id) ||
|
||||
@@ -825,6 +850,14 @@ export async function admitExplicitContinuationRetry(input: {
|
||||
inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"])),
|
||||
)).limit(1);
|
||||
if (superseding) return null;
|
||||
const [{ name: agentName } = { name: null }] = releasedClaim
|
||||
? await db.select({ name: agents.name }).from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)))
|
||||
: [];
|
||||
const claim = (executionRunId: string | null) => db.update(issues).set(executionRunId
|
||||
? { executionRunId, executionAgentNameKey: normalizeAgentNameKey(agentName), executionLockedAt: input.now }
|
||||
: { executionRunId: null, executionAgentNameKey: null, executionLockedAt: null })
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)));
|
||||
if (releasedClaim) await claim(parent.id);
|
||||
// Reuse dispatch's complete source/actor/receipt validation on the parent.
|
||||
// It proves only the parent; the new receipt below authorizes the successor.
|
||||
try {
|
||||
@@ -832,7 +865,10 @@ export async function admitExplicitContinuationRetry(input: {
|
||||
context, summary: null, exposeLowTrustRaw: false });
|
||||
} catch (error) {
|
||||
if (error instanceof Error && ["continuation_user_authorization_missing",
|
||||
"continuation_source_context_missing", "continuation_task_ownership_changed"].includes(error.message)) return null;
|
||||
"continuation_source_context_missing", "continuation_task_ownership_changed"].includes(error.message)) {
|
||||
if (releasedClaim) await claim(null);
|
||||
return null;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const continuation = { previousRunId: parent.id, ...authority.continuation };
|
||||
|
||||
@@ -1139,8 +1139,11 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
|
||||
|
||||
const scheduledRunId = randomUUID();
|
||||
if (contextSnapshot.explicitUserContinuation) {
|
||||
// A preparation that a shutdown or restart stopped before provider
|
||||
// work continues in the uncharged interrupted_preparation lane while
|
||||
// its failure budget is unspent, and in server_restart after that.
|
||||
const continuation = issueId && (retryReason === "transient_failure" || retryReason === SERVER_RESTART_RETRY_REASON
|
||||
|| isPreparationWaitRetryReason(retryReason)
|
||||
|| retryReason === INTERRUPTED_PREPARATION_RETRY_REASON || isPreparationWaitRetryReason(retryReason)
|
||||
|| retryReason === AGENT_DIRECTORY_BUSY_RETRY_REASON)
|
||||
? await admitExplicitContinuationRetry({
|
||||
db: tx as unknown as Db, companyId: run.companyId, issueId, agentId: run.agentId,
|
||||
|
||||
@@ -19,6 +19,8 @@ import { taskRetryRunAncestry } from "./recovery-run-ancestry.js";
|
||||
import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js";
|
||||
import { isInterruptedNativePreparation } from "./cancelled-native-startup.js";
|
||||
import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js";
|
||||
import { legacyControllerBootId } from "./legacy-controller-lease.js";
|
||||
import { currentControllerProcessStamp } from "./controller-process-identity.js";
|
||||
|
||||
describe("durable task Retry admission", () => {
|
||||
let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
@@ -192,6 +194,112 @@ describe("durable task Retry admission", () => {
|
||||
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.input.issueId)))
|
||||
.toHaveLength(0);
|
||||
});
|
||||
// Production shape (staging upgrades): a person's Retry admitted a run that was
|
||||
// still preparing, with no provider work started, when the upgrade stopped its
|
||||
// server gracefully. The interruption is not a failure and the Retry already
|
||||
// authorized this work, so recovery continues it with no new click.
|
||||
async function seedPreparingRetryAdmittedRun() {
|
||||
const f = await seedFailedRetryAdmittedRun();
|
||||
const [parent] = await db.update(heartbeatRuns).set({ status: "running", errorCode: null, finishedAt: null,
|
||||
controllerBootId: legacyControllerBootId, controllerProcess: currentControllerProcessStamp(),
|
||||
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '1 minute'`,
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
}).where(eq(heartbeatRuns.id, f.parent.id)).returning();
|
||||
return { ...f, parent };
|
||||
}
|
||||
const successorsOf = (runId: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId));
|
||||
it("continues a Retry-admitted run that a graceful shutdown stopped while it was preparing", async () => {
|
||||
const f = await seedPreparingRetryAdmittedRun();
|
||||
const drained = await heartbeatService(db).drainRunningRunsForShutdown("SIGTERM", new Date(), [f.parent.id]);
|
||||
expect(drained.interruptedRunIds).toEqual([f.parent.id]);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.parent.id)))[0])
|
||||
.toMatchObject({ status: "interrupted", errorCode: "server_shutdown_interrupted" });
|
||||
const successors = await successorsOf(f.parent.id);
|
||||
// The uncharged preparation lane: due at once, no failure retry spent.
|
||||
expect(successors).toEqual([expect.objectContaining({ status: "scheduled_retry",
|
||||
scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 })]);
|
||||
expect(drained.retryRunIds).toEqual([successors[0]!.id]);
|
||||
expect(successors[0]!.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0, interruptedPreparations: 1 });
|
||||
expect(successors[0]!.contextSnapshot?.explicitUserContinuation).toEqual({
|
||||
previousRunId: f.parent.id, commentId: null, taskRetryRequestId: f.retryRequestId });
|
||||
// The continuation holds the task's claim, and dispatch accepts it from its own receipt.
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]!.executionRunId).toBe(successors[0]!.id);
|
||||
await expect(dispatchRun(f.input, successors[0]!)).resolves.toMatchObject({ interruptedRunId: f.parent.id });
|
||||
// The replacement servers' own sweeps reuse the one continuation.
|
||||
await Promise.all([heartbeatService(db).reconcileStrandedAssignedIssues({ companyId: f.input.companyId }),
|
||||
heartbeatService(db).scheduleBoundedRetry(f.parent.id)]);
|
||||
expect(await successorsOf(f.parent.id)).toEqual([expect.objectContaining({ id: successors[0]!.id })]);
|
||||
});
|
||||
// Tasks a release without this fix already refused: the refusal released the
|
||||
// task's claim and left it in progress with no run. The periodic sweep admits
|
||||
// them now, once, with no manual step.
|
||||
async function seedRefusedShutdownContinuation(chained = false) {
|
||||
const f = await seedPreparingRetryAdmittedRun();
|
||||
let interruptedId = f.parent.id;
|
||||
if (chained) {
|
||||
// The person's Retry run failed transiently first; its automatic retry
|
||||
// (with its own receipt) is the run the shutdown stopped.
|
||||
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "remote_workspace_restore_failed", finishedAt: f.now })
|
||||
.where(eq(heartbeatRuns.id, f.parent.id));
|
||||
const first = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: f.now, delayMs: 0 });
|
||||
if (first.outcome !== "scheduled") throw new Error(`automatic retry not scheduled: ${first.outcome}`);
|
||||
interruptedId = first.run.id;
|
||||
}
|
||||
const finishedAt = new Date(Date.now() - 60_000);
|
||||
const [parent] = await db.update(heartbeatRuns).set({ status: "interrupted", errorCode: "server_shutdown_interrupted",
|
||||
error: "Interrupted by graceful server shutdown (SIGTERM)", startedAt: finishedAt, finishedAt,
|
||||
runtimeMode: "legacy", executionStage: "preparing", controllerLeaseExpiresAt: null, controllerBootId: randomUUID(),
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
}).where(eq(heartbeatRuns.id, interruptedId)).returning();
|
||||
await db.update(issues).set({ status: "in_progress", executionRunId: null, executionLockedAt: null })
|
||||
.where(eq(issues.id, f.input.issueId));
|
||||
return { ...f, parent };
|
||||
}
|
||||
it.each([{ chained: false, name: "Retry run" }, { chained: true, name: "automatic retry of a Retry run" }])(
|
||||
"continues a task an earlier release left waiting after refusing its shutdown continuation: $name", async ({ chained }) => {
|
||||
const f = await seedRefusedShutdownContinuation(chained);
|
||||
const scope = { companyId: f.input.companyId };
|
||||
await Promise.all([heartbeatService(db).reconcileStrandedAssignedIssues(scope),
|
||||
heartbeatService(db).reconcileStrandedAssignedIssues(scope)]);
|
||||
const successors = await successorsOf(f.parent.id);
|
||||
expect(successors).toEqual([expect.objectContaining({ status: "scheduled_retry",
|
||||
scheduledRetryReason: "interrupted_preparation" })]);
|
||||
expect(successors[0]!.contextSnapshot?.explicitUserContinuation).toEqual({
|
||||
previousRunId: f.parent.id, commentId: null, taskRetryRequestId: f.retryRequestId });
|
||||
// The released claim passes to the continuation, as on the shutdown path.
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]!.executionRunId).toBe(successors[0]!.id);
|
||||
await expect(dispatchRun(f.input, successors[0]!)).resolves.toMatchObject({ interruptedRunId: f.parent.id });
|
||||
});
|
||||
it.each(["provider_work_started", "no_bootstrap_proof", "operator_interrupt", "claimed_by_other_run", "reassigned",
|
||||
"superseded", "wrong_actor", "cancelled_task"] as const)(
|
||||
"does not continue a shutdown-interrupted Retry run without its original authority: %s", async kind => {
|
||||
const f = await seedRefusedShutdownContinuation();
|
||||
if (kind === "provider_work_started" || kind === "no_bootstrap_proof") await db.update(heartbeatRuns).set({
|
||||
resultJson: kind === "no_bootstrap_proof" ? {} : { executionRecovery: { kind: "bootstrap", providerWorkStarted: true } },
|
||||
}).where(eq(heartbeatRuns.id, f.parent.id));
|
||||
if (kind === "operator_interrupt") await db.update(heartbeatRuns).set({ errorCode: "operator_interrupted" })
|
||||
.where(eq(heartbeatRuns.id, f.parent.id));
|
||||
if (kind === "claimed_by_other_run") {
|
||||
const [other] = await db.insert(heartbeatRuns).values({ companyId: f.input.companyId, agentId: f.parent.agentId,
|
||||
status: "failed", finishedAt: f.parent.finishedAt, createdAt: new Date(f.parent.createdAt.getTime() - 1000),
|
||||
contextSnapshot: { issueId: f.input.issueId } }).returning();
|
||||
await db.update(issues).set({ executionRunId: other.id }).where(eq(issues.id, f.input.issueId));
|
||||
}
|
||||
if (kind === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.input.issueId));
|
||||
if (kind === "cancelled_task") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.input.issueId));
|
||||
if (kind === "superseded") await db.insert(heartbeatRuns).values({ companyId: f.input.companyId, agentId: f.parent.agentId,
|
||||
status: "succeeded", contextSnapshot: { issueId: f.input.issueId }, createdAt: new Date() });
|
||||
if (kind === "wrong_actor") await db.update(agentWakeupRequests).set({ requestedByActorId: "other-user" })
|
||||
.where(eq(agentWakeupRequests.id, f.retryRequestId));
|
||||
const [parent] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.parent.id));
|
||||
const result = await heartbeatService(db).scheduleBoundedRetry(parent.id, { now: new Date() });
|
||||
expect(result.outcome).not.toBe("scheduled");
|
||||
expect(await successorsOf(f.parent.id)).toHaveLength(0);
|
||||
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.input.issueId)))
|
||||
.toHaveLength(0);
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.input.issueId)))[0]!.executionRunId)
|
||||
.toEqual(kind === "claimed_by_other_run" ? expect.any(String) : null);
|
||||
});
|
||||
async function seedRefusedRetry() {
|
||||
const input = await seed();
|
||||
const now = Date.now();
|
||||
|
||||
Reference in new issue
Block a user