mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(recovery): let a task Retry run keep its bounded transient retry (#19384)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can click Retry on a blocked task. Paperclip then admits one new run that continues in the saved workspace > - That run can fail before the provider starts, for example when the download of the saved workspace snapshot gets a transient `ECONNRESET` > - The scheduler classifies this failure as transient and tries to schedule the normal bounded retry > - The automatic-retry admission accepts only continuations that a user comment started. It refuses a Retry continuation (`commentId: null`, `taskRetryRequestId` set) with `continuation_user_authorization_missing` > - So the task does not retry. Paperclip escalates the restore failure, and the task becomes blocked again after one network reset > - This pull request lets a plain task Retry renew its authority for the bounded transient retry, with its own receipt for each new run > - The benefit is that a transient setup failure after Retry uses the normal retry budget and does not block the task ## Linked Issues or Issue Description **What happened?** A run started by the task Retry button failed during setup: `remote_workspace_restore_failed: step=snapshot_download; storage=transport_closed code=ECONNRESET`. The run event said "The automatic retry could not revalidate the original user continuation." (retry reason `transient_failure`, attempt 1 of 2). No retry was scheduled, and the task became blocked. **Expected behavior** A transient setup failure of a Retry-admitted run gets the same bounded automatic retry as a run that a user comment started. **Root cause** `admitExplicitContinuationRetry` in `server/src/services/explicit-native-continuation.ts` required `explicitUserContinuation.commentId` to be a GUID. A Retry continuation has `commentId: null`. Also, dispatch in `server/src/services/execution-continuation.ts` bound a Retry receipt to exactly the run that it admitted, so a successor run could not prove its authority. Refs #19207 (it named remote restore failures and left a board recovery action) Refs #19340 (it retries quarantined tasks in their saved workspace) Refs #19326 (same bounded retry path for an out-of-date Runner) ## What Changed - `admitExplicitContinuationRetry` now accepts two kinds of authority. The comment path is unchanged and moved into `readCommentRetryAuthority`. The new `readTaskRetryRetryAuthority` accepts a plain task Retry only. The Retry request must be coalesced, from a user, with the same generation and actor, strategy `continue`, and no chat, queued-message, canonical-intent or fresh-workspace readiness. Interrupt and failed-run retries stay one-run. - The successor run gets its own `explicit_user_continuation_retry` receipt with the same `taskRetryRequestId` and actor and `automaticRetry.sourceRunId` set to the parent. The Retry request stays bound to the run that it admitted. - Dispatch (`buildExecutionContinuation`) accepts a Retry successor only from its own receipt, its exact `retryOfRunId` parent, the current `executionRunId` and the unchanged Retry intent. The direct path for the admitted run is unchanged. - Tests in `server/src/services/task-retry.test.ts`: one regression for the full retry chain, and eight refusal cases. ## Verification - Red: before the fix, the new test failed with `{ outcome: "not_scheduled", errorCode: "continuation_user_authorization_missing" }`. This is the same refusal as the incident. - Green: `pnpm --filter @paperclipai/server exec vitest run src/services/task-retry.test.ts` gives 58/58. - The regression test proves these points: the first retry is scheduled with `{ previousRunId: parent, commentId: null, taskRetryRequestId }`; the Retry request stays bound to the admitted run; dispatch accepts the successor; a repeated schedule call adopts the same successor without a second receipt; a second transient failure retries from its immediate parent; the budget then stops with `retry_exhausted`; dispatch refuses the successor when its own receipt is removed. - The refusal cases prove that no successor and no receipt are created after: wrong actor, unadmitted request, wrong generation, request bound to another run, chat Retry, fresh-workspace readiness, reassignment, superseding work. - Nearby suites pass (601/601): `execution-continuation`, `legacy-explicit-continuation-recovery`, `explicit-native-continuation`, `execution-continuation-provenance`, `quarantined-task-retry`, `issue-queued-comments-routes`, `task-retry`. - I did not run the layered server typecheck locally because of a disk limit in the sandbox. CI must confirm it. ## Risks - Behavior change: a plain task Retry run that fails transiently now retries automatically within the existing bounded budget (2 attempts). The scheduler still decides which failures are transient. The same rule already applies to runs that a user comment started. - Chat, queued-message and fresh-workspace Retries, failed-run retries and Interrupt keep their one-run contracts. - No schema or migration change. - Documentation gap: `doc/execution-semantics.md` (line 1210) says "One-run Retry and Interrupt intents retain their separate admission contracts; they are not renewable user-message authority." That sentence must change to describe this rule. The author's publication path cannot carry the full 320 KB file, so that one-sentence edit is still pending. The proposed text: "A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request itself stays bound to the run it admitted. Dispatch accepts that successor only from its own receipt, its exact retry parent, and the unchanged admitted Retry request, so a transient failure before or during that run, such as a reset saved-workspace download, uses the normal bounded retry budget instead of blocking the task. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority." > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code as a Paperclip agent, with tool use and code execution (shell, Vitest on embedded PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (the `doc/execution-semantics.md` sentence above is pending) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
96516a8622
commit
4033f7f17f
3 files changed
+195
-35
No files matched your search
@@ -458,7 +458,6 @@ export async function buildExecutionContinuation(input: {
|
||||
eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.reason, "task_retry"),
|
||||
eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.status, "coalesced"),
|
||||
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
||||
input.runId ? eq(agentWakeupRequests.runId, input.runId) : undefined,
|
||||
)) : [];
|
||||
const taskRetry = readTaskRetryIntent(taskRetryWake?.payload);
|
||||
const retryAuthorization = taskRetry && taskRetry.generationId === taskRetryWake?.id
|
||||
@@ -485,9 +484,15 @@ export async function buildExecutionContinuation(input: {
|
||||
input.runId ? eq(agentWakeupRequests.runId, input.runId) : undefined,
|
||||
)) : [];
|
||||
const authorization = continuationAuthorizations.find(value => taskRetryId
|
||||
? value.taskRetryRequestId === taskRetryId && taskRetryWake?.runId === value.runId &&
|
||||
taskRetryWake.requestedByActorId === value.actorId &&
|
||||
object(taskRetryWake.payload?.taskRetry).sourceRunId === value.previousRunId
|
||||
? value.taskRetryRequestId === taskRetryId && taskRetryWake?.requestedByActorId === value.actorId && (
|
||||
// The Retry receipt stays bound to the run it admitted. A bounded
|
||||
// automatic retry of that run carries its own admission receipt.
|
||||
value.automaticRetry
|
||||
? taskRetry?.generationId === taskRetryWake.id && taskRetry.actorId === value.actorId &&
|
||||
object(value.automaticRetry).sourceRunId === explicitUserSource && issue.executionRunId === input.runId &&
|
||||
priorRuns.some(run => run.id === value.runId && run.retryOfRunId === explicitUserSource)
|
||||
: taskRetryWake.runId === value.runId &&
|
||||
object(taskRetryWake.payload?.taskRetry).sourceRunId === value.previousRunId)
|
||||
: failedRunId
|
||||
? value.failedRunId === failedRunId && retryWakes.some(wake =>
|
||||
wake.runId === value.runId && wake.requestedByActorId === value.actorId &&
|
||||
|
||||
@@ -576,15 +576,81 @@ export async function admitExplicitNativeContinuation(input: {
|
||||
return continuation;
|
||||
}
|
||||
|
||||
type ContinuationRetryAuthority = {
|
||||
continuation: { commentId: string } | { commentId: null; taskRetryRequestId: string };
|
||||
actorId: string; sourceAuthorizationId: string; evidence: Record<string, unknown>;
|
||||
};
|
||||
|
||||
async function readCommentRetryAuthority(db: Db, input: {
|
||||
companyId: string; issueId: string; parentRunId: string; explicit: Record<string, unknown>;
|
||||
}): Promise<ContinuationRetryAuthority | null> {
|
||||
const { companyId, issueId, explicit } = input;
|
||||
if (!z.string().guid().safeParse(explicit.commentId).success) return null;
|
||||
const commentId = explicit.commentId as string;
|
||||
const [comment] = await db.select().from(issueComments).where(and(
|
||||
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
||||
eq(issueComments.id, commentId), eq(issueComments.authorType, "user"),
|
||||
isNull(issueComments.createdByRunId), isNull(issueComments.deletedAt),
|
||||
));
|
||||
if (!comment?.authorUserId || !comment.body.trim()) return null;
|
||||
const commentBodyHash = createHash("sha256").update(comment.body).digest("hex");
|
||||
const receipts = await db.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId' = ${input.parentRunId}`,
|
||||
));
|
||||
const receipt = receipts.find(row => {
|
||||
const auth = row.evidence.explicitUserContinuation as Record<string, unknown> | undefined;
|
||||
return auth && auth.previousRunId === explicit.previousRunId && auth.commentId === commentId &&
|
||||
auth.actorId === comment.authorUserId && !auth.failedRunId && !auth.queuedCommentInterruptId &&
|
||||
auth.commentUpdatedAt === comment.updatedAt.toISOString() && auth.commentBodyHash === commentBodyHash;
|
||||
});
|
||||
return receipt ? { continuation: { commentId }, actorId: comment.authorUserId, sourceAuthorizationId: receipt.id,
|
||||
evidence: { commentUpdatedAt: comment.updatedAt.toISOString(), commentBodyHash } } : null;
|
||||
}
|
||||
|
||||
/** A task Retry admits one plain continuation in the saved workspace. When that
|
||||
* run fails transiently, its bounded retry keeps the same actor and receipt;
|
||||
* chat, queued-message and fresh-workspace Retries keep their own contracts. */
|
||||
async function readTaskRetryRetryAuthority(db: Db, input: {
|
||||
companyId: string; issueId: string; agentId: string; parentRunId: string; explicit: Record<string, unknown>;
|
||||
}): Promise<ContinuationRetryAuthority | null> {
|
||||
const { companyId, issueId, explicit } = input;
|
||||
const requestId = z.string().guid().safeParse(explicit.taskRetryRequestId);
|
||||
if (explicit.commentId !== null || !requestId.success || explicit.queuedCommentInterruptId || explicit.readiness) return null;
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.id, requestId.data),
|
||||
eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.reason, "task_retry"),
|
||||
eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.status, "coalesced"),
|
||||
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
||||
));
|
||||
const intent = readTaskRetryIntent(wake?.payload);
|
||||
if (!wake?.requestedByActorId || !wake.runId || !intent || intent.generationId !== wake.id ||
|
||||
intent.actorId !== wake.requestedByActorId || intent.strategy !== "continue" ||
|
||||
intent.chatActionId || intent.queuedChatRequestId || intent.canonicalIntentId) return null;
|
||||
// The admitted run's authority may be its action ledger, an earlier
|
||||
// automatic-retry receipt, or the Retry receipt itself. Dispatch validation
|
||||
// of the parent below proves which one applies.
|
||||
const [receipt] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId' = ${input.parentRunId}`,
|
||||
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'taskRetryRequestId' = ${wake.id}`,
|
||||
)).limit(1);
|
||||
return { continuation: { commentId: null, taskRetryRequestId: wake.id }, actorId: wake.requestedByActorId,
|
||||
sourceAuthorizationId: receipt?.id ?? wake.id, evidence: {} };
|
||||
}
|
||||
|
||||
/** Re-admit a bounded automatic retry without lending it its parent's receipt.
|
||||
* The caller holds the task lock and creates the successor in this transaction.
|
||||
* Only unchanged user messages are renewable; Retry/Interrupt intents keep their
|
||||
* separate one-run contracts. The scheduler still owns retry and cleanup policy.
|
||||
* Unchanged user messages and plain task Retries are renewable; Interrupt and
|
||||
* failed-run retry intents keep their separate one-run contracts. The
|
||||
* scheduler still owns retry and cleanup policy.
|
||||
*/
|
||||
export async function admitExplicitContinuationRetry(input: {
|
||||
db: Db; companyId: string; issueId: string; agentId: string;
|
||||
parentRunId: string; successorRunId: string; now: Date;
|
||||
}): Promise<{ previousRunId: string; commentId: string } | null> {
|
||||
}): Promise<{ previousRunId: string; commentId: string | null; taskRetryRequestId?: string } | null> {
|
||||
const { db, companyId, issueId, agentId } = input;
|
||||
const [task] = await db.select().from(issues).where(and(
|
||||
eq(issues.companyId, companyId), eq(issues.id, issueId),
|
||||
@@ -602,28 +668,11 @@ export async function admitExplicitContinuationRetry(input: {
|
||||
await getExecutionBlocker(db, companyId, issueId)) return null;
|
||||
const context = parent.contextSnapshot;
|
||||
const explicit = context.explicitUserContinuation as Record<string, unknown> | undefined;
|
||||
if (!explicit || !z.string().guid().safeParse(explicit.previousRunId).success ||
|
||||
!z.string().guid().safeParse(explicit.commentId).success || explicit.failedRunId) return null;
|
||||
const commentId = explicit.commentId as string;
|
||||
const [comment] = await db.select().from(issueComments).where(and(
|
||||
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
||||
eq(issueComments.id, commentId), eq(issueComments.authorType, "user"),
|
||||
isNull(issueComments.createdByRunId), isNull(issueComments.deletedAt),
|
||||
));
|
||||
if (!comment?.authorUserId || !comment.body.trim()) return null;
|
||||
const receipts = await db.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId' = ${parent.id}`,
|
||||
));
|
||||
const receipt = receipts.find(row => {
|
||||
const auth = row.evidence.explicitUserContinuation as Record<string, unknown> | undefined;
|
||||
return auth && auth.previousRunId === explicit.previousRunId && auth.commentId === commentId &&
|
||||
auth.actorId === comment.authorUserId && !auth.failedRunId && !auth.queuedCommentInterruptId &&
|
||||
auth.commentUpdatedAt === comment.updatedAt.toISOString() &&
|
||||
auth.commentBodyHash === createHash("sha256").update(comment.body).digest("hex");
|
||||
});
|
||||
if (!receipt) return null;
|
||||
if (!explicit || !z.string().guid().safeParse(explicit.previousRunId).success || explicit.failedRunId) return null;
|
||||
const authority = explicit.commentId === null
|
||||
? await readTaskRetryRetryAuthority(db, { companyId, issueId, agentId, parentRunId: parent.id, explicit })
|
||||
: await readCommentRetryAuthority(db, { companyId, issueId, parentRunId: parent.id, explicit });
|
||||
if (!authority) return null;
|
||||
const [superseding] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, companyId),
|
||||
or(eq(heartbeatRuns.nativeIssueId, issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`),
|
||||
@@ -646,18 +695,18 @@ export async function admitExplicitContinuationRetry(input: {
|
||||
"continuation_source_context_missing", "continuation_task_ownership_changed"].includes(error.message)) return null;
|
||||
throw error;
|
||||
}
|
||||
const continuation = { previousRunId: parent.id, commentId };
|
||||
const continuation = { previousRunId: parent.id, ...authority.continuation };
|
||||
await db.insert(issueRecoveryActions).values({
|
||||
companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
||||
cause: "explicit_user_continuation_retry", fingerprint: input.successorRunId,
|
||||
status: "resolved", outcome: "retry_authorized", resolvedAt: input.now,
|
||||
nextAction: "The bounded retry has its own authorization for the unchanged user message.",
|
||||
nextAction: authority.continuation.commentId
|
||||
? "The bounded retry has its own authorization for the unchanged user message."
|
||||
: "The bounded retry has its own authorization for the same task Retry.",
|
||||
evidence: { explicitUserContinuation: {
|
||||
...continuation, actorId: comment.authorUserId, runId: input.successorRunId,
|
||||
commentUpdatedAt: comment.updatedAt.toISOString(),
|
||||
commentBodyHash: createHash("sha256").update(comment.body).digest("hex"),
|
||||
...continuation, actorId: authority.actorId, runId: input.successorRunId, ...authority.evidence,
|
||||
recordedAt: input.now.toISOString(), automaticRetry: {
|
||||
sourceRunId: parent.id, sourceAuthorizationId: receipt.id,
|
||||
sourceRunId: parent.id, sourceAuthorizationId: authority.sourceAuthorizationId,
|
||||
},
|
||||
} },
|
||||
});
|
||||
|
||||
@@ -83,6 +83,112 @@ describe("durable task Retry admission", () => {
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, source.id)))[0])
|
||||
.toMatchObject({ status: "failed", resultJson: source.resultJson });
|
||||
});
|
||||
// A run admitted by Retry can fail before provider dispatch, for example when
|
||||
// a saved-workspace download is reset. Its bounded transient retry carries
|
||||
// the same Retry authority, recorded for the new run.
|
||||
async function seedFailedRetryAdmittedRun() {
|
||||
const input = await seed();
|
||||
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, input.issueId));
|
||||
const [source] = await db.update(heartbeatRuns).set({ runtimeMode: "legacy", errorCode: "setup_failed",
|
||||
executionStage: "preparing", runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
}).where(eq(heartbeatRuns.id, input.observedRunId)).returning();
|
||||
const receipt = await requestTaskRetry(input), parentId = randomUUID();
|
||||
const continuation = await db.transaction(async tx => {
|
||||
await tx.select().from(issues).where(eq(issues.id, input.issueId)).for("update");
|
||||
const result = await admitExplicitNativeContinuation({ db: tx as unknown as Db,
|
||||
companyId: input.companyId, issueId: input.issueId, agentId: source.agentId,
|
||||
actorId: input.actorId, actorType: "user", reason: "task_retry", commentId: null,
|
||||
successorRunId: parentId, taskRetryRequestId: receipt.requestId!,
|
||||
});
|
||||
await tx.insert(heartbeatRuns).values({ id: parentId, companyId: input.companyId, agentId: source.agentId,
|
||||
issueId: input.issueId, status: "queued", contextSnapshot: { issueId: input.issueId } });
|
||||
await tx.update(agentWakeupRequests).set({ status: "coalesced", runId: parentId })
|
||||
.where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
return result;
|
||||
});
|
||||
expect(continuation).toMatchObject({ previousRunId: source.id, commentId: null, taskRetryRequestId: receipt.requestId });
|
||||
const now = new Date();
|
||||
const [parent] = await db.update(heartbeatRuns).set({ runtimeMode: "legacy", status: "failed", responsibleUserId: input.actorId,
|
||||
errorCode: "remote_workspace_restore_failed", executionStage: "preparing", startedAt: now, finishedAt: now,
|
||||
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
||||
remoteWorkspaceRestore: { step: "snapshot_download", retryable: true } },
|
||||
contextSnapshot: { issueId: input.issueId, wakeReason: "task_retry", taskRetryRequestId: receipt.requestId,
|
||||
previousRunId: source.id, explicitUserContinuation: continuation },
|
||||
}).where(eq(heartbeatRuns.id, parentId)).returning();
|
||||
await db.update(issues).set({ status: "in_progress", executionRunId: parent.id }).where(eq(issues.id, input.issueId));
|
||||
return { input, source, parent, now, retryRequestId: receipt.requestId! };
|
||||
}
|
||||
const dispatchRun = (input: { companyId: string; issueId: string }, run: typeof heartbeatRuns.$inferSelect) =>
|
||||
buildExecutionContinuation({ db, companyId: input.companyId, issueId: input.issueId, agentId: run.agentId,
|
||||
runId: run.id, context: run.contextSnapshot!, summary: null, exposeLowTrustRaw: false });
|
||||
it("retries a transient preparation failure of a Retry-admitted run under the same Retry authority", async () => {
|
||||
const f = await seedFailedRetryAdmittedRun();
|
||||
await expect(dispatchRun(f.input, f.parent)).resolves.toMatchObject({ interruptedRunId: f.source.id });
|
||||
const heartbeat = heartbeatService(db);
|
||||
const first = await heartbeat.scheduleBoundedRetry(f.parent.id, { now: f.now, delayMs: 1000 });
|
||||
expect(first.outcome).toBe("scheduled");
|
||||
if (first.outcome !== "scheduled") return;
|
||||
expect(first.run).toMatchObject({ retryOfRunId: f.parent.id, scheduledRetryAttempt: 1 });
|
||||
expect(first.run.contextSnapshot?.explicitUserContinuation).toEqual({
|
||||
previousRunId: f.parent.id, commentId: null, taskRetryRequestId: f.retryRequestId });
|
||||
// The Retry receipt stays bound to the run it admitted.
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.retryRequestId)))[0])
|
||||
.toMatchObject({ status: "coalesced", runId: f.parent.id });
|
||||
await expect(dispatchRun(f.input, first.run)).resolves.toMatchObject({ interruptedRunId: f.parent.id });
|
||||
// A repeated request adopts the same successor without another receipt.
|
||||
expect(await heartbeat.scheduleBoundedRetry(f.parent.id, { now: f.now })).toMatchObject({
|
||||
outcome: "scheduled", run: { id: first.run.id } });
|
||||
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.input.issueId)))
|
||||
.toHaveLength(1);
|
||||
// A second transient failure retries from its immediate parent, then the budget ends.
|
||||
const later = new Date(f.now.getTime() + 2000);
|
||||
const [failedFirst] = await db.update(heartbeatRuns).set({ status: "failed", errorCode: "remote_workspace_restore_failed",
|
||||
startedAt: later, finishedAt: later, resultJson: f.parent.resultJson }).where(eq(heartbeatRuns.id, first.run.id)).returning();
|
||||
await db.update(issues).set({ executionRunId: failedFirst.id }).where(eq(issues.id, f.input.issueId));
|
||||
const second = await heartbeat.scheduleBoundedRetry(failedFirst.id, { now: later });
|
||||
expect(second.outcome).toBe("scheduled");
|
||||
if (second.outcome !== "scheduled") return;
|
||||
expect(second.run).toMatchObject({ retryOfRunId: failedFirst.id, scheduledRetryAttempt: 2 });
|
||||
await expect(dispatchRun(f.input, second.run)).resolves.toMatchObject({ interruptedRunId: failedFirst.id });
|
||||
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "remote_workspace_restore_failed",
|
||||
startedAt: later, finishedAt: later }).where(eq(heartbeatRuns.id, second.run.id));
|
||||
await db.update(issues).set({ executionRunId: second.run.id }).where(eq(issues.id, f.input.issueId));
|
||||
expect(await heartbeat.scheduleBoundedRetry(second.run.id, { now: later })).toMatchObject({ outcome: "retry_exhausted" });
|
||||
// Removing the successor's own receipt reproduces the dispatch refusal.
|
||||
const receipts = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.input.issueId));
|
||||
expect(receipts).toHaveLength(2);
|
||||
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id,
|
||||
receipts.find(row => (row.evidence.explicitUserContinuation as Record<string, unknown>).runId === second.run.id)!.id));
|
||||
await expect(dispatchRun(f.input, second.run)).rejects.toThrow("continuation_user_authorization_missing");
|
||||
});
|
||||
it.each(["wrong_actor", "unadmitted", "wrong_generation", "other_run", "chat_retry", "fresh_workspace", "reassigned", "superseded"] as const)(
|
||||
"does not lend Retry authority to an automatic retry after %s", async kind => {
|
||||
const f = await seedFailedRetryAdmittedRun();
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.retryRequestId));
|
||||
const taskRetry = wake.payload!.taskRetry as Record<string, unknown>;
|
||||
if (["wrong_actor", "unadmitted", "wrong_generation", "other_run", "chat_retry"].includes(kind))
|
||||
await db.update(agentWakeupRequests).set({
|
||||
...(kind === "wrong_actor" ? { requestedByActorId: "other-user" } : {}),
|
||||
...(kind === "unadmitted" ? { status: "deferred_issue_execution" } : {}),
|
||||
...(kind === "other_run" ? { runId: f.source.id } : {}),
|
||||
...(kind === "wrong_generation" ? { payload: { ...wake.payload, taskRetry: { ...taskRetry, generationId: randomUUID() } } } : {}),
|
||||
...(kind === "chat_retry" ? { payload: { ...wake.payload, taskRetry: { ...taskRetry, chatActionId: randomUUID() } } } : {}),
|
||||
}).where(eq(agentWakeupRequests.id, f.retryRequestId));
|
||||
if (kind === "fresh_workspace") await db.update(heartbeatRuns).set({ contextSnapshot: { ...f.parent.contextSnapshot,
|
||||
explicitUserContinuation: { ...(f.parent.contextSnapshot!.explicitUserContinuation as Record<string, unknown>),
|
||||
readiness: { mode: "fresh_isolated_workspace", retryIntentId: f.retryRequestId, sourceRunId: f.source.id, retainedSourceIds: [] } } },
|
||||
}).where(eq(heartbeatRuns.id, f.parent.id));
|
||||
if (kind === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.input.issueId));
|
||||
if (kind === "superseded") await db.insert(heartbeatRuns).values({ companyId: f.input.companyId, agentId: f.parent.agentId,
|
||||
status: "succeeded", contextSnapshot: { issueId: f.input.issueId }, createdAt: new Date(f.now.getTime() + 1000) });
|
||||
const result = await heartbeatService(db).scheduleBoundedRetry(f.parent.id, { now: f.now });
|
||||
expect(result.outcome).not.toBe("scheduled");
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.parent.id))).toHaveLength(0);
|
||||
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.input.issueId)))
|
||||
.toHaveLength(0);
|
||||
});
|
||||
it("serializes concurrent clicks and remembers each key after completion", async () => {
|
||||
const input = await seed();
|
||||
const second = { ...input, idempotencyKey: randomUUID() };
|
||||
|
||||
Reference in new issue
Block a user