mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
fix(retry): never offer Retry while a live run owns the task (#22428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each task shows an interruption banner. The banner offers Retry when an attempt failed and nothing will move the task forward > - The banner reads the task's newest run. A continuation that the system cancels before it starts can be the newest run, while a different run owns the task and does the work > - In a restart-chaos wave, the banner offered Retry for about 140 seconds while the owning run worked. The task then finished with no click > - A Retry that the user does not need breaks the rule that the needed-Retry rate is 0% > - This pull request makes the banner and the Retry endpoint read the same observed run. That run is the live run that owns the task, or else the newest real attempt > - The benefit is that Retry shows only when nothing will advance the task, and a shown Retry is accepted ## Linked Issues or Issue Description **What happened?** A restart-chaos wave (three release swaps) showed a Retry banner on one task for 141 to 144 seconds. The reason code was `issue_execution_lock_changed`. Nobody clicked Retry, and the task completed. Timeline (UTC): - 14:32:09. Run A starts. - 14:51:12. A user follow-up wakes run B. Run B is queued behind run A. - 14:52:46. Run A succeeds. The system schedules continuation run C (`retryOfRunId` = A, reason `native_agent_blocker_continuation`). Run B starts and takes the task execution lock. - 14:53:16. The dispatch gate cancels C before it starts with `issue_execution_lock_changed`, because run B owns the lock. This is correct de-duplication. - 14:53:17. The banner shows `failed` with Retry. Run C was created after run B, so C is the newest run. The projection reads only the newest run and ignores that run B is running and owns the lock. - 14:55:41. Run B succeeds and the task is done. The banner goes away. Retry was never needed. Run B owned the task for the full banner window. **Expected behavior** The banner does not offer Retry while a run owns the task and is queued or running. A run that the system cancelled before it started because another run took the lock is not a failed attempt. When Retry is shown, pressing it is accepted for the run that the banner shows. **Steps to reproduce** 1. Create a task. Run A succeeds and leaves the task in progress. 2. Queue run B for the same task (for example, a user comment). Create continuation run C after run B. 3. Start run B and set it as the task execution lock owner. 4. Cancel run C before it starts with error code `issue_execution_lock_changed`. 5. Read the task interruption. Before this change: `state: failed`, `canRetry: true`, observed run C. After this change: `state: working`, `canRetry: false`, observed run B. **Paperclip version or commit** `fe0c4aa95d` (master at the time of the wave). ## What Changed - `server/src/services/task-retry-eligibility.ts`: add two shared conditions. - `taskAttemptCondition()`: a run cancelled before it started with `recovery_no_longer_needed`, `issue_closed_before_start` or `issue_execution_lock_changed` is not an attempt. The first two codes were already excluded inline in the projection. `issue_execution_lock_changed` is new. - `liveExecutionOwnerCondition()`: the run is the task's execution lock owner and it is queued or running. - `server/src/services/task-interruption-presentation.ts`: the observed run is the live execution owner when there is one. Otherwise it is the newest attempt, with the same native, first-class and legacy precedence as before. The owner lookup is a primary-key read, so the per-task run scan does not grow. - `server/src/services/task-retry.ts`: Retry admission, the refused chat Retry continuation and the superseded-Retry check use the same attempt filter. Admission also accepts the live execution owner when the request names it. The newest attempt stays acceptable, so the automatic Retry hooks (restart, provider limit, budget resume) keep their behavior. - `heldTaskRetryRefusals` uses the same attempt filter. - New test `server/src/__tests__/task-retry-banner-execution-owner.test.ts`: one state table. Each row asserts the list and detail projection, and when Retry is shown, presses it through `requestTaskRetry`. ### State table (tested as one table) | Row | Runs and lock | Banner | Observed run | Retry press | | --- | --- | --- | --- | --- | | 1 | Continuation scheduled; follow-up running and owns the lock | working, no Retry | follow-up | n/a | | 2 | Continuation cancelled unstarted (`issue_execution_lock_changed`); follow-up running and owns the lock (t032) | working, no Retry | follow-up | n/a | | 3 | Same as row 2, lock pointer cleared mid-transfer | working, no Retry | follow-up | n/a | | 4 | Scheduled continuation overdue past the 2-minute dispatch grace; follow-up running and owns the lock | working, no Retry | follow-up | n/a | | 5 | Newer attempt failed; older queued run owns the lock | working, no Retry | queued owner | automatic Retry of the failed run is still accepted | | 6 | Continuation superseded; follow-up then succeeded | resolved, no Retry | follow-up | n/a | | 7 | Continuation superseded; follow-up then failed | failed, Retry | follow-up | accepted | | 8 | Continuation superseded; predecessor failed; nothing else runs | failed, Retry | predecessor | accepted | | 9 | Continuation started, then lost the lock | failed, Retry | continuation | accepted | | 10 | Lock pointer left on a finished failed run | failed, Retry | that run | accepted | | 11 | Row 2 with native runs (`nativeIssueId`) | working, no Retry | native follow-up | Retry naming the owner is accepted | | 12 | Queued owner past its 15-minute move-forward wait; newer failed attempt | working, Retry (`move_forward`) | queued owner | accepted | Invariants that the table checks: - Retry is never shown while something will advance the task (rows 1 to 6, 11). Row 12 offers only `move_forward` on the owner. - A Retry press that names the live owner is accepted (rows 1, 2, 3, 5, 11, 12). - Retry is always shown when nothing will advance the task (rows 7 to 10). - The shared attempt filter is tested in `heldTaskRetryRefusals`, `retireSupersededTaskRetry` and the refused chat Retry continuation (`person_with_superseded_continuation` in `chat-channels.integration.test.ts`). - When Retry is shown, pressing it is accepted for the observed run (rows 7 to 10). ## Verification - Red: with the service changes removed, rows 1 to 8 fail (8 of 10). Rows 9 and 10 pass on both sides. - Green: `cd server && npx vitest run src/__tests__/task-retry-banner-execution-owner.test.ts` passes 14 of 14. - Related suites pass: `task-interruption-query.test.ts` (includes the query-plan bound test), `task-interruption-automatic-path.test.ts`, `legacy-continuation-authority.test.ts`, `native-closed-session-retry.integration.test.ts`, `interrupted-preparation-restore-retry.test.ts`, and the other server tests that call `taskInterruptionPresentation`, `requestTaskRetry` or `heldTaskRetryRefusals`. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. The change is read-side only. It does not change scheduling, dispatch, the lock, or the run-dispatch gate. - A run cancelled before start with `issue_execution_lock_changed` no longer shows as a failed attempt. If the lock was released and nothing else ran, the banner shows the previous attempt. A failed previous attempt still offers Retry (row 8). - The projection query has one more union branch. It is a primary-key lookup through `issues.execution_run_id`. The existing query-plan bound test passes. - No runner, migration or API contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Refs #22306 (the earlier fix for a due scheduled retry shown as failed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0e38c80323
commit
a0fe90f24b
5 files changed
+223
-17
No files matched your search
@@ -66112,7 +66112,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["person", "person_with_input_before_stop", "person_after_held_admission", "person_paused_after_accept", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
|
||||
it.each(["person", "person_with_input_before_stop", "person_after_held_admission", "person_paused_after_accept", "person_with_superseded_continuation", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
|
||||
const f = await currentGitHubTaskRetryFixture(initiator === "person_with_input_before_stop");
|
||||
try {
|
||||
const companyId = f.fixture.companyId, issueId = f.issue.id;
|
||||
@@ -66215,6 +66215,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(joined).toMatchObject({ accepted: true, requestId: request.requestId, chatActionId: request.chatActionId });
|
||||
}
|
||||
expect(request).toMatchObject({ accepted: true, stale: false, chatActionId: expect.any(String) });
|
||||
if (initiator === "person_with_superseded_continuation") await db.insert(heartbeatRuns).values({ companyId,
|
||||
agentId: f.fixture.assignedAgentId, status: "cancelled", errorCode: "issue_execution_lock_changed", runtimeMode: "legacy",
|
||||
issueId, retryOfRunId: sourceRunId, finishedAt: new Date(), contextSnapshot: { issueId } });
|
||||
const chatAction = async () => (await db.select().from(chatActions).where(eq(chatActions.id, request.chatActionId!)))[0];
|
||||
expect((await chatAction()).payload.sourceAuthorizationPending).toBe(true);
|
||||
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, agentWakeupRequests, companies, companyMemberships, createDb, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js";
|
||||
import { requestTaskRetry, retireSupersededTaskRetry } from "../services/task-retry.js";
|
||||
import { heldTaskRetryRefusals } from "../services/task-retry-eligibility.js";
|
||||
|
||||
vi.mock("../sentry.js", () => ({ captureTaskInterruptionFailure: vi.fn() }));
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
|
||||
type RunSpec = Partial<typeof heartbeatRuns.$inferInsert> & { ageSeconds: number };
|
||||
type Row = {
|
||||
name: string;
|
||||
predecessor?: Partial<typeof heartbeatRuns.$inferInsert>;
|
||||
followUp?: RunSpec;
|
||||
continuation?: RunSpec;
|
||||
lockOwner: "followUp" | "continuation" | null;
|
||||
expected: { state: string; canRetry: boolean; observed: "predecessor" | "followUp" | "continuation"; retryAction?: "move_forward" };
|
||||
retryWith?: "predecessor" | "followUp" | "continuation";
|
||||
native?: boolean;
|
||||
deliveredContinuation?: boolean;
|
||||
};
|
||||
|
||||
const minutesAgo = (minutes: number) => new Date(Date.now() - minutes * 60_000);
|
||||
const lockChanged = { status: "cancelled", errorCode: "issue_execution_lock_changed", startedAt: null,
|
||||
error: "Cancelled because another run owns task execution or checkout before replacement dispatch" } as const;
|
||||
const continuationRun = { scheduledRetryReason: "native_agent_blocker_continuation" } as const;
|
||||
|
||||
const STATE_TABLE: Row[] = [
|
||||
{ name: "continuation scheduled while the follow-up turn runs and owns the lock",
|
||||
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
|
||||
continuation: { ...continuationRun, status: "scheduled_retry", errorCode: "native_agent_blocker_continuation",
|
||||
scheduledRetryAt: new Date(Date.now() + 30_000), ageSeconds: 30 },
|
||||
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" } },
|
||||
{ name: "continuation cancelled because the follow-up turn took the lock (t032 banner)",
|
||||
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
|
||||
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.4), ageSeconds: 30 },
|
||||
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" } },
|
||||
{ name: "native continuation cancelled because the native follow-up turn took the lock",
|
||||
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
|
||||
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.4), ageSeconds: 30 },
|
||||
lockOwner: "followUp", native: true, expected: { state: "working", canRetry: false, observed: "followUp" } },
|
||||
{ name: "continuation cancelled by a lock transfer while the lock pointer is between owners",
|
||||
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
|
||||
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.4), ageSeconds: 30 },
|
||||
lockOwner: null, expected: { state: "working", canRetry: false, observed: "followUp" } },
|
||||
{ name: "overdue scheduled continuation behind a running lock owner",
|
||||
followUp: { status: "running", startedAt: minutesAgo(10), ageSeconds: 900 },
|
||||
continuation: { ...continuationRun, status: "scheduled_retry", scheduledRetryAt: minutesAgo(5), ageSeconds: 600 },
|
||||
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" } },
|
||||
{ name: "newer failed attempt while an older queued run owns the lock",
|
||||
followUp: { status: "queued", ageSeconds: 900 },
|
||||
continuation: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 200 },
|
||||
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" }, retryWith: "continuation" },
|
||||
{ name: "continuation queued past its move-forward wait behind a newer failed attempt",
|
||||
followUp: { status: "queued", ageSeconds: 1200 },
|
||||
continuation: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 200 },
|
||||
lockOwner: "followUp", deliveredContinuation: true,
|
||||
expected: { state: "working", canRetry: true, observed: "followUp", retryAction: "move_forward" } },
|
||||
{ name: "follow-up turn finished after superseding the continuation",
|
||||
followUp: { status: "succeeded", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 300 },
|
||||
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(2.5), ageSeconds: 200 },
|
||||
lockOwner: null, expected: { state: "resolved", canRetry: false, observed: "followUp" } },
|
||||
{ name: "follow-up turn failed after superseding the continuation",
|
||||
followUp: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 300 },
|
||||
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(2.5), ageSeconds: 200 },
|
||||
lockOwner: null, expected: { state: "failed", canRetry: true, observed: "followUp" }, retryWith: "followUp" },
|
||||
{ name: "superseded continuation over a failed predecessor with nothing else to run",
|
||||
predecessor: { status: "failed", errorCode: "adapter_failed", finishedAt: minutesAgo(1) },
|
||||
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.5), ageSeconds: 30 },
|
||||
lockOwner: null, expected: { state: "failed", canRetry: true, observed: "predecessor" }, retryWith: "predecessor" },
|
||||
{ name: "continuation that started before losing the lock is still a failed attempt",
|
||||
continuation: { ...continuationRun, ...lockChanged, startedAt: minutesAgo(1), finishedAt: minutesAgo(0.5), ageSeconds: 90 },
|
||||
lockOwner: null, expected: { state: "failed", canRetry: true, observed: "continuation" }, retryWith: "continuation" },
|
||||
{ name: "a lock pointer left on a finished run does not hide its failure",
|
||||
followUp: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 300 },
|
||||
lockOwner: "followUp", expected: { state: "failed", canRetry: true, observed: "followUp" }, retryWith: "followUp" },
|
||||
];
|
||||
|
||||
describe.skipIf(!support.supported)("Retry banner while another run owns the task (restart-chaos t032)", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("retry-banner-execution-owner-");
|
||||
db = createDb(temporary.connectionString);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
|
||||
|
||||
async function seed(row: Row) {
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Owner task", issuePrefix: `O${companyId.slice(0, 7)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Owner agent", adapterType: "process" });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user",
|
||||
status: "active", membershipRole: "member" });
|
||||
const [issue] = await db.insert(issues).values({ companyId, title: "Staging mix parent", status: "in_progress",
|
||||
assigneeAgentId: agentId, createdByUserId: "board-user" }).returning();
|
||||
const ids = { predecessor: randomUUID(), followUp: randomUUID(), continuation: randomUUID() };
|
||||
const base = { companyId, agentId, scopeKind: "issue" as const, contextSnapshot: { issueId: issue!.id },
|
||||
...(row.native ? { nativeIssueId: issue!.id, runtimeMode: "native" } : { issueId: issue!.id }) };
|
||||
await db.insert(heartbeatRuns).values({ ...base, id: ids.predecessor, status: "succeeded", livenessState: "advanced",
|
||||
createdAt: minutesAgo(30), startedAt: minutesAgo(30), finishedAt: minutesAgo(2), ...row.predecessor });
|
||||
for (const key of ["followUp", "continuation"] as const) {
|
||||
const spec = row[key];
|
||||
if (!spec) continue;
|
||||
const { ageSeconds, ...values } = spec;
|
||||
await db.insert(heartbeatRuns).values({ ...base, id: ids[key], createdAt: new Date(Date.now() - ageSeconds * 1000),
|
||||
...(key === "continuation" ? { retryOfRunId: ids.predecessor } : {}), ...values });
|
||||
}
|
||||
if (row.deliveredContinuation) await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue!.id,
|
||||
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: agentId,
|
||||
cause: "fresh_continuation", fingerprint: `fresh-continuation:${ids.predecessor}`, nextAction: "Continuing.",
|
||||
resolvedAt: minutesAgo(19), attemptCount: 1, evidence: { runId: ids.predecessor, continuationDelivery: "delivered",
|
||||
continuationRunId: ids.followUp, freshContinuation: { sourceRunId: ids.predecessor, recordedAt: minutesAgo(19).toISOString() } } });
|
||||
const [current] = await db.update(issues).set({ executionRunId: row.lockOwner ? ids[row.lockOwner] : null })
|
||||
.where(eq(issues.id, issue!.id)).returning();
|
||||
return { companyId, issue: current!, ids };
|
||||
}
|
||||
|
||||
it.each(STATE_TABLE)("$name", async row => {
|
||||
const f = await seed(row);
|
||||
const list = (await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id);
|
||||
const detail = await taskInterruptionPresentation(db, f.issue);
|
||||
for (const presentation of [list, detail]) {
|
||||
expect(presentation).toMatchObject({ state: row.expected.state, canRetry: row.expected.canRetry,
|
||||
observedRunId: f.ids[row.expected.observed], ...(row.expected.retryAction ? { retryAction: row.expected.retryAction } : {}) });
|
||||
expect(presentation?.retryAction).toBe(row.expected.retryAction);
|
||||
}
|
||||
if (row.expected.canRetry) {
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: "board-user",
|
||||
actorSource: "session", idempotencyKey: randomUUID(), observedRunId: list!.observedRunId,
|
||||
...(list!.retryAction ? { retryAction: list!.retryAction } : {}) });
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: list!.observedRunId });
|
||||
}
|
||||
if (row.lockOwner && row.expected.observed === row.lockOwner && !row.expected.canRetry) {
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: "board-user",
|
||||
actorSource: "session", idempotencyKey: randomUUID(), observedRunId: f.ids[row.lockOwner] });
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: f.ids[row.lockOwner] });
|
||||
}
|
||||
if (row.retryWith && !row.expected.canRetry) {
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: "board-user",
|
||||
actorSource: "session", idempotencyKey: randomUUID(), observedRunId: f.ids[row.retryWith] });
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: f.ids[row.retryWith] });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
describe("a held Retry beside an unstarted lock-changed cancellation", () => {
|
||||
async function seedHeldRetry(newer: "lock_changed" | "failed") {
|
||||
const row: Row = { name: "held", lockOwner: null,
|
||||
predecessor: { status: "failed", errorCode: "adapter_failed", finishedAt: minutesAgo(1) },
|
||||
continuation: newer === "lock_changed"
|
||||
? { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.5), ageSeconds: 30 }
|
||||
: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(0.8), finishedAt: minutesAgo(0.5), ageSeconds: 50 },
|
||||
expected: { state: "failed", canRetry: true, observed: "predecessor" } };
|
||||
const f = await seed(row);
|
||||
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.issue.assigneeAgentId!,
|
||||
source: "on_demand", reason: "task_retry", status: "deferred_issue_execution", requestedByActorType: "user",
|
||||
requestedByActorId: "board-user", payload: { issueId: f.issue.id, taskRetry: { sourceRunId: f.ids.predecessor },
|
||||
executionWait: { reason: "process_running" } } }).returning();
|
||||
return { ...f, requestId: wake!.id };
|
||||
}
|
||||
|
||||
it("keeps the held Retry's refusal current and does not retire the Retry", async () => {
|
||||
const f = await seedHeldRetry("lock_changed");
|
||||
expect((await heldTaskRetryRefusals(db, f.companyId, [f.issue.id])).get(f.issue.id)).toBe("process_running");
|
||||
await expect(db.transaction(tx => retireSupersededTaskRetry(tx as unknown as typeof db, { companyId: f.companyId,
|
||||
issueId: f.issue.id, requestId: f.requestId, sourceRunId: f.ids.predecessor }))).resolves.toBe(false);
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
|
||||
expect(wake).toMatchObject({ status: "deferred_issue_execution", error: null });
|
||||
}, 60_000);
|
||||
|
||||
it("still retires a held Retry that a newer real attempt superseded", async () => {
|
||||
const f = await seedHeldRetry("failed");
|
||||
expect((await heldTaskRetryRefusals(db, f.companyId, [f.issue.id])).has(f.issue.id)).toBe(false);
|
||||
await expect(db.transaction(tx => retireSupersededTaskRetry(tx as unknown as typeof db, { companyId: f.companyId,
|
||||
issueId: f.issue.id, requestId: f.requestId, sourceRunId: f.ids.predecessor }))).resolves.toBe(true);
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
|
||||
expect(wake).toMatchObject({ status: "skipped", error: "task_retry_superseded" });
|
||||
}, 60_000);
|
||||
});
|
||||
});
|
||||
@@ -2,7 +2,7 @@ import { ENVIRONMENT_PROVIDER_WAIT_KEY, ENVIRONMENT_PROVIDER_WAIT_RETRY_REASON,
|
||||
import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js";
|
||||
import { AGENT_DIRECTORY_WAIT_RETRY_REASON, INTERRUPTED_PREPARATION_RETRY_REASON } from "./execution-recovery-attempt.js";
|
||||
import { boundSetupCommandFailure, readSetupCommandFailure } from "./workspace-setup-command-failure.js";
|
||||
import { and, desc, eq, inArray, isNull, not, notInArray, sql, type SQL } from "drizzle-orm";
|
||||
import { and, desc, eq, inArray, isNull, not, notInArray, or, sql, type SQL } from "drizzle-orm";
|
||||
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
|
||||
import { nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
||||
import { agents, agentWakeupRequests, approvals, chatActions, companies, environmentLeases, heartbeatRuns, issueApprovals, issueRecoveryActions,
|
||||
@@ -26,7 +26,7 @@ import { canonicalExecutionUuid } from "./execution-identity.js";
|
||||
import { chatActionDeadline } from "./chat-action-work.js";
|
||||
import { NATIVE_CLEANUP_SETTLING_MESSAGE, NATIVE_SESSION_CLEANUP_QUARANTINED } from "./native-runtime/completed-turn-cleanup.js";
|
||||
import { isAcknowledgedLegacyUserStop, isQueuedDependencyWait, runStopPresentationJson } from "./run-cancellation.js";
|
||||
import { heldRetryOffersRetry } from "./task-retry-eligibility.js";
|
||||
import { heldRetryOffersRetry, liveExecutionOwnerCondition, taskAttemptCondition } from "./task-retry-eligibility.js";
|
||||
import { exhaustedNativeFinalizationCondition, savedResultRetryPendingCondition } from "./native-runtime/exhausted-finalization-retry.js";
|
||||
import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON,
|
||||
WORKSPACE_BUSY_RETRY_REASON } from "../modules/run-dispatch/index.js";
|
||||
@@ -457,18 +457,21 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
const requestedUuid = sql<string>`case when ${requestedIssueId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
||||
then ${requestedIssueId}::uuid else null end`;
|
||||
const legacyIssue = sql<string>`${heartbeatRuns.contextSnapshot}->>'issueId'`;
|
||||
const attempt = sql`not (${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null
|
||||
and coalesce(${heartbeatRuns.errorCode}, '') in ('recovery_no_longer_needed', 'issue_closed_before_start'))`;
|
||||
const candidate = (condition: ReturnType<typeof and>) => db.select({ id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition, attempt))
|
||||
const candidate = (condition: SQL | undefined, executionOwner = false) => db.select({ id: heartbeatRuns.id,
|
||||
createdAt: heartbeatRuns.createdAt, executionOwner: sql<number>`${executionOwner ? 1 : 0}::int`.as("execution_owner") })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition, taskAttemptCondition()))
|
||||
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
const nativeRun = eq(heartbeatRuns.nativeIssueId, requestedUuid);
|
||||
const firstClassRun = and(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.issueId, requestedUuid));
|
||||
const legacyRun = and(isNull(heartbeatRuns.nativeIssueId), isNull(heartbeatRuns.issueId), eq(legacyIssue, requestedIssueId));
|
||||
const candidates = unionAll(
|
||||
candidate(eq(heartbeatRuns.nativeIssueId, requestedUuid)),
|
||||
candidate(and(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.issueId, requestedUuid))),
|
||||
candidate(and(isNull(heartbeatRuns.nativeIssueId), isNull(heartbeatRuns.issueId), eq(legacyIssue, requestedIssueId))),
|
||||
candidate(and(liveExecutionOwnerCondition(companyId, requestedUuid), or(nativeRun, firstClassRun, legacyRun)), true),
|
||||
candidate(nativeRun),
|
||||
candidate(firstClassRun),
|
||||
candidate(legacyRun),
|
||||
).as("interruption_run_candidates");
|
||||
const latestRuns = db.select({ id: candidates.id }).from(candidates)
|
||||
.orderBy(desc(candidates.createdAt), desc(candidates.id)).limit(1).as("latest_interruption_runs");
|
||||
.orderBy(desc(candidates.executionOwner), desc(candidates.createdAt), desc(candidates.id)).limit(1).as("latest_interruption_runs");
|
||||
const wakeIssue = sql<string>`${agentWakeupRequests.payload}->>'issueId'`;
|
||||
const isRetryWait = sql<boolean>`coalesce(jsonb_typeof(${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}') = 'string', false)`;
|
||||
const agentIds = selected.flatMap(issue => issue.assigneeAgentId ? [issue.assigneeAgentId] : []);
|
||||
|
||||
@@ -1,5 +1,18 @@
|
||||
import { and, desc, eq, inArray, isNull, sql } from "drizzle-orm";
|
||||
import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
|
||||
import { agentWakeupRequests, heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
||||
|
||||
const UNSTARTED_SUPERSEDED_RUN_CODES = ["recovery_no_longer_needed", "issue_closed_before_start", "issue_execution_lock_changed"];
|
||||
|
||||
export function taskAttemptCondition(run: { status: SQL | typeof heartbeatRuns.status; startedAt: SQL | typeof heartbeatRuns.startedAt;
|
||||
errorCode: SQL | typeof heartbeatRuns.errorCode } = heartbeatRuns) {
|
||||
return sql`not (${run.status} = 'cancelled' and ${run.startedAt} is null
|
||||
and coalesce(${run.errorCode}, '') in (${sql.join(UNSTARTED_SUPERSEDED_RUN_CODES.map(code => sql`${code}`), sql`, `)}))`;
|
||||
}
|
||||
|
||||
export function liveExecutionOwnerCondition(companyId: string, issueId: SQL | string) {
|
||||
return sql`${heartbeatRuns.status} in ('queued', 'running') and ${heartbeatRuns.id} = (select ${issues.executionRunId} from ${issues}
|
||||
where ${issues.companyId} = ${companyId} and ${issues.id} = ${issueId})`;
|
||||
}
|
||||
|
||||
const REFUSALS_A_CLICK_CANNOT_CHANGE: ReadonlySet<string> = new Set([
|
||||
"cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required", "previous_execution_unverified",
|
||||
@@ -19,6 +32,7 @@ export async function heldTaskRetryRefusals(db: Pick<Db, "selectDistinctOn">, co
|
||||
where latest.company_id = ${companyId} and (latest.issue_id::text = ${agentWakeupRequests.payload}->>'issueId'
|
||||
or latest.native_issue_id::text = ${agentWakeupRequests.payload}->>'issueId'
|
||||
or latest.context_snapshot->>'issueId' = ${agentWakeupRequests.payload}->>'issueId')
|
||||
and ${taskAttemptCondition({ status: sql`latest.status`, startedAt: sql`latest.started_at`, errorCode: sql`latest.error_code` })}
|
||||
order by latest.created_at desc, latest.id desc limit 1)` })
|
||||
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
|
||||
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
|
||||
@@ -23,6 +23,7 @@ import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state
|
||||
import { deferredChatInputPredicate, resumeDeferredChatWake, unadmittedChatWakeupCondition } from "./durable-chat-wakeup.js";
|
||||
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
|
||||
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
||||
import { liveExecutionOwnerCondition, taskAttemptCondition } from "./task-retry-eligibility.js";
|
||||
import { undeliveredContinuation } from "./fresh-continuation-retry-fold.js";
|
||||
import { PersistedExecutionWaitRefusal } from "./deferred-execution-wait.js";
|
||||
import { rearmExhaustedNativeFinalization } from "./native-runtime/exhausted-finalization-retry.js";
|
||||
@@ -203,7 +204,7 @@ export async function retireSupersededTaskRetry(db: Db, input: {
|
||||
const [latest] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId),
|
||||
or(eq(heartbeatRuns.issueId, input.issueId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`), taskAttemptCondition(),
|
||||
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
if (latest?.id === input.sourceRunId) return false;
|
||||
const [wake] = await db.select({ agentId: agentWakeupRequests.agentId }).from(agentWakeupRequests).where(and(
|
||||
@@ -311,8 +312,10 @@ function admitTaskRetry(input: TaskRetryRequest) {
|
||||
}
|
||||
const [latest] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, issue.companyId),
|
||||
or(eq(heartbeatRuns.issueId, issue.id), eq(heartbeatRuns.nativeIssueId, issue.id),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`)))
|
||||
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`), taskAttemptCondition()))
|
||||
.orderBy(...(input.observedRunId ? [sql`case when ${heartbeatRuns.id}::text = ${input.observedRunId}
|
||||
and ${liveExecutionOwnerCondition(issue.companyId, issue.id)} then 0 else 1 end`] : []),
|
||||
desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
if (["done", "cancelled"].includes(issue.status) || !issue.assigneeAgentId || !latest ||
|
||||
latest.agentId !== issue.assigneeAgentId || input.observedRunId !== latest.id) {
|
||||
return { requestId: null, accepted: false, stale: true, observedRunId: latest?.id ?? null, chatActionId: undefined };
|
||||
@@ -542,7 +545,7 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
|
||||
const [latest] = await tx.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt, status: heartbeatRuns.status,
|
||||
startedAt: heartbeatRuns.startedAt, errorCode: heartbeatRuns.errorCode, resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, wake.companyId), or(eq(heartbeatRuns.issueId, issueId), eq(heartbeatRuns.nativeIssueId, issueId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`)))
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`), taskAttemptCondition()))
|
||||
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
if (latest?.id !== intent.sourceRunId || !latest.finishedAt) return false;
|
||||
if (intent.initiatedBy) {
|
||||
|
||||
Reference in new issue
Block a user