fix(recovery): continue a task whose system retry was refused at claim behind a saved-work hold (#21505)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, automatic recovery continues
the task, through the fresh-continuation fallback, in a new run that
starts from the saved work
> - A follow-up run (for example one started by a new comment or by
finished subtasks) can fail before any provider work and leave a
saved-work hold. The system then queues its own retry, and the dispatch
gate cancels that retry at claim with
`execution_reconciliation_required` because the hold is still in place
> - The fallback takes over the hold when that happens, but it continued
only "recent interruptions". That test required the failed run to be a
recovery run itself or to fail with a restart code. A first follow-up
run that failed with `adapter_failed` passed neither test, even though
the system had already decided to retry it. So the fallback answered "a
person must press Retry". The backlog pass used the same test, and the
assigned-work sweep never listed the task. The task stayed blocked with
no run, no pending path and no Retry button
> - This pull request treats a system retry refused at dispatch as proof
that the system continues the task. It also stops cancelled-at-dispatch
attempts from counting against the hourly budget, lets the sweep
continue a blocked task whose latest run was refused at dispatch, and
keeps recovery notices from asking for a Retry that the task does not
offer
> - The benefit is that such a task continues by itself in exactly one
fresh run from the saved work, whether the saved work was recovered
before or after the refusal, and a person's own failed Retry keeps
today's behaviour

## Linked Issues or Issue Description

Refs #21394, #21348, #21363, #21478

**What happened?**
A task's follow-up run, a legacy run that was not a recovery run, failed
before any provider work with `adapter_failed`. Its files could not be
restored, so it left a saved-work hold
(`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure:
restore_failed`). The system queued a retry of that run. The dispatch
gate cancelled the retry at claim with
`execution_reconciliation_required`, and the "older run hold" path
handed the hold to the fresh-continuation fallback
(`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback
found the refused run and asked `interruptionEligibility` whether the
interruption was recent. That predicate required
`continuesAutomaticallySql(source)` (a recovery run, or a restart,
deploy or bootstrap code), so it returned `person_owned` /
`retry_needed`. The stranded reconciler then blocked the task with
"Press Retry", but the saved-work hold reports that Retry is not
available. This happened twice in a local restart-chaos wave. In one
case the saved work was recovered two minutes before the claim. In the
other it was recovered thirty seconds after the claim.

**Expected behavior**
A task never rests blocked with no live run, no pending path and no
Retry. While the saved-work recovery runs, that recovery is the pending
path. When it finishes, one continuation starts from the saved work
through the fallback (isolation, the shared hourly cap). A run that was
cancelled at dispatch before it started does not use up the automatic
budget. A person's own failed Retry is still left for that person.

**Steps to reproduce**
1. Fail a follow-up run (wake reason `issue_children_completed`) before
provider work with `adapter_failed` and `workspaceRestoreFailure:
restore_failed`.
2. Queue the system's retry of that run (`retryOfRunId`, wake reason
`issue_continuation_needed`) and claim it. The dispatch gate cancels it
with `execution_reconciliation_required`.
3. Before this change, no continuation is recorded, the stranded
reconciler blocks the task, and nothing ever continues it.

**Paperclip version or commit**
master at 2756be2f32

## What Changed

- `fresh-continuation-fallback.ts`, `interruptionEligibility`: the
interruption counts as one the system continues when the failed run's
cause is system-owned and a later run that the dispatch gate refused
before it started is a system recovery run (`recoveryRunSql`, which
excludes a person's Retry). A failed run that was itself a person's
Retry (`task_retry` or `explicitUserContinuation`) never qualifies this
way. The 24-hour recency bound is unchanged. The claim-gate hand-off,
the backlog pass and its deadline query all use this one predicate.
- `recovery/continuation-ledger.ts`,
`recovery/task-automatic-attempts.ts`: one shared rule,
`startedOrPendingAutomaticRetry`. An automatic retry that the dispatch
gate cancelled before it started (`cancelled`, no `startedAt`,
`execution_reconciliation_required`) does not count toward the hourly
cap in either ledger.
- `recovery/assigned-work-sweep.ts`:
- When the latest run was refused at dispatch, the sweep continues from
the run it was refused for through the fallback. Before, it gave up.
- A blocked task with an active hold and a recent refused-at-dispatch
latest run is now listed as a suspect. The check hands it to the
fallback before it answers `left_to_hold_owner`. A pending recovery
becomes `waiting_on_recovery` with its recheck time.
- Holds that wait for a person's configuration or confirmation are still
excluded.
- Review follow-up (`recovery/task-automatic-attempts.ts`,
`recovery/assigned-work-sweep.ts`):
- The task ledger counts a sweep's fresh-continuation record (`trigger:
assigned_work`) itself. It no longer depends on a companion activity
entry, and an activity entry marked `continuedFromSavedWork` is not
counted a second time. Every sweep continuation counts once, whatever
path recorded it.
- When the sweep claims a continuation, it marks its claim entry
`continuedFromSavedWork` inside the transaction that records the fresh
continuation. A restart between the two writes can no longer leave one
attempt counted twice.
- The sweep's continuation behind a refused run spends the per-minute
repair budget (`ASSIGNED_WORK_REPAIRS_PER_MINUTE`). The excess is
deferred with a `retryAt`.
- `recovery/stranded-notice.ts`, `recovery/service.ts`: when the task's
execution blocker offers no Retry, the escalation notice and the
recovery action's next action leave out every Retry instruction. They
say that Paperclip is holding the task until its saved work is settled.
- No change to the dispatch gate (`heartbeat/queue.ts`, the restore-hold
predicate in `run-dispatch/adapters/postgres.ts`, or
`execution-blocker.ts`). The recorded continuation settles the hold, so
the delivered run is not refused again.

## Verification

Red before the change, green after (embedded Postgres, real services:
`terminalizeLegacyExecution`, the dispatch adapter's
`cancelStaleQueuedRun`, `reconcileStrandedAssignedIssues`,
`deliverReconciledExecutions`, `continueHeldExecutions`,
`recoveryService().assignedWork`):
- `server/src/services/fresh-continuation-fallback.test.ts`
- "continues a follow-up … when the saved work was recovered before the
system retry was refused at claim": red (`continuationDelivery:
"invalidated"`, no continuation). Green: the continuation is recorded at
the claim, and the stranded reconciler does not block the task. The
delivered run is queued and passes the claim gate (`not_stale`).
- "… recovered after the system retry was refused at claim": red (no
pending check). Green: the claim records `exact_recovery_pending` /
`saved_work_recovery_running`. When the recovery completes, the backlog
pass continues the task, and the delivered run passes the claim gate.
- "continues a follow-up … with its files in place, after its system
retry was refused at claim": red, then green. This is the control
without a restore failure.
- "still leaves a person's failed Retry of a follow-up for that person
…": stays `person_owned` / `retry_needed`, before and after.
- "counts only automatic retries that started, or still may, against the
task's hourly budget": new.
- `server/src/__tests__/failed-continuation-continues-again.test.ts`,
"the sweep continues a blocked task whose system retry was refused at
claim behind a recovered saved-work hold": red (the task was not a
suspect). Green: `repaired`, a pending continuation in the hold, the
task back in `todo`, and no wake enqueued. The next check reports a
pending intent.
- Same file, "a continuation the sweep records behind a refused run
counts once toward the task's hourly cap": red (5 counted after 5
attempts plus the new continuation), green (6, and the transient-retry
cap check refuses the next attempt).
- Same file, "continuations behind refused runs spend the sweep's
per-minute repair budget": 20 tasks. Red (20 continued in one pass),
green (10 continue, 10 are deferred with a `retryAt`, and the 10
continue in the next pass).
- Same file, "a sweep continuation interrupted right after it was
recorded still counts once": the continuation commits, then the
post-commit activity write fails (a simulated server stop). Red: 6
counted for 5 real attempts. Green: 5 counted, and the next automatic
retry is still allowed.
- `server/src/services/recovery/stranded-notice.test.ts`, "never tells a
person to press Retry when the task offers no Retry": new.

Local runs on this branch, one file at a time:
- `fresh-continuation-fallback.test.ts`: 77 passed. The suite's
`afterAll` teardown hit the 30 s hook timeout on a heavily loaded
machine.
- `failed-continuation-continues-again.test.ts`: 32 passed
- `assigned-work-invariant.test.ts`: 31 passed
- `stranded-notice.test.ts`: 33 passed
- CI on the first head (ce4431410b): every required check passed
- `server` `tsc --noEmit`: clean
- `pnpm verify --explain` selects most of the repository (2666 files),
so required CI supplies the broad coverage.

## Risks

- More tasks continue automatically. Each continuation goes through the
existing fallback. The 24-hour recency bound, the per-task hourly cap
(6), isolation of the next run, and every person-owned gate
(person-owned failure codes, a person's Stop, review, a person-owned
hold, a person's failed Retry) are unchanged.
- The hourly cap now ignores automatic retries that never started. A
task whose retries are refused at dispatch again and again is still
bounded: the refusal hands the hold to the fallback, and the fallback
records one continuation per source run.
- Known separate gap, not changed here: after an interrupted
preparation's holds are resolved (`settleInterruptedNativeBootstrap`),
`scheduleBoundedRetry` can report `exhausted` or `blocked`, and nothing
follows that up. The assigned-work sweep may now pick such a task up
through the refused-at-dispatch path, but this pull request does not
prove that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 22:01:39 -06:00
1 parent f7b1651799
commit 6b9e45787f
10 files changed
+364 -38

No files matched your search

@@ -3,10 +3,11 @@ import { and, eq, inArray, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { activityLog, agentWakeupRequests, agents, companies, createDb, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
import { recoveryService } from "../services/recovery/service.js";
import { ASSIGNED_WORK_CONTINUED_ACTION, assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js";
import { ASSIGNED_WORK_CONTINUED_ACTION, ASSIGNED_WORK_REPAIRS_PER_MINUTE, assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js";
import { taskAutomaticAttempts, transientRetryWithinTaskCap } from "../services/recovery/task-automatic-attempts.js";
import { FRESH_CONTINUATION_SCHEMA, continueHeldExecutions, ensureFreshContinuation, resetBacklogContinuationPacingForTests } from "../services/fresh-continuation-fallback.js";
import { PERMANENT_FAILURE_CODES, PERSON_OWNED_FAILURE_CODES } from "../services/continuation-cause.js";
import { AUTOMATIC_RETRY_REASONS, SHARED_CONTINUATION_WINDOW_MS } from "../services/recovery/continuation-ledger.js";
import { AUTOMATIC_RETRY_REASONS, SHARED_CONTINUATION_WINDOW_MS, sharedContinuationAttempts } from "../services/recovery/continuation-ledger.js";
import { SERVER_RESTART_RETRY_MAX_ATTEMPTS, SERVER_RESTART_RETRY_REASON, SERVER_RESTART_RETRY_WINDOW_MS } from "../services/execution-recovery-attempt.js";
import { FRESH_CONTINUATIONS_PER_HOUR } from "../services/fresh-continuation-fallback.js";
import { createSystemRecovery } from "../services/system-recovery-state.js";
@@ -414,4 +415,102 @@ describePostgres("a continuation that fails before doing any work is continued a
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).toMatchObject({ outcome: "repaired" });
expect(await continuationsOf(stopped.id)).toHaveLength(1);
}, 60_000);
async function blockedBehindRefusedRun(f: Awaited<ReturnType<typeof seed>>) {
const blockedAt = new Date(Date.now() - 20 * MINUTE);
const task = await f.task({ status: "blocked", blockedTransitionAt: blockedAt, statusVersion: 3 });
const source = await f.run(task.id, { errorCode: "adapter_failed", error: "native_instruction_materialization_handoff_unavailable",
finishedAt: new Date(blockedAt.getTime() - 500), resultJson: { workspaceRestoreFailure: "restore_failed" },
contextSnapshot: { wakeReason: "issue_children_completed" } });
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, kind: "active_run_watchdog",
status: "active", ownerType: "board", returnOwnerAgentId: f.agentId, cause: "legacy_execution_requires_reconciliation",
fingerprint: `legacy-execution:${source.id}`, nextAction: "Held.", createdAt: blockedAt,
evidence: { runId: source.id, originalFailureCode: "adapter_failed", workspaceRestoreFailure: "restore_failed",
systemRecovery: { ...createSystemRecovery(source.id), state: "complete" } } }).returning();
const refused = await f.run(task.id, { status: "cancelled", errorCode: "execution_reconciliation_required", startedAt: null,
retryOfRunId: source.id, finishedAt: new Date(blockedAt.getTime() + 1_000), createdAt: new Date(blockedAt.getTime() - 100), resultJson: {},
contextSnapshot: { wakeReason: "issue_continuation_needed", retryReason: "issue_continuation_needed" } });
return { task, source, hold: hold!, refused };
}
it("the sweep continues a blocked task whose system retry was refused at claim behind a recovered saved-work hold", async () => {
resetBacklogContinuationPacingForTests(db);
const f = await seed();
const { task, source, hold, refused } = await blockedBehindRefusedRun(f);
const queue = wakeQueue();
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).toContain(task.id);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id }))
.toMatchObject({ outcome: "repaired", verdict: { kind: "violation" } });
const [record] = await continuationsOf(source.id);
expect(record?.id).toBe(hold.id);
expect(record?.evidence).toMatchObject({ continuationDelivery: "pending", freshContinuation: { sourceRunId: source.id, trigger: "assigned_work" },
freshContinuationSettlement: { refusedAtDispatchRunId: refused.id, requestedBy: "system" } });
expect((await taskRow(task.id)).status).toBe("todo");
expect(queue.calls).toHaveLength(0);
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id }))
.toMatchObject({ outcome: "ok", verdict: { clause: "pending_intent" } });
}, 60_000);
it("a continuation the sweep records behind a refused run counts once toward the task's hourly cap", async () => {
resetBacklogContinuationPacingForTests(db);
const f = await seed();
const { task } = await blockedBehindRefusedRun(f);
for (const minutesAgo of [50, 45, 40, 35, 30]) await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system",
actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: task.id,
details: { outcome: "queued" }, createdAt: new Date(Date.now() - minutesAgo * MINUTE) });
expect(await transientRetryWithinTaskCap(db, f.companyId, task.id, new Date())).toBe(true);
const sweep = assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup });
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).toMatchObject({ outcome: "repaired" });
const attempts = await taskAutomaticAttempts(db, f.companyId, task.id, new Date());
expect(attempts.started).toHaveLength(FRESH_CONTINUATIONS_PER_HOUR);
expect(await transientRetryWithinTaskCap(db, f.companyId, task.id, new Date())).toBe(false);
}, 60_000);
it("continuations behind refused runs spend the sweep's per-minute repair budget", async () => {
resetBacklogContinuationPacingForTests(db);
const f = await seed();
const tasks = [];
for (let index = 0; index < 20; index++) tasks.push((await blockedBehindRefusedRun(f)).task);
let clock = new Date();
const sweep = assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup, now: () => clock });
const first = [];
for (const task of tasks) first.push(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id }));
expect(first.filter(check => check?.outcome === "repaired")).toHaveLength(ASSIGNED_WORK_REPAIRS_PER_MINUTE);
const deferred = first.filter(check => check?.outcome === "deferred_rate_limit");
expect(deferred).toHaveLength(20 - ASSIGNED_WORK_REPAIRS_PER_MINUTE);
expect(deferred.every(check => check!.retryAt!.getTime() > clock.getTime())).toBe(true);
clock = new Date(clock.getTime() + 66_000);
const second = [];
for (const task of tasks) second.push(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id }));
expect(second.filter(check => check?.outcome === "repaired")).toHaveLength(20 - ASSIGNED_WORK_REPAIRS_PER_MINUTE);
}, 120_000);
it("a sweep continuation interrupted right after it was recorded still counts once", async () => {
resetBacklogContinuationPacingForTests(db);
const f = await seed();
const task = await f.task();
const failed = await f.continued(task.id);
for (const minutesAgo of [50, 45, 40, 35]) await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system",
actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: task.id,
details: { outcome: "queued" }, createdAt: new Date(Date.now() - minutesAgo * MINUTE) });
const before = (await taskAutomaticAttempts(db, f.companyId, task.id, new Date())).started.length;
const crashAfterCommit = new Proxy(db, { get(target, property) {
if (property === "update") return (table: unknown) => {
if (table === activityLog) throw new Error("server stopped");
return target.update(table as typeof activityLog);
};
const value = Reflect.get(target, property, target);
return typeof value === "function" ? value.bind(target) : value;
} });
const sweep = assignedWorkSweep(crashAfterCommit, { enqueueWakeup: wakeQueue().enqueueWakeup });
await expect(sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).rejects.toThrow("server stopped");
expect(await continuationsOf(failed.id)).toHaveLength(1);
const attempts = await taskAutomaticAttempts(db, f.companyId, task.id, new Date());
expect(attempts.started).toHaveLength(before + 1);
expect(attempts.started).toHaveLength(5);
expect(await sharedContinuationAttempts(db, f.companyId, task.id, new Date())).toHaveLength(before + 1);
expect(await transientRetryWithinTaskCap(db, f.companyId, task.id, new Date())).toBe(true);
}, 60_000);
});
@@ -28,6 +28,8 @@ import { createPostgresRunDispatchAdapter } from "../modules/run-dispatch/adapte
import { createSystemRecovery } from "./system-recovery-state.js";
import { requestTaskRetry } from "./task-retry.js";
import { terminalizeLegacyExecution } from "./legacy-execution-recovery.js";
import { sharedContinuationAttempts } from "./recovery/continuation-ledger.js";
import { taskAutomaticAttempts } from "./recovery/task-automatic-attempts.js";
const support = await getEmbeddedPostgresTestSupport();
const suite = support.supported ? describe : describe.skip;
@@ -1351,4 +1353,125 @@ suite("fresh-continuation fallback", () => {
const [next] = await successors(f.runId);
expect(await readFreshWorkspaceReadiness(db, next!)).toMatchObject({ mode: "fresh_isolated_workspace" });
}, 60_000);
async function followUpFailedBeforeProviderWork(options: { restoreFailed?: boolean } = {}) {
const issueId = randomUUID(), runId = randomUUID();
const endedAt = new Date(Date.now() - 60_000);
await db.insert(issues).values({ id: issueId, companyId, title: "Summarize the subtasks", status: "in_progress", assigneeAgentId: agentId });
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: issueId, mode: "isolated",
strategyType: "git_worktree", name: "Task workspace", cwd: "/work/remote-task" }).returning();
await db.update(issues).set({ executionWorkspaceId: workspace!.id }).where(eq(issues.id, issueId));
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "automation", issueId,
runtimeMode: "legacy", status: "running", startedAt: new Date(endedAt.getTime() - 10_000),
createdAt: new Date(endedAt.getTime() - 60_000), responsibleUserId: userId,
contextSnapshot: { issueId, executionWorkspaceId: workspace!.id, wakeReason: "issue_children_completed" } }).returning();
const [environment] = await db.insert(environments).values({ name: `sandbox-${runId}`, driver: "sandbox" }).returning();
await db.insert(environmentLeases).values({ companyId, issueId, executionWorkspaceId: workspace!.id,
heartbeatRunId: runId, environmentId: environment!.id, status: "active", leasePolicy: "reuse_by_environment", provider: "daytona",
providerLeaseId: `sandbox-${runId}`, metadata: { remoteCwd: "/work/remote-task", driver: "sandbox",
workspaceRealization: { authoritativeRoot: "/work/remote-task" } } });
await terminalizeLegacyExecution({ db, run: run!, status: "failed", patch: { errorCode: "adapter_failed",
error: "native_instruction_materialization_handoff_unavailable", finishedAt: endedAt,
resultJson: options.restoreFailed === false ? { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }
: { workspaceRestoreFailure: "restore_failed" } } });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
return { issueId, runId, actionId: hold?.id ?? null };
}
async function savedWorkRecoveryState(actionId: string, runId: string, state: "running" | "complete") {
await db.update(issueRecoveryActions).set({ updatedAt: new Date(),
evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({ systemRecovery: { ...createSystemRecovery(runId), state } })}::jsonb` })
.where(eq(issueRecoveryActions.id, actionId));
}
async function systemRetryAtClaim(f: { issueId: string; runId: string }) {
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId,
runtimeMode: "legacy", status: "queued", retryOfRunId: f.runId,
contextSnapshot: { issueId: f.issueId, wakeReason: "issue_continuation_needed", retryReason: "issue_continuation_needed",
retryOfRunId: f.runId } }).returning();
const outcome = await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId, runId: run!.id,
expectedStatus: "queued", now: new Date() });
return { run: run!, outcome };
}
async function deliveredContinuationIsClaimable(f: { issueId: string; runId: string; actionId: string | null }, refusedRunId: string) {
await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId, issueId: f.issueId });
expect((await issue(f.issueId)).status).not.toBe("blocked");
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
await occupySlot();
await deliver();
const next = (await successors(f.runId)).filter(run => run.id !== refusedRunId);
expect(next).toHaveLength(1);
expect(next[0]!.status).toBe("queued");
expect(await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId, runId: next[0]!.id,
expectedStatus: "queued", now: new Date() })).toEqual({ outcome: "not_stale" });
return next[0]!;
}
it("continues a follow-up that failed before provider work with unrestored files, when the saved work was recovered before the system retry was refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork();
expect(f.actionId).not.toBeNull();
await savedWorkRecoveryState(f.actionId!, f.runId, "complete");
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({
freshContinuation: { sourceRunId: f.runId }, continuationDelivery: "pending",
freshContinuationSettlement: { refusedAtDispatchRunId: refused.run.id, requestedBy: "system" } });
await deliveredContinuationIsClaimable(f, refused.run.id);
}, 60_000);
it("continues a follow-up that failed before provider work with unrestored files, when the saved work was recovered after the system retry was refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork();
await savedWorkRecoveryState(f.actionId!, f.runId, "running");
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence.freshContinuationCheck)
.toMatchObject({ kind: "exact_recovery_pending", reason: "saved_work_recovery_running" });
await savedWorkRecoveryState(f.actionId!, f.runId, "complete");
await makeDue(f.actionId!);
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({ freshContinuation: { sourceRunId: f.runId },
freshContinuationSettlement: { refusedAtDispatchRunId: refused.run.id, requestedBy: "system" } });
await deliveredContinuationIsClaimable(f, refused.run.id);
}, 60_000);
it("continues a follow-up that failed before provider work with its files in place, after its system retry was refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork({ restoreFailed: false });
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({ freshContinuation: { sourceRunId: f.runId },
freshContinuationSettlement: { refusedAtDispatchRunId: refused.run.id, requestedBy: "system" } });
await deliveredContinuationIsClaimable(f, refused.run.id);
}, 60_000);
it("counts only automatic retries that started, or still may, against the task's hourly budget", async () => {
const f = await followUpFailedBeforeProviderWork();
for (let index = 0; index < FRESH_CONTINUATIONS_PER_HOUR; index++) {
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId, runtimeMode: "legacy",
status: "cancelled", errorCode: "execution_reconciliation_required", finishedAt: new Date(), retryOfRunId: f.runId,
scheduledRetryReason: "transient_failure", contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry" } });
}
expect(await sharedContinuationAttempts(db, companyId, f.issueId, new Date())).toHaveLength(0);
expect((await taskAutomaticAttempts(db, companyId, f.issueId, new Date())).started).toHaveLength(0);
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId, runtimeMode: "legacy",
status: "failed", errorCode: "adapter_failed", startedAt: new Date(), finishedAt: new Date(), retryOfRunId: f.runId,
scheduledRetryReason: "transient_failure", contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry" } });
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId, runtimeMode: "legacy",
status: "scheduled_retry", retryOfRunId: f.runId,
scheduledRetryReason: "transient_failure", contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry" } });
expect(await sharedContinuationAttempts(db, companyId, f.issueId, new Date())).toHaveLength(2);
expect((await taskAutomaticAttempts(db, companyId, f.issueId, new Date())).started).toHaveLength(2);
}, 60_000);
it("still leaves a person's failed Retry of a follow-up for that person after its successor is refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork();
await savedWorkRecoveryState(f.actionId!, f.runId, "complete");
await db.update(heartbeatRuns).set({ contextSnapshot: sql`${heartbeatRuns.contextSnapshot} || '{"wakeReason":"task_retry"}'::jsonb` })
.where(eq(heartbeatRuns.id, f.runId));
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence.freshContinuation).toBeUndefined();
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toEqual({ kind: "person_owned", reason: "retry_needed" });
}, 60_000);
});
@@ -26,7 +26,7 @@ import { notifyRoutineExecutionStatus } from "./routine-execution-status-notific
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
import { readRunCancellation } from "./run-cancellation.js";
import { readSystemRecovery } from "./system-recovery-state.js";
import { RESTART_CLASS_FAILURE_CODES, continuationCauseIsSystemOwned, continuesAutomaticallySql, isPersonOwnedFailureCode } from "./continuation-cause.js";
import { RESTART_CLASS_FAILURE_CODES, continuationCauseIsSystemOwned, continuationCauseIsSystemOwnedSql, continuesAutomaticallySql, isPersonOwnedFailureCode, recoveryRunSql } from "./continuation-cause.js";
import { sharedContinuationAttempts } from "./recovery/continuation-ledger.js";
export { AUTOMATIC_CONTINUATION_WAKE_REASONS, PERSON_OWNED_FAILURE_CODES, isPersonOwnedFailureCode, isRecoveryRun } from "./continuation-cause.js";
@@ -813,7 +813,10 @@ function interruptionEligibility(input: { now: Date; companyId: SQL; issueId: SQ
and retry.run_id is null and retry.payload->>'issueId' = ${input.issueId}::text and retry.requested_at > source.finished_at)
or exists (select 1 from ${heartbeatRuns} held where ${refusedLater}
and held.context_snapshot->>'wakeReason' = 'task_retry' and held.finished_at > ${since}))`);
const recent = source(sql`${continuesAutomaticallySql("source")}
const systemRetriedSource = sql`(${continuationCauseIsSystemOwnedSql("source")}
and coalesce(source.context_snapshot->>'wakeReason', '') <> 'task_retry' and not coalesce(source.context_snapshot ? 'explicitUserContinuation', false)
and exists (select 1 from ${heartbeatRuns} held where ${refusedLater} and ${recoveryRunSql("held")}))`;
const recent = source(sql`(${continuesAutomaticallySql("source")} or ${systemRetriedSource})
and greatest(source.finished_at, (select max(held.finished_at) from ${heartbeatRuns} held where ${refusedLater})) > ${since}`);
return { recent, personAsked };
}
@@ -38,8 +38,7 @@ import { attributeBlock, BLOCK_ATTRIBUTION_WINDOW_MS, blockedStatusDetailsSql }
import { ASSIGNED_WORK_CONTINUED_ACTION, PAUSED_FOR_AUTOMATIC_WORK, taskAutomaticAttempts } from "./task-automatic-attempts.js";
import { automaticRunCreationHeldUntil, readAutomaticWorkStatus, reserveAutomaticRunCreation } from "./automatic-run-brake.js";
import { isReattachPhaseActive, reattachPhaseRecheckAt } from "../heartbeat/run-admission.js";
import { ensureFreshContinuation, type FreshContinuationResult } from "../fresh-continuation-fallback.js";
import { canContinueRecoveryRunAutomatically } from "../continuation-cause.js";
import { continueSweepTaskFromSavedWork } from "./sweep-saved-work-continuation.js";
import { STRANDED_CONTINUATION_MAX_AGE_MS, STRANDED_KIND, strandedHoldContinuesAutomatically, type StrandedHoldRun } from "./stranded-hold-continuation.js";
import {
continuationActivityMessage,
@@ -499,6 +498,13 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
* Chat conversations and blocks older than the stale window are left out:
* the first wait on a person's message, the second is only reported.
*/
function latestRunRefusedAtDispatch(at: Date) {
return sql`coalesce((select lr.status = 'cancelled' and lr.started_at is null and lr.error_code = 'execution_reconciliation_required'
and lr.finished_at > ${new Date(at.getTime() - STRANDED_CONTINUATION_MAX_AGE_MS).toISOString()}::timestamptz
from heartbeat_runs lr where lr.company_id = ${issues.companyId} and (lr.context_snapshot->>'issueId') = ${issues.id}::text
order by lr.created_at desc limit 1), false)`;
}
async function listSuspects(scope: { companyId?: string; issueId?: string; createdAfter?: Date | null }, cursor: string | null, limit: number) {
const at = now();
const liveRun = sql`exists (select 1 from heartbeat_runs r where r.company_id = ${issues.companyId}
@@ -559,7 +565,9 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
-- except a recent board escalation, which the predicate judges by its failed run's cause
and not (ra.kind = ${STRANDED_KIND} and ra.owner_type = 'board' and exists (select 1 from heartbeat_runs sr
where sr.company_id = ra.company_id and sr.id::text = ra.evidence->>'latestRunId'
and sr.finished_at > ${new Date(at.getTime() - STRANDED_CONTINUATION_MAX_AGE_MS).toISOString()}::timestamptz)))`,
and sr.finished_at > ${new Date(at.getTime() - STRANDED_CONTINUATION_MAX_AGE_MS).toISOString()}::timestamptz))
and not (${issues.status} = 'blocked' and ra.cause not in ('configuration_incomplete', 'ownership_owner_unprovable')
and ${latestRunRefusedAtDispatch(at)}))`,
// An assignee that cannot take work (paused, being set up, held) is
// its own wait; the predicate names it when anything else asks.
sql`exists (select 1 from agents a where a.id = ${issues.assigneeAgentId} and a.company_id = ${issues.companyId}
@@ -602,6 +610,14 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
// hold's owner (the fresh-continuation fallback, same per-task cap). The
// sweep reports it and repairs only blocks that have no hold at all, so
if (issue.status === "blocked" && facts.activeRecoveryAction && !facts.activeRecoveryAction.continuedBySweep) {
const behindRefusedRun = facts.latestRun?.status === "cancelled" && facts.latestRun.errorCode === "execution_reconciliation_required";
const deferred = behindRefusedRun ? repairBudgetDeferral(at) : null;
if (deferred) return { verdict, outcome: "deferred_rate_limit", retryAt: deferred };
const refused = behindRefusedRun ? await continueFromSavedWork(issue, facts, at) : null;
if (refused?.kind === "continued") repairTimes.push(at.getTime());
if (refused?.kind === "continued" || refused?.kind === "already_continued") return { verdict, outcome: "repaired", retryAt: null };
if (refused?.kind === "backoff" || refused?.kind === "exact_recovery_pending")
return { verdict, outcome: refused.kind === "backoff" ? "deferred_cap" : "waiting_on_recovery", retryAt: refused.retryAt };
return { verdict, outcome: "left_to_hold_owner", retryAt: null };
}
// A stalled native run is ended by native recovery, not here: the sweep
@@ -623,11 +639,8 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
}
// A small process-wide budget keeps a post-deploy backlog from starting
// hundreds of runs at once. Deferred tasks come back on their deadline.
const minuteAgo = at.getTime() - 60_000;
while (repairTimes.length && repairTimes[0] <= minuteAgo) repairTimes.shift();
if (repairTimes.length >= ASSIGNED_WORK_REPAIRS_PER_MINUTE) {
return { verdict, outcome: "deferred_rate_limit", retryAt: new Date(repairTimes[0] + 60_000 + Math.floor(Math.random() * 5_000)) };
}
const budgetDeferral = repairBudgetDeferral(at);
if (budgetDeferral) return { verdict, outcome: "deferred_rate_limit", retryAt: budgetDeferral };
const heldUntil = facts.latestRun
? await automaticRunCreationHeldUntil(db, issue.companyId, { issueId: issue.id, sourceRunId: facts.latestRun.id }) : null;
if (heldUntil) return { verdict, outcome: "deferred_rate_limit", retryAt: heldUntil };
@@ -642,7 +655,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
await settleContinuationActivity(claimed, claimed.reason, "restored_to_review");
return { verdict: claimed.verdict, outcome: "repaired", retryAt: null };
}
const fresh = await continueFromSavedWork(issue, facts, at);
const fresh = await continueFromSavedWork(issue, facts, at, claimed);
if (fresh?.kind === "continued") {
await settleContinuationActivity(claimed, claimed.reason, "queued", { continuedFromSavedWork: "true" });
return { verdict: claimed.verdict, outcome: "repaired", retryAt: claimed.nextCheckAt };
@@ -683,6 +696,13 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
return { verdict: claimed.verdict, outcome: run ? "repaired" : "enqueue_refused", retryAt: claimed.nextCheckAt };
}
function repairBudgetDeferral(at: Date): Date | null {
const minuteAgo = at.getTime() - 60_000;
while (repairTimes.length && repairTimes[0] <= minuteAgo) repairTimes.shift();
return repairTimes.length >= ASSIGNED_WORK_REPAIRS_PER_MINUTE
? new Date(repairTimes[0] + 60_000 + Math.floor(Math.random() * 5_000)) : null;
}
function reportStalledRun(issue: IssueRow, runId: string, stall: NonNullable<AssignedWorkFacts["activeRuns"][number]["queuedStall"]> | null, at: Date) {
if (reportedStalledRuns.has(runId)) return;
if (reportedStalledRuns.size >= 1_000) reportedStalledRuns.clear();
@@ -725,20 +745,8 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) {
return { verdict: claimed.verdict, outcome: run ? "replaced_stalled_run" : "enqueue_refused", retryAt: claimed.nextCheckAt };
}
async function continueFromSavedWork(issue: IssueRow, facts: AssignedWorkFacts, at: Date): Promise<FreshContinuationResult | null> {
const source = facts.latestRun;
if (!source?.finishedAt || !["failed", "timed_out", "interrupted", "cancelled"].includes(source.status)
|| (source.status === "cancelled" && source.errorCode === "execution_reconciliation_required")) return null;
const [run] = await db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode, error: heartbeatRuns.error,
retryOfRunId: heartbeatRuns.retryOfRunId, scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
contextSnapshot: heartbeatRuns.contextSnapshot, runnerProfileJson: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, source.id)));
if (!run || !canContinueRecoveryRunAutomatically(run)) return null;
return ensureFreshContinuation(db, { companyId: issue.companyId, runId: source.id, trigger: "assigned_work", now: at })
.catch(error => {
logger.warn({ err: error, companyId: issue.companyId, issueId: issue.id }, "assigned-work continuation from saved work deferred");
return null;
});
function continueFromSavedWork(issue: IssueRow, facts: AssignedWorkFacts, at: Date, claim?: { activityId: string; reason: ViolationReason }) {
return continueSweepTaskFromSavedWork(db, { companyId: issue.companyId, issueId: issue.id, latestRun: facts.latestRun }, at, claim);
}
async function startSavedWorkRecovery(issue: IssueRow, facts: AssignedWorkFacts,
@@ -6,6 +6,11 @@ export const SCHEDULER_TRANSIENT_RETRY_REASON = "transient_failure";
export const AUTOMATIC_RETRY_REASONS = [SCHEDULER_TRANSIENT_RETRY_REASON, "server_restart", "interrupted_preparation"];
export const SHARED_CONTINUATION_WINDOW_MS = 60 * 60_000;
export function startedOrPendingAutomaticRetry() {
return sql`not (${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null
and coalesce(${heartbeatRuns.errorCode}, '') = 'execution_reconciliation_required')`;
}
export async function freshContinuationTimes(db: Db, companyId: string, issueId: string, since: Date) {
const rows = await db.select({ recordedAt: sql<string>`${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}'`,
trigger: sql<string | null>`${issueRecoveryActions.evidence}#>>'{freshContinuation,trigger}'` }).from(issueRecoveryActions).where(and(
@@ -25,7 +30,8 @@ export async function sharedContinuationAttempts(db: Db, companyId: string, issu
sql`coalesce(${activityLog.details}->>'continuedFromSavedWork', '') <> 'true'`)).limit(64),
db.select({ createdAt: heartbeatRuns.createdAt }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
inArray(heartbeatRuns.scheduledRetryReason, AUTOMATIC_RETRY_REASONS), gte(heartbeatRuns.createdAt, since))).limit(64),
inArray(heartbeatRuns.scheduledRetryReason, AUTOMATIC_RETRY_REASONS), gte(heartbeatRuns.createdAt, since),
startedOrPendingAutomaticRetry())).limit(64),
freshContinuationTimes(db, companyId, issueId, since),
]);
return [...plainContinuations.map(row => row.createdAt), ...retries.map(row => row.createdAt), ...fresh.map(record => record.at)]
+10
View File
@@ -176,6 +176,7 @@ import {
buildExecutionReviewParticipantRecoveryNoticeSeed,
buildExecutionReviewParticipantUnavailableNoticeSeed,
buildStrandedRecoveryEscalationNotice,
withoutRetryInstruction,
buildNoRunningExecutionNoticeSeed,
RETRY_FROM_SAVED_WORK_NEXT_ACTION,
type StrandedRecoveryNoticeSeed,
@@ -185,6 +186,7 @@ import {
isStrandedIssueRecoveryOriginKind,
} from "./origins.js";
import { withRecoveryContext } from "./status-only-context.js";
import { getExecutionBlocker } from "../execution-blocker.js";
import { currentRunFailureText } from "../native-runtime/native-exact-resume-failure.js";
import { isAutomaticRecoverySuppressedByPauseHold } from "./pause-hold-guard.js";
import { rehomeStrandedLegacyWorkspaceTasks } from "./legacy-workspace-rehome.js";
@@ -4251,6 +4253,13 @@ export function recoveryService(
recoveryCause,
successfulRunHandoffEvidence: input.successfulRunHandoffEvidence,
}, tx);
const executionBlocker = await getExecutionBlocker(tx as unknown as Db, input.issue.companyId, input.issue.id);
const retryOffered = !(executionBlocker && executionBlocker.canRetry === false);
if (!retryOffered && /\bretry\b/i.test(recoveryAction.nextAction ?? "")) {
recoveryAction.nextAction = withoutRetryInstruction(recoveryAction.nextAction!);
await tx.update(issueRecoveryActions).set({ nextAction: recoveryAction.nextAction })
.where(and(eq(issueRecoveryActions.companyId, input.issue.companyId), eq(issueRecoveryActions.id, recoveryAction.id)));
}
const isProviderQuotaWait =
recoveryCause === "provider_quota" &&
!recoveryAction.ownerAgentId &&
@@ -4334,6 +4343,7 @@ export function recoveryService(
fallbackBody: input.comment,
recoveryCause,
recoveryActionId: recoveryAction.id,
retryOffered,
continuesAutomatically: Boolean(input.latestRun) && recoveryAction.kind === STRANDED_KIND
&& recoveryAction.ownerType === "board" && input.latestRun?.agentId === input.issue.assigneeAgentId
&& strandedRunCauseIsTemporary({
@@ -401,3 +401,16 @@ it("tells the operator to switch a preconfigured Runner home to an isolated one"
expect(notice.nextAction).toContain("isolated Runner home");
expect(notice.nextAction).not.toContain("Choose an AI connection");
});
it("never tells a person to press Retry when the task offers no Retry", () => {
const sourceRun = { id: "run-1", status: "cancelled", errorCode: "execution_reconciliation_required" };
for (const seed of [buildNoRunningExecutionNoticeSeed({ status: "in_progress" }), null]) {
const offered = buildStrandedRecoveryEscalationNotice({ seed, recoveryActionId: "action-1", recoveryOwner: null, sourceRun });
expect(`${offered.body} ${JSON.stringify(allRows(offered.metadata))}`).toMatch(/Press Retry/);
const withheld = buildStrandedRecoveryEscalationNotice({ seed, recoveryActionId: "action-1", recoveryOwner: null, sourceRun,
retryOffered: false });
expect(withheld.body).not.toMatch(/retry/i);
expect(JSON.stringify(allRows(withheld.metadata))).not.toMatch(/retry/i);
expect(withheld.body).toContain("Paperclip is holding this task until its saved work is settled.");
}
});
@@ -27,6 +27,13 @@ export type StrandedRecoveryNoticeSeed = {
nextAction?: string;
};
export const SAVED_WORK_HELD_SENTENCE = "Paperclip is holding this task until its saved work is settled.";
export function withoutRetryInstruction(text: string): string {
const kept = text.split(/(?<=[.!?])\s+/).filter(sentence => !/\bretry\b/i.test(sentence));
return [...kept, SAVED_WORK_HELD_SENTENCE].join(" ");
}
export type StrandedRecoveryEscalationNotice = {
body: string;
presentation: IssueCommentPresentation;
@@ -270,6 +277,7 @@ export function buildStrandedRecoveryEscalationNotice(input: {
recoveryCause?: string | null;
recoveryActionId: string;
continuesAutomatically?: boolean;
retryOffered?: boolean;
recoveryOwner: { id: string; name: string | null } | null | undefined;
sourceRun: {
id: string;
@@ -292,7 +300,8 @@ export function buildStrandedRecoveryEscalationNotice(input: {
tone: "danger" as const,
} : SETUP_FAILURE_NOTICES[input.sourceRun?.errorCode ?? ""] ?? input.seed;
const fallbackBody = input.fallbackBody?.trim();
const body = seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY);
const honest = (text: string) => input.retryOffered === false ? withoutRetryInstruction(text) : text;
const body = honest(seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY));
const title =
STRANDED_RECOVERY_NOTICE_TITLES_BY_RUN_ERROR_CODE[input.sourceRun?.errorCode?.trim() ?? ""] ??
seed?.title ??
@@ -310,9 +319,9 @@ export function buildStrandedRecoveryEscalationNotice(input: {
),
keyValueRow(
"Next action",
seed?.nextAction ?? (input.recoveryOwner
honest(seed?.nextAction ?? (input.recoveryOwner
? "The assigned agent continues the task from the saved work"
: RETRY_FROM_SAVED_WORK_NEXT_ACTION),
: RETRY_FROM_SAVED_WORK_NEXT_ACTION)),
),
];
@@ -0,0 +1,51 @@
import { and, desc, eq, inArray, sql } from "drizzle-orm";
import { activityLog, heartbeatRuns, type Db } from "@paperclipai/db";
import { logger } from "../../middleware/logger.js";
import { continueFreshInTransaction, type FreshContinuationResult } from "../fresh-continuation-fallback.js";
import { canContinueRecoveryRunAutomatically } from "../continuation-cause.js";
import { continuationActivityMessage, type AssignedWorkFacts, type ViolationReason } from "./assigned-work-invariant.js";
type SweepTask = { companyId: string; issueId: string; latestRun: AssignedWorkFacts["latestRun"] };
type SweepClaim = { activityId: string; reason: ViolationReason };
function continueAndSettleClaim(db: Db, task: SweepTask, runId: string, at: Date, claim?: SweepClaim) {
return db.transaction(async tx => {
const result = await continueFreshInTransaction(tx as unknown as Db, { companyId: task.companyId, runId, trigger: "assigned_work", now: at });
if (result.kind === "continued" && claim) {
const update = { outcome: "queued", message: continuationActivityMessage(claim.reason), continuedFromSavedWork: "true" };
await tx.update(activityLog).set({ details: sql`coalesce(${activityLog.details}, '{}'::jsonb) || ${JSON.stringify(update)}::jsonb` })
.where(and(eq(activityLog.companyId, task.companyId), eq(activityLog.id, claim.activityId)));
}
return result;
});
}
/** The sweep's continuation of a task from saved work: from its failed latest run, or from the run a refused continuation was for. */
export async function continueSweepTaskFromSavedWork(db: Db, task: SweepTask, at: Date, claim?: SweepClaim): Promise<FreshContinuationResult | null> {
const source = task.latestRun;
if (!source?.finishedAt || !["failed", "timed_out", "interrupted", "cancelled"].includes(source.status)) return null;
if (source.status === "cancelled" && source.errorCode === "execution_reconciliation_required") {
const [refusedFor] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, task.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${task.issueId}`,
inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted", "cancelled"]), sql`${heartbeatRuns.finishedAt} is not null`,
sql`not (${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null
and coalesce(${heartbeatRuns.errorCode}, '') = 'execution_reconciliation_required')`))
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
if (!refusedFor) return null;
return continueAndSettleClaim(db, task, refusedFor.id, at, claim)
.catch(error => {
logger.warn({ err: error, companyId: task.companyId, issueId: task.issueId }, "assigned-work continuation behind a refused run deferred");
return null;
});
}
const [run] = await db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode, error: heartbeatRuns.error,
retryOfRunId: heartbeatRuns.retryOfRunId, scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
contextSnapshot: heartbeatRuns.contextSnapshot, runnerProfileJson: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, task.companyId), eq(heartbeatRuns.id, source.id)));
if (!run || !canContinueRecoveryRunAutomatically(run)) return null;
return continueAndSettleClaim(db, task, source.id, at, claim)
.catch(error => {
logger.warn({ err: error, companyId: task.companyId, issueId: task.issueId }, "assigned-work continuation from saved work deferred");
return null;
});
}
@@ -1,6 +1,6 @@
import { and, desc, eq, gte, inArray, sql } from "drizzle-orm";
import { activityLog, heartbeatRuns, type Db } from "@paperclipai/db";
import { ASSIGNED_WORK_CONTINUED_ACTION, AUTOMATIC_RETRY_REASONS, SCHEDULER_TRANSIENT_RETRY_REASON, freshContinuationTimes } from "./continuation-ledger.js";
import { ASSIGNED_WORK_CONTINUED_ACTION, AUTOMATIC_RETRY_REASONS, SCHEDULER_TRANSIENT_RETRY_REASON, freshContinuationTimes, startedOrPendingAutomaticRetry } from "./continuation-ledger.js";
import { CONTINUATION_CAP, CONTINUATION_CAP_WINDOW_MS } from "./assigned-work-invariant.js";
export { ASSIGNED_WORK_CONTINUED_ACTION, SCHEDULER_TRANSIENT_RETRY_REASON };
@@ -12,19 +12,23 @@ export async function taskAutomaticAttempts(db: Db, companyId: string, issueId:
db.select({ createdAt: activityLog.createdAt, outcome: sql<string | null>`${activityLog.details}->>'outcome'` }).from(activityLog).where(and(
eq(activityLog.companyId, companyId), eq(activityLog.entityType, "issue"), eq(activityLog.entityId, issueId),
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION), gte(activityLog.createdAt, since),
sql`coalesce(${activityLog.details}->>'continuedFromSavedWork', '') <> 'true'`,
sql`coalesce(${activityLog.details}->>${PAUSED_FOR_AUTOMATIC_WORK}, '') <> 'true'`))
.orderBy(desc(activityLog.createdAt)).limit(64),
db.select({ createdAt: heartbeatRuns.createdAt }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
inArray(heartbeatRuns.scheduledRetryReason, AUTOMATIC_RETRY_REASONS), gte(heartbeatRuns.createdAt, since)))
inArray(heartbeatRuns.scheduledRetryReason, AUTOMATIC_RETRY_REASONS), gte(heartbeatRuns.createdAt, since),
startedOrPendingAutomaticRetry()))
.orderBy(desc(heartbeatRuns.createdAt)).limit(64),
]);
const fresh = (await freshContinuationTimes(db, companyId, issueId, since))
.filter(record => record.trigger !== "assigned_work").map(record => record.at);
const freshRecords = await freshContinuationTimes(db, companyId, issueId, since);
const sweepFresh = freshRecords.filter(record => record.trigger === "assigned_work").map(record => record.at);
const fresh = freshRecords.filter(record => record.trigger !== "assigned_work").map(record => record.at);
const sweep = [...continuations.map(row => row.createdAt), ...sweepFresh];
const others = [...transientRetries.map(row => row.createdAt), ...fresh];
const all = [...continuations.map(row => row.createdAt), ...others];
const started = [...continuations.filter(row => row.outcome !== "not_started").map(row => row.createdAt), ...others];
return { continuations: continuations.map(row => row.createdAt), all, started };
const all = [...sweep, ...others];
const started = [...continuations.filter(row => row.outcome !== "not_started").map(row => row.createdAt), ...sweepFresh, ...others];
return { continuations: sweep, all, started };
}
export async function transientRetryWithinTaskCap(db: Db, companyId: string, issueId: string, at: Date) {