fix(recovery): continue only recent interruptions from the held backlog, newest first, paced (#21222)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server restarts or a deploy swaps servers, agent runs are
interrupted. The fresh-continuation fallback (#21135) continues each
interrupted task in a new run from its saved work.
> - The fallback also has a backlog pass. It continues tasks that older
releases left held by a recovery record.
> - The backlog pass had no age limit and no pacing. It took the oldest
records first, 25 per pass.
> - On the first hosted release with the fallback, it continued 176
tasks in 30 minutes. 74 of them had a source run that ended more than 7
days ago. The oldest ended four weeks ago. More than 100 runs were
queued, and one agent had about 130 runs.
> - The user-facing goal of the fallback is "an interruption is
invisible". It is not "run last week's failures again".
> - This pull request limits the backlog pass to recent interruptions,
newest first, at a paced rate.
> - The benefit is that a release no longer starts a herd of old,
unwanted runs, and recent interrupted work still continues by itself.

## Linked Issues or Issue Description

Refs #21135

**What happened?**
A hosted instance took its first release with the fresh-continuation
fallback. Within 30 minutes, the backlog pass continued 176 held tasks.
By source run age: 26 under 1 day, 76 from 1 to 7 days, 74 over 7 days.
The source end codes included `adapter_failed` (27),
`runner_remote_provider_artifact_incompatible` (14),
`native_provider_terminal_failed` (14) and `setup_failed` (8). These are
configuration or provider failures, not interruptions. The pass kept
firing at 12 to 36 continuations per 5 minutes, and the queued backlog
grew past 100 runs.

**Expected behavior**
The backlog pass continues a task only when the task was recently
interrupted by a restart, a deploy, a lost lease, a lost process or a
refused recovery. Old records and configuration or provider failures
stay as they are, for a person's Retry. The pass starts new runs at a
controlled rate, newest first.

**Steps to reproduce**
1. Have held recovery records that older releases saved, with source
runs that ended days or weeks ago, some with `adapter_failed`.
2. Deploy a release that contains #21135.
3. See that the disposition worker continues all of them, oldest first,
25 per pass, and queues a run for each.

## What Changed

- `continueHeldExecutions` and `heldExecutionsAwaitingFreshContinuation`
(server/src/services/fresh-continuation-fallback.ts):
- Continue a held record only if its source run ended within
`BACKLOG_CONTINUATION_MAX_AGE_MS` (24 hours) AND its error code is in
`BACKLOG_CONTINUATION_SOURCE_CODES` (restart, deploy, lease,
process-loss and recovery-refusal codes).
- Do not touch any other held record. Do not settle it and do not modify
it. A log line counts these records, at most once every 10 minutes.
  - Order by the source run's end time, newest first.
- Continue at most 5 records per pass and 10 per rolling minute per
server.
- The rule I chose, and why: 24 hours and interruption codes only. A
person who is waiting for a task notices within a day. A run that failed
for configuration or provider reasons fails again the same way, so a
person must change something and press Retry. The 7-day rule of the
assigned-work sweep is too wide for this pass: 76 of the 176 continued
tasks were from 1 to 7 days old.
- The live path does not change. Fresh interruptions come through
restart recovery and the disposition pass (`recovery_refused`), and
through continuation wakes. They are not paced and have no age rule.
- Upgrade safety: records that a release already continued carry
`freshContinuation` and are excluded as before. Queued continuation runs
are not changed.
- The 10-per-minute limit is per server. During a deploy overlap, two
servers can each continue 10 per minute. This is acceptable for a short
overlap, and it keeps this change free of shared state.

## Verification

- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts`: 53 passed.
- New test "continues only recent interruptions from the held backlog,
newest first, and leaves the rest untouched (staging herd)". It uses
mixed-age and mixed-cause records: 1 h and 6 h restart codes, 3-day and
28-day restart codes, and 2 h
`runner_remote_provider_artifact_incompatible` and `adapter_failed`.
Only the two recent restart-caused records continue, the 1 h record
first. The four other records keep the same status, evidence and
`updated_at` across repeated passes.
- New test "paces the held backlog to ten continuations a minute per
server": 12 eligible records give 10 continuations in the first minute
and the remaining 2 in the next minute.
- RED on origin/master: both new tests fail (`expected [] to have a
length of 1`; `expected 12 to be 10`).
- The existing tests for the live path (for example "continues tasks
that older releases left blocked by these codes", with a source run that
ended one minute ago) still pass.
- `pnpm --filter @paperclipai/server run typecheck`: clean.

## Risks

- A held record from 1 to 7 days ago, caused by an interruption, no
longer continues by itself. It stays as it was before #21135: the task
shows its recovery state and a person can press Retry.
- A record whose cause is not in the list (a future interruption code)
is not continued by the backlog pass. The live path still continues
fresh interruptions for any code that is not a person's.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 18:36:22 -06:00
1 parent b4ca19f3a8
commit 19f8ed1ea2
2 files changed
+139 -6

No files matched your search

@@ -10,6 +10,7 @@ import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./ex
import { getExecutionBlocker } from "./execution-blocker.js";
import {
FRESH_CONTINUATIONS_PER_HOUR, FRESH_CONTINUATION_SCHEMA, PERSON_OWNED_FAILURE_CODES, continueHeldExecutions, ensureFreshContinuation,
resetBacklogContinuationPacingForTests,
freshContinuationNote, heldExecutionsAwaitingFreshContinuation, nextFreshContinuationCheckAt, recordFreshContinuationCheck, FRESH_CONTINUATION_PENDING_BOUND_MS, FRESH_CONTINUATION_PROGRESS_BOUND_MS, FRESH_CONTINUATION_SAVED_WORK_BOUND_MS,
} from "./fresh-continuation-fallback.js";
import { heartbeatService } from "./heartbeat.js";
@@ -55,6 +56,7 @@ suite("fresh-continuation fallback", () => {
}, 60_000);
afterEach(async () => {
resetBacklogContinuationPacingForTests(db);
await db.execute(sql`truncate table issue_recovery_actions, native_run_finalizations, tool_invocations,
issue_approvals, approvals, issue_thread_interactions, activity_log, agent_wakeup_requests, environment_leases, execution_workspaces, heartbeat_runs, issues cascade`);
});
@@ -240,6 +242,55 @@ suite("fresh-continuation fallback", () => {
await expectOneFreshSuccessor(f);
}, 60_000);
/** A held record from an older release (settled blocked), its source run ended `ageMs` ago with `errorCode`. */
async function heldBacklog(errorCode: string, ageMs: number) {
const f = await unfinishedRun({ runtime: "native", errorCode, hold: "settled_blocked" });
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - ageMs) }).where(eq(heartbeatRuns.id, f.runId));
return f;
}
const continuedFrom = async (runId: string) => db.select().from(issueRecoveryActions)
.where(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${runId}`);
const evidenceOf = async (actionId: string) =>
(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, actionId)))[0]!;
it("continues only recent interruptions from the held backlog, newest first, and leaves the rest untouched (staging herd)", async () => {
// paperclip.staging, 22:53Z: the first release with the fallback continued 176 held tasks in 30 minutes;
// 74 had a source run that ended more than 7 days ago, many with configuration or provider failures.
const HOUR = 60 * 60_000;
const recentOld = await heldBacklog("native_restart_recovery_blocked", 6 * HOUR);
const recentNew = await heldBacklog("native_continuation_requires_reconciliation", 1 * HOUR);
const weekOld = await heldBacklog("native_restart_recovery_blocked", 3 * 24 * HOUR);
const monthOld = await heldBacklog("native_session_interrupted", 28 * 24 * HOUR);
const providerFailure = await heldBacklog("runner_remote_provider_artifact_incompatible", 2 * HOUR);
const adapterFailure = await heldBacklog("adapter_failed", 2 * HOUR);
const untouched = [weekOld, monthOld, providerFailure, adapterFailure];
const before = await Promise.all(untouched.map(f => evidenceOf(f.actionId!)));
expect(await continueHeldExecutions(db, new Date(), 1)).toMatchObject({ continued: 1 });
expect(await continuedFrom(recentNew.runId)).toHaveLength(1);
expect(await continuedFrom(recentOld.runId)).toHaveLength(0);
for (let pass = 0; pass < 3; pass += 1) await continueHeldExecutions(db, new Date());
expect(await continuedFrom(recentOld.runId)).toHaveLength(1);
for (const [index, f] of untouched.entries()) {
expect(await continuedFrom(f.runId)).toHaveLength(0);
const after = await evidenceOf(f.actionId!);
expect({ status: after.status, evidence: after.evidence, updatedAt: after.updatedAt })
.toEqual({ status: before[index]!.status, evidence: before[index]!.evidence, updatedAt: before[index]!.updatedAt });
}
}, 120_000);
it("paces the held backlog to ten continuations a minute per server", async () => {
const records = [];
for (let index = 0; index < 12; index += 1) records.push(await heldBacklog("native_restart_recovery_blocked", (index + 1) * 60_000));
const now = new Date();
for (let pass = 0; pass < 4; pass += 1) await continueHeldExecutions(db, now);
const started = (await Promise.all(records.map(f => continuedFrom(f.runId)))).filter(rows => rows.length > 0).length;
expect(started).toBe(10);
// The next minute has room again.
await continueHeldExecutions(db, new Date(now.getTime() + 61_000));
expect((await Promise.all(records.map(f => continuedFrom(f.runId)))).filter(rows => rows.length > 0).length).toBe(12);
}, 180_000);
const sweepWake = (issueId: string) => heartbeatService(db).wakeup(agentId, { source: "automation", triggerDetail: "system",
reason: "issue_continuation_needed", payload: { issueId }, requestedByActorType: "system",
requestedByActorId: "idle-task-sweep", contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_continuation_needed" } });
@@ -713,6 +764,19 @@ suite("fresh-continuation fallback", () => {
expect(await executionControlSchedule(db).disposition()).toBeNull();
}, 60_000);
it("schedules no disposition scan for a held record the backlog pass leaves alone, and still does for an eligible one", async () => {
const overdueCheck = (actionId: string) => db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
freshContinuationCheck: { kind: "backoff", checkedAt: Date.now() - 7_200_000, nextCheckAt: Date.now() - 3_600_000 } })}::jsonb` })
.where(eq(issueRecoveryActions.id, actionId));
const old = await heldBacklog("native_restart_recovery_blocked", 3 * 24 * 60 * 60_000);
await overdueCheck(old.actionId!);
expect(await nextFreshContinuationCheckAt(db)).toBeNull();
expect(await executionControlSchedule(db).disposition()).toBeNull();
const recent = await heldBacklog("native_restart_recovery_blocked", 60 * 60_000);
await overdueCheck(recent.actionId!);
expect(await nextFreshContinuationCheckAt(db)).not.toBeNull();
}, 60_000);
it("tells the agent the previous attempt was cut off and which actions are unverified", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
await db.insert(toolInvocations).values({ companyId, agentId, issueId: f.issueId, runId: f.runId, toolName: "github.create_pull_request",
@@ -665,11 +665,64 @@ export function heldExecutionsAwaitingFreshContinuation(db: Db, now: Date, limit
runId: sql<string>`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')` })
.from(issueRecoveryActions).innerJoin(issues, and(eq(issues.id, issueRecoveryActions.sourceIssueId),
eq(issues.companyId, issueRecoveryActions.companyId)))
.where(and(heldExecutionWhere(), freshContinuationCheckDue(now)))
.orderBy(asc(issueRecoveryActions.updatedAt), asc(issueRecoveryActions.id))
.where(and(heldExecutionWhere(), freshContinuationCheckDue(now), recentInterruptionSourceSql(now)))
// Newest interruption first: the most recent work is the work someone is waiting for.
.orderBy(desc(sql`(select source.finished_at from ${heartbeatRuns} source where source.company_id = ${issueRecoveryActions.companyId}
and source.id::text = coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId'))`),
asc(issueRecoveryActions.id))
.limit(limit);
}
/**
* The backlog pass continues a held record only when it is a recent
* interruption: its source run ended within BACKLOG_CONTINUATION_MAX_AGE_MS
* and ended with a restart, deploy, lease, process-loss or recovery-refusal
* code. A configuration or provider failure, or anything older, stays exactly
* as it is for a person's Retry: re-running last week's failures is not
* "an interruption is invisible". Fresh interruptions reach the fallback
* through restart recovery and the disposition pass, not through this list.
*/
export const BACKLOG_CONTINUATION_MAX_AGE_MS = 24 * 60 * 60_000;
export const BACKLOG_CONTINUATION_SOURCE_CODES = [
"native_restart_recovery_blocked", "native_continuation_requires_reconciliation", "native_session_interrupted",
"native_runner_process_exited", "server_shutdown_interrupted", "process_lost", "controller_lease_lost",
"execution_finalization_deadline_exceeded", "local_runtime_api_listener_unavailable",
"runner_remote_process_identity_unavailable", "continuation_user_authorization_missing",
] as const;
/** At most this many backlog continuations per server per rolling minute, and per pass. */
export const BACKLOG_CONTINUATIONS_PER_MINUTE = 10;
const BACKLOG_CONTINUATIONS_PER_PASS = 5;
/** Holds the disposition pass (unresolvedExecutionDispositions) handles itself, whatever their age. */
function liveDispositionHoldSql() {
return and(inArray(issueRecoveryActions.status, ["active", "escalated"]), eq(issueRecoveryActions.kind, "active_run_watchdog"))!;
}
/** The backlog pass's eligibility: the one predicate its listing and its deadline query share. */
function recentInterruptionSourceSql(now: Date) {
return sql`exists (select 1 from ${heartbeatRuns} source where source.company_id = ${issueRecoveryActions.companyId}
and source.id::text = coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')
and source.finished_at > ${new Date(now.getTime() - BACKLOG_CONTINUATION_MAX_AGE_MS).toISOString()}::timestamptz
and source.error_code in (${sql.join(BACKLOG_CONTINUATION_SOURCE_CODES.map(code => sql`${code}`), sql`, `)}))`;
}
/** Held records the backlog pass leaves for a person (older, or not an interruption). */
async function countHeldExecutionsLeftAlone(db: Db, now: Date): Promise<number> {
const [row] = await db.select({ count: sql<number>`count(*)::int` }).from(issueRecoveryActions)
.innerJoin(issues, and(eq(issues.id, issueRecoveryActions.sourceIssueId), eq(issues.companyId, issueRecoveryActions.companyId)))
.where(and(heldExecutionWhere(), sql`not ${recentInterruptionSourceSql(now)}`));
return Number(row?.count ?? 0);
}
/** Per database client: one per server process in production, one per test database in tests. */
const backlogContinuationTimes = new WeakMap<object, number[]>();
export function resetBacklogContinuationPacingForTests(db?: Db) {
if (db) backlogContinuationTimes.delete(db);
lastLeftAloneLogAt = 0;
}
let lastLeftAloneLogAt = 0;
const LEFT_ALONE_LOG_INTERVAL_MS = 10 * 60_000;
/** Held reconciliation records of a task's latest unfinished run (joined to its task). */
function heldExecutionWhere(options: { includeActive?: boolean } = {}) {
return and(
@@ -703,7 +756,7 @@ function heldExecutionWhere(options: { includeActive?: boolean } = {}) {
}
/** The earliest time a deferred check becomes due again, if any. */
export async function nextFreshContinuationCheckAt(db: Db): Promise<number | null> {
export async function nextFreshContinuationCheckAt(db: Db, now = new Date()): Promise<number | null> {
// The same records the held-run query and the disposition pass would select;
// a deadline on a record nothing would act on never schedules a scan.
// That includes active holds that backed off: the disposition pass rechecks
@@ -712,7 +765,10 @@ export async function nextFreshContinuationCheckAt(db: Db): Promise<number | nul
.from(issueRecoveryActions).innerJoin(issues, and(eq(issues.id, issueRecoveryActions.sourceIssueId),
eq(issues.companyId, issueRecoveryActions.companyId)))
.where(and(heldExecutionWhere({ includeActive: true }),
sql`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}' is not null`));
sql`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}' is not null`,
// A record the backlog pass leaves alone schedules nothing; active holds
// stay with the disposition pass, which rechecks them when due.
or(liveDispositionHoldSql(), recentInterruptionSourceSql(now))));
return held?.at ? Number(held.at) : null;
}
@@ -721,13 +777,26 @@ export async function nextFreshContinuationCheckAt(db: Db): Promise<number | nul
* releases, so tasks already waiting continue after deploy without a Retry.
* Runs on the execution-disposition worker. Bounded per pass.
*/
export async function continueHeldExecutions(db: Db, now = new Date(), limit = 25) {
const candidates = await heldExecutionsAwaitingFreshContinuation(db, now, limit);
export async function continueHeldExecutions(db: Db, now = new Date(), limit = BACKLOG_CONTINUATIONS_PER_PASS) {
if (now.getTime() - lastLeftAloneLogAt >= LEFT_ALONE_LOG_INTERVAL_MS) {
lastLeftAloneLogAt = now.getTime();
const leftAlone = await countHeldExecutionsLeftAlone(db, now).catch(() => null);
if (leftAlone) logger.info({ leftAlone, maxAgeHours: BACKLOG_CONTINUATION_MAX_AGE_MS / 3_600_000 },
"Held tasks left for a person: older than the backlog window or not an interruption");
}
// Per-server pacing: a backlog saved by older releases must not start a herd of runs.
let times = backlogContinuationTimes.get(db);
if (!times) backlogContinuationTimes.set(db, times = []);
while (times.length && times[0]! <= now.getTime() - 60_000) times.shift();
const room = Math.min(limit, BACKLOG_CONTINUATIONS_PER_MINUTE - times.length);
if (room <= 0) return { checked: 0, continued: 0 };
const candidates = await heldExecutionsAwaitingFreshContinuation(db, now, room);
let continued = 0;
const seen = new Set<string>();
for (const candidate of candidates) {
if (!candidate.runId || seen.has(candidate.runId)) continue;
seen.add(candidate.runId);
times.push(now.getTime());
try {
const result = await ensureFreshContinuation(db, { companyId: candidate.companyId, runId: candidate.runId,
trigger: "held_recovery", now });