perf(recovery): wait for an assignee that cannot run instead of retrying its continuation every pass (#21687)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops and recovery decides to continue the task in a
fresh run, it records a pending continuation. The execution-continuation
worker delivers it with `heartbeat.wakeup`.
> - When the task's assignee cannot take work (paused, terminated,
waiting for hire approval, or an invalid reporting chain), the wake
refuses. The worker recorded only an "attempted at" time and tried again
on the next 15-second pass, forever.
> - Each refused wake also saved a new skipped wake row. An idle server
with one such task did about 13,000 statements and 390 wake attempts
every 2 hours.
> - This pull request makes delivery check the assignee with the same
rule the wake uses. When the assignee cannot run, the continuation waits
on a stored retry time and is re-admitted at once when the assignee can
run again.
> - The benefit is that an idle server does almost no work for these
tasks, and a resumed agent still gets its continuation within seconds.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

`deliverReconciledExecutions`
(`server/src/services/execution-recovery-resolution.ts`) offers each
pending continuation (`continuationDelivery: "pending"`) to
`heartbeat.wakeup`. When the assignee is not invokable, the wake throws
"Agent is not invokable in its current state" and saves a skipped
`agent_wakeup_requests` row. The worker catches the refusal, records
`continuationDeliveryAttemptedAt`, and the continuation schedule
(`executionControlSchedule.continuation`) asks for another pass in 15
seconds because a pending row exists. Nothing ever changes, so the loop
never ends.

Measured on real PostgreSQL with the real coordinator and the real wake,
on a fake clock, for one pending continuation over 2 idle hours:

| Assignee status | Wake attempts (master) | Statements (master) |
| --- | --- | --- |
| paused | 380 | about 12,980 |
| terminated | 376 | about 12,820 |
| pending_approval | 395 | about 13,000 |

An agent in `error` is not affected: it is invokable, and the wake
delivers the continuation on the first pass. Only the statuses and
conditions that make an agent not invokable cause the loop.

**Expected behavior**

A continuation that cannot be delivered because its assignee cannot run
does not try again every pass. It waits for a real event (the agent can
run again) or a spaced backstop time.

**Steps to reproduce**

1. Create a task in `in_progress` assigned to an agent, with a failed
run.
2. Record a pending continuation for that run, as the fresh-continuation
fallback does.
3. Pause, terminate or un-approve the agent.
4. Run the execution-continuation worker. It calls the wake about every
18 seconds and saves a skipped wake row each time.

**Paperclip version or commit**

master at dcba9628c7.

## What Changed

- `server/src/services/execution-recovery-resolution.ts`: before the
wake, delivery calls `deferForUnwakeableAssignee`. When the assignee
cannot run, delivery skips the wake. The pending-page query skips
continuations whose stored retry time is in the future. The file stays
under 1,000 lines (997).
- `server/src/services/recovery/continuation-delivery-attempt.ts`:
`deferForUnwakeableAssignee` reads the assignee `FOR SHARE` and
evaluates it with `evaluateAgentInvokabilityFromDb`, the rule
`heartbeat.wakeup` uses (including the setup exception). In the same
transaction it stores `continuationDeliveryRetryAt` and a wait count on
the recovery action. The wait is 1 minute and doubles per refusal up to
1 hour. The row lock orders the wait against a concurrent status change,
so a resume cannot be missed. `continuationDeliveryDue` is the shared
"due now" condition. `readmitAgentContinuations` notifies the
continuation worker, then clears the stored time for one agent's waiting
continuations.
- `server/src/services/execution-control-work.ts`: the continuation
schedule asks for a pass in 15 seconds only when a continuation is due.
Otherwise it returns the earliest stored retry time.
- `server/src/services/agent-lifecycle.ts`: when an agent's status
changes from not invokable to invokable (resume, hire approval), the
existing hook also calls `readmitAgentContinuations`.
-
`server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts`:
five new tests on real PostgreSQL.

What the user sees does not change. The continuation stays pending and
the task stays assigned. The stranded-assignment check and the
assigned-work invariant still own the visible outcome for a task whose
assignee cannot run (the existing "can't run right now" escalation, or
the "agent paused" wait). When the agent can run again, the continuation
is delivered as before.

## Verification

- `pnpm exec vitest run
src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts` (in
`server/`):
- `waits instead of retrying every pass while the assignee is paused /
terminated / pending_approval`: 2 idle hours after the first minute.
    - master: 380, 376 and 395 wake attempts (red).
- this branch: 0 wake attempts, 68 statements, no skipped wake rows, the
task stays `in_progress` and assigned (green).
- `delivers the continuation within seconds once the paused assignee is
resumed`: after 3 idle hours, the agent is resumed through the real
agent lifecycle. Within 20 seconds the worker calls the wake once and
the record says `continuationDelivery: "delivered"` (green). On master
this test fails because the wake was already called during the idle
hours.
- `does not store a wait when the assignee is resumed while delivery is
checking it`: a resume transaction holds the agent row while delivery
checks. With the row lock, delivery sees the resumed agent and stores no
wait (green). Without the lock, it stores a wait (red).
- Neighbouring suites pass locally:
`automatic-run-brake-recovery-passes`, `recovery-wake-herd`,
`execution-control-work`, `fresh-continuation-move-forward-offer`,
`fresh-continuation-retry-delivery`, `fresh-continuation-fallback`,
`issue-recovery-actions`, `resource-lifecycle-events` (345 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. Only a continuation whose assignee is not invokable by the
wake's own rule waits. Every other refusal keeps the current behavior.
- A repair that makes the agent invokable without a status change (for
example a fixed reporting chain) does not fire the re-admit hook. The
backstop picks it up within at most 1 hour.
- A task reassigned while its continuation waits is invalidated on the
next backstop attempt instead of the next 15-second pass. The new
assignee gets its own assignment wake, so nothing waits on that.
- No schema or API change. The new fields live in the existing
`evidence` JSON.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621, #21674, #21675, #21676, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 19:41:35 -06:00
1 parent c66530da63
commit f4e025a1cf
5 files changed
+215 -12

No files matched your search

@@ -10,6 +10,7 @@ import { hasPendingNativeReplacements, nextNativeOwnershipProofDeadlineAt } from
import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js";
import { hasPendingPauseResume } from "./issue-tree-resume.js";
import { hasPendingBudgetResume } from "./budget-resume-continuation.js";
import { continuationDeliveryDue } from "./recovery/continuation-delivery-attempt.js";
export function registerExecutionControlWork(
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
@@ -94,11 +95,16 @@ export function executionControlSchedule(db: Db) {
return at === null ? null : deadline([new Date(at)]);
},
async continuation() {
const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
const pending = and(
eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
)).limit(1);
return rows.length || await hasPendingPauseResume(db) || await hasPendingBudgetResume(db) ? retry() : null;
);
const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
.where(and(pending, continuationDeliveryDue(new Date()))).limit(1);
if (rows.length || await hasPendingPauseResume(db) || await hasPendingBudgetResume(db)) return retry();
const [waiting] = await db.select({ at: sql<string | null>`min((${issueRecoveryActions.evidence}->>'continuationDeliveryRetryAt')::timestamptz)` })
.from(issueRecoveryActions).where(pending);
return deadline([waiting?.at ? new Date(waiting.at) : null]);
},
async status() {
const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)