Files
PaperClipAI/server
DottaandClaude Opus 5.5 45531b719e perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Stranded-work recovery checks tasks that are assigned to an agent
but have no live run. It runs as a scoped-deadline queue that a change
notification or a stored deadline wakes.
> - For a failed legacy run, the check calls
`terminalizeLegacyExecution` with the run's current status. When the
run's recovery decision is already recorded (for example "continue in a
fresh run from saved work"), that call changes nothing. But it still
updated the run row and notified about 18 delivery queues, including the
run's own recovery graph.
> - The graph notification re-checked the same task, which re-asserted
the same run again. The loop ran about once a second per waiting task,
forever, with zero runs active. On a hosted instance this shows up as
constant idle transactions.
> - This pull request makes a re-assertion that changes nothing return
before it writes or notifies.
> - The benefit is that an idle instance does no stranded-recovery work
for tasks that wait on a recorded decision. A real change (another run
fails, a comment, a status change) still notifies and recovers at once.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, the server runs many transactions per second. A
CPU profile groups most samples under `reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`). A local fixture on real
PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent,
whose latest legacy run failed and whose recovery decision is recorded.

Each check reaches the branch that terminalizes a failed legacy run that
needs reconciliation (`service.ts`, the
`legacyExecutionNeedsReconciliationWithEvidence` branch, and the same
call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution`
then:
- updates the run with the same status (only `updated_at` changes);
- calls `notifyExecutionRunWork`, `notifyInstructionCollection` and
`notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the
task and about 18 queues;
- finds the recorded decision and returns.

The graph hint re-checks the same task after the queue's 1-second pass
interval, so the loop never ends.

**Expected behavior**

A check that finds nothing to change stores nothing and wakes nothing.
Only a real change re-checks the task.

**Steps to reproduce**

1. Create a task in `in_progress` assigned to an agent, with a failed
legacy run (`adapter_failed`).
2. Start the issue-state recovery work. The boot pass records a
fresh-continuation decision for the run.
3. Leave the instance idle. On master: 591,045 statements in 30 minutes
for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update
"heartbeat_runs"` per task per second.

**Paperclip version or commit**

master at 7db92ea319.

## What Changed

- `server/src/services/legacy-execution-recovery.ts`:
- `terminalizeLegacyExecution` returns the unchanged run, without a
write or a notification, when all of these are true: the call re-asserts
the current status with no patch; the run has no workspace restore
failure or required workspace recovery; the task's execution and
checkout locks do not point at the run; and either the run needs no
reconciliation (`reconcileIfNeeded`), no assignee or reviewer owns a
decision, or the decision is already recorded.
- Why the short-circuit is safe: stranded-work and quota checks
re-assert the run's unchanged terminal status each time they revisit its
task. Once the decision is recorded, that re-assertion changes nothing.
If it wrote or notified anyway, the notifications would re-check the
same task, which would re-assert the run again, about once a second for
as long as the task waits.
- `findRecordedLegacyDecision` returns the resolved decision already
recorded for the run (a reconciliation for this run, an explicit user
continuation with a real successor, or a no-replay hold).
`decisionOwner` says whether the task's assignee or current reviewer
owns a decision about the run.
- The recorded-decision query moves into `findRecordedLegacyDecision`,
and the owner condition into `decisionOwner`. The full path uses both
helpers, so its behavior does not change.
- `server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts`:
two new tests on real PostgreSQL. They use the real coordinator, the
real issue-state recovery work and the production 5-minute backstop on a
fake clock.

## Verification

- `stays quiet for half an hour once each failed run has its decision`:
10 waiting tasks, zero runs. The test first checks that each failed run
has a recorded decision, then counts statements over 30 idle minutes.
  - master: 591,045 statements (red).
- this branch: 0 statements (green). No run is re-written or started,
and no task moves.
- `still recovers promptly when another run of a waiting task fails`:
after 10 idle minutes, a new legacy run of one task fails and notifies
like a finished run.
- this branch: within 5 seconds, recovery creates the run's
`legacy-execution:<run>` hold. Within the next 6 minutes, the
fresh-continuation fallback resolves that hold with
`executionReconciliation.runId` set to the new run. Over the next 30
minutes there is no `update "heartbeat_runs"` and fewer than 5,000
statements (the assigned-work invariant keeps its own spaced re-checks).
- master: 18,000 no-op `update "heartbeat_runs"` statements in the same
window (red).
- Neighbouring suites pass locally: `legacy-workspace-restore-recovery`,
`legacy-workspace-restore-lock-contention`,
`legacy-continuation-authority`, `heartbeat-process-recovery`,
`issue-recovery-assignments`, `recovery-deadline-ordinary-bootstrap`,
`issue-recovery-deadlines` (600 of 601 in one parallel run; `resolves a
run's task graph through index lookups` failed once under machine load
on a planner row count, and passes alone; it does not use the changed
code).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The short-circuit applies only when the full path would also
write nothing except `updated_at`. Every other case (status change,
patch, workspace restore evidence, an owned lock, no recorded decision)
takes the unchanged full path.
- A re-assertion no longer re-sends notifications. Queues that relied on
that repeated hint as a side effect lose it. The first terminalization
of a run still sends all notifications, and run, comment and status
writes still send their own.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21616, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:30:03 -06:00
..