Files
PaperClipAI/server
DottaandClaude Opus 5.5 dcba9628c7 perf(recovery): keep a finished task's settled attention settled and halve graph hint fan-out (#21676)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Issue recovery records "recovery actions" on tasks. A sweep resolves
the active action of a task that is already done or cancelled.
> - On an idle hosted instance, a debug-log run (from #21640) named a
ping-pong. A native worker re-surfaces a board attention for 13 finished
tasks. Recovery resolves it as `source_terminal`. The unique "one active
action" index then lets the worker insert it again, about every 26
seconds per task. That built 45,008 rows since 2026-10-21. Each insert
and each resolve also signals run dispatch and the task's recovery
graph, which wakes both sides again.
> - Separately, every task-graph recovery hint is published twice per
task queue: once as the graph hint and once as a `{companyId, backstop:
true}` twin. That is half of all idle signals, for a backstop that is
time-gated anyway.
> - This pull request breaks the ping-pong from the recovery side and
removes the backstop twin.
> - The benefit is that a finished task's settled attention stays
settled and wakes nothing. Each graph hint costs half the signals.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, recovery and native workers kept re-creating and
re-resolving the same board attention on 13 done or cancelled tasks. The
rows are `active_run_watchdog`, owner `board`, cause
`native_restart_recovery_blocked`. The loop added about one row per task
every ~26 s and about 2.6 work-signal commits per second.

**Expected behavior**

Once recovery settles a finished task's attention, the same attention
identity is not surfaced again, and settling it wakes nothing that a
finished task cannot use.

**Steps to reproduce**

1. Create a cancelled task.
2. Surface the native "controller process space unverified" attention
for it (same cause and fingerprint each time), then run
`reconcileActiveRecoveryActions`.
3. Repeat step 2. On master each cycle inserts a new active row and
resolves it, and each resolve signals the task queues and run dispatch.

**Paperclip version or commit**

master at 36101ebcb9.

## What Changed

- `server/src/services/issue-recovery-actions.ts`:
- **No-match skip.** `resolveActiveForIssue` first reads whether any
action matches. When none does, it returns `null` with no transaction
and no signal. Before, `writeExecutionControl` always published
run-dispatch, graph-recovery and execution hints first.
- **Settled-attention guard.** `upsertSourceScoped` returns the existing
resolved action instead of inserting when ALL of these hold:
    - the task is done or cancelled;
    - it has no active action;
- an action with the same cause and fingerprint was already resolved
with `source_terminal`.
The guard writes no row and sends no signal, and the caller gets that
resolved action. It covers only the exact identity that recovery itself
settled because the task finished. A new fingerprint (renewed controller
authority), a different cause, or a reopened task still creates an
action.
- **Quiet terminal resolve.** A new `sourceTerminal` resolve mode. It
updates the row only while the task is still done or cancelled. It still
notifies native recovery notices, unsafe-workspace export recovery, and
a preserved native finalization waiter. It does NOT signal run dispatch,
the task's recovery graph, or the execution disposition/continuation
queues: a finished task has no run to dispatch and no assignment to
recover.
- `server/src/services/recovery/service.ts`: the `source_terminal`
branch of `reconcileActiveRecoveryActions` uses `sourceTerminal: true`.
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery` publishes one graph hint per task queue and
no longer adds a `{companyId, backstop: true}` hint per queue. A new
`issueRecoveryBackstopHint(companyId)` builds that hint string.
- `server/src/services/recovery/work.ts`: when a task queue loads a
graph scope, it arms its own backstop in process
(`worker.invalidate(issueRecoveryBackstopHint(companyId))`). The #21552
guarantee stays: any graph activity arms the company backstop, which
runs at most once per backstop interval after the last full pass. It no
longer costs one signal per queue per hint.
- `server/src/__tests__/issue-recovery-actions.test.ts`: the fake-DB
unit tests accept the guard's extra read.
- `server/src/__tests__/issue-recovery-signal-fanout.test.ts` (new, real
PostgreSQL), six tests:
  - one graph hint per task queue and no backstop hint;
  - a resolve with no matching action returns null and signals nothing;
- an escalated action on a finished task is still resolved and still
signals;
- the ping-pong cycle: surface then reconcile, five times, leaves one
row and no stranded or active-recovery signals;
- settling a finished task's attention sends no task-queue, run-dispatch
or active-recovery signals;
- a renewed authority and a reopened task still get a new attention,
which stays active on a reopened task.

## Verification

- New test file:
  - master: 4 of 6 fail (red).
    - 8 task-queue hints per graph hint instead of 4.
    - A no-match resolve publishes 18 settled signals.
- Cycle 2 inserts a new active row; master builds a new row every cycle.
- Settling a finished task's attention signals the task queues and run
dispatch.
  - This branch: 6 of 6 pass (green).
- Regression on real PostgreSQL:
- heartbeat-native-restart-suspension, native-recovery-notice-work,
issue-recovery-deadlines (including the #21552 backstop tests),
assigned-work-invariant, execution-control-work, issue-recovery-actions,
issue-recovery-work-lifecycle, issue-recovery-assignments,
execution-recovery-resolution, automatic-run-brake-recovery-passes: all
pass (381 tests).
- Four unit tests in issue-recovery-actions use a hand-written fake DB.
They needed `innerJoin` on the fake query and an expected count of two
selects, because the guard adds one read.
  - recovery-deadline-acceptance: passes.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- The guard returns a resolved action to a caller that asked to surface
one. I checked the callers: the one that loops
(`surfaceNativeControllerNamespaceAttention`) ignores the return value.
The others only reach the guard for a finished task whose identical
attention recovery already settled, where re-surfacing it would be
resolved again within one sweep.
- The terminal resolve no longer wakes run dispatch or the task's graph.
Completing or cancelling a task already sends those signals itself.
- This does not stop the native worker from trying to surface attention
for finished tasks. That fix belongs in the native owner-release code,
which another change owns.
- The ~45k historical resolved rows on the hosted instance stay as they
are. A cleanup would need a migration or an operator script that deletes
resolved `native_restart_recovery_blocked` rows whose source is done or
cancelled, keeping the newest per (task, fingerprint) so the guard still
matches. That is not part of this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21621, #21640.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:38 -06:00
..