Commit Graph
6 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:29:22 -06:00
DottaandPaperclip e3fb4bc73a fix(native): recover saved listener startup failures (#19479)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native recovery must preserve saved work and respect current task
authority.
> - A closed local API listener can stop startup before a provider
starts.
> - #19462 gives new failures a bounded bootstrap retry.
> - Older saved failures still have a generic terminal code and a
recovery hold.
> - This pull request verifies the old producer outcome and uses the
existing safe replacement path.
> - Eligible tasks continue once. Unknown provider outcomes and user
holds remain blocked.

## Linked Issues or Issue Description

Refs: #19462

**What happened?**

A restart can leave an old listener startup failure in
`terminal_failure` with `native_session_interrupted`. Startup recovery
does not claim it after #19462. A completed file recovery can also
retain its no-replay hold.

**Expected behavior**

Continue automatically when durable records prove that no provider
started, the source stopped, and the current files were saved. Keep the
existing attempt budget and normal admission checks.

**Steps to reproduce**

1. Close the actual API listener after a fresh native coordinator claim.
2. Persist the old executor's generic failure. Save the workspace and
confirm source termination.
3. Restart recovery. The baseline schedules no successor. The repair
schedules one successor after all receipts pass.

**Paperclip version or commit**

Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`.

**Deployment mode**

Built from source. Local macOS arm64, Node 26.4.0, real embedded
PostgreSQL, independent filesystem execution. No external AI service is
used by the controlled acceptance.

## What Changed

- Recognize the exact saved local-listener guard through one proof
predicate shared by discovery and the locked transaction.
- Require a fresh first attempt, matching execution bindings, no prior
provider lineage, no provider events or tool calls, no checkpoint or
admission receipt, and no cancellation.
- Reuse current snapshot, exact stop, source block, attempt budget, and
successor idempotency checks.
- Wait for completed source file-recovery receipts. Retire only that
same source's verified preservation hold in the successor transaction.
- Bound discovery to 100 candidates per pass, keep completed history
quiet, and wake through existing transactional producer notifications.
- Retain retries after transient database and final ownership-proof
errors; advance the bounded cursor and force a rescan so a failing page
cannot hide later work.
- Let only the replacement duty discharge a proved-empty notification
during drain through its existing read-only readiness check, once per
notification generation. Keep real work and concurrent changes
outstanding without running replacement during drain.
- Wake replacement at canonical local-stop proof writes and the existing
dispatch eligibility boundary; these hints grant no new execution
authority.
- Add database regressions, scheduler quietness checks, real controller
handoff coverage, and an opt-in controlled Runner continuation.

## Verification

Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`.

Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest
5.0.3, frozen-lockfile dependencies. Tests execute freshly;
`--fsModuleCache` reuses only content-validated module transforms.

- Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18
seconds), including clean per-suite database disposal. All 62 assertions
also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook
timed out; that invocation is not counted as a pass. Docker's disk was
full, so the disposable PostgreSQL 17 data used a size-limited RAM
mount, with no host mounts. The owned container was removed afterward.
- Actual stopped-process producer suite: 31/31 PASS. A real exited child
produces the stop receipt, and its replacement notification settles only
after the outer transaction commits.
- Workspace and separate-process drain regressions: 33/33 PASS (75.90
seconds).
- Shared coordinator and execution-control lifecycle checks: 39/39 PASS
(16.43 seconds). Pending work, concurrent changes, cold startup,
shutdown, and passive drain boundaries remain covered.
- Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds),
using the unchanged test and timeouts. Its matching local RED reproduced
CI's 462-query, zero-pending-query, one-idle-hold timeout after an
admitted write committed during drain (69.46 seconds).
- Actual historical controller handoff: PASS (48.40 seconds). A separate
controller loads the original executor from `2f35eb1080`, writes its
generic listener failure, and exits. Recovery at the final head admits
exactly one successor. The real Rust Runner and controlled Codex process
complete governed `paperclip_finish`; attempt two commits and the saved
file remains intact. Only the child loads the old artifact, and both
isolated artifacts are restored afterward. Rust source subtree
`77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No
external AI service is used.
- Seven corrected existing fixtures pass: transactional stop-proof calls
(5 cases) and notification inventory (2 cases). Their earlier CI
failures exposed fixture calls outside the documented transaction and an
outdated expected queue set.
- Server typecheck: PASS.

Red/green proofs also cover the original saved failure on baseline
`0252421d5c` (zero successors instead of one), held-history quietness,
bounded forward scanning through 100 parked rows and concurrent wakes,
real snapshot/cleanup producer notifications, actual issue-row lock
timeout, and task/history eligibility notifications. An obsolete
prior-run question can be skipped to request another proof check; actual
skip while the task is paused or the source owns provider tool activity
schedules no successor.

The failed-page regression starts with 25 persistent verifier errors and
a healthy candidate 26. Baseline RED repeats the first page. Repair
GREEN advances the healthy candidate, revisits the original failures
after their fault clears, creates all 26 successors exactly once, and
becomes quiet.

Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21
UTC). Final-head CI is GREEN: 56 successful checks, three intentional
skips, one neutral result, and no pending or failing checks. [Attempt
two and
results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2)
include passing build, typecheck, Rust, native server, package, browser
E2E, serialized server, general server/chat tests, and final
verification. Five jobs in the first attempt were interrupted together
at 17:13:12 UTC; the native-server log records an explicit runner
shutdown signal. The single infrastructure retry reran only those five
jobs and dependent verification, preserving the 49 successful checks on
the unchanged head. No timeout was weakened, and setup failures or
interrupted invocations are not counted as passes.

Focused commands from the repository root:

```sh
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot
node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false
```

The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1`
and uses an original-executor child artifact with a `try/finally`
restore. Normal regressions require no Rust binary.

## Risks

- Compatibility is deliberately limited to positively proved fresh
remote startup failures. Other old ambiguous failures remain held.
- Existing provider work, a changed task block, incomplete preservation,
or revoked authority must prevent execution. Normal retry admission
remains in control.
- This PR does not deploy the change or mutate a hosted task. Hosted
recovery still requires the repaired revision and all current proof
gates.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact backend model identifier and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:49:28 -06:00
DottaandPaperclip 84e9c8f364 fix(retry): wait for admitted source cleanup before dispatch (#18898)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected-task Retry keeps the original request and its current
authority.
> - One durable Retry can already own a queued successor.
> - The old native environment can still have pending cleanup before
that successor starts.
> - The task execution lock points to the successor, so the old cleanup
can escape the queue claim check.
> - Dispatch then refuses the source state and fails the new run before
provider work.
> - This pull request keeps that same run queued and wakes dispatch when
cleanup changes.

## Linked Issues or Issue Description

Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs
cover preservation binding, interrupted preparation, old native
ownership, manual native holds or other setup guards. This change covers
an already admitted connected-task Retry whose native predecessor still
owes cleanup. It uses the shared cleanup predicate so the manual-hold
repair remains with its existing owner. Search found no duplicate
queued-source cleanup repair.

**What happened?**

A board Retry saved its exact chat request and admitted one queued
successor. The source lease was still pending cleanup when that
successor reached dispatch. A retained stop receipt did not settle the
full cleanup contract. The queue checked the task's current execution
owner, which was already the successor. The native authorization guard
then rejected the predecessor state and failed the new run before
provider dispatch.

**Expected behavior**

Keep the same canonical Retry and queued run while the exact source
cleanup is pending. After authoritative settlement, submit that run
through normal dispatch once. Do not infer safe execution from a
stop-and-retain receipt alone.

**Steps to reproduce**

1. Retry an authorized connected request from a failed native run.
2. Admit one successor while capacity keeps it queued.
3. Leave or restore the source lease to pending cleanup with a matching
retained stop receipt.
4. Release capacity and resume the ordinary queue.
5. The old claim path starts the successor and the native source guard
refuses it.

**Paperclip version or commit**

Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`.

**Deployment mode**

The regression uses the real HTTP route, chat authorization, queue
claims, disposable PostgreSQL and the production work coordinator on
macOS arm64. A controlled adapter supplies the final result. No hosted
provider replay or deployment is claimed.

## What Changed

- Validate the exact canonical Retry, chat action, company, task, agent,
source and successor before testing source cleanup.
- Keep the successor queued under the existing issue claim lock while
the shared cleanup predicate remains pending.
- Wake only the scoped queued-dispatch worker after cleanup or exact
workspace-owner release commits. Add no timer or background scan.
- Verify pending cleanup at intake, later cleanup invalidation, one
final execution, idempotency, Stop, changed input, unknown stop
receipts, a different live controller and signed sandbox absence
followed by owner release.
- Document the queued-wait contract. No schema, endpoint, credential,
model or provider policy changes are required.
- Align five native preparation acceptance fixtures with the existing
workspace publication contract. Use the real reservation and publication
owner. Retain a raw-root refusal case with no provider launch and
preserved files.

## Verification

- Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`
includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared
test helper preserves #18944's canonical retained-directory default, the
real publisher for the five fresh-workspace cases, and the explicit
raw-root refusal. Independent remerge review found no dropped upstream
behavior, guard or assertion.
- The latest head passes the focused six cases, all 580 complete
workspace/continuation tests, all 38 selected HTTP Retry cases, fresh
full recursive Node 24.22.2 types and full build. No affected tests
skipped. An initial cached test-layer attempt read old budget
declarations; regenerating them resolved all seven errors without
test/mock or production-export changes.
- All current-head CI is green at
`e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314)
and [ci /
e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086).
Greptile and Paperclip Review both rate this exact head 5/5, with no
findings or open review threads.
- Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges
master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts
were adjacent queue imports and two HTTP Retry test matrices. Both
imports and both complete matrices remain unchanged. Independent review
found no relaxed guard or assertion.
- Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases,
194 queue/cleanup/finalizer cases, module boundaries, recursive types
and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`.
- That sync head's [server
check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761)
failed five native acceptance cases in CI and locally with
`remote_workspace_published_path_invalid`. The test fixture supplied a
raw directory instead of a fresh publisher-generated workspace. The
guard from #18932 remains unchanged. The same five cases passed earlier
Linux CI before that guard was present.
- The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`
obtains the saved authority, path and run reference from real
`reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original
cases and a raw-root no-launch/no-result/files-preserved control pass.
Their native result, file, catalog and single-execution assertions
remain intact. An initial setup attempt was refused for a noncanonical
allocation root; the fixture now resolves that owned root before
publication.
- At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`,
the generated server test-layer typecheck passed. All 580 tests in the
complete workspace/continuation family pass, with no skips. That
test-only commit left production files byte-identical to the sync head.
- Previous published-head CI was green at
`dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci /
verify` and `ci / e2e`. The [Linux Node 24.22.2 server
shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638)
executed all five original acceptance cases and the new raw-root refusal
case successfully. Its whole workspace recovery file passed all 245
tests.
- Greptile reviewed previous published head
`dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or
open review threads. No unchanged-job rerun was used for the reproduced
sync-head failure.
- Causal red: the real HTTP regression started and failed the successor
at the native source-state guard while cleanup remained pending. A
second red proved that signed sandbox absence followed by copyback-owner
release lacked a queued-dispatch wake.
- Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner
release notification control and seven existing retry lineage controls
pass. The real coordinator resumes from committed cleanup or
workspace-owner release, without a manual queue restart. One controlled
adapter execution produces the result after settlement.
- Earlier branch verification passed full repository `pnpm -r
typecheck`, `pnpm build`, module boundaries and `git diff --check`.
- The full stale-queue file control passed all 64 cases with unchanged
observer/hook limits. The initial broader affected run had 192 passes
and one non-assignee native-owner case miss its existing five-second
adapter observer while already running with no error. The isolated case
passed; the unchanged base passed all 64 cases but hit its existing
30-second teardown timeout. Those original results are retained. The new
queue hint has no dispatch consumer in that fixture.
- Earlier-head CI was green: 54 success, three skipped and two neutral
checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was
cancelled by an external runner shutdown; its single unchanged-job rerun
passed. No source was changed to obtain that result.
- Greptile and Paperclip Review both rated earlier head
`6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or
unresolved threads. These results do not qualify the master-sync head.
- At initial verification, a complete local `pnpm test:run` was not
repeated while the related setup repair's full local run was active. The
full earlier-head CI suites passed. A full local suite has not been
repeated for the mechanical master sync. Local verification uses the
affected HTTP, producer, lineage and queue suites. No full local-suite
pass is claimed.
- The fresh clone's missing plugin SDK links were repaired with the
repository's own postinstall helper before verification. The embedded
PostgreSQL package's own symlink helper was also required.
- No staging task was submitted again with this repair, and no
deployment is claimed.

## Risks

- A pending source cleanup can keep the queued run waiting. Its cleanup
owner still owns reconciliation and its existing limits.
- Stop, changed source, unknown termination and live ownership retain
their existing authority checks. A cleanup notification grants no
execution authority.
- A new cleanup change after claim still faces the existing final
dispatch guard.
- Cleanup notifications can change dispatch timing in a running
controller. The real coordinator cases verify commit-bound wakeups; the
broader current queue-file control passed all 64 cases. The earlier
loaded-run and baseline teardown failures are retained in verification.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, tool use,
PostgreSQL and browser verification. This session does not expose a more
specific deployment model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 08:33:39 -06:00
DottaandPaperclip 9ca9af544a Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at 6a4c193906.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 16:00:35 -05:00
DottaandPaperclip 42fd568cf8 Drive issue recovery from committed changes and exact deadlines (#18402)
Reviewed and verified at 11b49125db.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 09:41:27 -05:00
DottaandPaperclip baf651c8d0 perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at f1c48c924d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 09:07:52 -05:00