Commit Graph
102 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
DottaandClaude Opus 5.5 89f486cdab fix(heartbeat): run one dispatch pass per agent, back off refused claims, and enforce the run limit in the claim (#21433)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. Each trigger (a
run finishes, a wake arrives, a sweep runs) asks for one dispatch pass
of that agent
> - An agent could run more runs than its `maxConcurrentRuns`, dispatch
passes could stack up without a limit, and every pass offered the same
unclaimable runs again
> - On hosted staging (e23549cd9c), one agent with limit 20 had 20–35
runs `running` and 170–270 queued. The logs showed 375 "offered a free
slot but could not be claimed" lines and 117 "claim was rejected" lines
in 10 minutes
> - This pull request runs one pass per agent at a time, folds triggers
into one follow-up pass, backs off refused claims, enforces the limit in
the claim transaction, and reads only the columns the pass needs
> - The benefit is that an agent never runs past its limit (also across
two servers during a deploy), and a burst of triggers costs one or two
passes, not one pass per trigger

## Linked Issues or Issue Description

Refs #21391, #21242, #21234, #21277, #21324, #21349

**What happened?**

On hosted staging (Core e23549cd9c), one agent ("Codie",
`maxConcurrentRuns` 20) had 170–270 queued runs and 20–35 runs
`running`, so it was over its limit. The logs showed a refused claim
offered again on every pass. This pull request is about correctness and
load. It is not the cause of the staging out-of-memory crashes: a
separate leak (`sessionToolAuthorityEpochs` in
`native-session-executor.ts`, about 3 MB per finished native run)
explains those, and another change fixes it.

Live again at 17:02Z on staging (Core 04f12fd2be, which includes
#21391): the same agent had 52 runs `running` against
`maxConcurrentRuns` 40. All 52 started between 16:55Z and 17:02Z, right
after a deploy released the backlog, with 112 runs still queued. So
overlapping passes exceed the limit with claimable runs, not only with
unclaimable ones.

**Why did it happen?**

1. `withAgentStartLock` chained each trigger behind the previous one and
measured its 30 s stale limit from when the predecessor was chained, not
from when it started. When triggers arrived faster than one per pass,
every trigger after about 30 s of backlog started its own pass next to
the others.
2. Each pass counted free slots by itself, outside any lock that other
passes or another server see. Overlapping passes and two servers during
a deploy swap each claimed up to the limit.
3. Each pass read every queued run with `select *`, including the large
`contextSnapshot`, and offered every refused run again with no backoff.

A local reproduction of the staging shape by another session (250
unclaimable queued runs with 8 KB snapshots, one trigger per second, on
e23549cd9c and again on master e7e28ba241): passes that run at the same
time went 1 → 12 → 39 → 79 → 139 at t = 30/45/75/116/178 s, only 4
passes finished in 178 s, and post-GC heap grew by about 2.8 MB per
stacked pass (519 MB → 1,061 MB at 190 passes, mostly the parsed
`heartbeat_runs` rows and the dispatcher facts each pass holds). 242 s
after the triggers stopped, 216 passes were still outstanding.

**Expected behavior**

At most one dispatch pass per agent runs at a time. Triggers that arrive
during a pass cause one follow-up pass. The `running` count never goes
above `maxConcurrentRuns`. A refused run is offered again after a
bounded wait, and no queued run is stranded.

**Steps to reproduce**

1. Give an agent a limit of 4, 12 aged queued runs that the chat-control
gate refuses, and 12 claimable runs.
2. Start a pass. While its first claim waits, send 40 triggers, each
after more than 30 s of the old lock's clock.
3. Count the passes, the refused offers, and the highest `running`
count.

## What Changed

- `server/src/services/agent-start-lock.ts`: replaces the stale-bypass
lock with `dispatchForAgent`. One pass runs per agent. Triggers during a
pass merge into one follow-up pass that all their callers share. A
trigger from inside the agent's own pass does not wait on itself. A pass
has a 60 s deadline on a monotonic clock, measured from when the pass
started (not from when a waiter was chained). At the deadline the pass
stops claiming (an abort signal) and the follow-up may start. A trigger
that arrives after the deadline also ends the pass, so a stalled timer
cannot keep an agent blocked. Post-pass effects run after the agent is
free, as before.
- `server/src/services/heartbeat/run-slot-admission.ts` (new):
`assertAgentRunSlotFree` counts the agent's `running` runs and compares
them with `maxConcurrentRunsSql()` inside the agent-scoped advisory lock
that every claim already takes (`withExternalAdmissionGuard`). It throws
`AgentRunSlotsFull` when the agent is full. The count leaves out the run
that the claimed run replaces (`retryOfRunId` or `previousRunId`), so a
successor that starts while its predecessor finishes is not refused.
- `server/src/services/heartbeat/queue.ts`:
- Every claim path (ordinary, queued comment, native review through
`native-review-dispatch.ts`) checks the slot inside its claim
transaction. A full agent leaves the run queued, and the pass stops.
- The pass reads only the columns the dispatcher needs, with `issueId`,
`wakeReason` and `previousRunId` taken from `contextSnapshot` in SQL. It
reads the full row only for the run it is about to claim.
- A refused claim (declined or 4xx) of a run that has waited more than
an hour (the existing aging limit) is not offered again for 5 s, then
10, 20, 40, and at most 60 s. A person's run of that age waits 2 s. A
run queued for less than an hour is never backed off, so a successor or
retry that waits for its predecessor starts as soon as the predecessor
ends. A backoff also ends when the run's task changes (status,
`updatedAt` or execution run) or when a run of that task ends in this
process. A backed-off run does not take the oldest-run share. The
queued-run sweep (`resumeQueuedRuns`, `resumeQueuedRunsAfterDrain`,
which the queued-run redrive calls) offers refused runs again regardless
of the backoff.
- A pass makes at most 24 claim attempts. When it reaches that limit it
asks for one follow-up pass that skips every run the chain already
tried, so the follow-up reaches runs further down the line. The chain
ends when a pass does not reach the limit, so refused runs cannot cause
a continuous chain of passes. Only this automatic follow-up carries the
tried list. A wake, a completion or another fresh trigger that merges
into it resets the list, so a run whose blocker cleared during the chain
is offered again.
- Admission, suppression and cutoff are read once per pass, not once per
trigger.
- Seam for a skip before the claim: in `runQueuedDispatchPass`
(`claimQueuedRunsForAgent`), right after the skip line (`if
(triedRunIds.has(candidate.id) || (!retryRefused &&
claimBackedOff(candidate.id, new Date(), taskStamp))) continue;`),
before the claim-budget check, `claimAttempts += 1` and the full-row
read. A `continue` there spends no claim attempt, reads no row, does not
enter the claim backoff or the aged-decline note, and does not call
`dispatcher.claimed`, so the run keeps its place and its oldest-run
share. The dispatcher exposes `dispatcher.personOrigin(run)` (the
person-origin fact it already derives) and `dispatcher.personRun(run)`.
The candidate rows keep `id`, `companyId`, `agentId`, `status`,
`invocationSource`, `retryOfRunId`, `wakeupRequestId`,
`scheduledRetryAt`, `scheduledRetryAttempt`, `scheduledRetryReason`,
`createdAt`, `startedAt`, and
`contextSnapshot.issueId/wakeReason/previousRunId`.
- Each pass is registered with the execution drain
(`drainActiveRunExecutions`) until it ends or reaches its deadline.
- Tests: new `server/src/services/agent-start-lock.test.ts` and
`server/src/__tests__/queued-run-dispatch-stacking.test.ts`.
`server/src/__tests__/heartbeat-start-lock.test.ts` is removed: it
asserted the 30 s stale bypass that this pull request removes.
`workspace-recovery-preparation-dispatch.test.ts` holds and counts
passes through the new function.

The chat-control fallback lookup from #21391 (`issueId = X OR
nativeIssueId = X OR contextSnapshot->>'issueId' = X`) was measured on
embedded Postgres with 200,000 runs in one company. Postgres uses a
BitmapOr of three existing indexes
(`heartbeat_runs_company_issue_created_idx`,
`heartbeat_runs_company_native_issue_contract_id_uq`,
`heartbeat_runs_company_ctx_issue_created_idx`), not a sequential scan:
0.33–0.45 ms per call for a recent previous run, a 30-day-old previous
run, and no previous run. It is not changed.

## Verification

Red/green, `server/src/__tests__/queued-run-dispatch-stacking.test.ts`
(embedded Postgres, real heartbeat queue, limit 4, 12 refused aged runs,
12 claimable runs, 41 triggers):

| Measure | master | this pull request |
|---|---|---|
| Passes | 42 | 2 |
| Peak passes at the same time | 40 | 1 |
| Peak claims in flight | 9 | 1 |
| Refused-claim offers per trigger | 12 | 0.29 (12 in total) |
| Peak `running` (limit 4) | 8 | 4 |
| Passes outstanding after the burst | — | 0 |

- "waits a bounded time before offering a declined/rejected run again":
on master 10 triggers offer the refused run 10 times. With the fix they
offer it once, and once the backoff passes it is offered again. The
rejected case uses the 422 "Unable to resolve responsible user" path
from the staging reproduction.
- "starts a run that becomes ready during a chain of passes when a fresh
trigger arrives": red at 3d48c4bed2, green now.
- "stops chaining passes once every refused run was tried": 60 refused
runs and one trigger cause exactly 3 passes, then no more.
- "starts runnable work queued behind more refused person runs than one
pass tries": 30 refused person runs ahead of 3 runnable runs, with free
slots. The runnable runs start after one trigger, in at most 3 passes.
At 68e64a2de9 the same 24 person runs were offered on every pass and the
runnable runs never started.
- `server/src/services/agent-start-lock.test.ts`: one pass for a burst
of 50 triggers plus one follow-up, no second pass after 45 s, a hung
pass stopped at its deadline without overlap, no self-wait, settle after
release, and errors passed to the caller.
- `node server/scripts/typecheck-layers.mjs`: pass.
- Vitest (embedded Postgres): `heartbeat/queue.test.ts` (102),
`recovery-wake-herd.test.ts` (35),
`heartbeat-queued-run-claim-isolation.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`issue-reassignment-queued-run.test.ts`,
`queued-comment-consumption.test.ts`,
`issue-queued-comment-queue.test.ts`, `chat-run-milestone-work.test.ts`.

## Risks

- Refusals of runs queued for less than an hour are not backed off. A
backoff there delayed successors that wait for in-process predecessor
ownership, which can end without a task change (two CI tests failed with
it). Per trigger, such a run costs at most one claim attempt.
- A refused run that has waited more than an hour now waits up to 60 s
(a person's run 2 s) before an event-driven pass offers it again, unless
its task changes. The sweep and the redrive ignore the backoff.
- A successor may start while the run it replaces is still `running`.
The limit check leaves that one predecessor out of the count.
- Claims of an agent that is at its limit now fail inside the
transaction. A run that `executeRun` claims directly (for example a
native resume) stays queued when the agent is full, and the next pass
starts it.
- A pass that passes its 60 s deadline can still finish one in-flight
claim next to the follow-up pass. The limit check in the transaction
keeps the `running` count within the limit.
- No schema or migration change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:55:38 -06:00
DottaandClaude Opus 5.5 e7e28ba241 feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns`
tasks at once. Other runs wait in the agent's queue with status `queued`
> - On hosted staging, one agent had all 20 slots full and 295 queued
runs. Some tasks waited 3 to 7 hours and never started
> - The UI showed these tasks as live or working: a pulsing "Live" badge
in the task header and in task rows, a spinning "Working … for 4 hours"
in the task chat, and "N live" counts that included every queued run
> - People could not tell that work was waiting, why it waited, or what
to change
> - This pull request makes a deep queue an honest, visible state:
"Waiting for Codie · 20 slots max · 41 of 295 in line · Configure"
> - The benefit is that operators see the real state and the one setting
that changes it

## Linked Issues or Issue Description

Refs #21391, #21349, #21242

**What happened?**

A queued run that never started looked the same as a running run. The
task header, the task rows in lists and the inbox, the task chat and the
sidebar counts all showed queued runs as live or working.

**Expected behavior**

A run that waits for one of its agent's concurrency slots shows that it
waits. It names the agent, the agent's limit and the run's place in
line, and it links to where the limit is set. The place in line is the
order in which the server starts the agent's queued runs.

**Why does it matter?**

With a deep queue, the UI said "working" for hours while nothing ran.
Operators could not see the cause or the fix.

## What Changed

- Server: a new module
`server/src/services/heartbeat/queued-run-line.ts` now holds the facts
that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The
dispatch path in `queue.ts` and the new "place in line" read use this
one builder, so the line the UI shows is the dispatch order. The logic
moved without change, including the recovery-share fix from #21391.
- Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit,
running count, position, queued count) and the label "Waiting for
<agent>" with cause `agent_at_capacity` to a queued, never-started run
when its agent has no free slot. It reads one batch of agents and
running counts. Then it builds one line per full agent (at most 8 agents
per read), using only the three context keys that ordering needs. No new
polling.
- Server: the place in line uses `queuedRunDispatchLine`, an O(n log n)
replay of the dispatcher. The plain replay rescans the queue after each
step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000,
600 ms at 10,000, all on the event loop). Dispatch itself is unchanged:
it stops after the free slots. Each agent's line is memoized for 5 s and
shared by concurrent reads. In a preview worktree, the line applies the
same creation-time cutoff as dispatch.
- Server: the task list summary adds `runningDescendantCount`
(descendants whose run started) next to `liveDescendantCount`.
`liveDescendantCount` still decides whether a blocked parent is covered.
The "n live below" chip uses the new count.
- Shared: the new `ExecutionAgentQueue` type and the optional field
`ExecutionProjection.agentQueue`.
- Server: `/companies/:companyId/live-runs` lists running runs before
queued runs, so a deep queue cannot push running work out of the 50-row
limit.
- UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20
slots max · 41 of 295 in line · Configure", with no spinner and no
elapsed timer. Configure opens the agent's Advanced view, where the run
policy (`maxConcurrentRuns`) is edited. The task chat (both renderers),
the run ledger and the Storybook story use it.
- UI: the task header and the task rows (task list, inbox, legacy views)
show a quiet "Waiting" instead of the pulsing "Live" when every live run
of the task has not started. The full sentence shows on hover. "n live
below" counts only started sub-tasks. A waiting sub-task still covers
its blocked parent.
- UI: the sidebar "N live" counts (Dashboard, agents) count only started
runs.
- UI: the task chat pill now names other server holds on a queued run
(for example "Waiting for maintenance") instead of "Working".
- Docs: DESIGN.md records the waiting-for-slot hold.
- Storybook: `runs-agent-queue-wait` stories (full line, task chat, task
row, one slot, mobile), with play checks.

## Verification

- Red/green server test:
`server/src/services/execution-projection-agent-queue.test.ts` (embedded
Postgres). Before this change, the projection has no `agentQueue`. Now
it checks the agent name, limit, running count, positions 1 to 4 in
dispatch order (a person's retry first, then priority, not creation
order), no wait while a slot is free, and isolation from other
companies' runs. It also checks that the positions equal the line that
the dispatch builder produces.
- `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`,
`recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and
`agent-live-run-routes` in `server/`.
- Runtime cost (rule `runtime-dependency-boundaries`): the assumption is
that the line stays cheap for a deep queue. `queued-run-line.test.ts`
checks that `queuedRunDispatchLine` equals the dispatcher's
claim-every-run order over 400 random queues (ages around the aging
boundary, recovery and person runs, blocked runs, deferred share runs,
all share counters). It also orders 10,000 runs in under 250 ms (the
quadratic replay takes about 600 ms). The DB test projects a 300-run
queue end to end with correct positions 1 to 300. Remaining limit: the
line is up to 5 s stale.
- Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts
`runningDescendantCount: 0` for a parent whose only live descendant is
queued and never started.
- UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep
"Live", no "running below" count for waiting sub-tasks) and
`TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the
Configure link, other holds named). The suites for the run ledger,
sidebar, sidebar agents, task detail, task columns and task chat also
pass.
- The Storybook stories render in headless Chromium and their play
checks pass.
- After deploy: open a queued task on staging and confirm the waiting
state.

## Risks

- Cost on read: a projection read that includes a queued run of a full
agent now reads that agent's queued runs (only small columns) and the
dispatch facts. This is at most one line per full agent every 5 s, for
at most 8 agents per read. Reads with no queued, never-started runs do
no more work.
- The place in line is the server's prediction of dispatch order. A
claim that is refused at start time (a gate, a blocker) can let the next
run start first. The position updates on the next refresh.
- The process-local "declined aged claim" hint that the dispatcher uses
is not visible to other servers, so the projection does not use it. This
can change only the aged-oldest share step.
- Configure opens the agent's Advanced view (the YAML run policy). The
agent's Runtime tab hides the run-policy fields today.
- Low risk otherwise: no schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless browser) and subagents for the UI survey.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:24:20 -06:00
DottaandClaude Opus 5.5 5c5a2fcebe fix(dispatch): start the assigned-work sweep's continuations and count only system recovery for the recovery share (#21391)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. #21234, #21277
and #21349 order them: person work first, a recovery share, aging, and a
share for the longest-waiting run
> - On hosted staging, one agent had about 250 queued runs. For 90
minutes after #21349 deployed, every run that started was a retry or
continuation with `retryOfRunId`. The oldest queued runs, from 15 hours
earlier, never started
> - The dispatch order was not bypassed. The oldest-run share offered
those runs, but the claim refused them every time, without a log at info
level or a visible state
> - The refusal came from the chat-control recovery gate. It needs
`retryOfRunId` to trace an automatic continuation. The assigned-work
invariant sweep queues continuations that name their source only in
context, so the gate returned "unresolved", and an unresolved claim
stays queued forever
> - This pull request lets the gate check such a continuation against
the task's previous run, and makes the recovery share count only system
recovery starts
> - The benefit is that the sweep's continuations start (it already told
people "you don't need to press Retry"), and system recovery gets its
share even when a person's recovery restarts keep starting

## Linked Issues or Issue Description

Refs #21234, #21277, #21324, #21349, #21242

**What happened?**

On hosted staging (build e23549cd9c), one agent with `maxConcurrentRuns`
20 had about 250 queued runs. From 12:06Z to 13:30Z, all 30 runs that
started had `retryOfRunId` set. The oldest queued runs were automatic
continuations (`issue_continuation_needed`) created from 22:53Z the day
before. Each task's activity shows "Continued automatically after an
interruption ... you don't need to press Retry" from the assigned-work
invariant sweep, at the time the run was queued. None of these runs
started. Assignments created at 07:51Z and later did start, in creation
order. Those starts came through the oldest-run share after the share
passed over the refused continuations.

**Why did it happen?**

1. `assignedWorkSweep` queues the continuation with `previousRunId` in
its context and no `retryOfRunId`. Its wake is a system wake
(`requestedByActorId` null) with reason `issue_continuation_needed`.
2. At claim, `withChatControlRecoveryGate` calls
`readChatControlRecoveryStop`. For an automatic continuation without
`retryOfRunId`, it returned `unresolved`. At the claim stage,
`unresolved` returns null: the run stays queued and nothing changes. So
every dispatch pass offered these runs and every claim refused them.
Retries and continuations with `retryOfRunId` pass the gate, so they
were the only recovery work that started.
3. Separately, the recovery share (#21234) read the agent's recent
starts and counted every start with a recovery wake reason as a recovery
start. A person's `issue_recovery_action_restored` restart ranks as
person work in the dispatcher, but it still reset the share. So the
recovery share almost never fired while those restarts kept starting.

**Free slots.** The "14 of 20 running" figure came from a capped run
list. A full read at 14:03Z showed 22 runs `running` on that agent, so
there were no free slots. The agent was over its limit, not under it.
Overlapping dispatch passes after the 30 s stale agent start lock are a
likely cause (each pass re-tried every refused run, so passes were
slow). This pull request removes the refused runs that made passes slow.
#21242 changes the claim-then-execute timing.

**Expected behavior**

An automatic continuation that the system queued starts in its turn. A
chat close of the task's previous run still stops it. The recovery share
goes to system recovery after three starts that were not system
recovery.

**Steps to reproduce**

1. Give an agent one slot. Interrupt a run on an in-progress task so
that the assigned-work invariant finds a violation.
2. Run the assigned-work check. It queues an `issue_continuation_needed`
continuation.
3. Queue fresher recovery runs on other tasks and dispatch.
4. See that the continuation stays queued on every pass.

## What Changed

- `server/src/services/chat-control-recovery-stop.ts`: for an automatic
continuation that has no `retryOfRunId`, the gate now walks to the
task's latest earlier run (by `issueId`, `nativeIssueId` or the context
task id, in the same company) and checks that run. If the task has no
earlier run, the result is `clear`. A continuation with `retryOfRunId`
is unchanged. A chat close of the previous run still stops the
continuation.
- `server/src/services/heartbeat/queue.ts`: the recovery-share counter
reads which recent recovery-reason starts were person-originated, with
the same source-chain query that the dispatcher already uses
(`listPersonOriginSourceRunIds`, at most 4 ids). Only system recovery
starts reset the share. This matches the in-pass rule
(`isSystemRecoveryRun`).

## Verification

- Red/green, embedded Postgres, real heartbeat queue
(`server/src/__tests__/recovery-wake-herd.test.ts`):
- "starts the assigned-work sweep's continuation of a task whose run was
interrupted, ahead of fresher retries": on master the continuation stays
queued for 60 s of dispatch passes; with the fix it starts.
- "gives system recovery its share when the agent's recent starts were a
person's recovery restarts": on master a person run starts first; with
the fix the system recovery run starts.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: "checks an
automatic continuation that names no retry source against the task's
previous run" proves that a chat close of the previous run still stops
the continuation (`stopped`) and that it is `clear` when the close is
not published.
- `node server/scripts/typecheck-layers.mjs`: passes.
- Related suites: `recovery-wake-herd`, `queued-run-dispatch-order`,
`assigned-work-invariant`, `run-authority-read-projections`,
`chat-run-milestone-work`, `legacy-preparation-retry-hold`,
`explicit-native-continuation`, `saved-work-retry-eligibility`,
`native-failed-turn-continuation.integration`,
`heartbeat-process-recovery` pass, except:
- `heartbeat-process-recovery` "schedules bounded retries for failed
accepted interaction continuation wakes" is flaky on master too (1 of 3
runs failed on master source, same line).
- `recovery-wake-herd` "delivers the continuation after a restart..."
failed once with a hook timeout under parallel load and passes alone.
- `runtime-selection.test.ts` fails locally in a nested worktree because
a bundled skill file path does not exist there (environment, not this
change).
- After deploy: the queued `issue_continuation_needed` runs on the
affected staging agent should start within the oldest-run share, and no
run should stay queued longer than the aging bound plus a few starts.

## Risks

- Behavior change: an automatic continuation without `retryOfRunId` was
refused at claim and failed at dispatch with
`chat_control_recovery_proof_unresolved`. It now inherits the chat-close
check of the task's previous run instead. A chat close still stops it.
- Already-queued stuck continuations on hosted instances will start
after deploy. On the staging agent that is about 30 runs. They go
through the normal order and shares, so they do not take all slots at
once.
- One extra indexed read per gate check of this shape, and one small
recursive query per dispatch pass when recovery runs are queued. No
schema change. Mixed versions are safe.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, built-in browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:51:33 -06:00
DottaandClaude Opus 5.5 6917d0568d fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server starts queued agent runs. A legacy-runtime run gets a 60
s controller lease when it is claimed. Only the run's own execution
renews that lease.
> - The queued-run dispatch claimed all the runs of one agent in a loop.
It started their executions only after the last claim finished.
> - When the database is slow (for example, while a flood of recovery
runs fills the queue), one claim can take tens of seconds. The earlier
claims' leases then expire before their executions start. The run fails
with `controller_lease_lost` and `renewalCount: 0`.
> - Also, the lease watch waited for its first 10 s tick before it
renewed, even when the claim-time lease was almost spent.
> - This pull request starts each run as soon as it is claimed. It also
makes the lease watch renew immediately when the lease is older than one
renewal interval.
> - The benefit is that a slow claim batch no longer makes runs fail
before they do any work.

## Linked Issues or Issue Description

Refs #21127, #21042, #21222

**What happened?**
On staging, legacy-runtime runs failed with `controller_lease_lost`.
Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`,
and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease
watch started on a lease that had already expired, and no renewal was
ever sent. In one run, the claim was at 23:20:02 and the loss was at
23:23:11. Losses came in groups on the same agent at the same moment.
For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all
lost at 23:23:11.6.

**Why did it happen?**
`startNextQueuedRunForAgent` claimed every run of a batch before it
called `executeRun` for any of them. Each claim commits a 60 s lease,
but nothing renews it until the run's execution starts its lease watch.
A queue flood (fixed separately in #21222) made each claim slow, so a
batch took minutes. This bug predates today's changes. The queue flood
exposed it. The run-start hand-over (#21127) did not cause it: no
renewal was queued on the controller-lease connection.

**Expected behavior**
A claimed run starts its execution, and renews its lease, without
waiting for other claims. A lease watch that starts late renews at once.
A lease that lapsed and was revoked, reaped or taken over is still lost
and is never revived.

## What Changed

- `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent`
starts each run's execution right after its own claim, not after the
whole batch. The concurrency cap, the agent start lock and the
run-admission check before each claim are unchanged. If a later claim
throws, the runs claimed before it are already executing, not stranded.
After merging master, the loop walks master's queued-run dispatcher
(#21234: person-initiated runs first, paced recovery delivery): each
successful claim is reported to the dispatcher (`dispatcher.claimed`)
and then started at once, so master's order and pacing are unchanged.
Master's long-queued-run bookkeeping (#21349) is kept too: a claim that
returns nothing is still recorded as declined (`noteDeclinedAgedClaim`),
and a successful claim still clears that record before the run starts.
- `server/src/services/legacy-controller-lease.ts`:
`watchLegacyControllerLease` renews immediately when the claim-time
lease has less than 50 s left (older than one renewal interval). It uses
the same conditional renewal statement. That statement extends only a
lease that this boot still holds on a running row. The deadline is
unchanged.
- Tests: red/green tests for both changes. A test also proves that a
late watch on a revoked, reaped or lapsed lease still loses it and does
not change the row.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass.
- Red on master (source reverted, new tests kept): "starts each claimed
run before claiming the next" fails (`executeRun` not called while the
second claim is in progress). "renews at once when the watch starts on a
lease older than one renewal interval" fails (aborted at the claim-time
deadline). The three "starts late ... still loses it" cases fail on
master because master loses them by deadline and does not check
ownership.
- After deploy: `controller_lease_lost` with `renewalCount: 0` should
drop to about zero on staging, also with a full queue. A late watch now
shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a
silent expired start.

## Risks

- Low. The executions of a batch now start while later claims of the
same batch are still in progress. Execution does not claim runs and is
not awaited by the loop, so the cap and the lock are not affected.
- A late lease watch sends one extra renewal at start. A normal claim
starts its watch within milliseconds with about 60 s left, so it sends
no extra statement and adds no wait to a turn start.
- No schema change. Mixed versions are safe: the renewal statement is
unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:08:38 -06:00
DottaandClaude Fable 5.2 3f030971dc fix(recovery): deliver a recorded continuation as the person's Retry run, keeping its isolation (#21378)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When Paperclip cannot resume a cut-off run exactly, the
fresh-continuation fallback records a continuation for the task. A
delivery pass later starts it as a run. Staging shows this pending state
can last 30 to 110 minutes for backlog items.
> - The fallback also decides how the next run is isolated from the old
writer. For `fresh_isolated_workspace`, the next run must prepare a new
workspace.
> - A person's Retry pressed while the continuation was recorded but not
yet delivered started its own run at once. That run did not carry the
recorded isolation, so it could use the shared directory next to a
writer that is not proven stopped. It also did not carry the
continuation's recovery context.
> - This pull request makes wake admission deliver the recorded
continuation as the person's run, under the task lock. It also folds a
saved Retry into a delivered continuation for every record kind.
> - The benefit is one run per interruption, whatever arrives first, and
the recorded isolation decision is never lost.

## Linked Issues or Issue Description

Refs #21348

**What happened?**
A person's Retry (`requestTaskRetry` + `resumeTaskRetry`), pressed while
a fresh continuation was recorded but not delivered, was admitted at
once as an ordinary `task_retry` run. That run had no
`freshContinuationIsolation` and no `recoveryActionId`. On a projectless
task the run used the shared directory that the fallback had decided to
isolate from. The later delivery pass marked the continuation as
superseded by that run. The reproduction showed exactly one run, so the
reported "two runs" did not happen in that order. The isolation was lost
for all three record kinds tested. For `writer_ended` and native
records, a Retry saved before delivery was not folded into the delivered
continuation, because the fold only covered `fresh_isolated_workspace`.

**Expected behavior**
With a recorded, undelivered continuation, the next run admitted for the
task is that continuation. It has the recorded isolation and the
person's Retry is recorded as answered by it.

**Steps to reproduce**
1. Record a fresh continuation (`ensureFreshContinuation`, delivery
`recorded`) for a projectless task whose legacy writer is on another
host.
2. Call `requestTaskRetry` and then `resumeTaskRetry`.
3. Before this change: one `task_retry` run without
`freshContinuationIsolation` or `recoveryActionId`, while the record is
still `pending`.

## What Changed

- `fresh-continuation-retry-fold.ts`:
- `undeliveredContinuation` reads the task's pending fresh-continuation
record under the task lock (`for update`). It returns nothing when a run
of the task was created after the record.
- `undeliveredContinuationContext` gives the run the continuation's
`recoveryActionId`, `retryOfRunId`, `forceFreshSession` and, for
`fresh_isolated_workspace`, `freshContinuationIsolation`.
- `deliverContinuationAsRun` marks the record `delivered` to the new run
in the same transaction, settles its coalesced holds, and folds other
waiting Retries.
- `foldWaitingRetriesIntoRecordedContinuation` folds a waiting Retry
into any delivered fresh continuation, not only an isolated one.
- `heartbeat/queue.ts` (wake admission, issue transaction): a
non-system, non-delivery wake with no execution blocker takes the
pending continuation. The run is inserted as before, then delivered as
the continuation. The fold after a delivery wake now applies to every
fresh-continuation kind. No change to ranking or to the claim loop.
- `execution-recovery-resolution.ts`: the delivery pass folds a saved
Retry for every fresh-continuation kind. The superseded-run fold keys on
`recoveryActionId`.
- The person's run keeps its `task_retry` wake reason and the user's
wake receipt, so the existing #21349 rules rank it as person work.
- Limits: the person-triggered delivery does not spend an automatic
attempt. It adds no `issue.continued_automatically` entry and no
scheduled transient retry, and it is not gated by the hourly cap or by
delivery-pass pacing. The record itself was already counted when it was
recorded. The agent's concurrency limit still applies at claim time.

## Verification

New file
`server/src/services/fresh-continuation-retry-delivery.test.ts`, with 12
tests. All cases cover `fresh_isolated_workspace`, `writer_ended` and
native (`remote_epoch`) records, through the real `requestTaskRetry` +
`resumeTaskRetry` path:
- A Retry during pending delivery gives exactly one run. That run is the
continuation, carries the recorded isolation (the workspace readiness
reads `fresh_isolated_workspace`), coalesces the Retry, and ranks as
person work. Later delivery passes add nothing.
- Delivery first, then a saved Retry: one run, and the Retry is
coalesced. A Retry on the delivered run joins it through the busy path.
- A Retry and two delivery passes race (concurrent transactions): one
run.
- After the continuation's own failure, a Retry gets its own run.
- The delivery pass is held by pacing (backlog source, agent queue
full), and the hourly cap and the shared attempt ledger are exhausted.
The person's Retry still starts the run.

Red before the fix: 8 of 12 failed. Green after: 12 of 12.

Local runs, one file at a time:
- `fresh-continuation-fallback.test.ts`: 71 of 72. The failure is "runs
a person's Retry pressed before delivery…", a deadlock in `afterEach`
truncate. It passes alone, and it fails the same way on unmodified
master under this machine's load.
- `__tests__/recovery-wake-herd.test.ts`: 29 passed
- `__tests__/execution-control-work.test.ts`: 27 passed
- `__tests__/issue-recovery-actions.test.ts`: 112 passed
- `workspace-recovery-fallback.test.ts`: 19 passed
- `heartbeat/queue.test.ts`: 101 passed
- `task-retry.test.ts`: 117 passed;
`legacy-preparation-retry-hold.test.ts`: 5 passed
- `server` `tsc --noEmit`: clean

## Risks

- A person's message, chat input or manual wake that arrives while a
continuation is pending also becomes that continuation, with a fresh
session and the recorded isolation. Before, such a wake ran without
isolation and superseded the record. This is the same rule for every
admission path, and it closes the shared-directory gap there too.
- A wake that takes the context and is then deferred by a later gate can
keep the isolation marker in its saved context. If the record is
delivered meanwhile, that later run prepares a new isolated workspace.
That is the safe direction.
- No schema change and no migration. Nothing is added before turn start
for healthy runs: the new read runs only at wake admission.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, file edit,
test runs). Model ID `claude-opus-5-5`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 07:13:49 -06:00
DottaandClaude Opus 5.5 5ede64f247 fix(recovery): settle an older run's saved-work hold, fold a waiting Retry into the isolated continuation, and plain recovery wording (#21348)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, a saved-work hold protects the files it
left. The fresh-continuation fallback then continues the task in a new
run, or asks a person to press Retry.
> - Staging found that an older run's saved-work hold made the dispatch
gate cancel every new run of the task. The hold was never settled, so
the task could not run again, and no person could act on the reason.
> - Recovery notices also showed internal codes and words such as
"board", "inspect", "repair" and "no live execution path". People cannot
act on these.
> - A person's Retry pressed while an isolated continuation was waiting
for delivery was saved, and then ran a second time after the
continuation.
> - This pull request settles an older run's hold through the fallback's
existing machinery, uses plain wording in every remaining recovery
notice (server and UI together), and folds a waiting Retry into the
isolated continuation that answers it.
> - The benefit is that a task held by an older run continues by itself,
or on one Retry, in a fresh isolated workspace. People read only what
happened and the one thing they can do, and a Retry never starts a
redundant second run.

## Linked Issues or Issue Description

**What happened?**
1. An earlier run's saved-work hold (for example "Automatic recovery of
saved work failed … Retry to try again") made the dispatch gate cancel
every new run of the task with `execution_reconciliation_required`. The
fresh-continuation fallback only took holds of the task's latest run.
Each refused run looked like a newer run, so the hold was never settled.
2. Recovery notices and recovery-task comments showed internal codes,
raw IDs, "board", "inspect", "repair" and "no live execution path". The
hold for remote files that could not be restored asked a person to
"verify its exact stop receipt" and to "record workspaceRepairEvidence".
3. On a projectless task whose old legacy writer might still be alive, a
person's Retry pressed after the isolated continuation was recorded, but
before it was delivered, stayed deferred. When the continuation run
ended, task release promoted that Retry into a second, sequential run.
Concurrent writers were already impossible on this path, because the
Retry waited behind the continuation. The second run was redundant: the
continuation was the answer to the person's Retry.

**Expected behavior**
An older run's hold does not block new runs indefinitely. The task
continues in an isolated workspace, by itself while the interruption is
recent, or on a person's Retry. Notices say what happened and the one
thing to do. A person's Retry answered by the continuation never starts
another run.

**Steps to reproduce**
1. Leave a hold from an older run whose saved-work recovery gave up,
then start a new run of the task. Before this change, the run is
cancelled at dispatch every time and the hold stays.
2. Open a task whose run was cut off and its retries exhausted. Before
this change, the notice reads "No live execution path … visible for
intervention".
3. On a projectless task with an isolated continuation recorded, press
Retry before delivery, then let the continuation end. Before this
change, a second run follows it.

## What Changed

- Older-run holds (`fresh-continuation-fallback.ts`,
`modules/run-dispatch/adapters/postgres.ts`):
- A run that was cancelled at dispatch (status `cancelled`, never
started, `execution_reconciliation_required`) never ran and follows no
run. The held-run listing and the fallback's successor check now ignore
it.
- The older run's hold goes through the fallback's existing machinery. A
saved-work recovery that is still in progress keeps its bounded wait.
Holds that a person owns stay. Past the bound, the hold is settled and
the task continues in a new isolated workspace. Held files are kept,
recorded as `kept_not_applied`, and never applied.
- One recency rule, `interruptionEligibility` (the #21222 rule), judges
recency by the newest run that the hold refused at dispatch. It no
longer uses the end time of the hold's own run. If the last attempt is
older than 24 hours, no automatic continuation starts. A person's Retry
always continues the task.
- After the dispatch gate refuses a run for another run's hold, it hands
that hold to the fallback in the same transaction. The normal dispatch
path is not changed.
- A person's Retry and the isolated continuation (`heartbeat/queue.ts`,
`execution-recovery-resolution.ts`, new
`fresh-continuation-retry-fold.ts`):
- The wake-admission transaction that inserts an isolated continuation
run (context `freshContinuationIsolation`) also coalesces the person's
waiting `task_retry` wake for that task, under the task row lock. This
applies only to a Retry with no message attached. The wake gets `status:
coalesced` and `runId` set to the continuation. A server stop after the
commit cannot leave the Retry unfolded.
- The delivery pass is idempotent about this. An action whose own
continuation is already queued is no longer hidden from the pass. When
the pass finds that continuation, it marks the action delivered and
folds any Retry that is still waiting.
- Before this change, the fold covered only a Retry that was already
waiting when the continuation was recorded. A Retry that carries a
person's message is not folded. Only a Retry whose source run is the
continuation's own source run is folded, and only while the continuation
is queued or running. A Retry for the continuation's own failure gets
its own run.
- Plain wording (server and UI together):
- "No live execution path" is now "Task could not continue". One seed
(`buildNoRunningExecutionNoticeSeed`) builds it for every producer.
- Exhausted saved-work recovery: "Paperclip could not recover the
agent's saved files … Press Retry to try again".
- The non-retryable continuation notice no longer shows the error code.
Recovery-task comments no longer show `stranded_issue_recovery`, raw IDs
or retry reason codes.
- `legacy-execution-recovery.ts` uses plain text for remote files that
could not be restored.
- `IssueRecoveryActionCard`: "Board decision required" is now "Waiting
for you".
- The UI still recognizes notices saved with the old text
(`adjacent-errors.ts`, `system-notice-humanizer.ts`).
  - The precise codes stay in the evidence and in the notice details.
- Merge with current master: `recovery/service.ts` keeps master's
three-way choice (retry exhausted / continuation refused /
non-retryable). The non-retryable branch now uses the plain notice
without the code.
- Code comments that the original commits added are removed
(CONTRIBUTING).

## Verification

Red-to-green tests:
- `server/src/services/fresh-continuation-fallback.test.ts`:
- "continues a task held by an older run's saved-work hold in a new
isolated workspace…"
  - "keeps the wait of a saved-work recovery that is still advancing…"
  - "keeps a hold a person owns blocking…"
- "leaves an old hold for a person's Retry, and that Retry runs
isolated" (refused run and waiting Retry)
- "continues a recent restart-stopped run whose prepared remote
workspace could not be restored…" (red on the wording)
- "leaves an older restart-stopped remote restore failure for a person's
Retry, which then runs isolated"
- Same file, "consumes a person's Retry exactly once when the server
stops right after admitting the isolated continuation": the test runs
the admission and nothing after it. The Retry is already coalesced. The
test then resets it to unfolded, as an older release would leave it, and
runs the next delivery pass. The result is exactly one run and the Retry
consumed. Red without the in-transaction fold, and red without the
delivery-pass fold.
- Same file, "never folds a Retry for the admitted continuation's own
failure into that ended continuation": the continuation ends (real
`cancelRun`), then the person presses Retry on it. The Retry gets its
own run. Red without the source-run and liveness guard.
- Same file, "runs a person's Retry pressed before delivery as the
recorded isolated continuation, never in the shared directory": a
projectless task, an old legacy writer that may still be alive, and an
isolated continuation pending. Delivery starts exactly one run, which
carries `freshContinuationIsolation` in a `fresh_isolated_workspace`.
Extended here: after the continuation run ends (real `cancelRun` release
path, then `resumeQueuedRuns` and delivery), no run comes from the
person's Retry. The Retry wake stays `coalesced` into the continuation.
Red without the delivery fold (the Retry became a second run), green
with it.

Local runs on this branch (based on master 566843820c, independent of
#21347), one file at a time:
- `fresh-continuation-fallback.test.ts`: 72 passed
- `__tests__/recovery-wake-herd.test.ts` 18,
`__tests__/issue-recovery-actions.test.ts` 112,
`workspace-recovery-fallback.test.ts` 19,
`__tests__/execution-control-work.test.ts` 27, `heartbeat/queue.test.ts`
101: all passed
- `recovery/stranded-notice.test.ts`: 32 passed
- `system-recovery-state.test.ts`: 11 passed
- `workspace-recovery-warm-handoff.test.ts`: 44 passed
- `modules/run-dispatch/adapters/postgres.test.ts`: 88 passed
- `__tests__/legacy-workspace-restore-recovery.test.ts`: 73 passed
- `__tests__/heartbeat-process-recovery.test.ts`: 457 passed
- UI: `IssueRecoveryActionCard.test.tsx` 70,
`task-chat/adjacent-errors.test.ts` 26,
`lib/system-notice-humanizer.test.ts` 10,
`task-chat/TaskChatSystemNotice.test.tsx` 14, all passed
- `server` `pnpm run typecheck` (CI's layered typecheck, including
tests) and `ui` `tsc --noEmit`: clean

Test expectations changed on purpose, because they asserted the old
wording or behavior:
- `stranded-notice.test.ts`: seed title "No live execution path" is now
"Task could not continue". "retried dispatch/continuation" is now "tried
to start/continue this task again". A new test covers the plain wording.
- `system-recovery-state.test.ts`: the exact exhausted text is now
"Paperclip could not recover the agent's saved files. Press Retry to try
again; …". "stopped after 6 attempts" is now "after 6 tries". The
refusal text no longer contains "(allocation_ownership_ambiguous)".
"names the last failure and its cause" is replaced by "keeps the failure
code and cause in the evidence".
- `workspace-recovery-warm-handoff.test.ts`: `/stopped after 6
attempts/` is now `/after 6 tries/`.
- `heartbeat-process-recovery.test.ts`: new wording replaces "bounded
retry budget" / "no live execution path" (x2), "retried continuation"
(x2), "retried dispatch" and "3x attempts". "non-retryable failure" +
"`budget_blocked`" is now "retrying alone would not fix", with no code
in the body. The recovery-task comment lines are now plain.
- UI: humanizer "Task paused - waiting on a recovery owner" is now "Task
could not continue". `IssueRecoveryActionCard` "Board decision required"
is now "Waiting for you".
- `fresh-continuation-fallback.test.ts` "runs a person's Retry pressed
before delivery…": it asserted that the Retry's wake stays
`deferred_issue_execution` after delivery. It now asserts that the wake
is `coalesced` into the continuation run.

## Risks

- An older run's hold is now settled automatically while the
interruption is recent. The continuation always runs in a new isolated
workspace, and the held files are never applied, so the old run's files
cannot be overwritten or reused.
- A person's Retry without a message, pressed while an isolated
continuation waits for delivery, does not start a run of its own. The
continuation is the answer to that Retry. A Retry that carries a message
still runs. Concurrent writers were already impossible on this path (the
Retry waited behind the continuation); this change removes the redundant
second run.
- Wording only changes what a person reads. The UI still recognizes
notices that older releases saved.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:59:52 -06:00
DottaandClaude Opus 5.5 0ec155ac31 fix(recovery): enqueue a fresh interruption's continuation at once and record why a folded wake did not start (#21354)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, for example when a server swap
reaches a sandbox late, the fresh-continuation fallback records one
continuation of the task from its saved work
> - The execution-continuation worker turns that record into a queued
run. It took one record per agent per pass, and it took the oldest held
record first
> - After a large swap, the backlog of old records is hours deep. A
record for a run that ended a few minutes ago waited behind the whole
backlog. The record also stayed outside the queue while the agent's
queue was deeper than its concurrency limit
> - Meanwhile the task showed "todo" with nothing running, and the
assigned-work sweep's wakes for it were folded into the pending record
without any trace
> - This pull request queues a fresh interruption's continuation on the
next pass, keeps a share of every pass for the old backlog, and records
a skipped wake row when a system wake is folded into a pending
continuation
> - The benefit is that an interrupted task gets its next run promptly,
it waits in the agent's queue where the dispatcher ranks it, and nothing
is lost or invisible

## Linked Issues or Issue Description

Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that
this builds on).

**What happened?**
On a hosted staging instance, five tasks had runs that ended after a
server swap ("The runner restored the conversation after process loss,
but the previous turn is no longer active."). The fallback recorded a
continuation for each task 3 to 6 minutes after the run ended. Each
record stayed `continuationDelivery: pending` for 30 to 110 minutes with
no delivery attempt. During that time the worker delivered continuations
whose source runs ended 4 to 7 hours earlier, one per agent per pass.
The tasks read as failed and needing Retry. The sweep tried 3 to 4 times
per task, and each wake returned `null` without a wake row.

**Expected behavior**
A recorded continuation for a recent interruption becomes a queued run
within about one delivery pass. Old backlog continues to drain and is
not starved. A wake that is not started records why.

**Steps to reproduce**
1. Give an agent a queue that is deeper than its concurrency limit.
2. Record pending continuations for older interruptions that are held
past the hold bound.
3. Record a continuation for a run that ended a few minutes ago.
4. Run `deliverReconciledExecutions`. Before this change the fresh
record is not delivered. The oldest held record is delivered first, one
per agent per pass.

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `deliverReconciledExecutions` now sorts pending continuations into two
classes by when the source run ended. "Fresh" means that the source run
ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same
value the dispatcher uses to tell fresh recovery from backlog.
- A fresh continuation is enqueued without the queue-depth room check.
It is limited only by the per-agent limit (one fresh continuation per
agent per pass) and the per-company cap per pass. Among fresh records,
the newest interruption goes first. A record whose wake was refused
moves behind records that were not attempted yet.
- Backlog continuations keep their earlier rules. These are the
queue-depth room check, the bypass after the 5-minute hold bound, oldest
held first, and one per agent per pass.
- Each pass runs three phases: fresh up to the company cap minus
`RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to
the cap, then fresh again up to the cap. When both classes wait, the
backlog gets at least 1 of the 5 slots per company per pass. The
per-company cap (5) does not change, so a swap that interrupts many runs
does not enqueue them all at once. All admissions still go through the
one `wake(...)` call.
- In `enqueueWakeup`, a system wake that is folded into the task's
pending fresh continuation now writes a `skipped` wake row with reason
`issue_continuation_pending`. Before this change it returned `null` with
no trace.
- Tests in `recovery-wake-herd.test.ts`:
- New: a fresh continuation is queued on the first pass while the
agent's queue is deeper than its limit.
- New: a fresh continuation is delivered ahead of an hours-old backlog
for the same agent, and the backlog keeps draining.
- New: many fresh continuations for one agent are enqueued one per pass,
newest first, and none is left behind.
- New: the backlog keeps a share of each pass while fresh continuations
fill the company cap.
- New: a refused fresh wake stays pending and is enqueued on the next
pass.
  - New: a folded system wake leaves a skipped row.
- Updated: existing tests that used a recent source run to mean "held
because the queue is full" now use backlog-age sources, and assertions
that depended on exactly one admission per agent per pass now check the
record under test.

## Verification

- Red before the fix: the 6 new tests failed. The 18 earlier tests still
passed.
- Green after the fix: `cd server && npx vitest run
src/__tests__/recovery-wake-herd.test.ts` gives 24 passed.
- `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27
passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed.
`src/services/fresh-continuation-fallback.test.ts` gives 62 passed.
`src/services/workspace-recovery-fallback.test.ts` gives 19 passed.
`src/services/native-runtime/native-safe-replacement.test.ts` gives 62
passed.
- `cd server && npx tsc --noEmit -p tsconfig.json` passes.

## Risks

- Behavior shift: an agent with a full queue now gets fresh recovery
continuations queued at once. They wait in its queue, where the
dispatcher ranking (person-initiated first, recovery share) decides when
they start. They are no longer held outside the queue. The per-company
cap per pass still limits the burst.
- Old backlog drains at a similar pace as before. It always gets at
least one slot per company per pass when it waits.
- The skipped wake row adds one small row each time the sweep retries a
task that has a pending continuation. The sweep spaces its retries, so
the volume is low.
- No migration. No API or UI contract change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, read-only staging API calls).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:46:12 -06:00
DottaandClaude Opus 5.5 ff971149c2 fix(dispatch): bound how long a queued run waits and keep a person's interrupted Retry person-first (#21349)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue decides which queued run an agent starts when a
slot frees up
> - Recent changes start person-initiated work first and give recovery
runs one start in four
> - A system continuation of a person's interrupted Retry ranked as
old-backlog recovery, and the recovery share always went to fresher
recovery runs
> - With a steady stream of new work, that continuation never started,
so the person's task sat queued for hours
> - This pull request adds aging and a longest-waiting share, keeps a
person's interrupted request person-first, and passes the turn on when a
long-queued run cannot be claimed
> - The benefit is that every ready queued run starts within a bounded
number of starts, and a person's Retry is not lost behind system work

## Linked Issues or Issue Description

**What happened?**

On a hosted staging instance, one agent had 28 runs running and kept
starting newer runs. The company had 67 queued runs, and 54 of them were
older than one hour. A person pressed Retry on a task. The Retry run was
interrupted by a server restart two minutes later. The system queued an
automatic continuation (`issue_continuation_needed`) with
`previousRunId` set to the interrupted Retry run. That continuation
stayed queued for more than 2.5 hours. The task showed as "working".
Other queued runs of the same person (some queued for more than 10
hours) also did not start.

Root cause:
- The dispatcher ranked the continuation as `recovery_backlog`, the last
tier, because it was a system wake and its source run ended more than 30
minutes ago. The person-first check only looked at wake requests
attached to the queued run itself. It did not look at the source run,
which was the person's `task_retry`.
- The 1-in-4 recovery share picked the first recovery run in sort order.
Fresh recovery runs sort before backlog runs, and fresh recovery runs
kept arriving, so the backlog run never got the share.
- Nothing aged a queued run. A run in a lower tier or with a lower
priority could wait forever.

**Expected behavior**

A system continuation of a person's Retry ranks as person-initiated.
Every ready queued run starts within a bounded number of starts, even
when fresh person and system work keeps arriving. A run that keeps
failing to claim is reported and does not block the next-oldest run.

**Steps to reproduce**

1. Give an agent one free slot and several queued runs.
2. Queue a person's `task_retry` run, start it, and interrupt it.
3. Queue a system continuation with `previousRunId` set to the
interrupted run, and let its source age past 30 minutes.
4. Keep adding fresh system and person work.
5. See that the continuation never starts.

**Paperclip version or commit**

Staging Core 7e7452e88f (contains #21234 and #21277). Fixed on top of
origin/master cb94108484.

**Deployment mode**

Hosted (authenticated, public exposure, managed cloud).

**Agent adapter(s) involved**

Any adapter. The bug is in queued-run dispatch order, before the adapter
starts.

Refs #21234, #21277

## What Changed

- `queued-run-dispatch-order.ts`: a ready run that has waited for one
hour (`QUEUED_RUN_AGING_MS`) moves up to at least the work tier.
- `queued-run-dispatch-order.ts`: every fourth start
(`OLDEST_DISPATCH_SHARE`) goes to the longest-waiting ready run that has
aged, whatever its class or priority. The dispatcher tracks this share
inside one pass.
- `queue.ts`: the dispatcher reads the agent's recent starts to know how
many starts passed over the oldest run. One query now serves both the
recovery share and the oldest-run share.
- `queue.ts`: a queued run is person-initiated when any run in its
source chain (`previousRunId`, then `retryOfRunId`, up to 8 runs, in one
recursive query) was a person's request: a `task_retry` or chat wake, a
run with a `taskRetryRequestId`, or a run whose wake request a user
made. A person's Retry stays first after repeated interruptions.
- `queue.ts`: when a run that has waited past the aging bound is offered
a slot and cannot be claimed, the server logs a warning with the reason
and the wait time. That run skips the oldest-run share for one aging
period, so the next-oldest run takes the turn. It is still offered a
slot in normal order on every pass.

## Verification

- `npx vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (38 passed).
- Red/green: with the old dispatcher and queue, these new tests fail and
the fix makes them pass:
- "starts an old queued continuation within a bounded number of starts
while fresh person and system work keeps arriving" (unit, continuous
stream of fresh person, work and recovery runs)
- "serves the longest-waiting run once every share even when fresher
work outranks it" (unit)
  - "lets an old-backlog recovery run age into the work tier" (unit)
- "starts the system continuation of a person's interrupted Retry ahead
of fresher system work" (embedded Postgres, the staging case)
- "gives the longest-waiting queued run a free slot after three fresher
starts passed it over" (embedded Postgres)
- "keeps a person's Retry first after its system continuation is
interrupted again" (embedded Postgres, two successive interruptions)
- "starts the oldest continuations of a long backlog within a bound
while later assignments and fresh recovery keep arriving" (unit, the
staging shape: 3 continuations 4-5 hours old behind 80 later
assignments)
- Added coverage that passes before and after: "starts a system
continuation that a person's Retry coalesced into ahead of fresher
system work", "passes the oldest-run share to the next oldest run while
the oldest one keeps failing to claim", and "does not use the oldest-run
share for a run that has not waited past the aging bound".
- Related suites pass: heartbeat-run-dispatch-work,
heartbeat-run-dispatch-lifecycle, heartbeat-queued-run-claim-isolation,
issue-reassignment-queued-run, heartbeat-startup-fresh-dispatch,
heartbeat-stale-queue-invalidation,
heartbeat-ephemeral-queued-migration, queued-comment-consumption (116
tests).
- `tsc --noEmit -p server` passes.
- Not related to this change: `heartbeat-stale-queue-invalidation`
"resumes a rejected connection_intent outcome and promotes the claimed
review task" also fails locally on the base commit without this change
(one queued run, so the new code does not run).

## Risks

- Behavior change: when runs have waited more than one hour, one start
in four goes to the oldest of them, even when fresher person work is
queued. A person's fresh work still gets the other three starts.
- More runs can rank as person-initiated: system continuations and
retries of a person's run. This is the intended outcome.
- One more small recursive query per dispatch pass (the source chain, at
most 8 hops per run, by primary key and the wake-request run index). It
only runs when more than one run is queued and some have a source run.
- The record of runs that failed to claim is in memory. After a restart,
a stuck run gets the oldest-run share again once, then is skipped again.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:59:56 -06:00
DottaandClaude Opus 5.5 566843820c fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.

## Linked Issues or Issue Description

Refs #21250, #21257 (merged), #21258, #21234

**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.

**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.

**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.

## What Changed

- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.

### Old → new wording

| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
&lt;remedy&gt;, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "&lt;what
happened&gt;, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |

Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.

### Deliberately changed test expectations

Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.

Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.

Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.

Local runs, on master after #21257 (e063587edd) plus these commits
(server `tsc --noEmit` also passes):
- `stranded-notice.test.ts`, `recovery/assigned-work-invariant.test.ts`,
`task-retry.test.ts` and `__tests__/assigned-work-invariant.test.ts`: 4
files plus `heartbeat/queue.test.ts`, 329 tests passed.
- `heartbeat-process-recovery.test.ts` and `recovery-wake-herd.test.ts`:
475 tests passed.

Code comments that the original commits added were removed to follow
CONTRIBUTING; the explanations are in this description.

## Risks

- The sweep now resolves board-owned stranded holds for recent temporary
failures. It is bounded to failures within the last 24 hours with a
restart-class or temporary-setup cause, one continuation per task under
the usual cap and spacing. Lasting causes stay with their owner.
- Lifting a block on explicit continuation is limited to blocks the
replaced interruption set. A declared unblock action or an open blocker
keeps the block.
- Wording only changes server text. The UI-recognized "No live execution
path" notice is unchanged.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:04:20 -06:00
DottaandClaude Opus 5.5 49f0030f06 fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:17:57 -06:00
DottaandClaude Opus 5.5 99d9e608f4 fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person reassigns a task, the old owner's native run is
cancelled and the new owner's run is queued
> - The old run's executor keeps closing its provider session for
several seconds after the run is marked cancelled
> - #21031 holds a queued run while the task's previous session close is
pending. That hold reads the runner's cleanup gate, which exists only
after the close begins
> - The stale-lock sweep cleared the task lock as soon as the old run
was terminal. The new owner's run was then claimed before the close
began, met the close as it escalated, and failed with
`native_session_cleanup_quarantined`
> - This pull request keeps the new owner's run queued while the old
owner's stopped run still settles in this process, and wakes it when
that executor finishes
> - The benefit is that a reassignment never fails the new owner's run
and needs no Retry

## Linked Issues or Issue Description

Refs #21031

**What happened?**
In the paid chat eval `tasks.edit-properties`, the old owner's run
`6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process
stopped only at 04:03:57. The stale-lock sweep cleared the task lock at
04:03:47. The new owner's run was claimed at 04:03:48.626. The runner
admission waited for the old close, which escalated to operator-required
at 04:03:56, and the new run failed with
`native_session_cleanup_quarantined`.

Why #21031 did not apply: its hold reads the runner's in-memory cleanup
gate. At claim time the old executor had not started its session close,
so no gate existed. The existing `predecessor_settling_in_process` hold
covers only the same agent, and a reassignment changes the agent. #21260
does not touch this path: its new checkpoint runs only before a turn in
a recovered session, and both sessions here were new.

**Steps to reproduce**
1. Run a native task for agent A.
2. Reassign it to agent B. A's run is cancelled, but its executor is
still settling.
3. The stale-lock sweep clears the task lock.
4. B's queued run is claimed before A's session close begins.

**Expected behavior**
B's run stays queued until A's executor has finished. It starts by
itself after that, then #21031's gate covers any close that is still
settling.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the
`predecessor_settling_in_process` hold in `claimQueuedRun` also holds
for another agent's native run on the same task when that run is
terminal but its executor is still active in this process. A live run of
another agent (for example a reviewer) does not hold it.
- `server/src/services/heartbeat/queue.ts`:
`notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch
signal when a native executor settles and another agent has a queued run
on the same task. It does nothing when no run waits.
- `server/src/services/heartbeat.ts`: the executor calls it after its
cleanup, next to the existing same-agent `startNextQueuedRunForAgent`.
- `server/src/services/heartbeat/queue.test.ts`: the exact ordering
(cancelled old-owner run still in the executor set, lock cleared, new
owner's run queued), a control for a live run of another agent, and the
dispatch wake.

## Verification

- `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t
"reassigned owner|another agent's live run|next owner once"` — the hold
and the wake tests fail on master and pass here.
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts` (all passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- A queued run on a task waits while another agent's terminal native run
on that task is still settling in this process. That wait ends when the
executor finishes, and the new wake re-drives dispatch. The queued-run
liveness redrive still covers a lost signal.
- The wake is event-driven: one indexed query when a native executor
settles, and one notification only when another agent waits on that
task. It adds no polling.
- Previous-release saved state: no persisted shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:49 -06:00
DottaandClaude Opus 5.5 7154030ef6 fix(heartbeat): draining/yielding servers claim no queued run, and closed tasks get no system continuation (#21258)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server drains. It must not start
new runs, because its own exit would cut them off.
> - The drain gate (run admission closed at SIGTERM) and the hand-over
gate (#21127, yield to a replacement that already dispatches) were
checked only by the queued-run dispatch passes.
> - Other start paths (executeRun of a queued run, delivered
continuations, immediate wake starts) went straight to `claimQueuedRun`,
so an outgoing server could still start runs after traffic moved.
> - Also, a system continuation queued while its task was open started
after another run had finished the task, and took an extra turn on a
done task.
> - This pull request makes `claimQueuedRun`, which every start path
uses, honour the drain and hand-over gate, and cancel a system
continuation whose task closed while it was queued.
> - The benefit is fewer runs cut off by a swap, and no extra turns on
finished tasks.

## Linked Issues or Issue Description

Refs #21127, #21135

**What happened?**
1. An outgoing server kept starting runs minutes after traffic moved to
its replacement. Each such run was cut off when the old server exited.
The run start went through `executeRun` of a queued run, which called
`claimQueuedRun` without the drain or hand-over check.
2. In release-swap waves, a continuation (execution reconciliation
delivery, invariant or recovery continuation, blocker-stall wake) was
queued while its task was open. Another run then finished the task. The
continuation still started and took an extra turn on a done task.

**Expected behavior**
A draining or yielding server leaves a queued run queued for the serving
server, on every start path. A system continuation does not start on a
task that was closed while it was queued. A person's message on a done
task still runs.

**Steps to reproduce**
1. Close run admission (drain) or set the hand-over yield on a server.
2. Start a queued run through `executeRun` (for example a delivered
continuation).
3. See the run claimed and started on the draining server.

## What Changed

- `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`) refreshes
and checks the run-admission gate first. When admission is closed
(draining or yielding), it leaves the run queued and returns. The
dispatch passes already did this check; now every start path does.
- `claimQueuedRun` re-reads the task for a system continuation
(`issue_recovery_action_restored`, `issue_continuation_needed`,
`issue_assignment_recovery`, blocker-stall wake). If the task is `done`
or `cancelled`, it cancels the run as `issue_closed_before_start`, skips
its wake, and releases the task. Other wake reasons, including a
person's message, are not affected.
- That cancel never consumes a person's message. Admission can attach
deferred comments to a queued system continuation without changing its
wake reason. The cancel locks the run's wake and the run (the order
admission's coalescing uses), reads the run's current comment ids, and
if any is a person's comment that no run of this agent has received yet
(read-only use of `deliveredQueuedCommentIds`, which includes the
history-read rule), the run is not cancelled: it is claimed as that
person's follow-up, which may reopen the task through the normal path. A
comment attached concurrently is either seen under the lock or, after
the cancel commits, saved behind the cancelled run by admission.
- Admission is checked again inside each claim transaction (ordinary
claim, queued-comment claim, native review claim), right after the write
that moves the run to `running`. If admission closed during the awaited
checks, the transaction rolls back: the run stays queued with no
`startedAt`, its wake and the task's execution lock are untouched, and
nothing is charged.
- Every cancellation `claimQueuedRun` makes from the caller's copy of
the run is now a compare-and-set from `queued`
(`setRunStatusFromLive(..., ["queued"])` through a new
`cancelRunIfStillQueued` helper): the closed-task cancel, the
refused-interaction cancel, the missing-@mention cancel, the daily-cap
cancel and the blocked-dependency cancel. A run another dispatcher
already started or finished keeps its status and result, and no wake or
release follows. The queue module no longer receives the unconditional
`setRunStatus`.
- The closed-task cancel takes its decision under the wake and run
locks, then cancels through `setRunStatusFromLive` fenced on the row
version (`xmin`) the decision read. That saves the same never-started
proof (`executionRecovery: { kind: "bootstrap", providerWorkStarted:
false }`) as every other queued-run cancel, so task release promotes a
person's message saved behind it. A message attached after the decision
changes the run, the cancel matches nothing and the decision is taken
again. `HeartbeatRecoveryDependencies.setRunStatusFromLive`
(`recovery.ts`) now types its existing `cancellationCondition` parameter
as `SQL`.
- The dispatch ranking in `startNextQueuedRunForAgent` is not changed.

### What I checked in `claimQueuedRun` for writes from the caller's copy

- Cancellations through `setRunStatus`: all five now conditional on
`queued` (above).
- The final claims (ordinary, queued-comment, native review) already
lock and re-read the run and update `where status = 'queued'`.
- The case-handoff cancel and the chat-control stop already lock the run
and update `where status = 'queued'`.
- The budget-stamp update already writes `where status = run.status`
(`queued`).
- `cancelRunInternal` (agent removed, agent not invokable, board-owned
budget stop, pause hold, budget cancel) re-reads the run and only
cancels a non-terminal status, so it cannot overwrite a finished result.
It can still stop a run another server has just started; that is a
policy stop (agent removed, budget, hold) that would apply to that run
anyway, and it is unchanged here.
- `cancelStaleQueuedRun` already takes `expectedStatus: "queued"`.

## Verification

Red-to-green tests in `server/src/services/heartbeat/queue.test.ts`:
- "claims no queued run once run admission is closed, whatever path asks
(draining or yielding server)": red before the gate was added to
`claimQueuedRun`, green after.
- "does not start a %s run whose task another run already finished":
`issue_continuation_needed` and `issue_recovery_action_restored` are
cancelled as `issue_closed_before_start`; `issue_commented` (a person's
message, carrying its `wakeCommentId`) is claimed and the stored run is
`running`.
- "answers a person's message attached to a system continuation on a
closed task once": still undelivered, the continuation is claimed and is
the only unfinished run (red when the cancel ignores comment ids, green
after); already received by the closing run, the continuation is
cancelled as `issue_closed_before_start` and no other run exists.
- "keeps the result of a continuation another dispatcher already started
and finished": the stale queued copy is passed after the run succeeded;
the stored run stays `succeeded` with its result, and no wake or release
follows. Red before the compare-and-set (the run was overwritten to
`cancelled`), green after.
- "returns the claim to the queue when run admission closes while the
claim is being checked": admission closes inside an awaited check; the
run stays `queued` with no `startedAt`, the task's execution lock and
the wake are unchanged. Red before the in-transaction re-check, green
after.

Red-to-green test through the real release path, in
`server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`:
- "delivers a person's message saved behind a system continuation
cancelled at claim because its task closed": a never-started legacy
continuation on a done task, with a person's deferred message behind it.
Red before the fix: the cancelled run had `resultJson: null`, with no
never-started proof. Green after: the proof is present, the wake is
skipped, and the message is promoted and runs once.

Test expectations changed because they asserted the old behaviour:
- `server/src/services/agent-interaction-delivery.test.ts` "cancels
refused queued runs through the public queue claim without dispatch":
now expects the cancel through `setRunStatusFromLive(runId, "cancelled",
["queued"], …)` instead of `setRunStatus`.
- `server/src/services/heartbeat/queue.test.ts` (refused-assignment held
case): asserts `setRunStatusFromLive` is not called, instead of
`setRunStatus`.
- The `issue_commented` case of "does not start a %s run…" now asserts
the run is claimed and `running` (it previously asserted only that the
error code was not `issue_closed_before_start`).

Local command, on this branch based on current master (includes #21234;
the dispatch ranking it changed is untouched here):
- `cd server && pnpm exec vitest run
src/services/heartbeat/queue.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 102 tests passed. After the
proof fix, `queue.test.ts` again passed 97/97, and the new stale-queue
test passed. (The file's `afterAll` database cleanup hook times out on
this loaded machine on master too; it is not related to this change.)

Why the local release-swap wave does not show the yield fix:
- The "yield to the replacement" signal exists only when the run-start
handover is created (`server/src/index.ts`, about line 1320). That
happens only when a direct deployment listen URL exists, that is, with
`PAPERCLIP_EPHEMERAL_SERVER=true` and `PAPERCLIP_DATABASE_LISTEN_URL`
(the volume-free configuration).
- The local short form of the wave leaves `--ephemeral-server` out, so
the handover is disabled and `isRunAdmissionClosed()` becomes true only
at SIGTERM. The wave therefore cannot exercise the yield.
- The evidence for this fix is the unit test above and the reasoning
that every start path now reaches the same gate in `claimQueuedRun`.
Hosted confirmation must come from a volume-free instance.
- The closed-task fix was found in release-swap waves (an extra turn on
a done task in two tasks).

## Risks

- `claimQueuedRun` can now await an admission refresh. The refresh
returns at once when the cached state is fresh, so the normal path adds
no wait before `turn.start`.
- A system continuation for a closed task is cancelled. This matches
admission, which already refuses such wakes for a closed task.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:36:05 -06:00
DottaandClaude Opus 5.5 6cb0dd0c6b fix(recovery): continue twice-interrupted tasks and waiting follow-ups after a release swap (#21250)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off by a restart or a release swap, the
fresh-continuation fallback continues the task in a new run.
> - The continuation goes through normal admission, or through a
person's saved follow-up when one is waiting.
> - In release-swap waves, three gaps left tasks stopped with no run:
admission refused an isolated continuation of a task with no project, a
person's saved follow-up behind the replaced run was never promoted, and
the fallback relied on a saved follow-up that the cut-off run had
already read.
> - This pull request closes those three gaps.
> - The benefit is that a twice-interrupted task, and a task with a
waiting follow-up, continue by themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. A task with no project was interrupted twice. The second continuation
used a fresh isolated workspace, which clears the task's workspace
binding. The admission preflight still required that binding for an
isolated task with no project, so it refused the continuation as
`workspace_worktree_requires_project` about every 15 s. Each refusal
re-blocked the task and added a comment (329 to 428 skipped wakes per
wave). The runtime itself supports this case: it prepares a new
workspace (`create_new`).
2. The fallback continued a task with delivery `saved_input` (a person's
deferred follow-up is the next turn). The stranded-queue pass refused to
promote that follow-up, because the replaced native run's finalization
was `terminal_failure` (restart recovery ended it), not `committed`. The
release path would also have re-blocked the task with a reconciliation
incident. The follow-up never ran.
3. The fallback chose `saved_input` because a deferred user wake
existed, but the cut-off run had already taken that message into its
turn. Promotion settled the message as delivered, and no run followed.
The task showed "working".

**Expected behavior**
An interrupted task continues by itself in each case, with no person
action and no repeated comments.

**Steps to reproduce**
1. Run a release-swap wave with projectless tasks in isolated mode and
person follow-ups during runs.
2. Interrupt the same task twice, or interrupt a run that has a deferred
follow-up behind it.
3. See the task stay blocked or "working" with no run.

## What Changed

- Admission (`server/src/services/execution-workspace-policy.ts`,
`server/src/services/heartbeat/queue.ts`,
`server/src/services/fresh-continuation-fallback.ts`):
`isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The
admission preflight sets it from the new
`isFreshIsolatedContinuationWake`, which returns at once (no query)
unless the wake context carries `freshContinuationIsolation`. This does
not touch the dispatch ranking.
- Follow-up promotion
(`server/src/modules/wake-queue/adapters/postgres.ts`): the
stranded-queue pass and the release path skip the native finalization
gate and the reconciliation incident for a run that the fallback
superseded (`agentRunSupersededByFreshContinuation`).
- Saved-input choice
(`server/src/services/fresh-continuation-fallback.ts`): a deferred user
wake counts as saved input only when it has no comment ids, or still has
a message that the agent has not received
(`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the
fallback records its own continuation.

- Review round 1: (a) whenever the old writer is not proven stopped
(`fresh_isolated_workspace`), the fallback records its own isolated
continuation first, also when the task had a binding; saved input
follows it on the new workspace. (b) Release skips the manual-admission
hold and the pending-cleanup check for a run the fallback replaced with
an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`,
via `freshContinuationIsolatesRun`); other replacements still wait.
- Review round 2 (the isolated continuation must run before the waiting
follow-up): while the fallback's `fresh_isolated_workspace` continuation
is still pending delivery, release leaves the person's follow-up
deferred. Otherwise the follow-up would become the task's next run
without the isolation marker, and delivery would count it as the
continuation and skip the isolated run. The continuation's own release
promotes the follow-up behind it. `freshContinuationIsolatesRun` now
also reports `continuationPending`.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/execution-workspace-policy.test.ts`: admission
keeps a fresh isolated continuation of a projectless task runnable.
- `server/src/modules/wake-queue/adapters/postgres.test.ts`
"restart-ended native run": saved input behind a run the fallback
replaced is promoted.
- `server/src/services/fresh-continuation-fallback.test.ts` "relies on
saved input only while it still holds an undelivered message" (taken:
true gives `recorded`, taken: false gives `saved_input`).
- `postgres.test.ts` "promotes saved input behind an exhausted restart
hold ... (released, continuation pending)": the follow-up stays deferred
while the isolated continuation is pending, then is promoted when the
continuation's run is released. It was red before the fix (`expected
'promoted' to be 'released'`).
- Local, round 2: `cd server && npx vitest run
src/modules/wake-queue/adapters/postgres.test.ts
src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests
passed.

Local command, on this branch based on current master (includes #21222
and #21234):
- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts
src/__tests__/execution-workspace-policy.test.ts
src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests
passed (re-run after rebasing on #21234).

Local release-swap cells:
- L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered,
including a second one on the same task; 0
`workspace_worktree_requires_project` rows (before: 329 to 428 skipped
wakes per wave).
- R2 (6 tasks, 3 swaps): a task interrupted three times had its second
and third continuations delivered. The same cell found gap 3 (a
follow-up already read by the cut-off run), which the last commit fixes.
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds these fixes and the related recovery fixes: 11/11 tasks
finished hands-off (master: 9/11). The lost-continuation and
follow-up-promotion fixes were visibly working in it.

Review-round tests: "moves the task off the old writer's workspace and
runs the isolated continuation before saved input" (changed expectation:
`recorded` instead of `saved_input`; red on the previous head);
"promotes saved input behind an exhausted restart hold once a %s
continuation replaced the run" (isolated: promoted, red before;
`writer_ended`: held). 3 files, 224 tests passed.

Conflict note: the last commit was resolved onto current master. It no
longer imports `taskHistoryReadsForRun`, which belongs to a separate
opening-note change.

## Risks

- An isolated projectless continuation now passes admission. It runs in
a new workspace that the run prepares, so it never shares a workspace
with the cut-off run.
- A superseded run's follow-up is promoted without its native
finalization committed. The superseded run is fenced (its writes are
refused), so there is still one writer.
- No migration. No schema change. The normal admission path adds no
query.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:10:47 -06:00
DottaandClaude Opus 5.5 1c33275ada fix(heartbeat): start the new owner after reassignment away from a queued creator run (#21256)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent often creates a task for itself and then reassigns it
to another agent
> - The creator is busy in its own chat run, so the new task's
assignment run stays queued and does not take the task's execution lock
> - #21030 releases the outgoing assignee's never-started run on
reassignment, but only when that run holds the execution lock
> - Here the run did not hold the lock, so it stayed queued. The new
owner's wake then adopted it as the active run and was deferred behind
it
> - When the queue later withheld the stale run as
`issue_assignee_changed`, nothing promoted the deferred wake, and the
stranded-wake sweep ignores assignment wakes. The new owner never ran
> - This pull request closes all three gaps
> - The benefit is that a reassigned task starts its new owner at once,
and recovery still starts it if a release is missed

## Linked Issues or Issue Description

Refs #21030

**What happened?**
In the paid chat evals `onboarding.first-work` and
`questions-and-approvals.duplicate-answer` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master b1ef1f3022), the
CEO created a task for itself while its chat run was active, then
reassigned it to Morgan. The CEO's queued assignment run was withheld at
the execution gate with "issue assignee changed before the queued run
could start; the new owner will be woken instead". Morgan's wake stayed
`deferred_issue_execution` until teardown, about 4.5 minutes later.
Morgan never ran.

**Steps to reproduce**
1. Give an agent a running run on another task.
2. Create a task assigned to that agent, so its assignment run is queued
without the execution lock.
3. Reassign the task to a second agent.
4. The second agent's wake is deferred behind the stale run and is not
promoted when that run is withheld.

**Expected behavior**
The stale run is released and the new owner gets exactly one run for the
task, promptly.

## What Changed

- `server/src/routes/issues.ts`: on reassignment, when no live run holds
the task's lock, also release the outgoing assignee's never-started
`issue_assigned` run for that task. This uses the existing
`issue_reassigned` stop from #21030. The run is read through a new
`heartbeat.getQueuedAssignmentRunForIssue` accessor in
`heartbeat/run-state.ts`; route test mocks of the heartbeat service gain
that method.
- `server/src/services/heartbeat/queue.ts`: when the queue gate
withholds a queued run as `issue_assignee_changed`, it now releases the
task and promotes the new owner's deferred wake. The promoted run starts
after the claim lock is released (`deferredPostCommitEffects`), like the
neighboring cancellation branches. A failed promotion is logged and left
to recovery.
- `server/src/modules/run-dispatch/adapters/postgres.ts`: the same
cancellation hints each deferred wake of the task to the
deferred-receipt worker in the same transaction, so recovery runs even
if the process stops before promotion.
- `server/src/services/heartbeat/queue.ts`, `deferred-receipt-work.ts`,
`deferred-execution-notifications.ts`: the stranded-wake sweep now
includes deferred `issue_assigned` wakes of the current assignee. It
promotes them only when no queued, running or retrying run remains for
the task, through the normal release admission.
- `server/src/__tests__/heartbeat-reassignment-deferred-wake.test.ts`:
the cohort's state through the real wake path. One case covers the gate
withholding the stale run. One case covers a wake left deferred after
the blocking run finished. Both expect exactly one run by the new owner,
executed.
- `server/src/__tests__/issue-reassignment-queued-run.test.ts`: the
#21030 case now also runs when the queued run does not hold the lock,
and expects exactly one new-owner run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-reassignment-deferred-wake.test.ts
src/__tests__/issue-reassignment-queued-run.test.ts` passes. On master,
both heartbeat cases fail (the wake stays `deferred_issue_execution`)
and the no-lock route case fails (the stale run stays queued). With only
the gate change removed, the gate case fails and the recovery case still
passes.
- Focused suites: `deferred-execution-receipt-work`,
`heartbeat-lock-release-on-reassignment`,
`heartbeat-stale-queue-invalidation`,
`issue-agent-mutation-ownership-routes`, `src/modules/run-dispatch`,
`src/modules/wake-queue` pass.
- `cd server && npx tsc --noEmit` passes.

## Risks

- Behavior change: reassignment now cancels the outgoing assignee's
queued `issue_assigned` run for the task even when it does not hold the
lock. The queue gate already cancelled that run later as
`issue_assignee_changed`; now the new owner does not wait for it.
- The stranded-wake sweep now also reads deferred assignment wakes. It
skips any wake whose task still has a live run, so a wake waiting behind
a real owner is not promoted early.
- Upgrade safety: no schema or saved-state shape changes. Wakes already
saved as `deferred_issue_execution` by the previous release are promoted
by the sweep after deploy.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:36:05 -06:00
DottaandClaude Opus 5.5 ade7e0643f fix(recovery): pace recovery continuations and start person-initiated work first (#21234)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues tasks that a restart or a lost run left
unfinished
> - Recent recovery changes restored many held tasks at the same time,
and each restored task got a queued run on its agent
> - The agent's queue starts runs oldest first, so a person's Retry or
chat message waited behind every older recovery run
> - On staging, one agent had 131 queued runs, and person Retries waited
more than 50 minutes
> - This pull request paces how many recovery continuations enter an
agent's queue, and it starts person-initiated work before system
recovery work
> - The benefit is that a burst of recovery cannot starve people, and
every recovery run still runs

## Linked Issues or Issue Description

Refs #21222, #21139, #21135

**What happened?**

Staging deployed #21139 and #21135 at 22:53Z. Within one hour, the
company had 201 queued runs and 78 running runs. 122 queued runs had the
recovery wake reason `issue_recovery_action_restored`. Each one was on a
different task. They all came from the reconciled-execution delivery
pass (`source: execution.reconciled`), at a rate of 5 to 15 per minute
from 22:57Z to 23:51Z. One agent had 131 queued runs. The oldest queued
run had waited since 22:53Z. A person's Retry at 23:14Z was still queued
4 minutes later. Six person Retries on that agent, queued from 23:08Z to
23:14Z, were still queued at 00:04Z, more than 50 minutes behind
recovery runs. At 00:04Z, that agent's newest start was a recovery run
created at 22:57Z. In the last 400 runs, 25 succeeded.

**Expected behavior**

A person's Retry, chat message, or assignment starts before system
recovery work on the same agent. Recovery work still drains at a steady
rate. All recovery runs run eventually, and none is dropped. A task
whose continuation waits shows that the task continues automatically. It
does not show that the task is idle, and it does not ask for Retry.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Queue many recovery continuations for the agent
(`issue_recovery_action_restored`), for example after a server restart
with many held tasks.
3. Click Retry on another task of the same agent.
4. See that the Retry stays queued until all older recovery runs start.

**Paperclip version or commit**

Staging build 2026.1106.0+2187.git.57aeda190

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native and legacy runtimes). The problem is in the
shared run queue, so it applies to all adapters.

## What Changed

- New module
`server/src/services/heartbeat/queued-run-dispatch-order.ts` decides the
order in which an agent's queued runs start. Person-initiated runs start
first: `task_retry`, external chat messages, and any wake that a user
requested. Other work comes next, together with recovery runs for
interruptions that ended in the last 30 minutes. Old-backlog recovery
runs come last. Inside each class, the existing order applies:
readiness, then priority, then age. Runs whose dependencies are not
ready stay last.
- Recovery runs get a bounded share. When the agent started 3 other runs
after the oldest queued recovery run arrived and none of them was
recovery work, the next slot goes to a recovery run. The share applies
across the whole batch when one dispatch fills many slots. Thus people
cannot starve recovery.
- `startNextQueuedRunForAgent` in
`server/src/services/heartbeat/queue.ts` uses the new order. It changes
only the ranking. It does not change `claimQueuedRun`, the admission
preflight, or the promoter.
- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts` paces delivery.
It leaves a continuation pending when the agent already has a queued
recovery run, or when the agent's queue is at or above its concurrency
limit. One pass delivers at most 5 continuations per company and at most
1 per agent. The pass reads pending continuations in pages and leaves
out agents and companies that it already handled, so continuations for
busy agents cannot hide continuations for other agents. A continuation
that has waited 5 minutes or more is admitted even when ordinary work
keeps the queue full, and it goes ahead of fresher continuations,
longest wait first. Otherwise the most recent interruption goes first.
The existing 15-second continuation schedule delivers the remaining
items. This schedule runs only while pending continuations exist, so an
idle server does not poll.
- The assigned-work sweep
(`server/src/services/recovery/assigned-work-sweep.ts`) counts a pending
continuation as a pending wake for its task. Thus the invariant does not
see a held-back task as stranded, and it does not enqueue a second wake.

## Verification

- `pnpm exec vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 17 tests pass.
- Red/green: with the source changes reverted to master, all tests in
`recovery-wake-herd.test.ts` fail. Each review fix also has a test that
fails on the head before that fix. With this change, they pass.
- `recovery-wake-herd.test.ts` covers these cases with embedded Postgres
and the real heartbeat queue:
- Six recovery wakes are queued, then a person's request. The first
dispatch starts the person's run. All six recovery runs start later.
- Four pending continuations for one agent: one pass delivers one. The
others stay pending and are delivered on later passes. The continuation
schedule stays armed while items are pending and returns null after all
items are delivered.
- A continuation stays pending while the agent's queue is at its
concurrency limit.
- The per-company cap applies, and the most recent interruption is
delivered first.
- A continuation that has waited past the bound is admitted while
ordinary work keeps the queue full.
- New interruptions arrive for the same agent on every 15-second pass,
and the older continuation is still admitted within the bound.
- Continuations for four busy agents (32 rows, more than one page) do
not hide an idle agent's continuation.
- A held-back continuation counts as a `queued_wake` pending intent for
the assigned-work invariant, and the task shows `working` with
`canRetry: false`.
- `queued-run-dispatch-order.test.ts` simulates dispatch: a Retry starts
ahead of 40 recovery wakes, and all 40 still run. With a person Retry
arriving at every step, every window of 4 starts includes 1 recovery
run. A fresh interruption's continuation is not put behind old-backlog
recovery runs. When one dispatch fills 8 slots, the order is PPPRPPPR (P
is a person run, R is a recovery run).
- Related suites pass: fresh-continuation fallback, issue recovery
actions, completed-turn cleanup hold, issue monitor scheduler, workspace
recovery fallback, assigned-work invariant, execution control work,
execution recovery resolution, and the heartbeat queue.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- Behavior change: when a person's run waits behind recovery runs, it
now starts first. Recovery runs start a little later, but they still get
at least 1 of every 4 starts while people send work.
- Delivery of continuations is slower during a burst: at most 1 per
agent and 5 per company every 15 seconds. This is intentional. A
continuation that waits shows as continuing automatically, and the
invariant does not repair it twice.
- Ranking adds up to three small indexed reads when an agent has more
than one queued run. The reads run only when the queue has recovery runs
or wake requests to classify.
- No schema or migration changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:30:56 -06:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
DottaandClaude Opus 5.5 dfc510b887 fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud replaces the server process on each release. The old
server keeps serving until the new one passes health checks and takes
the traffic.
> - During that overlap the old server keeps starting new runs right up
to SIGTERM, although the new server can already start them.
> - SIGTERM then cuts off exactly those runs while they still prepare.
In one audited swap the old server started 30 runs after the new server
booted, and 23 of them ended `process_lost`.
> - Admission must never pause, so the fix must hand over who starts new
runs, not stop admission.
> - This pull request makes the new server record when it starts runs.
The old server then leaves new run starts to it, and starts them again
if the new server stops or dies.
> - The benefit is that a deploy no longer starts runs on a server that
is about to stop, and the company always has one server that starts
runs.

## Linked Issues or Issue Description

Refs #20852 (preserves preparing and launching turns that a deploy
interrupts; this PR reduces how many such turns the old server starts).
Related: #21046, #21107 (shared controller-instance owner proof), #20986
(queued-run dispatch signal).

**What existing behavior does this improve?**

The start of new runs while two servers overlap during a Cloud deploy.

**Subsystem affected**

Server run admission (`server/src/services/heartbeat/queue.ts`), the
controller instance table, and startup and shutdown in
`server/src/index.ts`.

**Current behavior**

The old server starts queued runs until it receives SIGTERM. Runs that
it starts in the last minutes of the overlap are interrupted while they
prepare. Nothing tells the old server that its replacement already
starts runs: `controller_instances.epoch` is written at process start,
and that happens minutes before the new server can start runs.

**Proposed behavior**

The new server records `dispatching_at` and a short
`dispatch_lease_expires_at` on its `controller_instances` row right
after its startup queued-run pass. The old server reads this through an
in-memory flag at run admission and leaves new run starts to the new
server. Runs it already started continue and are handed over at SIGTERM
as today. If the new server stops, drains or dies, the old server starts
runs again within seconds.

**Reason and benefit**

Fewer runs are cut off by each swap. There is never a moment without a
server that starts runs.

**Breaking changes**

None. The migration adds two nullable columns. Releases without this
code ignore the columns, and their rows never carry the marker, so a
mixed-version overlap behaves as today.

## What Changed

- `controller_instances`: new nullable columns `dispatching_at` and
`dispatch_lease_expires_at` (migration
`0471_controller_instances_dispatch_marker`, two `ADD COLUMN`
statements, no lock beyond the catalog update).
- New `server/src/services/run-start-handover.ts`. It is one worker on
the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`).
Its renewal and recheck are coordinator deadlines, and a drain change or
expiry wakes it.
- Replacement side: `begin()` records the intent and writes the marker
(it creates the boot's row if the row is missing). If the server drains
at that moment, the worker writes the marker when the drain ends. The
worker then publishes a `run-start-handover` hint and renews the 20 s
lease every 5 s. It renews only while an older boot that is not stopped
or fenced may still serve, and for at most 30 min. All marker writes run
in one serialized chain. `release()` at shutdown lands after any renewal
already in flight. A clear that fails is retried, and shutdown waits for
the release. While the server cannot dispatch (drain, quiesce, admission
closed), the worker clears the lease.
- Old-server side: `isYielding()` is an in-memory read. The hint
refreshes it, and so does one read that a dispatch pass waits for (at
most 1 s) when the last read is more than 2 s old. A read that gets no
answer within 1 s never keeps the server yielding. The server admits,
and the yield also ends when the replacement's last seen lease runs out
on the old server's own clock. Reads are a single flight on the
dedicated connection. While it yields, the server rechecks every 3 s,
calls `onResume` when the marker is gone, and sends a coalesced
queued-run-dispatch hint for skipped starts.
- It yields only to a newer epoch with a live marker. It never yields to
an older boot, to itself, or to a row without the marker, and it does
not yield without a row of its own.
- **Database invariant:** every statement of the hand-over runs in a
small transaction on the dedicated one-connection controller-lease pool
(`controllerLeaseDb`) with a 1 s statement timeout. That covers reads,
marker writes, the `run-start-handover` notification (commit-bound, in
the marker write's own transaction) and the skipped-start hint. None of
it touches the shared pool or registers tracked database work. The pool
is shared only with legacy controller-lease renewals, and never with the
liveness pool that native lease renewals use. A hand-over statement
waits at most one legacy batch (10 s), inside the 15 s between a marker
renewal and the marker's expiry. It delays a legacy renewal (60 s lease,
renewed every 10 s) by at most 1 s. I did not piggyback on the
controller-instance lease renewal, because that renewal runs only while
the server holds work and only every 30 s.
- **Import audit of `run-start-handover.ts`:**
  - `drizzle-orm` `sql`: builds statements, no I/O.
- `controllerLeaseDb`: the dedicated connection, used by every statement
here.
- `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs
`pg_notify` only on the transaction it is given and registers no intent,
so the coordinator's `reconcileDatabaseWork` has nothing of ours to
check.
  - `logger`: no I/O to the database.
  - `controller-process-identity`: in-memory boot id and host hash.
- `delivery-work-coordinator`: type only. The worker is registered for
lifecycle and deadlines; its `run` awaits only the serialized marker
chain, and reads are not awaited.
  - `DELIVERY_QUEUES`: constants.
  - `task-admission` drain subscriptions: in-memory events.
- `heartbeat.ts`: run admission is also closed while the server yields.
A `runStartHandover` service option (default: the process's registered
hand-over) lets one process host two assembled services in tests.
`queue.ts`: `startNextQueuedRunForAgent` and
`resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That
call returns `null` (nothing to await) unless the hand-over read is
stale.
- `index.ts`: creates the hand-over only when the instance lease is on,
the deployment notification channel exists, and the scheduler is
enabled. It calls `begin()` after the RunDispatch startup pass and wakes
that queue. It releases the hand-over when run admission closes at
shutdown, and stops it before the lease stops.
- `deployment-notifications.ts`: keeps the LISTEN connection open while
a hand-over involves this server, so skipped starts and lease changes
reach the other server at once.
- `doc/execution-semantics.md`: new section "Run starts during a deploy
overlap". It covers the marker, who yields, failure handling, what
counts as starting a new run, and what a post-deploy audit should show.

**Marker contract for other components (for example the deployer):** a
replacement starts runs when its `controller_instances` row has an
`epoch` greater than the old boot's, and `dispatching_at is not null and
dispatch_lease_expires_at > now() and stopped_at is null and fenced_at
is null`. `dispatching_at` stays set after the lease lapses, as the time
the boot began starting runs.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test
with two `heartbeatService`s, each with its own pool, listener,
coordinator and boot. A wake that the old server accepts stays queued
there. The replacement's real queued-run dispatch starts it, and claims
are observed per pool. After a replacement release, the old server
starts the run itself. Red: with the yield removed from
`isRunAdmissionClosed`, the old server claims the run.
- `cd server && npx vitest run
src/__tests__/run-start-handover.test.ts`: 19 tests pass against
embedded PostgreSQL. Each test uses two servers, each with its own pool,
boot id, LISTEN connection and delivery coordinator. The tests cover:
- the old server is ready and the new server starts; the new server sets
the marker; the old server yields; the queued run is claimed only by the
new server, and a skipped start wakes the new server's dispatch
- the old server learns from its own dispatch path without a
notification: a read older than 2 s is refreshed before the start
- the new server dies after it sets the marker: the old server resumes
and runs its queued-run pass
  - the new server shuts down: the old server resumes at once
- the new server drains: it releases the marker and takes it back
afterwards
- a replacement that starts up during a drain takes over when the drain
ends
  - a shutdown release during an in-flight renewal stays released
- a clear that fails at shutdown (statement timeout behind a row lock)
is retried by a later release
- an older server that outlives the lease TTL keeps yielding while the
replacement renews
- a yield that the old server cannot confirm within 1 s (its only
connection is busy) does not stop starts (red/green)
- a yield ends when the replacement's lease runs out, even while every
read waits for a connection (red/green)
- renewals blocked past the TTL hand starts back once, then the
replacement takes them again (bounded flapping)
- a clear that fails when the replacement starts draining is retried
well inside the lease (red/green)
- with both servers' shared pools blocked past the TTL, the marker is
written, renewed and released, and the old server yields and resumes,
all before the block ends (red when hand-over statements use the shared
pool)
  - an older-release row without the marker changes nothing
- a server never yields to an older boot or to itself, and does not
yield without a row of its own
  - a lone server writes once, never renews, and admits from memory
  - without a hand-over (self-hosted) there is nothing to await
- Red: on `origin/master` the test file cannot run, because the module
and the columns do not exist. The old server has no signal and keeps
starting the handed-over run.
- `cd server && npx vitest run
src/__tests__/deployment-notifications.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts
src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`:
all pass.
- `cd packages/db && pnpm check:migrations`: safety and compatibility
checks pass. `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify` asks for broad verification because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Run starts move from the old server to the new server during the
overlap. If the new server cannot dispatch, the old server resumes on
the lease clear (about 3 s) or the lease lapse (up to about 23 s after a
crash). If a hand-over read fails, the server keeps admitting.
- If the old server was not listening for notifications, it can start
runs for up to about 2 s after the marker (the age of its last read). On
a dispatching server, the pre-dispatch read adds one small bounded query
at most every 2 s.
- An older boot that has been idle for more than 30 min counts as gone.
The new server then does not renew, and the old server admits again as
today.
- The migration adds nullable columns only. The deployer can apply it
while the previous release serves.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:09:53 -06:00
DottaandClaude Opus 5.5 940d681974 fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. When a run
stops, its session close continues in the server process
> - The runner refuses every new session of the task until that close
settles. A close that cannot prove the old runner suspended needs a
stopped-process proof before it is released
> - A reassignment stops the old assignee's run mid-turn. The new
assignee's run was dispatched one second later into that gate, and it
failed with `native_session_cleanup_quarantined`
> - Only a Retry continued the task. Recovery must work without a
person, and the needed-retry rate must be 0%
> - This pull request keeps such a run queued until the close settles or
its old owner is proven stopped
> - The benefit is that the new run starts by itself after cleanup,
without a failed run or a Retry

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `tasks.edit-properties` (and
`tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a
task while the old assignee's native run was mid-turn, with a Paperclip
tool call in flight. The reassignment cancelled that run. Its session
close then failed with `native_session_close_unrecoverable: runner did
not durably suspend before checkpoint`, and the runner kept an
operator-required cleanup gate for the task. The new assignee's run had
started 1 second after the cancel. The runner's admission waited for the
in-flight close. When the close escalated, admission refused the run
with `native_session_cleanup_quarantined`. No run was queued again.

**Steps to reproduce**
1. Run a native task and stop its run while its session close cannot
settle at once.
2. Queue a run of another agent for the same task.
3. The run is dispatched into the cleanup gate and fails before it
launches.

**Expected behavior**
The run waits for the cleanup, and starts once the close settles or the
old owner is proven stopped. It never fails, and no Retry is needed.

## What Changed

- `server/src/services/native-runtime/native-cleanup-admission-hold.ts`:
`nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an
operator-required gate it first tries
`retireStoppedSessionCleanupGates`, which retires the gate when the old
owner's runner and provider processes are proven stopped. This is the
same proof that Retry uses.
- `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an
unstarted `paperclip_runner` run queued while its task has a settling or
unproven close. This matches the existing
`predecessor_settling_in_process` hold, which covers only the same
agent. A reassignment changes the agent.
- `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a
released close now also wakes queued-run dispatch. A close that
escalates to operator-required also wakes it, so admission tries the
stopped-process proof at once. The queued-run liveness redrive covers
any later proof.
-
`server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`:
an assembled heartbeat test. It is red on master (the run is claimed
into a pending close and into an operator-required close). It is green
here (the run stays queued, then is admitted after the close settles). A
control case shows that a run for another task is not held.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3
passed; 2 red on master)
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts
src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts
src/services/queued-run-dispatch-liveness.test.ts` (180 passed)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- A run whose task has an operator-required gate that is never proven
stopped now stays queued instead of failing. Before, that run failed,
and Retry was blocked by the same proof. The liveness redrive re-checks
it with back-off.
- The hold applies only to unstarted `paperclip_runner` runs of the
gated task. Other tasks and agents are not affected.
- Previous-release saved state: no persisted shape changes. The gate
lives in process memory, and runs saved by the previous release are
claimed by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:54 -06:00
DottaandClaude Opus 5.5 5a7378af56 fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued runs. The run-dispatch worker
re-runs the queue only when a write sends a "queued run dispatch"
signal.
> - A follow-up run can queue while its predecessor still holds its
sandbox lease. The claim then declines the run without a log line, and
the run stays `queued`.
> - After a controller restart, nothing sent a new signal once that
lease was settled. The run stayed `queued` forever. The issue had no
Retry, and the reconciler counted the queued run as a valid next action.
> - This pull request sends a dispatch signal from the native
finalization writes that can clear a predecessor's cleanup gate, logs
why a claim was declined, and bounds the wait. The dispatch worker
re-drives an undispatched queued run after 60 s as a backstop, and the
ownership reconciler reports and re-drives one still queued after 5
minutes.
> - The benefit is that a follow-up starts within seconds of its
predecessor settling, a lost signal delays it by about a minute instead
of forever, and a run that still cannot start is visible.

## Linked Issues or Issue Description

Refs #20909, #20935, #20953, #20970

**What happened?**

Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` +
#20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold,
`--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the
outage with an accepted result. The next controller finalized it at
22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z
(`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source
`native_status_decision`). The run stayed `queued` with `claimed_at =
null` until the wave ended 19 minutes later. The agent stayed `running`
with no running run. The task had no Retry. The grade was
`R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`.

**Root cause**

`claimQueuedRun` (`server/src/services/heartbeat/queue.ts`,
`lockIssueExecutionClaim`) declines a queued run while
`hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is
true for its predecessor. After the restart, the predecessor's warm
`reuse_by_environment` lease was still `active` when the continuation
queued, so the first claim was declined. The decline is correct, but
nothing owned the retry. `registerRunDispatchWork`
(`server/src/services/heartbeat/dispatch-work.ts`) has `next: async ()
=> null`: the queue is only re-run by a later dispatch signal. At wave
end the lease was `retained` with `cleanupStatus: success`, which does
not block a claim. A dispatch pass in that state starts the run
(verified locally). So no pass ran after the gate cleared. The ownership
reconciler listed the run as `queued_run` with owner `run_dispatch`
(always "alive"), and I7 treats any queued run as a next action, so
nothing reported it.

This is not the #20935 throttle. The wave base predates #20935, and a
throttled intent keeps its wake intent `queued` with a `nextAllowedAt`
deadline. Here the run row existed and its claim was never retried. On
current master #20935 parks this specific `response_wake` self-wake, but
any continuation, comment wake or Retry that queues behind a settling
predecessor after a restart has the same gap. The red test uses a
`same_agent` continuation, which master still enqueues.

**Steps to reproduce**

`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
"starts a continuation that queued behind its predecessor's sandbox
release after a graceful restart".

## What Changed

- Event-driven dispatch. A lease release already signals queued-run
dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`,
stop-and-retain settlement). I enumerated every `environment_leases`
status writer: each one that moves a lease out of a blocking state
already signals. The writers that do not signal only acquire a lease or
set `pendingCleanupInFlight`, so they add a block and do not clear one.
The native coordinator side had no signal, so:
- `notifyNativeFinalizationWork` (the shared notifier of coordinator
commits and terminal transitions) now also signals queued-run dispatch
for the company.
- The coordinator writes that clear `leaseOwner` or end the phase
outside that notifier now signal in the same transaction:
`releaseLapsedCoordinatorOwner`, native startup cleanup,
observed-workspace preservation, `record_finalization_error` in the
interaction bridge, the wake-queue terminal recovery, and the
reconciler's ambiguous-state terminal failure.
- Not changed: three non-transactional pre-dispatch closes in
`run-completion.ts` and `agent-execution-stop.ts`. Their callers queue
or settle the successor after the write, so the successor's own enqueue
starts the queue.
- `claimQueuedRun` now logs the reason when it declines a queued run at
the issue execution claim (`predecessor_cleanup_pending`,
`predecessor_still_executing`, `issue_execution_owned_by_another_run`,
case handoff, task chat cleanup), when a predecessor is still settling
in this process, and when a temporary budget hold keeps it queued.
- New `server/src/services/queued-run-dispatch-liveness.ts`:
`listUndispatchedQueuedRuns` reads queued runs in active companies whose
ready, non-paused agent has no running run. It reads every page with a
keyset cursor until the set is exhausted, so older held runs cannot hide
a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each
such run 60 s after it was created, then every 2, 4, 8 minutes, capped
at 15 minutes. Run ids that leave the set are dropped.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch worker's `next` now plans those re-drives. It re-reads only the
companies the pass just dispatched, keeps the deadlines of every other
company, and runs no query when the pass dispatched nothing. Only an
instance-wide pass (startup or explicit wake) reads every company. It
adds each due company to the dispatch scopes and returns the next due
time.
- `packages/db`: new partial index `heartbeat_runs_queued_created_idx
(company_id, created_at, id) WHERE status = 'queued'` (migration
`0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in
`client.ts`). It returns `null` (no deadline, no SQL until a signal)
when no run is undispatched or scheduling is suppressed. Every re-drive
goes through the normal `startNextQueuedRunForAgent` and
`claimQueuedRun` gates, so it never bypasses a hold, budget, pause or
cleanup gate.
- `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns`
into the worker (two lines).
- `server/src/services/recovery/ownership-invariant-report.ts`: new
check. A queued run still undispatched 5 minutes after creation is an I7
violation (`kind: queued_run`, `reason: queued_run_not_dispatched`),
written to the activity log once. A younger one sets the report's next
due time.
-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
`applyOwnershipReportBackstops` sends a queued-run dispatch signal once,
for each company with a newly found violation. Repeat passes do not
signal again, so a held run does not wake the report every 30 s.
- `tests/restart-chaos/README.md`: records the wave and the fix in the
CR results table. The existing CR1 cell (#20909) and the enforced U1
check already cover this class in the harness.

## Verification

- RED on master (`23909077ad`):
- `heartbeat-native-restart-suspension.test.ts` "starts a continuation
that queued behind its predecessor's sandbox release after a graceful
restart": the continuation is `queued` behind the predecessor's `active`
warm lease. The run-dispatch worker is registered, the lease then
settles to `retained`/`success` with no dispatch signal, and the run
stays queued: `AssertionError: expected null not to be null` after 75 s.
- `ownership-reconciliation-report.test.ts` "reports and re-drives a
queued run that no dispatch started while its agent had nothing
running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no
violation for a run queued 19 minutes).
- `ownership-reconciliation-report.test.ts` "signals queued-run dispatch
when finalization work settles or a finished coordinator's lapsed owner
is released": `expected [] to include 'settled'`.
- GREEN on this branch: all pass. The restart test has two cases. When
the lease is released through `releaseLease`, the continuation starts
within 5 s. When the lease is settled by a write with no signal, the
backstop starts it within 75 s. A new report test puts three older held
runs ahead of a stranded run with a page size of 2, finds the stranded
run, and checks that the backstop signals once.
- `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap,
and going quiet.
- Focused suites pass (22 files, 495 tests):
recovery-deadline-acceptance, live-recovery-work,
heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work,
deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the
ownership kinds suites, and others.
`heartbeat-native-restart-suspension` and
`ownership-reconciliation-report` pass in full. `tsc --noEmit` for the
server passes.

## Risks

- A queued run that is held on purpose (for example a temporary budget
hold or a case handoff waiting for release) is now re-checked at most
every 15 minutes after the first minute. Each re-check is one indexed
query and the normal claim gates. Such a run is also reported once as an
I7 violation.
- An instance with an undispatched queued run keeps a dispatch deadline,
so it does not idle while that run waits. An instance with no such run
is unchanged.
- Schema: one additive partial index, built concurrently. No data
change. State saved by an older version is read as is. Nothing runs
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:04:58 -06:00
DottaandClaude Opus 5.5 dfd2a6efca fix(native): restart a follow-up the closing server cut before it attached, instead of reattaching to its predecessor's Runner (#20979)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs on a remote sandbox keep a warm Runner between turns. A
follow-up run takes over its settled predecessor's provider session and
Runner.
> - When a deploy stops the server, a follow-up that is still preparing
fails with `local_runtime_api_listener_unavailable`. The executor saves
this failure as a same-run bootstrap retry (#20786). The saved failure
proves that the run did no provider work of its own.
> - After the restart, restart recovery saw a checkpoint and a provider
session, and chose `reattach_remote_runner`. But the checkpoint and the
Runner identity belong to the predecessor. The Runner identity check
then failed with `native_remote_recovery_lease_mismatch:
stage=runner_verification field=identity_run_id`. The run ended terminal
and the task needed Retry.
> - This pull request recognizes that durable shape in restart recovery
and restarts the run as a fresh bootstrap on its warm session. It does
not try to reattach to a Runner that belongs to a different run.
> - The pull request also stops the queue from claiming a queued run
after shutdown closed run admission during the queue scan.
> - The benefit is that a deploy during a follow-up's preparation no
longer ends the run terminal with a Retry.

## Linked Issues or Issue Description

Refs #20786, #20928, #20852, #20904

**What happened?**
In three Codex runs across two Daytona restart-chaos waves (warm
sandboxes, CR1 and K15 with overlap), a warm follow-up run
(`automation`, `retryOfRunId` = its predecessor) started 0–2 s before
its server got SIGTERM. Its bootstrap failed with
`local_runtime_api_listener_unavailable` and saved a `bootstrap_retry`
with `providerSessionEstablished: false`, `providerEventsExist: false`.
After the restart, `native.recovery.transition` chose
`reattach_remote_runner`. `verifyRemoteRunnerReattachment` then threw
`identity_run_id`, because the warm Runner's identity still named the
predecessor. The run ended with
`native_continuation_requires_reconciliation`. In every sandbox capture
the Runner was cleanly suspended, with no stderr.

**Expected behavior**
The run did no provider work and never wrote its own Runner identity.
The next server must restart it without a Retry.

**Steps to reproduce**
1. Finish a remote warm native run, so that its provider session and
Runner identity are saved.
2. Start a follow-up run. Admission carries the predecessor's provider
session and Runner process identity into the follow-up.
3. Send SIGTERM while the follow-up prepares, so that it fails with
`local_runtime_api_listener_unavailable`.
4. Restart the server. Restart recovery chooses
`reattach_remote_runner`, and the run fails with `identity_run_id`.

## What Changed

- `native-restart-recovery.ts` (`claimNativeRestartRecoveries`): add the
`remoteFollowUpNeverAttached` proof. It is true only when all of these
are true:
- the run is a failed, settled `bootstrap_retry` whose original failure
is `local_runtime_api_listener_unavailable`, with
`providerSessionEstablished: false`, `providerEventsExist: false`, no
successor and no lease owner;
- the run is on a prepared remote sandbox workspace with the current
preparation fencing version, it is not a plugin task, and it has no
Runner controller checkpoint and no staged attachment of its own;
- its session checkpoint is only the carried admission checkpoint (no
cursor, turn, result, terminal or pending request), and a settled
earlier run of the same session saved that exact provider session;
- its Runner process identity is absent or is exactly the identity of a
settled predecessor;
- it has no provider event, no Runner-originated event, and no
`native.process_identity_recorded` event of its own.
With that proof, the run gets a `bootstrap_incomplete` claim with the
reason `remote_follow_up_never_attached`, and it passes the same
current-admission check as the other settled bootstrap retries.
- `verifyRemoteRunnerReattachment` is unchanged. It still refuses a
Runner that names a different run.
- `heartbeat/queue.ts` (`startNextQueuedRunForAgent`): check run
admission again before each claim. A run that a scan reaches after
shutdown closed admission stays queued for the next server.
- Tests:
- `native-runner-restart-recovery.integration.test.ts`: eight cases for
the staging shape. Three restart (inherited Runner identity; inherited
identity plus a Runner request; a Runner request with no identity). Five
stay held (own recorded Runner identity; a Runner-originated event; a
carried active turn; a provider session that no settled run saved; a
saved failure that established a provider session).
- `queue.test.ts`: a queued run stays queued and is not executed when
admission closes during the scan.

### How this relates to #20852 and #20904

- #20852 hands a preparing run to the next controller and persists
launch intents, so that preparation itself survives a deploy. It does
not change restart recovery classification for a settled
`bootstrap_retry`, and it does not touch the run-admission gate. This PR
covers runs that already failed and saved a `bootstrap_retry` (including
state that older releases saved), so the two are complementary.
- #20904 changes startup readiness for native-controller preservation
obligations. It does not touch either code path in this PR.

### Finding about the "draining server" admission

In both waves the follow-up was claimed before the process observed
shutdown. In CR1 (seed 9511) the run was created at 21:54:04.695Z and
started at 21:54:04.854Z; the server logged "shutdown signal received"
at 21:54:05 (the harness sent SIGTERM at 21:54:03.256Z, under load
average 10). In the K15 overlap (seed 9513) the runs started at
23:31:10.9Z and 23:31:11.5Z, and the old server got SIGTERM at
23:31:13Z. During the overlap the old server is fully live and has no
signal that a newer deploy serves traffic. So the existing admission
gate (`closeProcessRunAdmission`) was not bypassed. The only gap in the
gate was the check-then-claim window inside
`startNextQueuedRunForAgent`, which this PR closes. Holding work on a
superseded but not yet signaled server needs a new "superseded deploy"
signal; this PR does not add one.

## Verification

- Red on master (`23909077ad`): `node cli/node_modules/tsx/dist/cli.mjs
packages/db/src/test-postgres-command.ts -- node
node_modules/vitest/vitest.mjs run --project @paperclipai/server
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "closed the runtime listener before it attached"`: the three restart
cases failed with `"kind": "reattach_remote_runner"` (the staging
disposition); the five held cases passed.
- Red on master: `queue.test.ts -t "closes admission during the queue
scan"` failed (`expected [ { …(81) } ] to deeply equal []`).
- Green with the fix: both filters pass.
- `native-runner-restart-recovery`, `native-runner-restart-preparation`,
`native-startup-retry` and `native-controller-recovery` integration
files: 267 passed, 14 skipped.
- `queue.test.ts`: 89 passed.
- `cd server && pnpm typecheck`: passed.
- Chaos waves: see the PR comments for the reruns of seeds 9511 (CR1)
and 9513 (K15 overlap).

## Risks

- The new proof applies only to the closed-listener `bootstrap_retry`
that #20786 writes, on a remote sandbox, with a carried session that a
settled predecessor saved. Every other shape keeps its earlier
classification.
- A `bootstrap_incomplete` claim sends the run through the existing
bootstrap path, which verifies the prior owner of the session root
before it starts. The s9205 follow-ups already use this path with an
inherited Runner identity.
- Upgrade safety: no persisted shape changes. The proof reads only
fields that the current and recent releases write. A row saved without
these facts keeps the earlier verdict.
- No wait or file work is added before `turn.start`. The new queries run
only in restart recovery.
- The queue change only stops claiming after admission closed. Runs stay
queued for the next server, the same as the existing entry check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:36:45 -06:00
DottaandClaude Opus 5.5 edd00526e7 fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.

## Linked Issues or Issue Description

**What happened?**

Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.

Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.

**Expected behavior**

A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.

**Steps to reproduce**

Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.

## What Changed

- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.

## Verification

- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.

## Risks

- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:40:44 -06:00
DottaandClaude Fable 5.2 a56b648602 fix(upgrade-readiness): report restartable runs and a handoff verdict that blocks only on lost work (#20910)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deployer can replace the server while agent runs continue. Before
the swap, it reads the upgrade-readiness report of the old server
> - The version 1 report is ready only when every active run is
adoptable and no wake admission is in flight
> - On a busy instance, new runs arrive each minute. Some runs are
always in preparation, so the report is almost never ready and the
instance does not upgrade
> - A run in preparation has done no agent work. Restart recovery starts
it again. A swap loses nothing
> - This pull request adds a `handoff` object to the report. It
separates recoverable, restartable and blocking runs
> - The benefit is that a deployer can wait only for work that a swap
would lose

## Linked Issues or Issue Description

Refs #20852 (reference only; this change does not depend on it and does
not use its version 2 shape).

**What happened?**

A busy instance had 16 running runs and 1 queued run. Fourteen had a
live process. Two or three native runs had not launched a process yet.
One was a legacy run. `upgradeReadiness.ready` needs `blockingRuns === 0
&& pendingWakes === 0` in one probe. That state almost never occurs, so
the deployer never swaps the server.

**Expected behavior**

The report tells the deployer which runs a swap would lose. Runs that
only wait for a launch, and wake admissions in flight, do not block a
graceful replacement.

**Steps to reproduce**

1. Keep a steady flow of new runs on an instance.
2. Read `GET /api/instance/task-drain?upgradeReadiness=1` repeatedly.
3. See `ready: false` each time, because one or more runs are in
preparation or a wake admission is in flight.

## What Changed

- `summarizeUpgradeReadiness` returns an additive `handoff` object:
`version`, `ready`, `recoverableRuns`, `restartableRuns`,
`blockingRuns`, `pendingWakes`, `unsavedWakes`.
- `handoff.ready` is true when the instance is not draining, the census
is complete, and `handoff.blockingRuns` and `handoff.unsavedWakes` are
0. Saved pending wakes do not change it.
- The wake queue tracks each public wake admission until its wake
request and queued run commit. `unsavedWakes` is the count of admissions
before that commit. Without a valid count, every pending wake is
unsaved.
- The top-level version 1 fields keep their values and meaning. `ready`
and `blockingRuns` are unchanged, so a current consumer sees no looser
verdict.
- The census query marks a run `restartable` only with all of these:
`paperclip_runner` adapter; no process identity; no provider session
before or after; no checkpoint header; no dispatched runner task; no
execution-start event. It must also be a native run in the `observed`
phase with preparation fencing and an unsettled finalization record, or
a legacy run in the `preparing` stage under a recorded controller.
- All other runs without adoption evidence stay blocking. This includes
local adapters, legacy runs with a child process, follow-up runs that
carry a provider session, and runs the census could not observe.
- The OpenAPI description and `doc/testing-native-remote-continuity.md`
describe the new fields and their limits.

Why these classes are correct on master:

- Restartable native runs: restart recovery claims a run with no
checkpoint and no provider evidence as `bootstrap_incomplete` and
executes the same run again (`native-restart-recovery.ts`
`classifyNativeRunnerRecoveryEvidence`; `heartbeat/recovery.ts` executes
each claim). A graceful shutdown suspends the run and notifies recovery.
Existing tests: "recovers an unclaimed remote startup once without
requiring a provider checkpoint" and "recovers an unclaimed remote
startup that a graceful shutdown suspended during preparation".
- Restartable legacy runs: process-loss recovery retries a `preparing`
run with `providerWorkStarted: false`. Existing test: "continues a chain
whose preparations were killed twice and then interrupted again, without
a Retry".
- Pending wakes: the count is wake admissions in flight in the process,
not queue rows. A wake commits its wake request and its queued run in
one transaction. After the commit, the next server dispatches the queued
run at startup, so a saved wake does not block. Before the commit,
nothing is saved. Shutdown waits at most 30 seconds and then exits, so
an unsaved wake can be lost and blocks.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/upgrade-readiness.test.ts
src/services/heartbeat/upgrade-readiness-post-result.test.ts`: 28
passed.
- Red/green: before the implementation, the 7 new tests failed (6
summary tests, 1 PostgreSQL census test). After it, all pass.
- Red/green for unsaved wakes: a test holds the agent row lock while a
wake admission waits. Without the fix, `handoff.ready` was true with no
saved wake. With the fix, it is false with `unsavedWakes: 1`, and the
wake saves after the lock releases.
- New cases: only recoverable runs; recoverable plus unlaunched runs
plus pending wakes (`handoff.ready` true, top-level `ready` false and
`blockingRuns` unchanged); a started run without recovery evidence; an
unobserved execution; draining; truncated census.
- The PostgreSQL test changes one fact at a time (process identity,
provider session, phase, fencing, checkpoint, adapter, each
execution-start event) and expects the run to become blocking.
- `cd server && npx vitest run src/__tests__/openapi-routes.test.ts
src/__tests__/instance-settings-routes.test.ts`: 157 passed (187 with
the two readiness files).
- `cd server && npx tsc --noEmit`: passed.
- `pnpm verify` exited 2 (a documentation path is a global CI input).
Required CI supplies the broad coverage.
- Not run: a live server replacement. The classification relies on the
existing recovery tests named above.

## Risks

- Low risk for current consumers. The change is additive and the version
1 fields are unchanged.
- `handoff.ready` is an observation at one instant. A wake admission
that starts after the probe is unsaved until it commits.
- Restart recovery applies its own proofs and a bounded attempt budget
before it starts a restartable run again. The report is an observation,
not a guarantee for each run.
- The probe adds one indexed `not exists` on run events per running row,
inside the existing two-second statement timeout.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`), with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 13:55:47 -06:00
DottaandClaude Opus 5.5 7e8901aa60 fix(recovery): let Retry and the person's answer continue a held task with a pending decision (#20872)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process.
Recovery then saves the sandbox files and puts an execution hold on the
task
> - The hold offers Retry. A person can also continue the task with a
new message
> - Many held tasks also have a pending decision: the question that a
parked turn asked, or an older connection request
> - Continuation admission refuses every continuation while a decision
is pending ("decision_pending"). The answer to that decision is not a
message, so admission defers it behind the hold ("execution_recovery")
> - So Retry waits for the answer and the answer waits for the hold. The
task shows a Retry that does nothing, and nothing continues it
> - This pull request lets a person's Retry pass an open question, and
lets the person's own answer continue the task through the hold
> - The benefit is that a task with a lost Runner and a pending decision
always has a control that works

## Linked Issues or Issue Description

**What happened?**
On staging, the same-run retry of six remote native runs found the
Runner process gone (`process_not_running`, lifecycle `ready`). Each run
failed with `runner_remote_process_identity_unavailable`. Saved-work
recovery then recovered the sandbox files, and the task showed "Saved
work recovered. Retry or send a message to continue." with `canRetry:
true`. Four of these tasks also had a pending decision: the question
that the parked turn asked, or an older pending connection request. For
those tasks, Retry is deferred with `decision_pending`, and an answer to
the decision is deferred with `execution_recovery`. No control continues
the task.

**What did you expect to happen?**
Retry starts one new run. The person's answer to the pending decision
also continues the task.

**Steps to reproduce**
1. A remote native run asks the user a question and parks the turn.
2. The Runner process exits. The same-run retry fails with
`runner_remote_process_identity_unavailable`.
3. Saved-work recovery recovers the files. The task offers Retry.
4. Click Retry: the Retry waits with "A pending approval or question
must be resolved before this message can start."
5. Answer the question: the answer waits with "Waiting for execution
recovery. Your message is saved."

Refs #20838, #20853

## What Changed

- `admitExplicitNativeContinuation`
(`server/src/services/explicit-native-continuation.ts`): a task Retry is
no longer held by a plain pending question (`ask_user_questions`) or
connection request (`connection_intent`). Any interaction with approval
semantics still holds Retry: a plan review (`issue_document` target), a
governed tool action, a secret proposal, a connection authorization, and
other confirmations. A pending approval still holds every continuation.
A new message supersedes a plain question, as on master, and waits for a
decision with approval semantics.
- `admitExplicitNativeContinuation`: the person's own resolution of an
interaction (`mutation: "interaction"`, answered/accepted/rejected,
resolved by the same user) is explicit user input. It continues the held
task like a new message: a fresh session, the hold resolved as
`explicitUserContinuation` with the `interactionId`, and
`issue.execution_recovery_settled` activity. It never replays the failed
run. Governed tool-action confirmations are excluded. The existing
`readQueuedInteractionResponse` reads the answer.
- `heartbeat/queue.ts`: both admission calls pass the wake payload to
the admission.
- `buildExecutionContinuation`
(`server/src/services/execution-continuation.ts`): dispatch rechecks the
user authorization. It accepted an interaction receipt only through a
saved interrupt queue, so the answered successor failed with
`continuation_user_authorization_missing`. It now also accepts the
person's own answer: the interaction named by the durable authorization
was resolved by that user and not by an agent, and the authorization
names this successor run. The check does not depend on the run's current
trigger, so a later answer that replaces the trigger cannot invalidate
it.
- New API-level test
`server/src/services/governed-wait-runner-loss.test.ts` with the two
staging shapes (a parked question from the failed run; an older
connection request on a monitor wake). It checks that Retry starts
exactly one run and leaves the decision pending, that the person's
answer starts exactly one run and settles the hold, and that an answer
resolved by a different user keeps the hold.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- `cd server && pnpm exec vitest run
src/services/governed-wait-runner-loss.test.ts`: 5 passed. The answer
tests now also build the successor's continuation as dispatch does and
check that the agent receives the answer.
- Red for dispatch (first head, ac17aebe): both answer tests fail with
`continuation_user_authorization_missing`.
- Ponytail follow-ups: a two-answer test (the second answer waits behind
the successor; dispatch also holds with a replaced trigger) and 'keeps
Retry on hold while %s is pending' for four approval-bearing kinds. On
head 60beb773, 5 of these fail. Two more tests show that a new message
supersedes a plain question and waits for a plan review (both unchanged
from master). All 12 tests in the file pass now. The PR adds no code
comments; the rationale is in `doc/execution-semantics.md` and here.
- Continuation suites (execution-continuation,
execution-continuation-provenance, legacy-continuation-authority,
explicit-native-continuation, task-retry, governed-wait-runner-loss):
634 passed.
- Red on master (product hunks removed, test kept): 4 of 5 fail. Retry
tests: `successors: 0`, deferred `decision_pending`. Answer tests:
`successors: 0`, deferred `execution_recovery`. Every case asserts
`runError: "runner_remote_process_identity_unavailable"`.
- Related suites: explicit-native-continuation, task-retry,
workspace-recovery-admission, retry-workspace-continuity,
workspace-recovery-retry-copyback, workspace-recovery-fallback,
native-workspace-reconstruction, native-recovery-preparation-failure,
issue-thread-interactions-service. 957 of 959 passed. The 2 failures are
30 s `afterEach` hook timeouts in native-workspace-reconstruction; all
51 of its tests pass alone, and the same file on unchanged master shows
5 hook timeouts on this machine.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Medium. A Retry can now start while a plain question or connection
request is pending; decisions with approval semantics still hold it. The
new run sees the open decision in its task context, and the decision's
answer still wakes the agent later. Approvals keep holding Retry, so
approval gates are unchanged.
- An answer now continues a held task in a fresh session. All other
admission gates still run: stop proof, saved-work recovery, workspace
repair, active execution, pause, budget and ownership. Only the user who
resolved the interaction can continue this way.
- This does not make recovery automatic. A task whose Runner was lost
still rests on the hold until a person clicks Retry or answers.
Automatic settlement of a parked governed wait is a separate change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:42:36 -06:00
DottaandClaude b93df4ca38 fix(chat): consume a saved chat wake whose message a completed run answered (#20689)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - External chat (for example GitHub review requests) wakes the
assigned agent with a saved chat message
> - When the agent is already running, the chat wake is saved behind
that run and released after it finishes
> - The running agent often reads the new message as part of its current
turn, so the message is already answered when the saved wake is released
> - The release kept that chat wake anyway, removed its now-delivered
message ids, and started a reviewed chat run with no message
> - A reviewed chat run with no message cannot be attested, so it failed
with `reviewed_chat_execution_binding_not_authorized` and the task
became blocked
> - This pull request treats a chat message wake like a comment wake:
when a completed run already received all its messages, the saved wake
is consumed
> - The benefit is that a message that arrives during an active run no
longer blocks the task after it was answered

## Linked Issues or Issue Description

**What happened?**
A GitHub chat message arrived for a task while the assigned agent's run
was active. The wake was saved behind that run. The run read the message
and answered it. When the run finished, the saved chat wake was still
promoted. Its message ids were removed because a completed run already
received them. The new run had source `chat:<provider>` and no message,
so reviewed chat attestation denied it with `context_mismatch`. The run
failed with `reviewed_chat_execution_binding_not_authorized`.
Review-path recovery inherited the same context and failed too, and the
task was blocked.

**Steps to reproduce**
1. Assign a task in review to an agent that is running.
2. Deliver an external chat message for the task. The wake is saved
behind the active run.
3. Let the active run receive the message as wake input and succeed.
4. Release the task's execution lock.
5. See that the saved chat wake is promoted with no message, instead of
being consumed.

**Expected behavior**
The saved chat wake is consumed with "Queued messages were already
delivered to a completed run". No new run starts. The chat does not get
a "not started" notice for a message that was answered.

**Paperclip version or commit**
`origin/master` at 6e524d8a69.

**Additional context**
This was seen on a staging deployment on five runs across three tasks in
one day. It is a message timing race, not a deploy or authorization
problem.

## What Changed

- `server/src/modules/wake-queue/adapters/postgres.ts`: a saved wake
whose reason is `External chat message received` and whose payload
mutation is `chat_message_received` is treated as message-only. If a
completed run already received all of its messages, the release consumes
it, the same as a comment wake. If any message is still unread, it is
promoted as before.
- `server/src/services/chat-inbound-wakeup-publications.ts`: a chat
receipt consumed for that reason does not publish "This follow-up was
not started". Other declines keep that notice.
- `server/src/services/queued-comment-delivery.ts`: constants for the
chat message wake reason and the delivered-message reason.
- New test
`server/src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`.

## Verification

- `cd server && npx vitest run
src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`
- RED on `origin/master`: the release promotes the consumed chat wake
("Unexpected promotion of a consumed chat wake"), and the receipt
resolves to a `not_started` notice.
- GREEN with this change: the wake is cancelled with the delivered
reason, no run starts, no "not started" notice, and a chat wake with an
unread message is still promoted.
- Related suites: `src/modules/wake-queue`,
`heartbeat-reviewed-chat-binding.integration.test.ts`,
`durable-chat-wakeup.test.ts`,
`heartbeat-native-status-context.test.ts`,
`external-chat-wait.integration.test.ts`,
`heartbeat-held-comment-wake.test.ts`.
- `npx tsc --noEmit -p server` passes.

## Risks

- A chat wake that merged with another kind of wake takes the merged
wake's reason and mutation. Only a wake whose current reason and
mutation are both the chat message values is treated as message-only, so
other merged intents keep promotion.
- A visible "Your follow-up is queued." notice is not followed by a "not
started" notice when the message was already answered. That matches the
existing behavior when the wake is promoted.
- Not changed: a separate case where messages from two different chat
conversations are merged into one saved wake. That case fails with
`binding_denied` and needs its own change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, read-only API access). Extended
reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:15:13 -06:00
DottaandPaperclip 5bad074baa feat: add individual agent interaction permissions (#20411)
Add eleven sparse incoming and outgoing restrictions, human CLI/API administration, shared mutation enforcement, and pending-delivery checks. Preserve existing defaults, agent settings, and existing authorization boundaries.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-09 20:34:04 -06:00
DottaandPaperclip b09f17fbfd fix(agents): repair queued setup through task connection cards (#20361)
## Thinking Path

> - Paperclip manages AI agents doing work for people.
> - Agent verification must finish before an accepted task can start.
> - A missing AI account default can prevent verification before a
provider starts.
> - Lifecycle policy treated that owned configuration refusal as a
temporary failure and retried each minute.
> - The accepted turn remained Queued without explaining the
prerequisite.
> - Park account configuration failures and reuse the task's existing
inline connection card.
> - Successful repair rechecks verification and releases the original
queued turn under its original authority.

## Linked Issues or Issue Description

**What happened?** An OpenCode agent selected the responsible user's
OpenRouter default, but that default did not exist. Verification retried
unchanged configuration indefinitely and its accepted task stayed
Queued.

**Expected behavior:** Preserve accepted work, explain the setup
prerequisite, and let the appropriate person repair it using the
existing task connection UI. Completion must verify the agent before
starting the same queued turn. Genuine pause, approval and access gates
remain effective.

**Reproduction:** Create an OpenCode Runner agent with a
responsible-user OpenRouter binding and no personal default; assign a
task while verification is pending. The base retries verification and
shows an unexplained queue.

Related: #18702 handles pending task connection recovery after account
repair. This PR handles the earlier agent-verification boundary. #20330
addresses a separate Codex governed-wait timeout and is already merged.

## What Changed

- Share owned account-selection failure evidence through a pure library
contract. Lifecycle policy parks those configuration failures with
fixed, redacted guidance; temporary infrastructure errors retain their
retries.
- Batch setup-state reads for unstarted queued turns. Distinguish manual
repair from automatic setup retries and preserve Stop, started/history
state and lifecycle holds.
- Reuse the existing `connection_intent` card, account selection and
completion APIs. Its controller-only `agentSetup` marker accepts only
never-started legacy turns; runtime tokens still require a running run.
- Address the card to the lifecycle verifier's account owner, preserving
the task's execution principal and private-task visibility. Attach
missing cards on admission, verification, startup and full recovery,
including retained retries and continuations. Repeated delivery and
publication remain bounded. A new queued turn supersedes the setup card
of a stopped turn; generic connection cards retain cross-run reuse.
- Recheck lifecycle verification after usable-account completion, then
release the existing turn through the normal readiness path. Preserve
decline, Stop, reassignment and changed setup-owner gates. Fence an
in-flight verification failure from overwriting a repair; add no
provider wake or recurring worker.
- Park owned account refusals during resume as well as initial
verification. Bind setup cards to the lifecycle operation, settle
obsolete deliveries, and keep explicit Retry guidance accurate.
- Reuse the router’s existing lifecycle pin for setup repair without
rerouting the task, advancing the pool cursor, or replacing its saved
binding. Legacy direct-account cards remain readable.
- Hide the incorrect Resume banner during setup, retaining genuine
manual/budget pause controls. Document the execution and retry contract.

## Verification

- Red: eight original behavior assertions failed on the base, including
the real selector's retry decision and the real batched queue
projection. Six owned selection cases failed classification.
Temporary-error presentation, different task/verifier users, retained
retry/continuation recovery and the setup Resume banner also have
failing-before/passing-after regressions.
- Green: 430 affected backend, lifecycle, shared-contract and UI tests
passed before the final review corrections. Subsequent focused runs
cover one inline card, real account selection/completion, duplicate
delivery/publication, verification before readiness, unchanged run
authority, private tasks, no extra provider wake,
decline/Stop/reassignment, company isolation, invalid runtime tokens,
transient retries and redacted diagnostics.
- Final setup/card and generic connection-service suites: 49 tests
passed. The stopped-turn case failed before the correction, then passed
with real account completion, the stopped source remaining cancelled,
and the replacement source remaining queued.
- After rebase, all 129 focused assertions passed. The local queue
suite’s 88 execution assertions passed repeatedly, but its
database-fixture teardown exceeded both 30-second and 120-second hooks.
An isolated queue case completed, and the setup/connection suites fully
passed. The original queue test and cleanup remain unchanged; broad CI
supplies the full-suite result.
- Server/UI typechecks, module boundaries, token gates and diff checks
passed. `pnpm verify --explain` requested broad CI because the
execution-contract document changed; no redundant full local suite was
substituted.
- Real source acceptance: macOS arm64, Node 26.4.0, isolated embedded
PostgreSQL and local board identity. Created an OpenCode/OpenRouter
agent and task through the API with no default. Admission retained one
unstarted turn and produced the existing inline Connect OpenRouter card.
The browser opened its credential form and closed it without entering
credentials. The same card/run persisted after server restart and
reload, with the setup label and no incorrect Resume. The browser also
reproduced Stop followed by a new queued turn retaining the stale card
on the prior head. On `12f609c98372fd1e50644028db75cfcc8deb210d`,
startup superseded the stale card and retained the same replacement
turn, with the original turn still cancelled and no provider execution.
- The earlier CI case-search failure also reproduces on exact base
`0e3fdebea8257bdff11573081c76b306f12db230` without this PR. Its repair
is now in master via #20371; this branch is rebased onto it and contains
no API-search changes.
- Staging's actual Runtime test returned “Connect an account and choose
your personal default” before provider startup. The original turn later
started independently on deployed code, then hit a separate
workspace-export failure. Automatic file recovery subsequently marked
that run successful; its task remains Blocked and the trial’s useful
output is unverified. This PR is not deployed; provider authentication
through its card and downstream publication remain unverified. No
account, credential, default, permission or binding was changed during
diagnosis.
- Previous revision review: Greptile 5/5 on
`12f609c98372fd1e50644028db75cfcc8deb210d`, with all four inline
findings addressed and resolved. [Full CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42713494508)
on that revision: 59 successful checks, one neutral check and three
intentional skips, with no pending or failed checks.

- Latest revision `976bcae993805a8a2222395bbff9394f6791b424`: resume,
router and Retry-copy regressions failed before repair. Green: 92
lifecycle/setup/shared tests, 83 setup/generic-connection/router tests,
and 137 UI/shared tests. Server/UI typechecks, module boundaries and
token gates passed. [Full CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42720731137)
on this exact revision: 59 successful checks and three intentional
skips, with no pending or failed checks. Greptile reviewed this head at
5/5; all six review findings are addressed and resolved. Real Stop/retry
acceptance retained the same replacement turn and current-operation card
through two actual server restarts, with no provider execution.

## Risks

- Known account configuration failures now require repair or explicit
Retry. Temporary failures keep automatic retries; accepted work and
authority remain intact.
- The projection adds one company-scoped batch read only for unstarted
queued turns. Card attachment examines at most 100 active candidates and
adds no timer, worker, schema or migration.
- The optional stored setup-card marker requires an aware reader during
rollback. No run/session/checkpoint identity format changes. Setup
presentation grants no access and cannot bypass verification or other
gates.

## Model Used

- OpenAI GPT-6 (Codex), with reasoning, tool use, code editing and test
execution. The exact serving model identifier and context-window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 17:49:34 -06:00
DottaandPaperclip 3cc1032f6a fix: deliver saved answers to dependency-blocked tasks (#20363)
## Thinking Path

> - Paperclip lets people direct agent work through tasks and structured
answers.
> - Dependencies stop task execution until prerequisite work finishes.
> - A person must still be able to answer or redirect the assignee of a
blocked task.
> - Ordinary comments already get a bounded answer/triage turn. A saved
question response has no comment ID, so the same gates reject it.
> - Later dispatch also requires an active task and tries to check it
out. That conflicts with bounded interaction on a blocked task.
> - This pull request verifies the stored human response at each gate
and keeps the turn in bounded interaction mode.
> - The assignee can receive the answer without starting dependency work
or removing its blockers.

## Linked Issues or Issue Description

Refs #19923. That change covers dependency waits and comment retries.
This change covers stored interaction responses that have no comment ID.

**What happened?**

A parent task waits on a parked child. The assignee asked a setup
question. After a person answers, the response stays queued. Sending
that saved response is refused as `issue_dependencies_blocked`. Ordinary
new messages can reach the same assignee.

Passing admission alone is insufficient. The dispatch status check
rejects a blocked task, and automatic checkout then tries to start work
whose prerequisites are unfinished.

**Expected behavior**

Deliver the stored human answer or confirmation to its requesting
assignee as a bounded answer/triage turn. Keep the task blocked and keep
its dependency relationships. Preserve current-owner, pause, approval,
budget, recovery, and execution gates.

**Steps to reproduce**

1. Assign a parent task and make a backlog child block it.
2. Save a question on the parent and answer it as a person.
3. Dispatch its resolved-interaction wake, or send the saved answer from
the task queue.
4. Observe that the answer does not reach the provider while an ordinary
comment can run in answer/triage mode.

**Paperclip version or commit**

Reproduced on master at `653ab8b18f76`. This branch is rebased onto
master at `0e3fdebea8`.

## What Changed

- Verify the stored response by company, task, requesting agent, status,
kind, human resolver, and resolution time. A context flag alone is
insufficient.
- Allow a verified response through dependency admission and claim in
the existing bounded interaction mode.
- Recheck that response during scheduled retry promotion and final
provider dispatch.
- Keep bounded responses from checking out the blocked task or setting
it to `in_progress`, including when the last blocker finishes during
preparation.
- Preserve continuation classification through workspace and AI-account
resource waits, so the final response and owner checks still run.
- Correct the task-case API summary so the existing `case for task`
discovery regression passes on the first search page. This fixes a
pre-existing failure exposed by full CI, without changing search ranking
or assertions.
- Ignore obsolete Retry notices after newer conversation work, without
acknowledging saved input or changing execution authority.
- Document the response contract. Add real PostgreSQL and
assembled-heartbeat regressions.

## Verification

- Final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`: [required CI
passed, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/42706363120/attempts/2),
including typecheck, production build, server/chat tests, Runner checks,
and all 8 browser shards. PR status: 59 passing checks and 3 normal
skips.
- Red: answered questions and accepted/rejected confirmations return no
admitted run before the fix. All three new positive cases fail at the
dependency gate. A resource-wait retry also reproduces as refused before
its gate is fixed.
- Green: at delivery revision `9375a603e4`, focused heartbeat
integration and classification passed 14 tests; retry scheduling and
dispatch safety passed 241 tests. These use real PostgreSQL and
controlled adapter and resource-availability boundaries. The wider
initial dependency, native-continuation, and queue suites also passed
before the final refinements.
- Negative cases keep pending, missing, wrong-task, and agent-resolved
responses blocked. Changing the response proof immediately before
dispatch cancels the run without calling the provider. Policy controls
retain the current assignee, task status, and pending approval checks.
- Positive cases execute once, leave the parent blocked, preserve the
dependency edge, and leave the child in backlog with no child execution.
The same saved response survives a resource wait; removing its human
proof refuses promotion.
- The resource-wait provider-boundary and blocker-completion regressions
also failed before their fixes. All 14 focused integration and
classifier cases passed at delivery revision `9375a603e4`, including a
changed response during retry preparation.
- Red: a retained Retry for an older run makes a newer successful
conversation display as failed. Green: all 78 database-backed query and
interruption-presentation tests pass, including a later real failure and
independent queued input controls.
- Server layered typecheck and module-boundary checks passed at
`11992aad0c7fbaf13e7548d71ee633560bd8bb6f`, before the one-line API
summary correction. That CI passed build, typecheck and every browser
shard, but exposed the pre-existing case-search assertion and a
report-assets test timeout. Final-head CI passed as linked above.
- Fresh Greptile review is 5/5 on final commit
`ba4b337cf1bb0236b7c0a742665b222ee894ff51`, with no unresolved threads.
- The existing API discovery suite reproduced its first-page assertion
before the summary correction; all 3 cases pass afterward. The timed-out
report-assets privacy test passed locally in isolation and the full
support job passed on the final-head rerun. Runner shutdowns interrupted
the first attempt; affected jobs were rerun once without code changes. A
separate onboarding timeout also passed on that rerun.
- `pnpm verify --explain` requests broad CI because the execution
contract is a global input. The final CI run supplied that coverage.
- Separately recovered the reported staging conversation through an
ordinary message and the existing steering control. The agent
acknowledged the message and the saved-answer queue drained. The fresh
Notebook action subsequently succeeded. This was on the previously
deployed revision; it does not claim this code is deployed.

## Risks

- This changes admission and dispatch. Each gate checks the persisted
response; no caller-supplied marker grants the exception.
- The bounded turn does not check out the task, resolve a dependency, or
approve another action. Existing pause, ownership, budget, approval, and
stopped-execution controls still apply.
- Each additional lookup is one primary-key read and runs only for a
response on a dependency-blocked path. There is no new polling worker.
- No migration is required. Older queued contexts get the exception only
when their stored human response still verifies. No prior run or action
is replayed by this change.
- This PR does not roll out the code to staging.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:27:34 -06:00
DottaandClaude Opus 5.5 93f7c3b5d3 fix(tasks): keep a budget-held Retry's failure visible and refresh the open task (#20338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a task's automatic recovery stops, the person's Retry rechecks
the task's gates. If a gate refuses, for example a budget hard stop, the
Retry is saved and starts by itself once the gate allows it
> - After a person pressed Retry under a budget hold, the task page went
blank: no failure, no notice and no Retry. A person had no way to see
why nothing started
> - Two gaps caused it. The task notice drew only system-owned states,
and a budget hold presents as a person-owned state with no card of its
own. Also, admission saved the wait reason without publishing anything
that an open page listens to
> - This pull request keeps the failure visible and names the budget
wait, and it refreshes an open task when admission records why a Retry
waits
> - The benefit is that "failure remains visible if the new attempt
cannot start" holds without the person reloading the page
> - These two commits were pushed to #20306 after it was squash-merged,
so they are not on master

## Linked Issues or Issue Description

Refs #20306 (merged without these two commits) and #20270.

**What happened?**
In the chat eval
`chat-features.runner-opencode.local.recovery.retry-exhausted`, a person
pressed Retry on a task whose native recovery was exhausted while the
agent's budget was held. The Retry was accepted (`POST
/api/issues/{id}/retry` returned 202). The task page then stayed blank
for more than 60 seconds on the open page.

**Expected behavior**
The task keeps the failure visible and says that it is waiting for the
budget. Once the budget allows new work, the saved Retry starts once by
itself.

**Steps to reproduce**
1. Let a task's run fail, and put the assignee over a budget hard stop.
2. Open the task and press Retry.
3. The page shows nothing about the failure or the wait.

**Paperclip version or commit**
master at 76254ead22

## What Changed

- `ExecutionBlockerNotice.tsx`: a Retry held by a budget hard stop
presents as `waiting_for_user` with `userAction: "budget"`. The notice
now shows "Couldn't finish this attempt. Your message is saved. It is
waiting for the budget to allow new work. Raise the budget in Costs."
(`data-testid="task-execution-waiting"`). It shows no second Retry,
because the saved Retry starts by itself. Other person-owned decisions,
such as approvals and run-stopped confirmation, keep their own surfaces.
- `deferred-execution-wait.ts`: when the wait reason of a deferred
request changes, its task records an `issue.execution_waiting` activity.
Its live event refreshes an open task page. Callers inside a transaction
(`heartbeat/queue.ts`) pass their post-commit publication list, so the
event is published only after commit.

## Verification

- A probe on the real deferred-receipt fixture showed that a budget-held
Retry presents as `waiting_for_user` / `budget` / `canRetry: false` at 0
and 11 seconds. Before this change, the notice rendered nothing for that
state.
- `ExecutionBlockerNotice.test.tsx`: 18 tests pass. The budget-held
notice is covered, it offers no duplicate Retry, and approvals still
render nothing here.
- `deferred-execution-wait.test.ts` (new, embedded PostgreSQL):
- One live event is published for each new wait reason. The same reason
again publishes nothing.
  - Inside a caller's transaction, nothing is published before commit.
  - A request that is no longer deferred records nothing.
  - Two of these cases fail when the change is reverted.
- `deferred-execution-receipt-work.test.ts`: passes (49 tests together
with the new file).
- `pnpm check:token-gates`: clean.
- Paid live run: `recovery.retry-exhausted` on an eval branch that
includes both commits. The run ID will be added below.

## Risks

- Low. The new notice renders only for `waiting_for_user` with
`userAction: "budget"`, which rendered nothing before.
- The activity row is written only when a deferred request's wait reason
changes, so a gate that refuses repeatedly adds one row, not one row for
each evaluation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:52:49 -06:00
DottaandClaude Opus 5.5 d10c6e0e8b fix(chat): hold a chat message at the agent's daily limit instead of dropping it (#20292)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent Chat sends each person's message to the conversation agent
through a durable outbox (`deliverConversationComments`). The outbox
keys each wake receipt as `conversation-comment:<commentId>`.
> - The outbox treats any receipt with that key as delivered. Queue
admission writes a `skipped` receipt when the agent reached its daily
run or cost limit.
> - So a chat message sent after the agent used its daily allowance was
consumed silently. No run started, the chat showed nothing, and nothing
resumed it.
> - A person's message must never be silently dropped. The daily limit
is an operator control, so the message must wait honestly, not bypass
the limit.
> - This pull request makes the outbox hold the message at the daily
limit, like it already does for a paused agent, and resume it when the
limit resets.
> - The benefit is that a chat message at the daily limit is saved,
shown with its reason, and sent automatically at 00:00 UTC.

## Linked Issues or Issue Description

No public issue exists. Inline bug report:

**What happened**

In a chat eval, the conversation agent (CEO) had `maxDailyRuns: 3`.
After three chat turns and a separate failed-model phase on another
agent, the person sent a fourth message. The comment was saved (POST
201), but no CEO run started. The conversation stayed `active` with no
`executionRunId`. The ownership reconciler later reported
`I7_visible_next_action`. The server log shows no wake error. Queue
admission had written a `skipped` receipt (`heartbeat.daily_run_limit`)
under the outbox key, so the outbox counted the message as delivered.

**Expected behavior**

A chat message that reaches the agent's daily limit is not consumed. It
stays saved, the chat shows why it waits, and it is sent automatically
when the limit resets. The daily limit still applies: the message does
not start a run early.

**Steps to reproduce**

1. Set a conversation agent's `runtimeConfig.heartbeat.maxDailyRuns` to
N.
2. Send N chat messages and let each turn finish.
3. Send one more chat message.
4. See that no run starts, the chat shows no waiting state, and the
message is never answered (a `skipped` wake receipt with reason
`heartbeat.daily_run_limit` exists for `conversation-comment:<id>`).

## What Changed

- `server/src/services/agent-conversations.ts`: add
`conversationDeliveryHold`, which applies the same daily run/cost limit
check as queue admission. The outbox returns `deferred` at the limit and
writes no receipt, so the message stays owed. It reports the next UTC
day start through a new optional `onDeferredDeadline` option.
- `server/src/services/recovery/service.ts`: stranded-work recovery
passes its `onDeferredDeadline` to the outbox, so the held message is
retried when the limit resets.
- `server/src/services/heartbeat/queue.ts`: if the limit is reached
while an outbox wake is in admission (a race), admission throws a 409
with `reason: heartbeat.daily_*` for `conversation-comment:` keys
instead of writing a `skipped` receipt. The outbox already maps a
matching 409 to `deferred`. Other wakes keep the current skip behavior.
- `server/src/routes/issues.ts` (`GET /issues/:id/queued-comments`):
when a chat has no queue receipt but has messages held by the daily
limit, return them as a `deferred` queue with `executionWait` (reason
and message). Edit, send and discard stay off because the outbox owns
delivery.
- `ui/src/components/task-chat/TaskChatQueuedMessages.tsx`: show the
`executionWait` message for such an outbox-held queue (no queue id and
no target run). Receipt-backed queues keep the current behavior and do
not duplicate the task status.
- Tests: three real-database tests in
`agent-conversation-delivery.test.ts` (hold and resume after the limit
resets, recovery deadline, and a limit reached during real
`heartbeatService.wakeup` admission) and one UI test.

## Verification

- `cd server && pnpm exec vitest run
src/__tests__/agent-conversation-delivery.test.ts`: 10 passed. Without
the source change, the three new tests fail (`expected 'delivered' to be
'deferred'`; the real queue path writes the skipped receipt).
- `cd server && pnpm exec vitest run
src/__tests__/issue-recovery-conversations.test.ts
src/__tests__/agent-conversations.test.ts
src/__tests__/heartbeat-undispatched-execution-wait.test.ts
src/services/heartbeat/queue.test.ts
src/__tests__/agent-chat-messages.test.ts`: 215 passed.
- `cd ui && pnpm exec vitest run
src/components/task-chat/TaskChatQueuedMessages.test.tsx`: 36 passed.
- `pnpm exec tsc --noEmit -p server/tsconfig.json`: clean. `node
scripts/check-token-gates.mjs`: clean.
- Manual: give a chat agent `maxDailyRuns: 1`, send two messages. The
second message shows in the queued messages bar with "<agent> reached
its daily run limit. Your message is saved and sends automatically when
the limit resets at 00:00 UTC."

## Risks

- Behavior change for chat only: a chat message at the daily limit now
waits in the outbox instead of being consumed. It is sent after 00:00
UTC. If an operator raises the limit, the message is sent on the next
recovery pass for that chat; the UI message does not promise an earlier
send.
- Saved state from the previous release: no persisted shape changes.
Messages that an earlier release already consumed with a `skipped`
receipt stay consumed. They are not replayed later, which avoids sending
old messages without notice.
- The queued-comments read adds one agent read and one daily-limit count
for a chat with no queue receipt. This is the same query that admission
runs.
- The recovery deadline is in-process; instance-sleep scheduling does
not get a new deadline from this change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:03:28 -06:00
DottaandClaude Opus 5.5 c5c587f162 fix(issues): start a deferred children-completed wake the release drain missed (#20180)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent that delegates waits for its children, and is woken when
they finish (`issue_children_completed`)
> - Restart chaos t06 ended with both children done and the parent
blocked forever
> - The children finished while the parent's run was still going, so the
wake was deferred as a follow-up. That run then ended with the agent
setting the parent blocked, and the deferred wake was never started
> - The durable re-drivers for deferred wakes cover only
`issue_blockers_resolved`, so nothing else picks up a missed
`issue_children_completed`
> - This pull request makes both re-drivers cover children-completed
wakes too
> - The benefit is that a parent always hears that its children
finished, even when the release drain misses the deferred wake

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261109T041248-s9501`, task t06 (claude, M1, K2), Daytona per-turn,
master fa7950121d.

**Problem:** parent CHA-10 (all times UTC):
- 04:29:57: run 9cf49f52 started.
- 04:32:35 and 04:32:43: its children CHA-13 and CHA-14 finished.
- 04:32:43: `issue_children_completed` merged into the running turn.
- 04:32:53: that wake was deferred as a follow-up
(`issue_execution_deferred`, `deferred_issue_execution`, no run).
- 04:33:13: the turn ended with the agent setting the parent `blocked`,
a decision made before it saw the children's results.

The deferred wake stayed deferred for the rest of the wave, including
three later restarts. The parent sat blocked with no Retry. The
deferred-receipt worker (`deferred-receipt-work.ts`) and the
stranded-queue resume (`queue.ts`) select deferred dependency wakes with
`issueBlockersResolvedWakePayloadSql`, which matches only
`issue_blockers_resolved`.

## What Changed

- `server/src/services/issue-dependency-wakeups.ts`:
`dependencyResolvedWakePayloadSql` and `isDependencyResolvedWakePayload`
replace the blockers-only predicates. They match both
`issue_blockers_resolved` and `issue_children_completed` in
`_paperclipWakeContext.wakeReason`.
- The deferred-receipt worker and the stranded-queue resume use them, so
a deferred children-completed wake is started once the parent has no
live run, the same way a blockers-resolved one is.
- No code comments added.

## Verification

Red/green:
- `deferred-execution-receipt-work.test.ts` "starts a deferred
children-completed wake left behind when its parent's run blocked the
parent (restart chaos t06)". It sets up a blocked parent whose last run
succeeded, a done child, and the deferred `issue_children_completed`
receipt as t06 left it.
  - On master: the receipt stays `deferred_issue_execution` (red).
- With this PR: the receipt completes with exactly one
`issue_children_completed` run.
- `deferred-execution-receipt-work` and
`heartbeat-dependency-scheduling` pass (60 tests). `tsc --noEmit -p
server` is clean.

## Risks

- The workers now also select deferred children-completed wakes.
Admission is unchanged: they still pass the queue's own gates, and they
start only when the issue has no live run, the same as blockers-resolved
wakes.
- No schema change.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#20069, #20108)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user docs affected)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:14:47 -06:00
DottaandClaude Opus 5.5 5a51a7685d fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop cancels the scope's running work; the board
resolves it with Raise budget and resume
> - In the restart-chaos H1 scenario, the cancelled tasks did not
continue after resume and each needed a manual Retry
> - Resume is the board's explicit decision to continue, so the work it
stopped should continue once, as if the board clicked Retry
> - This pull request files that Retry automatically, keyed by the
cancelled run, once the run's stop has settled
> - The benefit is that a budget raise restores the work without manual
Retries, without repeating completed work, and without charging the
run's attempt budget

## Linked Issues or Issue Description

Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101
(superseded: split into #20143 and this PR)

**What happened?**
Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802
(H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run
(enforcement stop) and CHA-1's run (agent lifecycle pause from the
budget hold) stayed cancelled until someone clicked Retry.

**Expected behavior**
Resuming continues each task its budget stopped, exactly once, with no
attempt charged, an activity entry, and no repeat of work that already
continued. A run a person stopped stays stopped.

**Steps to reproduce**
Set a budget below spend while a task runs; let the stop cancel the run;
resolve with Raise budget and resume. Before this change, no new run is
queued.

## What Changed

- New table `budget_resume_claims` (migration 0451). One row per
cancelled run, outside the run's result JSON, so a Stop that settles
during the resume cannot erase it.
- `markBudgetCancelledWorkForResume` (called by `resolveIncident` before
the incident closes): claims every task run its budget stopped since the
incident opened. That covers runs budget enforcement fenced, including
stops still in flight, and runs the agent lifecycle pause stopped for
this budget (`pauseStop`, carried from the lifecycle operation's
`pauseCause` so a resume before the Stop keeps the link). Runs a person
stopped are excluded. Two indexed passes led by `(company_id, status)`.
- `deliverBudgetResumeContinuations`: once a claimed run's stop settles,
files the resolving board member's Retry (`requestTaskRetry`,
idempotency key `budget-resume:<run>`, with the member's session source
and the chat Retry callback). The Retry path checks, under the task
lock, that the run is still the task's latest and the task is open, so
continued or completed work is never repeated. Claim settlement and its
activity entry commit together; retryable refusals and errors retry.
- Delivery runs in the always-on budget-enforcement worker after each
pass's Stop delivery. Its schedule includes deliverable claims and the
earliest stale-claim expiry. The execution-continuation worker is a
backstop.
- `agent.paused` / `agent.resumed` logging is in #20113.
`budget.cancelled_work_resume_requested` (incident) and
`budget.cancelled_work_continued` / `_not_continued` (task) are recorded
here.

## Verification

- New `budget-resume-continuation.test.ts` (Postgres, real
`resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry`
into a queued run):
- each task continues once, including a blocked task; repeat delivery or
resolve saves nothing new;
- a stop still in flight at resume, whose terminal write replaces the
result JSON, is still continued;
  - a 30-task scope is fully continued;
- a lifecycle-pause stop caused by the budget is continued, and one
caused by a manual pause is not;
- continued work, done tasks, a run that finished before its stop, and
person-stopped runs (including ones enforcement also fenced) are not
continued.
- `lifecycle.test.ts`: the pause records its cause on the operation and
on stopped runs, including when the budget is resumed before the pause's
Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery,
and a wake at claim expiry.
- 287 tests across the affected suites passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun (Phase 3 runs it after merge).

## Risks

- Resume now continues budget-stopped tasks automatically. A board that
wants a task stopped can Stop it again; person-stopped runs are never
continued.
- A new table and one index (migration 0451).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:56:28 -06:00
DottaandClaude Opus 5.5 c49ab6b9ee feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task delegates to children and waits on blockers. Its agent
learns of progress only through wakes
> - Today only completion (`issue_children_completed`) and resolution
(`issue_blockers_resolved`) wake a waiting issue
> - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on
a recovery action that needs the board, and its parent sat silently
blocked for 50+ minutes
> - This pull request wakes the waiting agent once per such stall, tells
it what it can do, and shows the stall on the parent's page with the
child's Retry
> - The benefit is that a stuck child or blocker no longer silently
strands its parent

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan; restart chaos campaign). Restart
chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude).

**Problem**

From the kept database:
- CHA-8 `blocks` CHA-5.
- CHA-8's board-owned recovery action (`native_session_retry_exhausted`)
was created at 18:44:02.909 and resolved with outcome `blocked` at
18:44:08, when CHA-8 moved to `blocked` for the board.
- CHA-5's only wake in the whole wave was its `issue_assigned` at
18:20:48. Its agent never heard about CHA-8.

**Expected behavior**
- When a child or blocker stalls on the board, the waiting agent is
woken once. It can then re-delegate, escalate, or block with a reason.
- The board sees the stall on the waiting task, with the child's Retry.

## What Changed

- **Shared:**
- `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`),
`ISSUE_BLOCKER_STALL_RELATIONS`, and the types
`IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`;
  - `Issue.stalledDependencies`.
- **Server, `services/recovery/blocker-stall-wakeups.ts`:**
- A stall is an issue that is not done or cancelled, whose latest
recovery action needs the board. That means either open
(`active`/`escalated`) and board-owned or escalated, or board-owned and
resolved as `blocked` while the issue is still `blocked`. The second
case is what t05 needed: recovery resolves its action that way when it
hands the issue to the board.
- Waiters are the parent, and every issue the stalled one `blocks`, when
they are open, visible and agent-assigned.
- Each waiter gets one wake per stall. The idempotency key
`issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked
through the indexed `(company_id, payload->>'issueId')` path. A later
stall has a new action id, so it wakes again.
  - Each wake writes activity `issue.blocker_stalled_wake`.
- The pass runs from the recovery-action sweep
(`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep
(`reconcileResolvedDependencyWakeBackstop`, which visits each blocked
issue). No new worker.
- A waiter's agent that cannot read the stalled issue (`issue:read`,
which honors private visibility) is woken without its title, identifier
or recovery details; the activity entry withholds them too.
- **Database:** migration `0450` adds the partial unique index
`agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id,
idempotency_key)` for `issue_blocker_stalled:` keys in delivered or
pending statuses, so two concurrent sweeps cannot both deliver a stall.
It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed
in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`).
- **Queue:** `allowsDependencyBlockedWake` (in
`run-dispatch/domain/wake-context.ts`) lets a stall wake run while the
waiter's blockers are unresolved, at both dependency gates. Without it
the wake was parked until the stalled blocker resolved, which is never.
Pause holds are unchanged.
- **Agent guidance:** the wake payload carries `blockerStall`, and
`renderPaperclipWakePrompt` adds a "Stalled dependency" section. It
names the task and its recovery action and the board's next action, then
the three choices: re-delegate, escalate with a comment, or block with a
reason naming the stalled task.
- **Other reason lists:** `issue_blocker_stalled` joins the
responsible-user wake reasons, the conversation execution-wake reasons
(it never drives a chat turn) and the wake diagnostic's known reasons.
- **Board surface:**
- `GET /issues/:id` returns `stalledDependencies`, readable issues only
(`readableIssueEdgeIds`).
- `StalledDependencyNotice` shows on the issue's chat notices, next to
`IssueBlockedNotice`: each stalled child or blocker, its board next
action, a link, and the child's own Retry (`useTaskRetry`), offered only
when the child's interruption allows it.

## Verification

Red/green:
- `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests).
  - On master: no `issue_blocker_stalled` wake is raised.
- With this PR: the parent is woken once per stall, and re-woken after a
new stall. A blocker's dependent is woken.
- The t05 shape (an action resolved as `blocked`) wakes once, and
nothing more after a Retry.
- No wake for an agent-owned action, a done, unassigned or hidden
waiter, or an action resolved another way.
  - `listStalledDependencies` lists children and blockers.
- `heartbeat-dependency-scheduling.test.ts` "admits an
issue_blocker_stalled wake while a blocker is unresolved". On master the
wake was parked (`null`); once admitted, the run-start gate cancelled
it. With this PR it runs with `dependencyBlockedInteraction`. The file:
14 passed.
- `packages/adapter-utils` `server-utils.test.ts`: the new prompt test
failed on master (no `blockerStall`). The file: 141 passed.
- `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed.
`IssueChatThread.test.tsx`: 109 passed.

Regression and static checks:
- Nearby server suites (recovery durable wait, dependency wake routes,
recovery actions, run-preparation): 149 passed.
- `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`.
- `pnpm check:token-gates` is clean.

Review round 1 (Greptile, Superagent), each with a regression:
- A backlog waiter is never woken. The waiter's status and assignee are
guarded at enqueue.
- One stall per run: no coalescing, so a merged run cannot keep only the
last stall's context.
- Only a delivered or pending wake ends a stall. A refused one
(wake-on-demand off) is retried after 10 minutes, and an agent that
takes no wakes is skipped without writing a row.
- Company-wide passes resume after the last stall a capped pass reached.
The per-issue sweep also checks the issue's own children and blockers,
which covers an unassigned stalled blocker.
- The dependency gates no longer trust the reason string.
`verifyBlockerStallWake` requires a system request, the stalled issue's
current stall action, and a real child or blocker edge. An
agent-requested wake with the same context is parked (tested).
- The notice shows a refused Retry's reason, and a lost response as
"Taking longer than usual…", with Retry still available (tested).

Review round 2 (Greptile), each with a regression:
- Race: two passes claiming one stall leave one delivered row (unique
index; the loser's insert is skipped).
- The company recovery sweep selects companies whose only stall is a
resolved-`blocked` board action, so an unassigned stalled blocker with a
todo waiter is reached.
- Private stalled issue: the waiter's wake and activity carry no title,
identifier or recovery details.
- `GET /issues/:id` always returns `stalledDependencies`, so a cleared
stall replaces the cached list.
- The notice shows a failed child read with a Reload button (tested).
- `pnpm --filter @paperclipai/db check:migrations` passes.

Review round 3 (Greptile), each with a regression:
- A stall wake that fails to enqueue is retried: the recovery-action
pass reports `blockerStallWakeFailures` and the dependency pass counts
it in `enqueueFailed`, so the worker throws and retries at its backoff
(`issue-recovery-work-lifecycle.test.ts`).
- The notice drops a stall once the child's refreshed status has moved
on (`StalledDependencyNotice.test.tsx`, 7 tests).

## Risks

- A deploy wakes the waiters of every issue that is stalled today, once
each. That is intended (they are stranded now), and it's bounded at 200
stalls per pass.
- A stall wake may run while blockers are unresolved, like a comment
wake. It is bounded to one per stall, and pause holds still apply.
- The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet
list the new reason. The wake prompt carries the guidance.
- One additive index, built concurrently; no persisted run state change.
Existing rows cannot violate it before deploy, since no
`issue_blocker_stalled:` keys exist yet.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 18:26:07 -06:00
DottaandClaude Opus 5.5 c4ba3974a7 fix(heartbeat): close run admission before a graceful shutdown closes its listener (t09) (#20102)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A graceful restart (K2) drains the control plane: it closes HTTP
admission, suspends running runs, and the next boot resumes them
> - Run admission stayed open while the HTTP listener closed.
Heartbeat's shutdown flag was set only later, inside hot-restart
preparation
> - A run that finished in that window released its issue and promoted
the queued comment wake. That started a new run that had no callback
listener
> - A remote warm continuation then failed with
`local_runtime_api_listener_unavailable`, and the task needed a Retry
(restart chaos t09)
> - This pull request closes run admission before the listener closes,
so the promoted wake stays queued for the next boot
> - The benefit is that a comment sent during a deploy continues on its
own after the restart

## Linked Issues or Issue Description

Refs #19596

**What happened?**
In restart-chaos wave w-20261108T192459-s9501 (Daytona warm, Codex C1,
K2), a K2 SIGTERM started a graceful drain at 19:42:52 UTC, and the HTTP
listener closed first. At 19:43:03 the running turn finished and
released its issue. That promoted the queued comment wake, and
`startNextQueuedRunForAgent` started a new run about 11 s into the
drain. Backend setup resolved the local runtime API URL, found the
listener closed, and threw `local_runtime_api_listener_unavailable`. The
remote target is excluded from the closed-listener handoff, so the run
failed, and the task waited for a manual Retry.

**Expected behavior**
No queued run starts once a graceful shutdown has begun. A wake promoted
during the drain stays queued, and the next boot starts it with no
Retry.

**Steps to reproduce**
1. Run a remote warm task and post a comment while its turn runs, so the
comment wake queues behind it.
2. Send SIGTERM to the control plane just before the turn finishes.
3. Before this change, a new run starts during the drain and fails with
`local_runtime_api_listener_unavailable`.

**Paperclip version or commit**
master at d007afbf86 (the wave ran on c5a172a183).

## What Changed

- `heartbeat/run-admission.ts` (new): a process-wide run admission gate
(`closeProcessRunAdmission`, `isProcessRunAdmissionClosed`). HTTP routes
build their own heartbeat instances, so the gate cannot live in one
instance.
- `heartbeat/queue.ts`: a new optional dependency,
`isRunAdmissionClosed`, which defaults to the process-wide gate.
`startNextQueuedRunForAgent`, `resumeQueuedRuns` and
`resumeQueuedRunsAfterDrain` start nothing (and read nothing) once it
returns true. Wakes are still recorded and promoted; only the start is
held.
- `heartbeat.ts`: also honors its own `shutdownInProgress` flag.
- `shutdown.ts`: `closeHttpListenerForShutdown` takes
`closeRunAdmission` and calls it first, before it checks or closes the
listener. `index.ts` passes `closeProcessRunAdmission`.

## Verification

- Red/green, new tests:
- `queue.test.ts`: "starts no run once shutdown closed run admission"
(for `startNextQueuedRunForAgent` and `resumeQueuedRuns`), and "honors
the process-wide run admission gate in every heartbeat instance".
- `shutdown.test.ts`: "closes run admission before it stops accepting
requests".
- `pnpm exec vitest run src/services/heartbeat/queue.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/services/native-runtime/native-startup-retry.integration.test.ts
src/shutdown.test.ts` (server): 5 files, 277 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (server): passed.

## Risks

- Low risk. Only queued-run starts are held during a shutdown that has
already begun; wakes, drains, suspension and finalizers are unchanged.
The next boot's queued-run dispatch starts held runs.
- Known remaining window: a run admitted just before shutdown can still
reach backend setup after the listener closed. A local runner hands off
to restart recovery there. A remote warm continuation still fails,
because the remote target is excluded from that handoff (#19596) and the
bootstrap retry does not cover warm continuations. This change removes
the t09 sequence, where the run started after shutdown had begun; it
does not remove that narrower window.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution, a read-only research subagent).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:59:55 -06:00
DottaandClaude Opus 5.5 8475826d2d fix(wake-queue): deliver user comments whose wake a restart lost or a stale continuation dropped (#19992)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user's follow-up comment on a task must reach the agent exactly
once, through restarts (invariant 4)
> - The final unmasked local chaos storm (96 tasks, 31 restarts) passed
94 tasks. The 2 failures were both lost user comments
> - In t69, a restart killed the server between saving the comment and
writing its wake. In t72, a comment's own run was cancelled as stale,
because the earlier run completed the task first without reading the
comment
> - This pull request adds a reconciler rule that gives a lost comment
its wake, and saves the comment before a stale cancellation
> - The benefit is that a user's message is never silently dropped when
a restart or a completion race hits at the wrong moment

## Linked Issues or Issue Description

Refs #19956 (work-intent outbox), #19944, #19831 (plan, invariant 4).

**What happened?**
In chaos wave `w-20261108T055444-s9401` (local, unmasked, codex and
claude, 30-min K13 storm, master `46bf913fb9`), 94 of 96 tasks passed.
Two failed:
- **t69, R1 `rested_not_done`.** The user's comment was saved at
06:14:59.358 and reopened the task to `todo`. A K1 SIGKILL fired 28 ms
later, before the wake was written. No wake row exists. Recovery skips a
`todo` task whose last run succeeded, so the task sat with no run until
the wave ended.
- **t72, R4 `concurrent_runs`.** A comment posted during a run queued
its own run. That run was claimed at 06:16:04.874. The first run's
completion marked the task `done` 2 ms later. The comment's run was then
cancelled with `continuation_task_ownership_changed` before dispatch,
and the comment was lost.

**Expected behavior**
Each accepted user comment is delivered once, unless the task is
cancelled or reassigned, or a person closes it.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--replay w-20261108T055444-s9401/t69` (or `.../t72`).

## What Changed

- `recovery/ownership-invariant-report.ts`: a new reconciler-owned I4
rule, `user_comment_without_wake`. It finds a user comment that meets
all of these conditions:
  - it is newer than every run of the task;
- the task is assigned, is not a conversation, has no live run or lock,
and has at least one earlier run;
  - no wake request carries the comment.

A comment younger than 30 s is not reported yet, because it may still
get its normal wake. Its end-of-grace time is returned as `nextDueAt`.
- `native-runtime/native-finalization-reconciler.ts`: the backstop step
is now exported as `applyOwnershipReportBackstops`. For each such
comment it saves the missing outbox receipt (`executionWait: wake_lost`)
and promotes it.
- `work-intent-outbox.ts`: new `saveUndeliveredUserComments`, which the
reconciler rule and the stale-run path both use. It saves only live user
comments that no completed run received. It writes nothing for a task
that is reassigned, is a conversation, or has a status the caller does
not allow.
- `heartbeat/queue.ts`: the saved-receipt pass that already sends
pause-saved messages also sends `wake_lost` receipts. It uses the same
per-comment consumption check and the same chat-input exclusion.
- `heartbeat/stale-comment-continuation.ts` and `heartbeat.ts`: before a
stale continuation is cancelled, its undelivered user comments are saved
as a deferred receipt. The release drain then reopens the `done` task
and delivers them once. This applies only when an agent run that started
before the comment completed the task after it. A task that a person
marks done is left closed.
- `tests/restart-chaos/grader.ts`: R4 ignores runs cancelled before
dispatch (`continuation_task_ownership_changed`) when it looks for
overlapping execution. Those runs never executed. Comment consumption
still counts them.

## Verification

Each focused file ran alone in one process (`--maxWorkers=1`):
- `heartbeat-held-comment-wake.test.ts` (22 pass). This includes two new
cases:
- t69 shape: the reconciler rule fires, the backstop delivers the
comment once, and a second pass does nothing;
- t72 shape: the stale run is cancelled, the task reopens, and the
comment is delivered once.

  Both fail with the rule or the save disabled, and pass with them.
- `ownership-reconciliation-report.test.ts` (23 pass). It includes the
rule's boundaries: a carried comment, a too-young comment (deadline
only) and an already-read comment.
- `tests/restart-chaos/grader.test.ts` (17 pass). The new R4 case fails
with the old grader. A cancelled run that is not a pre-dispatch
cancellation still fails R4.
- Unchanged and passing: comment wake batching (27), refused assignment
wake (3), wake-queue postgres adapter (103), stale queue invalidation
(68), deferred receipt work (45). The first version also reopened a task
that a person had marked done, which broke one deferred-receipt test.
That case is now excluded.
- The server package typecheck and the restart-chaos harness typecheck
pass.
- Same-session chaos context on master: K6 crash loop
`w-20261108T063326-s9402`: 6 of 6 pass. The K4 long-hold wave is still
running. After merge, I'll replay only the targeted fault shapes (t69
and t72).

## Risks

- The reconciler rule runs on each report pass. It is bounded by the
1-hour lookback and the per-kind row cap, and it excludes conversation
tasks, so chat input keeps its own authorization.
- A comment younger than 30 s waits up to the end of its grace period
before the backstop acts.
- The stale-run save reopens a `done` task only for a comment that the
completing run could not have read.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 02:14:45 -06:00
DottaandClaude Opus 5.5 d33b3ac263 fix(wake-queue): save temporarily refused work intents and promote them through one path (#19956)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Users send work to agents as assignments, comments and Retries. The
wake queue delivers each one as a run
> - When an admission check refused a wake for a temporary reason, the
wake often ended as a terminal skip, so the work was lost
> - Restart-chaos waves found dropped comments and Retries (RC-37,
s7532) and double delivery (RC-53). Each fix covered one hold type
> - The durable ownership plan (section E) asks for one outbox rule: a
closed set of terminal outcomes, a saved deferral for every temporary
refusal, and one promotion path
> - This pull request adds that rule. It saves messages refused by a
pause, promotes deferred intents from every hold-release event through
one function, and adds a reconciler backstop
> - The benefit is that an accepted message is delivered exactly once
after a pause lifts, and the reconciler reports any intent that is stuck
or dropped

## Linked Issues or Issue Description

Refs #19831 (plan, section E), #19838 (inventory, wake-queue rows and
section 1b), #19927 (Phase 2 data: the I4 false positive), #19702,
#19741, #19829, #19826. Builds on #19944 (section D, merged).

**What happened?**
- A user commented while the agent was paused, or while the company was
paused. Admission refused the wake. Nothing saved the message as
pending, so when the pause lifted, nothing delivered it.
- Hold-release events woke different subsets of workers. A saved intent
then waited out a 30-second refusal backoff after its hold cleared.
- The I4 reconciler check matched free text with a regular expression.
Phase 2 found a false positive: a delivered retry whose error said
"retrying once".

**Expected behavior**
An accepted work intent ends only as delivered, consumed by a later run,
task cancelled, or hard stop. Every temporary refusal saves the intent
as `deferred` with its reason and its release trigger. Every
hold-release event promotes deferred intents through one function
(invariant 4).

**Steps to reproduce**
Pause an agent. Post a user comment on its task. Resume the agent. On
`master` the comment is never delivered as a wake.

## What Changed

- New `server/src/services/work-intent-outbox.ts`:
- `WORK_INTENT_TERMINAL_OUTCOMES`: the closed set of terminal outcomes.
- `TEMPORARY_REFUSAL_TRIGGERS`: maps each temporary refusal code to its
release trigger.
- `promoteDeferredWorkIntents`: the one promotion function. It removes
the 30-second refusal backoff from the matching deferred intents. It
then wakes run dispatch, the saved-receipt worker and stranded-work
recovery, and hints each saved intent of the issue.
- `heartbeat/queue.ts`: the #19829 save-as-deferred block is now a
shared helper. A user's message refused by a paused agent, a paused
company, or a budget pause that is not a reached hard stop is saved with
`executionWait: { reason, trigger, message }`. The caller still gets a
409. The message is saved only when the task's latest run is terminal,
because the release drain promotes a saved message behind that run. A
task that never ran keeps the message in the task, and stranded-work
recovery starts the task once the pause lifts. A `pending_approval`
agent and a task-tree hold are not changed here: the release drain still
fails a promoted wake for a pending approval, and tree holds have their
own resume path.
- Hold-release events call `promoteDeferredWorkIntents`:
  - accounting settled (`run-cost-accounting.ts`);
  - environment lease released (`environments.ts`, two sites);
  - native run projected terminal (`native-run-projection.ts`);
  - budget pause lifted (`budgets.ts`);
  - company reactivated (`companies.ts`);
  - task-tree hold released (`issue-tree-control.ts`);
  - agent lifecycle ready (`agent-lifecycle-driver.ts`).
- Reconciler (`recovery/ownership-registry.ts`,
`ownership-invariant-report.ts`, `native-finalization-reconciler.ts`):
- registers three new kinds: `wake_request_deferred`, `queued_run` and
`scheduled_retry_run`;
- I4 now reads the closed refusal code (the wake `reason`) of accepted
intents, not free text;
- a new I4 check reports a deferred pause refusal whose pause has lifted
for more than 5 minutes. The worker then promotes it as a backstop;
- a lifted pause inside its 5-minute grace window returns its deadline
through `nextDueAt`, and the worker also wakes on agent-lifecycle
signals (company and agent pause changes). So a missed release event is
still checked.
- `doc/SPEC-implementation.md`: adds the rule "Accepted work intents are
delivered exactly once".

## Verification

Each focused file ran alone in one vitest process (`--maxWorkers=1`) on
embedded Postgres. Results at `2875c85fd2`:

| Question | Loop | Elapsed | Result |
|---|---|---|---|
| Is a comment refused by a pause saved, then delivered once after the
pause lifts? | `heartbeat-held-comment-wake.test.ts -t "pause lifts"` |
8 s | 2 pass. With `queue.ts` from `master`, both fail: no intent is
saved |
| Does I4 use the closed code, and does it catch a stuck deferral? |
`ownership-reconciliation-report.test.ts` | 45 s | 20 pass. This
includes the Phase 2 "retrying once" false positive and the case
"paused, then lifted" |
| No regression on the paths that changed | held-comment (15),
archived-company guard (8), undispatched execution wait (32), deferred
receipt work (45), recovery pause durability (16), native run projection
work (55), tree-control routes and services (19, 1, 7) | 8–334 s each |
All pass |

- Exactly-once: each pause test runs two promotions (a release event,
then the reconciler backstop) and many dispatch passes. It asserts one
delivery. The existing consumption tests (RC-53) still pass.
- Server package typecheck passes.
- Speed: nothing new runs before `turn.start`. The new admission work
happens only on a refusal. Promotion runs in the release transaction and
in the background reconciler.
- Not run here: chaos waves. The coordinating session schedules them on
the shared harness.

## Risks

- `promoteDeferredWorkIntents` moves `updated_at` back for deferred
intents that were refused in the last 30 seconds. A busy company with
many release events can retry refused intents more often. Each retry
re-runs normal admission, and the refusal resets the backoff.
- More release sites now wake run dispatch and stranded-work recovery.
These notifications are coalesced per company.
- Lifting a pause is now the operator's explicit authorization to send
what the pause refused, as Send is after a Stop. When the pause stopped
the task's run, the lifted pause issues the task's next explicit wake,
which adopts the saved message once. Other messages queued behind an
acknowledged Stop keep the existing rule. If Stop recovery still holds
the task, the message waits for that recovery.
- An agent pause's stop sweep keeps the messages that the pause deferred
(`preservePauseDeferred`). Terminate and cleanup stops still end them.
- The backstop check reads only pause refusals (`company.inactive`,
`agent.not_invokable`), because those holds can be checked exactly.
Other deferred kinds keep their existing promotion paths.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:43:17 -06:00
DottaandClaude Opus 5.5 18644a3263 feat(recovery): instrument 20 durable write boundaries with crash points (#19938)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - #19937 added `crashPoint(name)` and a registry of 20 named
boundaries from the Phase 0 inventory
> - A name without a call site cannot fire, so the crash-point suite
cannot test it
> - This pull request puts one call at each registered boundary and adds
a test that keeps the registry and the call sites in step
> - The benefit is that the on-demand suite (next pull request) can kill
the control plane at every known write-ahead gap

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead
coverage"). Stacked on #19937 (helper and registry). Refs #19861
(ownable kind ids).

**Problem**
The registered crash points have no call sites.

**Expected behavior**
Each registered point is called at its gap: after the first durable
write or outside effect returns, before the next one starts. Disarmed
(always in production) each call is one comparison.

**Scope**
Call sites and a coverage test only. The suite runner and CI workflow
follow.

## What Changed

- Server call sites (`@paperclipai/shared/crash-points`):
- `sandbox-creation-operations.ts`: around the provider `create` call
(intent → provider → lease).
- `native-session-executor.ts`: after `native.process_start_requested`,
before spawn.
- `paperclip-control-plane-port.ts`: after the result-accept
transaction, before finalization starts (only when a result was
inserted).
- `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before
the run projection.
- `workspace-operations.ts`: `workspace_finalize` only — after the
running row, and after `settle()` before the done row.
- `environment-runtime.ts`: provider release → lease row; teardown claim
→ destroy; reusable destroy → lease release; stop-and-retain claim →
stop.
- `heartbeat.ts`: lease release → runtime services; budget reservation
hold → dispatch.
- `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`.
- `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit →
spool unlink; receipt spool → DB row → unlink.
- `packages/paperclip-runner/src/crash-point.ts`: a small copy of the
helper for controller code in this package, which does not depend on
`@paperclipai/shared` (adding that dependency would change the runner
content ID). It arms only `runner.*` names. runnerd never sees the
variable: its environment is an allowlist.
- Runner call sites: after the runnerd spawn, before its identity is
published (both spawn paths); `turn.start` result received → controller
journal saved → next send.
- `server/src/__tests__/crash-point-call-sites.test.ts`: every
registered point has a call site, every call names a registered point,
and every `kinds` entry is an ownable kind id of the Phase 1 registry.

## Verification

- `cd server && npx vitest run
src/__tests__/crash-point-call-sites.test.ts` → 3 passed.
- `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter
@paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean.
- Behavior with nothing armed is unchanged: each call returns at its
first comparison (measured in #19937: about 3.7 ns).

## Risks

- Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT`
names it and `NODE_ENV` is not `production`.
- Points on the path before `turn.start` (process start, runner spawn)
cost nanoseconds, within plan invariant 8.
- `usage-receipts.ts`: the persist-then-unlink one-liner became a block
with the same statements in the same order.
- The runner package change gives the runner a new content ID, as any
runner source change does. No persisted run state changes
(rules/upgrade-safe-run-state.mdc).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits) and a read-only exploration subagent that
mapped the boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:56:33 -06:00
DottaandClaude Opus 5.5 a94881d1a7 fix(heartbeat): save a comment refused by a temporary hold and route post-quarantine input to its own gate (#19829)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user's follow-up comment or Retry must be delivered once,
eventually, even when a temporary company admission hold (accounting
still settling) is active
> - #19741 kept claimed runs queued through such a hold and woke held
work when it cleared
> - A restart-chaos wave still lost a comment, because its wake was
refused before admission and written as a terminal skip
> - The same wave left a Retry waiting forever for a stop record of a
process that never existed
> - This pull request saves the refused comment as a deferred receipt,
and routes input after a pre-launch quarantine failure to the
quarantine's own gate
> - The benefit is that the comment is delivered after the hold, and the
task asks for the action that actually unblocks it instead of offering a
Retry that can never start

## Linked Issues or Issue Description

Refs #19741 and #19702.

**What happened?**
In a restart-chaos wave, a failed run's accounting held the company for
600 s.
1. A user commented while the task's run was live. The enqueue budget
gate refused the comment's wake before it reached the deferral behind
the live run. Because it was a user request with no saved receipt, it
was written as `skipped` (`budget.blocked`). The live run finished
without it and the task ended `done`, so the comment was lost.
2. A Retry created a run that the native cleanup-quarantine fence
refused before it requested any runner launch
(`native_session_cleanup_quarantined`, no process events). A second
Retry then waited with `process_identity_missing` ("The previous run has
no verified stop record") for a process that never existed. The task
kept offering Retry, and that Retry could never start.

**Expected behavior**
A comment refused by a temporary hold stays saved and is delivered once
later. Input after a run that was refused before launch is handled by
the quarantine gate: Retry is not offered (send a new message), and a
new message gets the verified-stop check of the previous provider
session.

## What Changed

- `heartbeat/queue.ts`: when the enqueue budget gate refuses a user's
comment wake and the block is temporary (not a reached hard stop, per
`isBoardOwnedBudgetBlock`), the wake is saved as a
`deferred_issue_execution` receipt. The receipt carries the comment ids
and an `executionWait` of `budget.blocked`, and the gate notifies the
deferred-receipt worker and throws `PersistedExecutionWaitRefusal`. It
is not written as `skipped`. Normal promotion (when the live run ends)
and #19741's hold-release wakeups deliver it.
- `native-local-process-stop.ts`: new `nativeRunRefusedBeforeLaunch`. It
returns true for a failed native run with
`native_session_cleanup_quarantined`, no recorded process, and no
launch-request or identity event.
- `explicit-native-continuation.ts`: such a run passes the
process-identity check, so the existing quarantine branch decides. A
Retry gets `cleanup_quarantined` (shown with `canRetry: false` and "send
a new message"). A new message gets
`verifyStoppedNativeSessionForContinuation`. Any launch request or
identity record keeps the original stop proofs.

- `native-session-executor.ts`
(`verifyStoppedNativeSessionForContinuation`): when the source has no
process record and no launch event, the verifier relies on the previous
owner's stop evidence. Before this change it required the source to
share the owner's process identity, so a new message stayed held even
after the owner stopped. A source that carries a process identity must
still match the owner exactly.

## Verification

Each new test fails on `master` and passes with this change, on embedded
Postgres:
- `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: a comment
whose wake the hold refuses at enqueue is saved with a `budget.blocked`
wait and is not skipped. It is delivered exactly once after accounting
settles.
- `server/src/services/explicit-native-continuation.test.ts`:
- A run refused by the quarantine before launch routes a new message to
the quarantine's stop verification and a Retry to `cleanup_quarantined`.
These failed on `master` with `process_identity_missing`.
- Guards (green on both): a quarantined run that requested a launch, and
another failure without a process record, still require
`process_identity_missing`.
- `server/src/services/quarantined-task-retry.test.ts`: a never-launched
quarantined source is certified once the previous owner has stopped (red
on `master`). It stays held if the source requested a launch or the
owner is still running. All 31 tests pass, and the `stopped native
conversation physical cleanup` suite passes (46).
- Related suites pass locally (6 files, 620 tests): explicit native
continuation, held comment wake, queued comments routes, comment wake
batching, refused assignment wake, deferred receipt work. Two
`deferred-execution-receipt-work` tests failed once under parallel load
and passed when run alone.

## Risks

- The saved-comment path applies only to user comment wakes refused by a
temporary budget hold. Hard stops, system wakes, case handoffs, durable
chat requests, and existing saved receipts are unchanged.
- The pre-launch rule is limited to the quarantine failure code with no
launch evidence. Every other missing process identity still requires the
existing stop proof.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution,
kept restart-chaos instance inspection).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 08:54:14 -06:00
DottaandPaperclip 384fa2b7be Allow context compaction while Agent Chat waits (#19801)
## Thinking Path

> - Paperclip manages agents and their work.
> - Agent Chat can wait for a user between useful turns.
> - The task Properties panel already lets that user request context
compaction.
> - The queue treated that maintenance request as an ordinary wake and
refused it while the chat waited.
> - Compaction must maintain that session without starting a new
conversation turn.
> - This change admits the authenticated request, preserves the waiting
state during execution, and prevents competing requests from saving
extra maintenance work.
> - Ordinary and unauthorized wakes remain blocked.

## Linked Issues or Issue Description

Refs #19378, which introduced the existing Compact now action. Refs
#19377, whose review identified this independent queue admission gap.
Searches for `context_compaction` with `waiting` and `Compact now` with
`waiting` found no existing fix for this admission bug.

**What happened?**

An owner clicks Compact now for a waiting Agent Chat conversation. The
real compaction service reaches the heartbeat queue, receives no run,
and reports that compaction did not start.

**Expected behavior**

The authenticated owner can request one separate compaction run while
the chat waits. The request must keep its existing actor, task, agent,
budget, and execution checks. Ordinary waiting-chat wakes must stay
blocked.

**Steps to reproduce**

1. Use an open Agent Chat conversation whose agent supports compaction.
2. Wait until the conversation is waiting for its owner.
3. Request Compact now from the task Properties panel.
4. Observe the refusal before this fix. The new database regression
calls the real compaction service and queue to reproduce it.

## What Changed

- Admit only the existing complete manual compaction envelope. Require a
user actor, no coalescing, the compaction intent, and matching nonempty
request IDs.
- Preserve the waiting conversation state, session generation, reply
boundary and unanswered questions in the existing conversation
preparation owner.
- Recheck active and deferred maintenance under the existing issue
admission lock. Competing clicks cannot save a second maintenance
request.
- Add execution coverage with and without a pending question. Add
concurrent admission and deferred-request controls, including admission
after the pending receipt is cancelled. Keep the existing authorization
and ordinary conversation controls.

## Verification

- The original service-to-queue request failed before the admission fix.
- The execution regression then caught the review finding: compaction
succeeded, but both waiting-state controls failed. The same complete
controls passed after the conversation preparation fix.
- A real concurrent request test recorded two durable maintenance
receipts before the atomic admission fix. It counts deferred authority
as well as admitted runs.
- Current integrated candidate: normal current Runner core and surface
TypeScript exports passed. All seven normal App typecheck programs
passed. The whole queue/execution suites passed 86/86, the whole
context-compaction suite passed 5/5, and 11 selected ordinary
conversation controls passed. There were 102 actual passes; the 29
unselected conversation cases are not counted as passes.
- Database controls use fresh dedicated PostgreSQL 17 databases, Node
24, one test worker and pools capped at four. Private fixture adapters
use the existing dedicated database and normal migrations. They keep
test bodies, assertions and time limits intact.
- Earlier fixture and setup failures remain recorded separately from
product failures. These include an omitted external URL, an invalid
closed-task fixture, a missing crypto import, an incorrect
result-envelope matcher, and an unavailable test-library helper. Each
was corrected at its owning boundary. A test-output text matcher also
missed Vitest's abbreviated failure; the raw assertion stack and failure
were retained.
- Local session tests exercise the actual admission, native Runner
lifecycle, accepted result and persisted conversation state through the
supported test backend. They do not prove live provider compaction, UI
rendering or deployed suspension.
- Exact-head remote CI passed: 59 successful checks, three intentional
skips and one neutral check. Greptile and Ponytail both reviewed this
head at 5/5 with no actionable findings. Ponytail explicitly marked it
Ship.
- Master advanced during review. A local merge evaluation preserves
every unrelated master change and all four PR files. The current App
programs and all three execution controls passed again on that projected
squash tree to check the workspace-finalization seam.

## Risks

The change affects admission and preparation for one maintenance
request. A broad exception could start ordinary work or reopen a waiting
chat. The request therefore keeps the complete existing manual envelope
and authorization checks. Conversation preparation still checks the
session generation. The existing issue lock owns duplicate admission.
There is no new timer, database migration or provider protocol.

## Model Used

OpenAI Codex based on GPT-6 managed the change and reviewed the shared
queue fix. OpenAI `gpt-6.1-sol` authored the focused regression
coverage. Both used reasoning, repository tools and local code
execution. The runtime did not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing Compact now documentation remains accurate; no new
configuration is added)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 05:11:44 -06:00
Devin FoleyandPaperclip bf12b2d8ad fix: retry reviewed chat setup after temporary contention (#19787)
Retry reviewed chat preparation with the original accepted input.

Bound retries for authorization proof waits that expire before provider
work. Preserve the original answer receipt and queue later wakes
separately. Keep plan confirmation semantics by assigning the interaction
marker only to question answer delivery and legacy question payloads.

Verification: 433 tests across four backend suites, all 17 project-chat
browser tests, and all server source/test typecheck layers passed. The
plan approval regression was reproduced on the previous head before the
fix. All applicable CI passed and exact-head Greptile Apex scored 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-07 02:14:42 -08:00
DottaandClaude Opus 5.5 e620ce3678 fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can comment, click Retry, or rely on automatic recovery while
an agent works on a task
> - Restart-chaos waves showed accepted follow-ups being dropped, or
delivered twice, around restarts and temporary company admission holds
(accounting still settling after a restart)
> - A dropped comment means the user's request is never acted on. A
duplicate reopens work that is already done
> - This pull request makes accepted work wait through a temporary hold
and resume when it clears. It also stops a comment from being promoted
again after a completed run already received it
> - The benefit is that each accepted follow-up is delivered exactly
once, eventually, unless the task is cancelled or a reached hard stop
applies

## Linked Issues or Issue Description

Refs #19702 (never-started tasks during a temporary hold), #19700
(bounded accounting hold), #14910 (stale reopen from a deferred wake; no
file overlap).

**What happened?**
Restart-chaos waves found these shapes:
1. **Comment wake cancelled.** A comment was saved behind a run and
promoted when the run ended. A temporary company hold refused the
unstarted queued run, and claim cancelled it terminally. The recovery
wake that followed was cancelled the same way, and the task rested
`blocked` without the comment.
2. **Dead Retry.** A Retry click during the hold stayed
`deferred_issue_execution` after the hold cleared. Nothing woke it, and
the issue held no execution lock.
3. **Refused assignment.** A new task's first wake was refused during
the hold. Nothing woke stranded-work recovery when the hold cleared, so
the task stayed `todo` with no run.
4. **Silent comment after recovery.** A comment was saved behind a run
that restart recovery finalized. A company-scoped hint waited out the 30
s deferred-receipt rescan, so the task looked done with the message
unread.
5. **Comment delivered twice.** A run whose provider process was lost
had a saved comment. Its retry started after the comment, received it in
its prompt (task history), did the work, and finished the task. The
saved wake was then promoted anyway, which reopened the done task.

**Expected behavior**
An accepted follow-up (comment, Retry, recovery) is delivered exactly
once, eventually, unless the task is cancelled. A temporary refusal
never ends in a terminal cancel, skip, or block. Reaching a hard stop
remains the board's decision.

## What Changed

- `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued
run stays queued when the block is temporary. A reached hard stop or
blocked unpriced usage still cancels it.
- `recovery/service.ts`: stranded-work recovery waits out a temporary
hold for any task. It does not escalate to `blocked`. This extends
#19702 beyond never-started tasks.
- `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved
out of the recovery service. `budgets.ts` is unchanged.
- `run-cost-accounting.ts`: settling a run's accounting wakes queued-run
dispatch, the deferred-receipt worker, and stranded-work recovery. Held
work waits passively; this is what resumes it.
- `deferred-execution-notifications.ts`: new
`notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run
reaching a terminal status) and `environments.ts` (lease release) now
hint each saved wake of the issue by id, so delivery does not wait out
the company rescan interval.
- `wake-queue/adapters/postgres.ts`: release treats a saved comment as
delivered only when a succeeded run of the wake's agent on that issue
started after the comment's latest revision, and either received the
comment as its own wake input (`wakeCommentIds`) or was a retry of the
interrupted turn whose task history included it. In that case it cancels
the saved wake with "Queued messages were already delivered to a
completed run" and leaves the task as it is. Task history alone never
consumes a message saved for its own turn. One query checks the whole
batch.
- `run-cost-accounting.ts`: settling late receipt debt also sends the
hold-release notifications.
- `doc/execution-semantics.md`: documents the rule.

## Verification

Red on `master` and green with this change, on embedded Postgres:
- `server/src/__tests__/heartbeat-held-comment-wake.test.ts`:
- A promoted comment refused by the hold stays pending, with no
cancelled run and no `blocked` task. It is delivered once after the hold
clears through real `accountRunCost`.
- A Retry clicked during the hold starts exactly once after accounting
settles, which wakes its owner.
- A new task whose assignment was refused during the hold starts once
accounting settles, which wakes stranded-work recovery.
- A comment that a later succeeded retry already received as history, or
that a later run received as wake input, is consumed without reopening
the task.
- Guard tests (green on both):
- A comment is still delivered in each of these cases: the later run
never received it, it was edited after that run started, it was posted
while the run was already running, or a non-retry run only saw it as
history.
  - A task whose continuation was refused waits out the hold.
  - A reached hard stop is still refused.
- `server/src/__tests__/native-run-projection-work.test.ts`: projecting
a recovered run terminal hints the saved comment's own wake (red on
`master`).
- Related suites pass locally (14 files, 482 tests): wake-queue, queued
comments routes, refused assignment wake, native run projection.
- Live replays (local, unmasked, kept instances):
- `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The
comment's run was created at 03:56:24 during the hold, stayed queued,
started at 04:06:24 when the hold cleared, and succeeded. It is the only
run carrying the comment.
- `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it
reproduced shape 4. Run 592d772f was killed 250 ms after
`run.result.accepted`, released and finalized at restart, and the task
rested `done` with the comment still saved. Calling the existing sweep
on the kept database promoted it, which confirmed that the missing piece
was the trigger.

## Risks

- A queued run can now sit queued for the length of a temporary hold,
which #19700 bounds. It does not hold the issue's execution lock, so
later comments still coalesce into it.
- The delivered-comment rule depends on the run's recorded prompt
context. A succeeded run that received the comment in its history counts
as delivery, even if the agent did not act on it. This matches
exactly-once delivery semantics.
- Settling accounting now emits three extra notifications per run. They
are coalesced per company.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution,
live restart-chaos replays).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 00:24:09 -06:00
DottaandPaperclip c9391720a1 fix(heartbeat): send saved follow-ups after task completion (#19600)
## Thinking Path

> - Paperclip manages agents and their work.
> - Task messages remain in a durable queue while an earlier run saves
its result.
> - That result can mark the task done before the user sends the saved
messages.
> - Send records current user intent, but its wake guard allowed only
active task states.
> - The old queue therefore remained pending with an obsolete
finalization message.
> - This change admits that exact Send receipt and reopens the task
through the issue service in the successor transaction.
> - The agent can receive the saved input once while the completed
result remains unchanged.

## Linked Issues or Issue Description

**What happened?**

Send on a saved message can leave the message queued after its native
predecessor succeeds and commits. The old wait message remains visible.
Allowing admission alone also fails: continuation construction rejects a
task that remains done.

**Expected behavior**

Explicit Send must deliver the saved input once through normal
admission. It must retain the previous run and its result.

**Steps to reproduce**

1. Save a follow-up while a native run finishes its result.
2. Let that run succeed, commit its accepted result, and mark the task
done.
3. Send the saved follow-up. Before this change, the receipt stays
deferred.

Related public work: Refs #19542. That change selects recovered
workspace files. This change handles the later saved-message admission.
Refs #19602 for the independent workflow fixture repair required by CI.

## What Changed

- Allow the exact explicit Send wake guard to accept a done task. Keep
automatic recovery and cancelled task guards unchanged.
- Reopen the task through the company-scoped issue service after
admission gates pass. Commit reopening, queue consumption and the
successor together. Publish the audit after commit.
- Add a native succeeded/committed predecessor regression with unchanged
saved work and one successor after two resume calls.
- Verify real heartbeat startup under the clicking operator. Keep
automatic input, cancelled tasks and paused agents held.
- Check workspace preflight for the prospective reopened task. Use the
normal company-scoped transition if preflight blocks a done task, so its
completion metadata is cleared.
- Check a successful native predecessor for its committed accepted
result, current ownership and pending cleanup. Keep approvals and
questions effective even without a recovery action.
- Document the saved Send contract and update the queue test dependency.

## Verification

- Final merge qualification for
`675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: all 56 applicable status
checks pass, with no pending or failed checks. [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/42307623559)
includes typecheck, build, all applicable test and E2E shards, and
aggregate verification. Current-head Greptile is 5/5; all five review
threads are resolved. Any paused or queued application review is not
claimed as a new completed review.

- Final base qualification at
`675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: official preflight passes
(1.5s); all six integrated model-normalization cases pass (19.07s); both
affected test typecheck layers pass. The integration changes no
production server/shared/Runner/adapter inputs, so the preceding 12-case
Send proof, two ordinary status guards, and unchanged production
typecheck layers remain applicable. Fresh final-head CI supplies broad
coverage.

- New review regression: both failed and cancelled native predecessors
on a task manually marked Done fail before the repair with
`continuation_task_ownership_changed` through real
`resumeQueuedCommentInterrupt` admission. At
`bfb0eff7941090ac3ac43ce4366ba2e3d46612e9`, all 12 affected
Send/source/cleanup/workspace/decision cases pass (246.80s), ordinary
Done/cancelled rejection cases pass (2; 100.31s), and all seven server
typecheck layers pass. History validation uses the prospective reopened
status only for an already-verified explicit native Send. No status is
changed during a dry run.
- Prior `e64e4bb1` CI passed the affected serialized server suites,
including the complete saved-message route suite, typecheck and build.
General server shard 4 still failed two unrelated stale default-model
expectations (`gpt-5.6-sol` versus the new `gpt-6.1-sol` default). The
equivalent repair has already merged in #19610 as
`8621bcf53ee2e4bbefb7d193add42dd544ca06e8`; the separately prepared
#19645 is closed as superseded. This source fix does not weaken or edit
those assertions.

- Original RED: the committed-source and completed-task startup cases
left the saved receipt deferred.
- Review RED: Done plus an unrunnable isolated worktree created one
successor instead of zero. A successful committed predecessor with
pending lease cleanup also created one instead of zero. The new
Done-to-blocked path retained stale completion time. Pending approvals
and questions each allowed a successor. All failures were captured
before their respective repairs.
- Integrated production revision
`99018264dd1ef002ff3500378184bcd167d9206b`: 417 affected cases pass
across the complete native continuation, responsible-user startup, and
workspace preflight suites (386 + 18 + 13). These include the original
successful startup case and six cleanup/finalization faults that hold
the exact saved receipt until settlement, then deliver it once.
- `pnpm --filter @paperclipai/server typecheck` passes all seven layers
on that integrated head, including the local Runner/SDK dependency
build. Module-boundary, whitespace, and added-code-comment checks pass.
- Normal master integrations preserve branch history. No force push or
assertion weakening was used. Fresh CI and automated review are running
on `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`, which normally integrates
the already-merged #19610 fixture repair after the reproduced inherited
CI failure.
- The earlier CI attempt failed in five unchanged workflow fixture cases
because IMAGE_ONLY was unset. The merged #19602 repairs that fixture. A
later inherited duplicate default in that independent fixture has a
separate repair; this PR does not edit it.
- Follow-up route-fixture RED: three existing native queue delivery
cases failed because their fixture marked a run successful without an
accepted result or committed finalization. The same three cases pass
after the fixture records the complete native success contract (42.08s);
delivery assertions and production gates are unchanged. The affected
test-tree TypeScript layer also passes. The full route run passed all
119 assertions but failed its 30-second database teardown hook, so it is
not recorded as a suite pass. That result is retained. One unchanged
bounded rerun was stopped after more than 15 minutes without a final
verdict; it is inconclusive, not a pass.
- Final integrated head `e64e4bb1`: all 13 focused route and
completed-task admission cases pass across two real-PostgreSQL suites
(188.95s), including the three repaired route fixtures and every new
cleanup/finalization/decision hold. Fresh preflight passes; all seven
server typecheck layers pass. Required final CI supplies the broader
route and server coverage.
- These tests use real PostgreSQL and normal admission, cleanup, queue
consumption, and startup services. The startup suite controls the
provider boundary. Hosted provider acceptance remains pending normal
deployment.

## Risks

- Explicit Send can reopen a done task. This is limited to its validated
saved receipt and current assignee. A cancelled task remains held.
- Reopening uses the existing company-scoped issue transition and shares
the successor transaction. A failed admission does not reopen the task.
Provider Stop, workspace, approval, pause and budget checks retain their
existing owners.
- Cleanup and pending decisions retain the done task and saved receipt.
An unrunnable workspace uses the existing blocked-task diagnostic. No
schema or runtime package changes.

## Model Used

- OpenAI Codex, GPT-6 family; exact serving snapshot/context window not
exposed; repository analysis, code editing, tool execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 17:15:07 -06:00
DottaandPaperclip 5d25117ffa fix(heartbeat): recover deferred dependency completion wakes (#19457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks wait for other tasks through saved dependency edges and wake
requests.
> - A dependency can finish while its assignee answers a question about
the blocked task.
> - Paperclip saves the completion wake behind that active turn.
> - Recovery did not find that saved wake unless it also contained a
comment or an interaction response.
> - This pull request restores that wake through the existing release
and dispatch checks.
> - The task can then continue after the active turn and its cleanup
finish.

## Linked Issues or Issue Description

**What happened?**

A task stayed blocked after its dependency finished. The completion wake
was saved while an audit turn was active. The wake stayed deferred after
that turn ended. It had no comment or interaction response, so both
recovery queries and the input guard excluded it.

**Expected behavior**

Recover the saved dependency wake when the prior turn and its cleanup
finish. Start one dependency continuation under the current execution
rules.

**Steps to reproduce**

1. Run a question response on a dependency-blocked task.
2. Finish the dependency while the response is active.
3. Save the completion wake behind that response.
4. Finish the response and its environment cleanup, or restart the
workers.
5. Observe that the saved wake has no run.

**Paperclip version or commit**

Reproduced against `70d23612dd`. Rechecked against `2f35eb1080` with the
final regression fixture.

**Deployment mode**

Source checkout on Linux x64, Node 25.9.0, pnpm 9.15.4, and isolated
embedded PostgreSQL.

Related work: #16673 added recovery for saved messages. #17288 and
#16966 address assignment wakes after cleanup. This change covers
dependency completion receipts with no message.

## What Changed

- Include saved `issue_blockers_resolved` receipts in worker
reconstruction and stranded queue selection.
- Let these receipts pass the input guard without a comment or
interaction response.
- Use the existing release transaction and dispatch checks. Add no
timer, migration, or direct task status change.
- Add process-backed regression cases for restart, cleanup release,
cancellation, agent pause, pending cleanup, and unresolved blockers.
- Update the dependency wake recovery document.

- Share the dependency-wake payload predicate and SQL fragment across
reconstruction and stranded receipt admission, so these eligibility
checks have one owner.

## Verification

- The final regression fails on the base code: the receipt stays
`deferred_issue_execution` and has no successor. The same test passes
with the fix.
- Six focused cases pass. They use real PostgreSQL and a real process
adapter. They check saved output, one dependency continuation, clean
task locks, repeat recovery, and execution gates. They do not call a
model provider.
- Three nearby suites pass: 125 tests across queue wiring, dependency
scheduling, and dependency recovery.
- Final-head server layered typechecking passes with
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1`. Whitespace and module boundary
checks pass. Two parallel checker attempts reached the sandbox memory
limit; the completed low-memory check covers all source and test layers.
- The full deferred-receipt suite passes: 45 tests. Combined with the
nearby suites, 170 local tests pass. These results cover head
`8f9d3aa6ac19a6d2532bd869eb8adda7cf122a5a`. Required CI remains in
progress.
- Local workspace typechecking and build stop at the Runner Rust step
because Cargo is absent. The full local test suite was not run. Required
CI must supply broad coverage.
- The hosted incident has not been recovered or deployed from this
branch.

- Review follow-up at `182385e9bc433adac06440997de7e4cfbf889a72`: normal
merge of master `0d03610d2f73b7cd799547e43929bb2303319024`; no conflict
or history rewrite. All 54 receipt/helper cases pass on macOS arm64,
Node 26.4.0, real embedded PostgreSQL and process adapters. The six
dependency continuation and safety cases also pass separately. Module
boundaries and whitespace checks pass. No assertion, timeout, execution
gate or retry policy changed for this extraction.
- Current shared, SDK and adapter-utils source builds pass. The first
broad local server typecheck read stale borrowed adapter declarations;
rebuilding them exposed nominal type duplication between the borrowed
workspace and this checkout. It is not counted as a server typecheck
pass. All 30 SDK and 169 adapter runtime JavaScript outputs match the
previously tested artifacts. Final revision
`182385e9bc433adac06440997de7e4cfbf889a72` now passes all 55 applicable
CI checks (three skipped, one neutral review), including clean-workspace
typecheck, build and broad tests. [CI
result](https://github.com/paperclipai/paperclip/actions/runs/42262173960).
Greptile and Paperclip review both give 5/5 with no open findings.

## Risks

- Recovery now considers a saved dependency receipt with no message.
Current ownership, source finalization, cleanup, pause, cancellation,
and dependency checks still control execution.
- The scan remains bounded and uses existing transaction notifications
and restart reconstruction. The integration checks verify that it
becomes quiet after settlement.

## Model Used

- OpenAI Codex, based on GPT-6. The exact provider model ID and context
size are not exposed in this session. Used reasoning, tool calls,
TypeScript edits, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:49:38 -06:00
DottaandClaude Opus 5.5 8c578668b8 fix(heartbeat): start new queued runs promptly after restart and name the wait (#19431)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued agent turns. Restarts and deploys
happen often.
> - After a restart, new queued turns waited minutes before they
started. The task showed only "Queued", with no reason.
> - The cause is startup order. The queued-run dispatch worker registers
only after many unrelated first passes finish one after another: tool
reviews, delivery workers, the terminal workspace sweep, native restart
recovery, and residual and issue recovery.
> - The server binds its listener before this recovery, and
`/api/health` answers during it. A deployment can route traffic to the
new server for minutes before queued work starts.
> - This pull request starts queued turns that never executed as soon as
plugin workers and adapters are loaded. It also names the wait while the
server starts, recovers, or shuts down.
> - The benefit is that tasks resume quickly after a restart and show a
visible reason while they wait.

## Linked Issues or Issue Description

**What happened?**
On a hosted instance, the server restarted twice for deploys. Both
times, queued runs on idle agents started about 3.5 to 4 minutes after
the restart. Before the restart, the runs also waited about 5 minutes
while the deploy drained the old server. During the wait after the
restart, the execution presentation showed `phase: queued`, `label:
"Queued"`, `cause: null`.

**Expected behavior**
Queued work resumes automatically and quickly after a restart. While it
waits, the task shows a reason.

**Steps to reproduce**
1. Start a task drain (`POST /api/instance/task-drain`). Assign issues
to idle agents. The runs stay queued.
2. Stop the server (SIGTERM) and start it again.
3. Make any startup first pass slow (a large instance does this
naturally). Queued runs start only after every earlier startup pass
finishes.

**Paperclip version or commit**
master at 7e3caa7bac.

**Deployment mode**
Hosted Railway deployment with fleet deploy drain. Reproduced locally
with embedded PostgreSQL and an injected 20-second startup step.

## What Changed

- `heartbeat/queue.ts`: add `resumeFreshQueuedRunsAtStartup`. It starts
queued runs on agents with no run left `running` and no queued retry or
continuation (`started_at`, `retry_of_run_id`,
`process_loss_retry_count`, `continuation_attempt`). Every run still
passes the ordinary per-agent claim and dispatch gates, and scheduling
suppression (including the task drain) still applies.
- `heartbeat.ts`: expose it. Count it as a pending wake, like
`resumeQueuedRunsAfterDrain`, so drain status and upgrade readiness see
it.
- `index.ts`: run that pass at startup after plugin workers and external
adapters load. It does not wait for the unrelated startup first passes.
It skips while a drain, warm standby, or shutdown is active. Log when it
completes and when the full queued-run pass completes, with process
uptime, so a hosted restart shows both timings.
- `execution-projection.ts`: while the server is `starting`,
`recovering`, or `draining` (shutdown), a queued run shows **Waiting for
server restart** (cause `server_restarting`) with a next action. A task
drain keeps its **Waiting for maintenance** reason.
- `doc/execution-semantics.md`: describe the startup fast path and the
new presentation.
- The full queued-run pass, its position after native restart recovery,
and the drain behavior are unchanged.

## Verification

- Red, then green, local restart with a 20-second injected startup step
(embedded PostgreSQL, three process agents, runs queued under a task
drain, SIGTERM, restart):
- Before: the listener bound at 09:41:50.6. Runs started at 09:42:10.8,
20.2 s later, after the slow step ended.
- After: runs started 0.2 to 0.3 s after the listener bound, while the
slow step still ran. Each original run executed once.
- During the before run, `GET /api/issues/:id/execution` showed `Waiting
for server restart` / `server_restarting` until dispatch.
- Red, then green: `execution-projection.test.ts` restart-phase cases
failed on the old code (`label: "Queued"`, `cause: null`) and pass now.
- New `heartbeat-startup-fresh-dispatch.test.ts` (real PostgreSQL): a
fresh run starts; an agent with a `running` row from the previous
process and an agent with a queued retry stay queued; nothing starts
while a task drain is active.
- `npx vitest run` for the new test, `heartbeat-task-drain*.test.ts`,
`execution-projection*.test.ts`, `startup-recovery-timing.test.ts`,
`heartbeat/queue.test.ts`, `liveness.test.ts`: 132 passed.
- `npx tsc --noEmit -p server/tsconfig.json` passed.

## Risks

- New queued runs can start before native restart recovery finishes. The
fast path excludes any agent with a `running` row or a queued retry or
continuation, and the claim checks still enforce issue ownership,
predecessor cleanup, budgets, pauses, and concurrency. A new wakeup
during startup already starts runs at this point, so this adds no new
class of early execution.
- The drain is unchanged. It still holds new runs while the deploy waits
for in-flight runs (the Cloud wait budget is at most 5 minutes). This PR
does not change Cloud.
- The new label depends on the process-local startup phase. It shows
only on the process that is restarting.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local server runs). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 04:00:42 -06:00
DottaandClaude Opus 5.5 140fcf98ed fix(retry): name a declined task Retry and let a new click replace it (#19426)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task interrupted by a restart or deploy must resume automatically,
or offer a Retry that works on first use
> - A board Retry saves an intent first. Dispatch then rechecks the
requester's authority and saved source before it starts a run
> - When that recheck refused the intent, dispatch returned a bare
"deferred". The intent stayed saved with no reason, and the task still
showed Retry
> - Each later click with a new key joined that same intent, so Retry
returned 202 and never started a run
> - This pull request names the refusal on the intent and lets a new
click replace an intent that can never start
> - The benefit is that a Retry button that is shown starts one run, or
explains why it cannot

## Linked Issues or Issue Description

**What happened?**
A native task in a managed sandbox lost its runner twice. Saved-work
recovery recovered the files. The task showed "Saved work recovered.
Retry or send a message to continue." and `canRetry: true`. A Retry
click returned 202 and logged `issue.retry_requested`, but no run
started. A second click with a new idempotency key returned the same
request ID and `state: failed`. No run started.

**Expected behavior**
If Retry is offered, a click starts exactly one run. If the Retry cannot
start, the task names the reason, and a later click is a fresh attempt.
A Retry accepted during the pre-deploy drain starts after the restart.

**Steps to reproduce**
1. Recover the files of a failed native run so the hold offers Retry.
2. Save a Retry intent whose dispatch-time authority recheck fails (for
example, the requester lost access).
3. Dispatch it: no run starts, and the intent has no wait reason.
4. Click Retry again with a new key: the response returns the first
request ID, and no run starts.

**Paperclip version or commit**
Reproduced on `b400914345` (staging served `4d49a31286`; the affected
code is the same).

Related: #19399 (automatic Retry after a lost runner), #19422.

## What Changed

- `heartbeat/queue.ts`: when dispatch declines a task Retry because the
requester can no longer change the task, or its saved board message
changed, record a named wait (`retry_authority_denied`,
`retry_source_changed`) and log it. Before, it returned "deferred" with
no record.
- `task-retry.ts`: a new click with a new key replaces an unadmitted
intent that dispatch declined for its requester's access
(`retry_authority_denied`). The old intent becomes `skipped` /
`task_retry_replaced`, and activity records `replacedRequestId`. Waits
on shared state (cleanup, recovery, an active run, preservation) still
coalesce. The restart hook never replaces a user's click. Chat-owned
intents keep their own owner.
- `task-interruption-presentation.ts`: fixed public messages for the two
new wait reasons. `retry_authority_denied` keeps Retry, because a new
click is a fresh attempt. `retry_source_changed` is terminal: an edited
saved message cannot be replayed, so Retry is hidden and the user sends
a new message.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New regressions in `server/src/services/workspace-recovery.test.ts`
use the real recovery, Retry and dispatch services on embedded Postgres.
The fixture runs the automatic Retry from a lost runner, then that
Retry's own run loses its runner, and recovery saves its files:
- `starts exactly one run when Retry is pressed after the automatic
Retry's own run lost its runner` (also checks that a replayed dispatch
and a second click start no second run).
- `does not let a Retry that could not start swallow a later Retry`: red
before the fix in two steps. With no fix, the declined intent has no
wait reason. With only the wait fix, the second click returned the first
request ID. Green after the fix: one run, bound to the second click.
- `dispatches a Retry accepted during the pre-deploy drain once the
restarted server resumes`: green before and after. The drain does not
lose a saved Retry.
- `npx vitest run src/services/workspace-recovery.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/services/quarantined-task-retry.test.ts
src/services/explicit-native-continuation.test.ts`: 892 passed.
- Not verified on staging. After deploy, an intent that dispatch refuses
shows a named `retryWait` reason.

## Risks

- Low risk. Only intents that dispatch declined for their requester's
access are replaced. All other coalescing stays.
- A replaced intent's late dispatch finds it `skipped` and starts
nothing, so a double click still starts at most one run.
- The exact gate that held the staging intent is not confirmed from
records. If it was a shared-state wait, the new code names it only if it
is one of the two reasons above.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:52:21 -06:00
DottaandClaude Opus 5.5 7e3caa7bac fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.

## Linked Issues or Issue Description

Refs #19394

**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.

**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.

**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.

**Paperclip version or commit**
master at 4d49a31286 (includes #19394).

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `issue-tree-control.ts`: add `ISSUE_PAUSED_ERROR_CODE` and
`pauseSuppressionResult()`. A gate records `treeHoldSuppression: {
holdId, rootIssueId }` on a run that it refused for a pause.
- `findOwedPauseResume` now treats the latest run as owed when it is an
`issue_paused` cancellation under a pause hold. It uses the recorded
hold. For an unmarked run (a scheduled retry cancelled before this
change), it uses a snapshot pause that was active at cancellation. The
operator Stop rule does not change.
- Scheduled-retry gate (`run-dispatch/adapters/postgres.ts`,
`cancelSuppressedRetryInTx`): locks the pause row `FOR SHARE` and
records the covering hold. A concurrent release either sees the refusal,
or the retry sees no pause and runs normally.
- Queued-run claim gate (`heartbeat/queue.ts`): cancels with
`issue_paused` and the hold marker, after the agent start lock is
released. Before, the cancel re-entered that lock from inside it, and
the agent's queue stalled until the 30-second stale-lock timeout. After
the refusal is saved, the gate re-checks the hold and resumes the task
itself if release already finished (`resumeIfPauseAlreadyReleased`).
- Release (`issue-tree-resume.ts`, route): revisits the snapshot members
and every task with a run refused for that hold, also when `wakeAgents`
is set. This covers tasks created under the tree after the snapshot and
tasks moved out before release.
- `packages/db`: partial index
`heartbeat_runs_tree_hold_suppression_idx` (migration 0436, `CREATE
INDEX CONCURRENTLY`, registered as a top-level migration) for that
lookup.
- Docs: `doc/execution-semantics.md` (gate-refused work, race handling,
non-assignee and pre-fix scope) and `doc/DATABASE.md` (index).

## Verification

- Red before the fix, green after: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/issue-tree-pause-resume-run.test.ts` (21
tests).
- "resumes a parked retry the pause suppressed exactly once after
release / resume / legacy": before, `expected [] to have a length of 1`
(no resumed run). After, one `issue_tree_resumed` run with
`interruptedRunId` = the suppressed retry, no execution blocker, and the
task finished. `reconcileStrandedAssignedIssues` added no run during the
hold or after release.
- "resumes a queued run the claim gate refused during the hold exactly
once": before, the test timed out at 15 s (start-lock re-entry). After,
the run is cancelled with `issue_paused` and the hold marker, and
release re-dispatches it once.
- "resumes a refused retry on a task moved out of the paused tree before
release" and "... on a task outside the snapshot when release also wakes
members": review findings, one resume each.
- "leaves a due retry parked when the pause is released while the gate
refuses it": the test waits until the gate blocks on an uncommitted
release. The retry is not cancelled and runs once. Without the `FOR
SHARE` lock the gate does not wait and the test fails.
- "resumes work a gate refused after the release that should have
resumed it had finished": the late-refusal re-check resumes once, and a
repeated check does not dispatch again.
- "does not resume a paused-code cancellation that no pause hold
covered": negative case.
- Related suites pass locally: `src/__tests__/issue-tree*`,
`src/modules/run-dispatch`, `src/services/heartbeat/queue.test.ts`,
`heartbeat-dependency-scheduling`, `heartbeat-workspace-busy`, and
`packages/db` `client.test.ts`.
- I did not run a live staging check here. Staging APIs are not
reachable from this environment.

## Risks

- Migration 0436 adds a partial index on `heartbeat_runs`. It is built
concurrently and covers only cancelled runs with the marker, so it is
small.
- The scheduled-retry gate now takes a `FOR SHARE` row lock on the pause
hold before it cancels. Release updates that row, so the two serialize
briefly. Release does not lock issues or runs, so the lock order cannot
deadlock with the gate's issue-then-run locks.
- The queued-run claim gate now records `issue_paused` in place of the
generic `cancelled`. Stranded recovery already classifies `issue_paused`
as non-retryable. The pause resume owns that work, so recovery does not
add a second run.
- Scope limits (documented): a non-assignee run refused by a pause is
not replayed, the same as non-assignee wakes that admission drops during
a hold. Among pre-fix rows, only scheduled retries carry `issue_paused`,
so only those fall back to the snapshot. A pre-fix row whose hold is
already released does not self-recover; the staging task that showed
this bug needs one manual wake, or a new pause and release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:03:15 -06:00
DottaandPaperclip 6364fa7327 fix(heartbeat): wait for retry predecessor cleanup (#18989)
## Thinking Path

> - Paperclip manages agent work and preserves its task ownership.
> - Heartbeat admission controls when a queued run can start.
> - A failed run can still own workspace saving and environment cleanup.
> - Automatic recovery can transfer the task execution lock to its
queued retry before that cleanup finishes.
> - The old queue check then sees the retry itself and misses the
predecessor's remaining work.
> - This pull request checks the exact retry predecessor before claiming
the queued run.
> - Existing cleanup notifications can then resume that same run through
normal admission.

## Linked Issues or Issue Description

Refs #18898 and #18691. The first repair covers canonical connected-task
retries with a native source. The second covers live remote workspace
writers. This change covers ordinary automatic continuation when the
predecessor failed before native runtime selection. The search found no
open repair for that path.

**What happened?**

A preparation failure saved a terminal legacy run row while its remote
workspace save and environment cleanup were still active. Generic
continuation created one queued successor and moved the task execution
lock to it. The queue checked that current lock but did not check the
successor's predecessor. The new run started and failed at the existing
unfinished-writer guard.

**Expected behavior**

Keep that same successor queued until its exact predecessor finishes
execution and cleanup. Check company, agent and task identity before
using retry provenance as a wait condition. Preserve Stop, reassignment,
failed cleanup and independent recovery gates.

**Steps to reproduce**

1. Save a failed legacy preparation run with an active environment
lease.
2. Queue its ordinary automatic continuation and bind the task execution
lock to the successor.
3. Claim the queued run before the predecessor finishes cleanup.
4. The old code starts the successor. The fixed code retains it in the
queue.
5. Commit successful cleanup through the existing cleanup owner. The
ordinary dispatcher starts that same run once.

**Paperclip version or commit**

The causal regression fails on master
`86814c6550f52e99ede150a883bdc773cbb540e3`.

**Deployment mode**

Source checkout on macOS arm64 with disposable PostgreSQL, the assembled
heartbeat service and the production work coordinator. The adapter
boundary is controlled. No hosted provider replay or deployment is
claimed.

## What Changed

- Check the current task execution owner and the retry's exact
predecessor under the existing task claim lock.
- Use the saved native task binding, ordinary task binding and
historical context fallback in that order. Ignore unrelated retry
provenance.
- Reuse the existing live-executor and pending-cleanup checks. Reuse
committed cleanup notifications and the existing dispatcher.
- Add real PostgreSQL claim controls and assembled cleanup-event
acceptance. Keep the original source failure, saved files and current
task ownership intact.
- Document the queued predecessor-cleanup rule.

## Verification

- RED: the unchanged queue claimed the successor and wrote a new
controller lease while its legacy predecessor retained an active remote
environment lease. The regression expected the successor to remain
queued.
- GREEN: the same regression passes after the change. The complete queue
suite passes all 71 tests.
- The assembled heartbeat suite passes all six tests. The added cases
cover one continuation after committed cleanup, saved-file reuse, final
task lock release, board Stop, reassignment and failed cleanup.
- Scope controls cover a deleted source after scan, another company,
another task, another agent, canonical native task binding and the
existing current-owner hold. A failed cleanup and a still-live local
executor retain the wait.
- Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check`
pass.
- Initial fixture attempts failed on database constraints and observer
timing. The final fixtures use valid immutable bindings, the board Stop
attribution and a completed real dispatcher call after reading the exact
committed lease state. Production ownership guards were retained
throughout.
- Full low-memory repository typecheck passed at `5cddb49322`. The sync
to master `04a24f4e06` adds only the independent retained-allocation
test fixture from #18984; production code is unchanged.
- Full `pnpm build` passes on current head
`c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`.
- The post-sync run executes all 80 behavior assertions successfully
across the queue, assembled cleanup and updated upstream retention
suites. Its queue `afterAll` then exceeds the existing 30-second
database shutdown hook. That failed suite result is retained. The
controlled post-build queue rerun passes all 71 tests with normal
teardown. The earlier shutdown cause remains undetermined. Its owned
queue-test PostgreSQL process has exited. The separate local default
`pnpm test:run` was intentionally stopped after full unfiltered CI
passed on the exact head. Four local chat partitions completed before
that stop; this is not a local full-suite pass. The verified owned
process group and its PostgreSQL fixture have exited. Logs and retained
fixture data remain available. The timeout has not been increased.
- Full unfiltered CI passes on current head
`c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`: all 12 server shards, all
six internal chat partitions, all nine serialized server shards, the
workspace groups, native Runner lanes, typecheck, build and browser E2E.
Both required [ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711606613)
and [ci /
e2e](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711263146)
pass. The file selector is skipped, so this is the full CI test matrix.
The head has 55 successful checks and three expected skipped checks. The
ready-for-review contributor trust and security scan also pass.
- Current-head Greptile is 5/5, its check is successful, and no inline
review threads remain. The ordinary review check also passes. No merge,
deployment or original task retry was performed.

## Risks

- A predecessor with unfinished cleanup keeps its successor queued. Its
existing cleanup owner remains responsible for settlement.
- A task claim reads at most one additional predecessor row. A matching
source also uses the existing ownership and cleanup queries. The change
adds no timer, schema, credential, provider or model policy.
- A committed cleanup signal only requests normal admission. Current
ownership and independent execution gates still apply.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and
PostgreSQL verification. The session does not expose a more specific
serving model ID or context-window limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused suites pass; the
separate local full-suite attempt is explicitly incomplete above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:16:54 -06:00
DottaandPaperclip 84e9c8f364 fix(retry): wait for admitted source cleanup before dispatch (#18898)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected-task Retry keeps the original request and its current
authority.
> - One durable Retry can already own a queued successor.
> - The old native environment can still have pending cleanup before
that successor starts.
> - The task execution lock points to the successor, so the old cleanup
can escape the queue claim check.
> - Dispatch then refuses the source state and fails the new run before
provider work.
> - This pull request keeps that same run queued and wakes dispatch when
cleanup changes.

## Linked Issues or Issue Description

Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs
cover preservation binding, interrupted preparation, old native
ownership, manual native holds or other setup guards. This change covers
an already admitted connected-task Retry whose native predecessor still
owes cleanup. It uses the shared cleanup predicate so the manual-hold
repair remains with its existing owner. Search found no duplicate
queued-source cleanup repair.

**What happened?**

A board Retry saved its exact chat request and admitted one queued
successor. The source lease was still pending cleanup when that
successor reached dispatch. A retained stop receipt did not settle the
full cleanup contract. The queue checked the task's current execution
owner, which was already the successor. The native authorization guard
then rejected the predecessor state and failed the new run before
provider dispatch.

**Expected behavior**

Keep the same canonical Retry and queued run while the exact source
cleanup is pending. After authoritative settlement, submit that run
through normal dispatch once. Do not infer safe execution from a
stop-and-retain receipt alone.

**Steps to reproduce**

1. Retry an authorized connected request from a failed native run.
2. Admit one successor while capacity keeps it queued.
3. Leave or restore the source lease to pending cleanup with a matching
retained stop receipt.
4. Release capacity and resume the ordinary queue.
5. The old claim path starts the successor and the native source guard
refuses it.

**Paperclip version or commit**

Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`.

**Deployment mode**

The regression uses the real HTTP route, chat authorization, queue
claims, disposable PostgreSQL and the production work coordinator on
macOS arm64. A controlled adapter supplies the final result. No hosted
provider replay or deployment is claimed.

## What Changed

- Validate the exact canonical Retry, chat action, company, task, agent,
source and successor before testing source cleanup.
- Keep the successor queued under the existing issue claim lock while
the shared cleanup predicate remains pending.
- Wake only the scoped queued-dispatch worker after cleanup or exact
workspace-owner release commits. Add no timer or background scan.
- Verify pending cleanup at intake, later cleanup invalidation, one
final execution, idempotency, Stop, changed input, unknown stop
receipts, a different live controller and signed sandbox absence
followed by owner release.
- Document the queued-wait contract. No schema, endpoint, credential,
model or provider policy changes are required.
- Align five native preparation acceptance fixtures with the existing
workspace publication contract. Use the real reservation and publication
owner. Retain a raw-root refusal case with no provider launch and
preserved files.

## Verification

- Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`
includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared
test helper preserves #18944's canonical retained-directory default, the
real publisher for the five fresh-workspace cases, and the explicit
raw-root refusal. Independent remerge review found no dropped upstream
behavior, guard or assertion.
- The latest head passes the focused six cases, all 580 complete
workspace/continuation tests, all 38 selected HTTP Retry cases, fresh
full recursive Node 24.22.2 types and full build. No affected tests
skipped. An initial cached test-layer attempt read old budget
declarations; regenerating them resolved all seven errors without
test/mock or production-export changes.
- All current-head CI is green at
`e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314)
and [ci /
e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086).
Greptile and Paperclip Review both rate this exact head 5/5, with no
findings or open review threads.
- Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges
master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts
were adjacent queue imports and two HTTP Retry test matrices. Both
imports and both complete matrices remain unchanged. Independent review
found no relaxed guard or assertion.
- Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases,
194 queue/cleanup/finalizer cases, module boundaries, recursive types
and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`.
- That sync head's [server
check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761)
failed five native acceptance cases in CI and locally with
`remote_workspace_published_path_invalid`. The test fixture supplied a
raw directory instead of a fresh publisher-generated workspace. The
guard from #18932 remains unchanged. The same five cases passed earlier
Linux CI before that guard was present.
- The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`
obtains the saved authority, path and run reference from real
`reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original
cases and a raw-root no-launch/no-result/files-preserved control pass.
Their native result, file, catalog and single-execution assertions
remain intact. An initial setup attempt was refused for a noncanonical
allocation root; the fixture now resolves that owned root before
publication.
- At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`,
the generated server test-layer typecheck passed. All 580 tests in the
complete workspace/continuation family pass, with no skips. That
test-only commit left production files byte-identical to the sync head.
- Previous published-head CI was green at
`dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci /
verify` and `ci / e2e`. The [Linux Node 24.22.2 server
shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638)
executed all five original acceptance cases and the new raw-root refusal
case successfully. Its whole workspace recovery file passed all 245
tests.
- Greptile reviewed previous published head
`dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or
open review threads. No unchanged-job rerun was used for the reproduced
sync-head failure.
- Causal red: the real HTTP regression started and failed the successor
at the native source-state guard while cleanup remained pending. A
second red proved that signed sandbox absence followed by copyback-owner
release lacked a queued-dispatch wake.
- Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner
release notification control and seven existing retry lineage controls
pass. The real coordinator resumes from committed cleanup or
workspace-owner release, without a manual queue restart. One controlled
adapter execution produces the result after settlement.
- Earlier branch verification passed full repository `pnpm -r
typecheck`, `pnpm build`, module boundaries and `git diff --check`.
- The full stale-queue file control passed all 64 cases with unchanged
observer/hook limits. The initial broader affected run had 192 passes
and one non-assignee native-owner case miss its existing five-second
adapter observer while already running with no error. The isolated case
passed; the unchanged base passed all 64 cases but hit its existing
30-second teardown timeout. Those original results are retained. The new
queue hint has no dispatch consumer in that fixture.
- Earlier-head CI was green: 54 success, three skipped and two neutral
checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was
cancelled by an external runner shutdown; its single unchanged-job rerun
passed. No source was changed to obtain that result.
- Greptile and Paperclip Review both rated earlier head
`6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or
unresolved threads. These results do not qualify the master-sync head.
- At initial verification, a complete local `pnpm test:run` was not
repeated while the related setup repair's full local run was active. The
full earlier-head CI suites passed. A full local suite has not been
repeated for the mechanical master sync. Local verification uses the
affected HTTP, producer, lineage and queue suites. No full local-suite
pass is claimed.
- The fresh clone's missing plugin SDK links were repaired with the
repository's own postinstall helper before verification. The embedded
PostgreSQL package's own symlink helper was also required.
- No staging task was submitted again with this repair, and no
deployment is claimed.

## Risks

- A pending source cleanup can keep the queued run waiting. Its cleanup
owner still owns reconciliation and its existing limits.
- Stop, changed source, unknown termination and live ownership retain
their existing authority checks. A cleanup notification grants no
execution authority.
- A new cleanup change after claim still faces the existing final
dispatch guard.
- Cleanup notifications can change dispatch timing in a running
controller. The real coordinator cases verify commit-bound wakeups; the
broader current queue-file control passed all 64 cases. The earlier
loaded-run and baseline teardown failures are retained in verification.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, tool use,
PostgreSQL and browser verification. This session does not expose a more
specific deployment model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 08:33:39 -06:00
DottaandPaperclip d8c1b9d4b5 fix(runtime): reclaim failed remote checkpoint retries (#18877)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must preserve the original execution and provider
state.
> - A failed remote run can retain its provider PID after heartbeat
releases the task execution lock.
> - Normal resumption excludes that PID, while restart recovery rejects
the released lock.
> - Recovery must reclaim that exact execution only after current
admission and durable ownership checks pass.
> - Failed retries and admission holds need separate finite budgets.
Neither can prove a provider stopped.

## Linked Issues or Issue Description

**What happened?**

A failed remote native run remains in retryable failure without a task
execution owner. Normal resumption and restart recovery both reject it
before inspecting the preserved remote provider. The current remote
workspace reference format also misses the due-retry routing path.

**Expected behavior**

A due exact-checkpoint retry reclaims its original task execution only
when current ownership and safety gates permit it. Newer task input
prevents an old claim. Lost or ambiguous provider state still prevents
replay or replacement. Ineligible recovery eventually becomes a quiet
manual reconciliation obligation.

**Steps to reproduce**

1. Start a remote native run and persist its provider checkpoint and
encrypted Runner authority.
2. Fail with exact-checkpoint recovery and a retained remote PID.
3. Release the task execution lock at heartbeat completion.
4. Run ordinary resumption and restart recovery after the retry
deadline.
5. Observe that both reject the same eligible run before remote
inspection.

**Paperclip version or commit**

Reproduced from source at `d59791828482`; integrated with upstream
`db9be4c5a6`.

**Deployment mode**

Remote workspaces with PostgreSQL. Local regressions use disposable
PostgreSQL and encrypted checkpoint persistence.

Related public repairs #18846, #18854, #18860 and #18841 cover finished
per-turn owners, pre-provider failures, unstarted follow-ups and storage
migration. They do not reclaim this failed run's released execution
lock. Merged #18879 also adds scheduled-resume lock retention and
reclaim. This PR keeps that compatibility and requires encrypted Runner
authority, canonical source ordering and a charged failed retry before
reclaim. Its scheduled-resume regression now uses a real encrypted
authority and retains a snapshot-only refusal control.

## What Changed

- Share normal admission checks and one guarded execution-binding
restore/event helper between normal and remote resumption.
- Keep agent-name normalization in a module with no imports. Preserve
heartbeat's existing export. Native admission no longer loads heartbeat
orchestration for a string key.
- Recheck company, assignment, agent eligibility, Stop, pause, approval,
dependencies, execution and checkout ownership before reclaim.
- Reject newer task input using PostgreSQL creation-time and ID
ordering, preserving timestamp precision.
- Require the recorded remote target, encrypted Runner authority, exact
checkpoint identity and existing controller fence.
- Commit task binding, controller lease, run status and recovery event
atomically. Recognize both remote workspace reference formats.
- Charge failed reattachments against the existing
three-provider-attempt budget, including retained-lock crash windows.
Active controller adoption preserves its charged attempt.
- Use the existing five-retry/30-minute system-recovery policy for
admission holds. Ended tasks and newer sources exhaust it immediately.
Preserve PID, checkpoint and task bindings, suppress resource
settlement, and assign manual reconciliation with executor stop/state
explicitly unverified.
- Register recovery intent before deadline writes; preserve live/future
deadlines quietly and roll back zero-row writes. Record changed hold
reasons once and terminal dispositions separately.
- Keep historical provider-budget exhaustion behind current
admission/source/controller/authority checks. Make an existing
orphan-cleanup fixture explicitly due.
- Include exact plugin task leases that have no workspace reference.
Reuse every reclaim and retry-budget guard, with a bounded
company/task/run lease projection.
- Share strict manual ownership proof across stale-lock cleanup,
adoption, queue release and wake admission. Require termination receipts
for every exact remote lease before normal cleanup can release either
binding.
- Preserve governing Stop through the existing audited cancellation
path. Its acknowledgement does not release an unverified provider.
Ordinary Retry can wait for exact Stop and file preservation, then queue
one continuation.

## Verification

- Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its
[current CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858)
is terminal and clean: 54 successful, three skipped and two neutral
contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on
this exact head with no open findings. No fresh Paperclip 5/5 assessment
is claimed.
- The [prior complete CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388)
passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and
Paperclip both 5/5 and no open threads.
- All 233 tests across the six affected heartbeat, restart-isolation,
remote-admission, scheduled-resume and session-binding families pass on
production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under
Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and
plugin task leases, competing controllers, exact rollback, Stop,
assignment/company boundaries, attempts, work-intent ordering, finite
backoff and quiet held state.
- The merge preserves #18931's per-run audit-failure isolation and this
PR's unchanged-transition hold. Two initial composition cases failed
because their setup or expected result described the older contract. The
rollback test now expects the exact per-run deferral while retaining
full coordinator equality and no work intent. The routine fixture writes
a real encrypted checkpoint through the normal owner before reaching the
actual routine unique constraint. No production admission guard or
rollback assertion was weakened. A separate agent reviewed this exact
merge and fixture delta without finding a blocker.
- Full recursive `pnpm -r typecheck` and `pnpm build` pass on production
merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits
change only three test files. Both the two-file `a5104fa` and final
`5d7baf` server test-layer typechecks pass.
- Four production-identical `a5104fa` probes pass with the rebuilt
actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL
checkpoints under Node 24.22.2. Workspace-reference and plugin-lease
admission each cover surviving and lost providers. Replacement preserves
one provider launch, one original side effect, the charged attempt and
restored task execution binding. Lost providers reject snapshot recovery
without replacement. This is protocol-boundary evidence, not a live
Claude or hosted provisioning claim.
- Initial `a5104fa` CI found one additional routine integration fixture
missing the persisted `exact_checkpoint_resume` mode. Its encrypted
checkpoint was already present. The exact case fails locally before
repair and passes after recording that mode and expecting the charged
provider attempt. Every exact routine conflict, rollback, checkpoint,
task binding, successor-count and sibling-startup assertion remains.
Final head changes only that fixture. The correct native-server group
passes all nine files: 175 executed cases and 14 container-only skips,
matching the now-successful native-server CI gate.
- The earlier canonical HTTP Retry acceptance passed on the prior
reviewed head with real PostgreSQL, the recovery worker and real files.
Pending Stop protects both bindings and starts no successor. Exact
termination and copyback queue one preserved-workspace continuation. No
browser or live-model outcome is inferred from that test.
- The predecessor's full local `pnpm test:run` failed in an unchanged
Discord Gateway fixture with a 200 ms lease and a 422 initial ownership
error. Focused predecessor and untouched-base controls pass; that
full-run cause remains unproved. The required command was also started
on production-identical `a5104fa`. All six local chat shards passed,
covering 1,850 cases. The duplicate local invocation was interrupted
with SIGINT while the general-server group was still running after this
head's complete CI and the combined repair qualification had passed. The
wrapper exited 1 on cancellation, no process retained its test log, and
no full local pass is claimed. An earlier attempt to select the native
lane through the CI wrapper instead selected local package sharding:
2,631 cases passed and 13 failed. Those separate failures are retained
without causal attribution or unrelated source changes; that run is not
counted as native-server qualification. The direct native group and
current complete CI matrix pass.
- Live Claude Opus 5.5 continuation, hosted plugin provisioning,
deployment and recovery of historical tasks remain unverified. No
original task was retried or mutated for this verification.

## Risks

- Restoring a task binding grants remote inspection; it does not prove
that the provider survived.
- Unknown effects, lost providers and missing semantic results retain
their existing reconciliation requirements.
- Exhausted admission holds preserve execution bindings and resource
evidence for manual reconciliation. They do not authorize a fresh
executor or provider/resource release.
- The source guard is conservative: newer task input prevents recovery
of an older failed source.
- No database migration or provider profile change is required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and
shell execution. The runtime does not expose the exact model snapshot or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:18:07 -06:00
Devin FoleyandPaperclip 1a39dea9aa fix(heartbeat): keep a routine claim queued while its slot is held (#18933)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A routine creates one task per tick. Only one open task of a routine
can hold an execution lock (`issues_open_routine_execution_uq`).
> - Since #18879, a failed native run that waits for a scheduled resume
keeps its task's lock. Routine dispatch counts only queued, running or
scheduled runs as live, so it can queue a newer task of the same routine
meanwhile.
> - When that newer run is claimed, stamping its execution lock violates
the unique index. The raw database error is not a deferrable claim
rejection, so it escapes the queue loop. The queue code notes that this
can stop the agent's queue and startup recovery.
> - This pull request turns that exact violation into a 409 claim
rejection, so the run stays queued until the slot is free.
> - The benefit is that a busy routine slot delays one run instead of
stopping the queue or startup.

## Linked Issues or Issue Description

Refs #18879 (keeps a failed run's lock while its resume is pending),
#18916 (defers the same unique-index conflict in restart recovery's lock
reclaim) and #18931 (open; keeps one run's restart-recovery failure from
stopping startup recovery). This is the same index conflict on a
different path: the heartbeat queue claim, not restart recovery. It does
not change #18916 or #18931. Its doc paragraph is separate from the one
#18931 edits.

**What happened?**

While auditing the lock paths from #18879 for this index, I found that a
queued run of a newer routine task can be claimed while an older task of
the same routine still holds the slot through a failed run with a
pending resume. The claim's lock stamp then raises `duplicate key value
violates unique constraint "issues_open_routine_execution_uq"`.
`isDeferrableClaimRejection` accepts only `HttpError` 4xx, so the error
escapes `startNextQueuedRunForAgent`.

**Expected behavior**

The claim is rejected as a conflict and rolls back. The run stays queued
and is claimed after the older task releases the slot. Other errors
behave as before.

**Steps to reproduce**

1. Create two open tasks of the same routine (same origin and
fingerprint).
2. Give the older task an execution lock held by a failed run.
3. Queue a run for the newer task and claim it. On `master` the claim
throws the unique violation.

The new queue test runs these steps against PostgreSQL.

**Paperclip version or commit**

`master` at `0dedbf23c35`.

**Deployment mode**

Any server with routines whose tasks run on native runners.

## What Changed

- `bindClaimedIssueExecution` in
`server/src/services/heartbeat/queue.ts`: if stamping the execution lock
violates `issues_open_routine_execution_uq`, it throws
`conflict("Another execution for this routine is already in progress")`.
The claim transaction rolls back, and the queue loop keeps the run
queued as it does for other 4xx claim rejections. Every other error is
rethrown unchanged. The issue checkout route already maps this violation
to the same 409.
- `doc/execution-semantics.md`: one paragraph on the behavior.

## Verification

- Failing first: `queue.test.ts` "keeps a claim queued while another
task of its routine holds the execution slot". On `master` the claim
raised the `issues_open_routine_execution_uq` violation. Now it rejects
with 409, the run stays `queued`, neither task's lock changes, and
nothing executes.
- Failing first, review round 1:
`heartbeat-queued-run-claim-isolation.test.ts` "keeps a routine run
queued while another task holds the routine slot, then runs it once the
slot is free" goes through `resumeQueuedRuns`. On `master` it rejected
with the unique violation. Now the routine run stays queued with no
error code while another queued run for the same agent executes. After
the slot is released, the routine run executes exactly once (executions
are counted per run, because the finished task can queue its own
follow-up run).
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: 70 passed.
- `npx vitest run src/services/heartbeat/recovery.test.ts
src/__tests__/routines*.test.ts`: 4 files, 205 passed.
- `pnpm --filter @paperclipai/server typecheck`: pass.
- Not verified: the queue loop end to end on a real instance. The loop's
handling of a 4xx claim rejection is existing behavior.

## Risks

- While the older task holds the slot, the newer run stays queued and
the loop logs "queued heartbeat run claim was rejected" on each pass. It
is claimed once the older task's resume finishes or its lock is
released.
- Only this one constraint is mapped. Other database errors still escape
as before.
- No schema migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, test runs). The runtime does not expose the context
window or reasoning configuration.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 21:33:00 -08:00