Commit Graph
6 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 9c230ace94 fix(heartbeat): settle deferred chat wakes that can never resume instead of retrying every pass (#21678)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat deferred-receipt worker resumes wakes that waited
behind a running task execution
> - A chat message that arrives while its task runs is saved as a
deferred chat wake
> - When the task finishes (done or cancelled) first, the saved wake can
never resume, but the sweep retried it forever and touched `updated_at`
on every pass
> - An idle instance did one write and one failed resume about 4 times a
second, and the log line did not say why
> - This pull request gives each such wake one recorded outcome, and
makes every other failure wait on a stored, spaced, bounded next attempt
> - The benefit is an idle database stays idle, and a person sees the
existing "not started" result instead of silence

## Linked Issues or Issue Description

**What happened?**
The stranded deferred-queue sweep in
`server/src/services/heartbeat/queue.ts` selects
`deferred_issue_execution` chat-inbound wakes. On every pass it runs
`update agent_wakeup_requests set updated_at = now()` for each wake.
Then `resumeDeferredChatWake` fails because the task is done or
cancelled, so the chat authority denies the work start
(`task_assignment`). Nothing records the failure, so the next pass
repeats it.

**Steps to reproduce**
1. Save a chat message on a task while a run is active, so the message
becomes a deferred chat wake.
2. Let the run finish and mark the task done.
3. Leave the instance idle. The log shows "saved chat source could not
resume after recovery" again and again for the same request ids.

**Expected behavior**
A wake that can never resume gets one terminal outcome with a stored
reason. A wake that can resume later waits on a stored time. An idle
instance runs no statements for these wakes.

**Paperclip version or commit**
master at 36101ebcb9 and later.

**Relevant logs or output**
On the hosted instance with no active runs: 1,405 failed resumes in 6
minutes (3.9/s) across 363 request ids, plus one write per attempt. With
the test harness, 12 stale wakes over 30 idle minutes caused 1,344
statements, 84 wake writes and 84 failure log lines in one process.

## What Changed

- `heartbeat/queue.ts`: the stranded-queue sweep no longer touches
`updated_at` on chat-inbound wakes. Non-chat saved wakes keep the touch.
It moves `updated_at`, which spaces the worker's 30-second deadline for
wakes with an execution wait, and a board chat continuation
(`ai-personal-connection-recovery` e2e) depends on that spacing. The
chat branch now records the outcome of each resume. The log line now has
`reason`, `outcome` and `attempts`. Unscoped sweeps skip a wake whose
stored next attempt time is in the future.
- `durable-chat-wakeup.ts`: add `recordDeferredChatWakeDecline`,
`clearDeferredChatWakeRetry` and `deferredChatWakeDeclineForError`.
- Settle: set the wake to `skipped` with the reason in `error`, set
`finishedAt`, mark the chat delivery with the existing "Input saved. Its
work request was skipped…" text, and notify the inbound wake notice
worker. That worker shows the existing "This follow-up was not started"
notice where a "queued" notice is visible. Wakes older than 24 hours
settle silently (see Risks).
- Retry: store `payload.chatResume = { attempts, reason, nextAttemptAt
}` and the reason in `error`. Delays are 1 min, 10 min, 1 h, 6 h and 24
h. After five attempts the wake settles as `retry_exhausted:<reason>`.
- Permanent authority reasons (`source_shape`, `delivery_binding`,
`conversation_binding`, `destination_binding`, `source_comment`) settle.
Other reasons, such as principal access, runtime fence, privacy or
infrastructure errors, retry.
- `chat-channels/inbound/wakeups.ts` (the deferred chat wake owner): for
the recovery sweep only, return a decline instead of a silent return or
an authority error:
- The task is done or cancelled and the wake is older than
`CLOSED_TASK_CHAT_WAKE_REOPEN_WINDOW_MS` (1 hour): settle with
`task_closed` and start no run.
- The task is done or cancelled and the wake is recent: reopen the task
to `todo` in the same transaction as the authorization check, then
resume through the normal chat admission (`queueIssueAssignmentWakeup`).
This is what ingress does today for a message on a finished task:
`chat-channels.ts` reopens the bound task and admits the message. An
authorization failure rolls back the reopen.
- Several recent saved messages from the same conversation on the
finished task start one follow-up run. The wake that reached the owner
is authorized first, and any decline is recorded on that wake. Each
other recent message is authorized in a savepoint, and a message that
fails keeps its own receipt. The newest authorized message is admitted,
and the run's `wakeCommentIds` holds the whole batch. Inside the
admission transaction, the durable request's `authorize` callback locks
the other receipts. If another run took one of them, it refuses the
whole batch (`chat_inbound_wakeup_batch_changed`, stored retry).
Otherwise `onAdmitted` marks them `coalesced` with the run id in the
same transaction. A restart or a refused admission leaves them deferred.
- The sweep acts only on `inbound_wakeup` chat sources. Task-retry chat
receipts that share the `chat-inbound:` key keep their own owner.
- A refused start ends in the state that admission already gives a
refused chat message. A paused agent saves `executionWait` and resumes
when the pause lifts. A budget hard stop or a non-invokable agent
produces a skipped receipt or a saved wait. Nothing retries on a timer.
- The task is missing or reassigned: settle. The task is in backlog, or
the chat source is still in ingress: retry.
  - Other callers (task retry, cleanup) keep their old behavior.
- `issue-assignment-wakeup.ts`: add the optional `wakeCommentIds` input.
It goes into the run context so that the follow-up run carries every
folded message.
- `heartbeat/deferred-receipt-work.ts`: the deferred worker schedules a
wake at its stored `nextAttemptAt`. A notification that names the exact
wake (for example, the run it waited on finished) still resumes it at
once.
- Tests: add `deferred-chat-wake-sweep.test.ts`. Update three scenarios
in `chat-channels.integration.test.ts` whose source is permanently gone
and which now settle.

## Verification

- New `server/src/__tests__/deferred-chat-wake-sweep.test.ts` (embedded
PostgreSQL, real deferred-receipt worker and heartbeat queue, fake
clock, statement counter):
- 12 stale wakes on done or cancelled tasks, 30 idle minutes. Before
(master): 1,344 statements, 84 wake updates, 84 failure log lines in the
idle window. After: 0 statements, 0 updates, 0 log lines. Each wake is
`skipped` with `error = task_closed` after one attempt, and no run
starts.
- A recent wake (5 minutes old) on a done task reopens the task and
starts one run.
- Two recent wakes on one done task start exactly one run. That run's
`wakeCommentIds` holds both comments, and both wakes record its run id.
- A recent wake whose start is refused (paused agent) keeps the saved
pause wait, makes 0 statements in 30 idle minutes, and starts one run
when the pause lifts.
- A chat wake whose task row no longer exists settles once as
`task_missing`.
- When another run takes a batched message before admission, the batch
is refused with no run. The next attempt answers the remaining message
alone.
- A 3-day-old and a 3-hour-old wake on finished tasks both settle. Only
the 3-hour-old one gets a delivery text and one "not started" notice. A
second notice pass and a fresh notice worker (a restart) add nothing,
and the old one is never a notice candidate.
- When a newer saved message on the same finished task lost its source
comment, the older valid message still gets its run, and the newer one
is never folded into it.
- A retryable failure (paused endpoint) makes 3 attempts in 30 idle
minutes (at 0, 1 and 11 minutes), with the reason in `error`. Then the
endpoint is unpaused and a notification names the wake, and it starts a
run at once.
- A wake deferred behind a running run makes 0 statements in 10 idle
minutes. It resumes at once when that run finishes.
- Red/green: on master the first three cases fail, and on this branch
all four pass.
- Focused suites pass locally: `queue.test.ts`,
`deferred-execution-receipt-work.test.ts`,
`heartbeat-held-comment-wake.test.ts`, `human-directed-work.test.ts`,
`owner-chat-instruction-authority.test.ts`,
`explicit-native-continuation.test.ts`, and the 20 "reconsiders the same
GitHub reply after stopped native workspace cleanup" cases in
`chat-channels.integration.test.ts`.
- `pnpm verify --explain` selected 2,728 files, so I ran only focused
suites on this shared machine. CI supplies the broad coverage.

## Risks

- On first deploy, the existing stuck wakes drain on their own. On the
hosted instance, the 359 chat-inbound wakes on done or cancelled tasks
are all older than the 1-hour window, so they settle once as
`skipped`/`task_closed` and start 0 runs. All but 2 of the 364 are older
than a day, and none is younger than an hour, so the deploy starts 0
follow-up runs. The 5 on blocked or todo tasks get one resume attempt
each. They then resume, wait on a persisted execution wait, or follow
the stored retry schedule and settle within about 31 hours.
- Product decision: the owner chose "close silently" for old saved
messages. A wake older than `SILENT_CHAT_WAKE_SETTLEMENT_AFTER_MS`
settles with its reason in `error` and `finishedAt`, and has no visible
chat effect. Its delivery text is not written, no notice is signaled,
and the receipt gets `payload.silentSettlement = true`. Both the notice
candidate query (`wakeup-notices.ts`) and `resolveInboundWakeReceipt`
skip such wakes, so a later pass or a restart never edits an old
"queued" notice. The value of 24 hours from the wake's `created_at` was
picked by the reviewer, not the owner, so it is easy to change.
- A wake between the two bounds (older than 1 hour, newer than 24 hours)
keeps the existing visible result. Its "Your follow-up is queued."
notice changes once to "This follow-up was not started."
- Visible edits on deploy: on the hosted instance, only wakes younger
than 24 hours can get one. There were 2 at measurement time, and likely
0 by deploy. The other wakes close silently.
- Product decision: the owner chose "yes, if recent" for a message that
arrived just as its task finished. The freshness bound is
`CLOSED_TASK_CHAT_WAKE_REOPEN_WINDOW_MS` in `durable-chat-wakeup.ts`.
Its value of 1 hour from the wake's `created_at` was picked by the
reviewer, not the owner, so it is easy to change. Set it to 0 to settle
every closed-task wake without a run.
- Non-chat saved wakes keep the per-pass touch. The other 75 deferred
wakes on the hosted instance are not chat-inbound, so this PR does not
change their timing.
- No change to authorization, budget, pause or approval checks. The
reopen runs in the same transaction as the full chat authorization, and
the authorization still decides. No wait is added before a run's first
turn.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:31:40 -06:00
DottaandClaude Opus 5.5 1c33275ada fix(heartbeat): start the new owner after reassignment away from a queued creator run (#21256)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent often creates a task for itself and then reassigns it
to another agent
> - The creator is busy in its own chat run, so the new task's
assignment run stays queued and does not take the task's execution lock
> - #21030 releases the outgoing assignee's never-started run on
reassignment, but only when that run holds the execution lock
> - Here the run did not hold the lock, so it stayed queued. The new
owner's wake then adopted it as the active run and was deferred behind
it
> - When the queue later withheld the stale run as
`issue_assignee_changed`, nothing promoted the deferred wake, and the
stranded-wake sweep ignores assignment wakes. The new owner never ran
> - This pull request closes all three gaps
> - The benefit is that a reassigned task starts its new owner at once,
and recovery still starts it if a release is missed

## Linked Issues or Issue Description

Refs #21030

**What happened?**
In the paid chat evals `onboarding.first-work` and
`questions-and-approvals.duplicate-answer` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master b1ef1f3022), the
CEO created a task for itself while its chat run was active, then
reassigned it to Morgan. The CEO's queued assignment run was withheld at
the execution gate with "issue assignee changed before the queued run
could start; the new owner will be woken instead". Morgan's wake stayed
`deferred_issue_execution` until teardown, about 4.5 minutes later.
Morgan never ran.

**Steps to reproduce**
1. Give an agent a running run on another task.
2. Create a task assigned to that agent, so its assignment run is queued
without the execution lock.
3. Reassign the task to a second agent.
4. The second agent's wake is deferred behind the stale run and is not
promoted when that run is withheld.

**Expected behavior**
The stale run is released and the new owner gets exactly one run for the
task, promptly.

## What Changed

- `server/src/routes/issues.ts`: on reassignment, when no live run holds
the task's lock, also release the outgoing assignee's never-started
`issue_assigned` run for that task. This uses the existing
`issue_reassigned` stop from #21030. The run is read through a new
`heartbeat.getQueuedAssignmentRunForIssue` accessor in
`heartbeat/run-state.ts`; route test mocks of the heartbeat service gain
that method.
- `server/src/services/heartbeat/queue.ts`: when the queue gate
withholds a queued run as `issue_assignee_changed`, it now releases the
task and promotes the new owner's deferred wake. The promoted run starts
after the claim lock is released (`deferredPostCommitEffects`), like the
neighboring cancellation branches. A failed promotion is logged and left
to recovery.
- `server/src/modules/run-dispatch/adapters/postgres.ts`: the same
cancellation hints each deferred wake of the task to the
deferred-receipt worker in the same transaction, so recovery runs even
if the process stops before promotion.
- `server/src/services/heartbeat/queue.ts`, `deferred-receipt-work.ts`,
`deferred-execution-notifications.ts`: the stranded-wake sweep now
includes deferred `issue_assigned` wakes of the current assignee. It
promotes them only when no queued, running or retrying run remains for
the task, through the normal release admission.
- `server/src/__tests__/heartbeat-reassignment-deferred-wake.test.ts`:
the cohort's state through the real wake path. One case covers the gate
withholding the stale run. One case covers a wake left deferred after
the blocking run finished. Both expect exactly one run by the new owner,
executed.
- `server/src/__tests__/issue-reassignment-queued-run.test.ts`: the
#21030 case now also runs when the queued run does not hold the lock,
and expects exactly one new-owner run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-reassignment-deferred-wake.test.ts
src/__tests__/issue-reassignment-queued-run.test.ts` passes. On master,
both heartbeat cases fail (the wake stays `deferred_issue_execution`)
and the no-lock route case fails (the stale run stays queued). With only
the gate change removed, the gate case fails and the recovery case still
passes.
- Focused suites: `deferred-execution-receipt-work`,
`heartbeat-lock-release-on-reassignment`,
`heartbeat-stale-queue-invalidation`,
`issue-agent-mutation-ownership-routes`, `src/modules/run-dispatch`,
`src/modules/wake-queue` pass.
- `cd server && npx tsc --noEmit` passes.

## Risks

- Behavior change: reassignment now cancels the outgoing assignee's
queued `issue_assigned` run for the task even when it does not hold the
lock. The queue gate already cancelled that run later as
`issue_assignee_changed`; now the new owner does not wait for it.
- The stranded-wake sweep now also reads deferred assignment wakes. It
skips any wake whose task still has a live run, so a wake waiting behind
a real owner is not promoted early.
- Upgrade safety: no schema or saved-state shape changes. Wakes already
saved as `deferred_issue_execution` by the previous release are promoted
by the sweep after deploy.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:36:05 -06:00
DottaandClaude Opus 5.5 e620ce3678 fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can comment, click Retry, or rely on automatic recovery while
an agent works on a task
> - Restart-chaos waves showed accepted follow-ups being dropped, or
delivered twice, around restarts and temporary company admission holds
(accounting still settling after a restart)
> - A dropped comment means the user's request is never acted on. A
duplicate reopens work that is already done
> - This pull request makes accepted work wait through a temporary hold
and resume when it clears. It also stops a comment from being promoted
again after a completed run already received it
> - The benefit is that each accepted follow-up is delivered exactly
once, eventually, unless the task is cancelled or a reached hard stop
applies

## Linked Issues or Issue Description

Refs #19702 (never-started tasks during a temporary hold), #19700
(bounded accounting hold), #14910 (stale reopen from a deferred wake; no
file overlap).

**What happened?**
Restart-chaos waves found these shapes:
1. **Comment wake cancelled.** A comment was saved behind a run and
promoted when the run ended. A temporary company hold refused the
unstarted queued run, and claim cancelled it terminally. The recovery
wake that followed was cancelled the same way, and the task rested
`blocked` without the comment.
2. **Dead Retry.** A Retry click during the hold stayed
`deferred_issue_execution` after the hold cleared. Nothing woke it, and
the issue held no execution lock.
3. **Refused assignment.** A new task's first wake was refused during
the hold. Nothing woke stranded-work recovery when the hold cleared, so
the task stayed `todo` with no run.
4. **Silent comment after recovery.** A comment was saved behind a run
that restart recovery finalized. A company-scoped hint waited out the 30
s deferred-receipt rescan, so the task looked done with the message
unread.
5. **Comment delivered twice.** A run whose provider process was lost
had a saved comment. Its retry started after the comment, received it in
its prompt (task history), did the work, and finished the task. The
saved wake was then promoted anyway, which reopened the done task.

**Expected behavior**
An accepted follow-up (comment, Retry, recovery) is delivered exactly
once, eventually, unless the task is cancelled. A temporary refusal
never ends in a terminal cancel, skip, or block. Reaching a hard stop
remains the board's decision.

## What Changed

- `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued
run stays queued when the block is temporary. A reached hard stop or
blocked unpriced usage still cancels it.
- `recovery/service.ts`: stranded-work recovery waits out a temporary
hold for any task. It does not escalate to `blocked`. This extends
#19702 beyond never-started tasks.
- `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved
out of the recovery service. `budgets.ts` is unchanged.
- `run-cost-accounting.ts`: settling a run's accounting wakes queued-run
dispatch, the deferred-receipt worker, and stranded-work recovery. Held
work waits passively; this is what resumes it.
- `deferred-execution-notifications.ts`: new
`notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run
reaching a terminal status) and `environments.ts` (lease release) now
hint each saved wake of the issue by id, so delivery does not wait out
the company rescan interval.
- `wake-queue/adapters/postgres.ts`: release treats a saved comment as
delivered only when a succeeded run of the wake's agent on that issue
started after the comment's latest revision, and either received the
comment as its own wake input (`wakeCommentIds`) or was a retry of the
interrupted turn whose task history included it. In that case it cancels
the saved wake with "Queued messages were already delivered to a
completed run" and leaves the task as it is. Task history alone never
consumes a message saved for its own turn. One query checks the whole
batch.
- `run-cost-accounting.ts`: settling late receipt debt also sends the
hold-release notifications.
- `doc/execution-semantics.md`: documents the rule.

## Verification

Red on `master` and green with this change, on embedded Postgres:
- `server/src/__tests__/heartbeat-held-comment-wake.test.ts`:
- A promoted comment refused by the hold stays pending, with no
cancelled run and no `blocked` task. It is delivered once after the hold
clears through real `accountRunCost`.
- A Retry clicked during the hold starts exactly once after accounting
settles, which wakes its owner.
- A new task whose assignment was refused during the hold starts once
accounting settles, which wakes stranded-work recovery.
- A comment that a later succeeded retry already received as history, or
that a later run received as wake input, is consumed without reopening
the task.
- Guard tests (green on both):
- A comment is still delivered in each of these cases: the later run
never received it, it was edited after that run started, it was posted
while the run was already running, or a non-retry run only saw it as
history.
  - A task whose continuation was refused waits out the hold.
  - A reached hard stop is still refused.
- `server/src/__tests__/native-run-projection-work.test.ts`: projecting
a recovered run terminal hints the saved comment's own wake (red on
`master`).
- Related suites pass locally (14 files, 482 tests): wake-queue, queued
comments routes, refused assignment wake, native run projection.
- Live replays (local, unmasked, kept instances):
- `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The
comment's run was created at 03:56:24 during the hold, stayed queued,
started at 04:06:24 when the hold cleared, and succeeded. It is the only
run carrying the comment.
- `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it
reproduced shape 4. Run 592d772f was killed 250 ms after
`run.result.accepted`, released and finalized at restart, and the task
rested `done` with the comment still saved. Calling the existing sweep
on the kept database promoted it, which confirmed that the missing piece
was the trigger.

## Risks

- A queued run can now sit queued for the length of a temporary hold,
which #19700 bounds. It does not hold the issue's execution lock, so
later comments still coalesce into it.
- The delivered-comment rule depends on the run's recorded prompt
context. A succeeded run that received the comment in its history counts
as delivery, even if the agent did not act on it. This matches
exactly-once delivery semantics.
- Settling accounting now emits three extra notifications per run. They
are coalesced per company.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution,
live restart-chaos replays).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 00:24:09 -06:00
51a65af9b8 fix(recovery): continue preserved preparation after process loss (#19464)
## Thinking Path

> - Paperclip manages AI agents and their execution state.
> - A server restart can stop workspace preparation before provider
execution.
> - Workspace recovery must preserve files before the task can continue.
> - Recovery handled interrupted preparations but skipped failed
process-loss records with the same no-provider certificate.
> - A withheld first retry could also hide the remaining retry budget.
> - This change resumes certified preparations through normal admission
after file recovery.

## Linked Issues or Issue Description

**What happened?**
A process-loss attempt remained failed after workspace recovery
completed. Its first retry had been cancelled before dispatch by the
workspace hold. The task had no running attempt or automatic
continuation.

**Expected behavior**
Continue once after the server proves that provider work did not start,
file preservation completed, and the exact leases stopped. Retain the
existing failure budget and all task admission gates.

**Steps to reproduce**
1. Stop a Runner preparation before native runtime selection. Record the
restart recovery bootstrap certificate.
2. Retain the workspace and withhold the first automatic retry before
dispatch.
3. Complete file preservation and restart the recovery worker.
4. Observe that the old code schedules no successor. The regression test
reproduces this with PostgreSQL and real filesystem operations.

**Paperclip version or commit**
Reproduced against master at `2f35eb1080`. Related prior work: #19422
continues saved user input after verified copy-back. This change handles
automatic continuation of certified process-loss preparation.

## What Changed

- Accept failed process-loss preparation only with the durable
no-provider bootstrap certificate and existing execution, lease and file
checks.
- Discover and settle the recovered source through the existing
preparation recovery path.
- Continue from a verified withheld retry without rewriting its history
or resetting its failure count. Adopt it only as the sole retry child;
multiple-child history reuses the newest existing child
deterministically.
- Exercise automatic dispatch, preserved-file readback, one provider
turn, accepted result and task completion in the local native-session
fixture.
- Keep a durable continuation receipt after file-hold settlement until a
successor is recorded. Known gates wait for existing eligibility events.
Unresolved delivery keeps its shared recovery deadline.
- Document the recovery and retry-budget rules.

## Verification

- Red: both new PostgreSQL/filesystem regressions failed with zero
scheduled successors where one was required.
- Green: 54 focused recovery, automatic-dispatch, interruption,
multiple-child history and admission safety cases pass on `13c27c2c2f`.
The full PostgreSQL admission adapter suite passes all 83 tests; server
source declaration typecheck also passes. Local environment: Darwin
arm64, Node 26.4.0, PostgreSQL 18.1, Vitest 5.0.3.
- Additional red/green: interruption after committed settlement,
pause/resume, failed enqueue after cursor advancement, and absence of an
earlier retry all failed before their repairs. Restart before enqueue,
restart after the successor claim, and pause/resume now preserve one
continuation. A rolled-back eligibility change cannot start execution.
Deferred and delivered records have no retry deadline.
- Local acceptance uses the real heartbeat dispatcher, isolated files
and PostgreSQL, with a simulated native provider. It reads the preserved
file, writes continued work, accepts the structured result and completes
the task once.
- Negative cases retain no-certificate, provider-evidence, live-owner,
Stop, missing-stop-receipt, wrong-snapshot, human-block, pause,
reassignment, approval and exhaustion fences.
- CI fixture regression: 4 error-propagation/concurrent-resolution cases
pass after adding the new obligation read to their transaction stubs;
existing assertions remain unchanged.
- Existing idle-drain and notification contract regressions: 4 tests
pass after restricting recovery wakes to actual durable obligations.
Late approvals block scheduled, queued and final dispatch.
- Additional existing retry-scheduling suite: 62 cases pass locally. The
pre-provider quota case exceeds its existing 5-second terminal wait on
this loaded Darwin host, including when tested alone and with the
preceding retry implementation. It remains a local timing limitation;
the assertion and timeout were not changed. All Linux CI server shards
pass on the combined head, including this unchanged suite.
- Existing run-control, legacy-recovery and shared execution-control
lifecycle suites: 128 tests pass with the review repair. Required CI
supplies final-revision broad coverage.
- Real-Codex acceptance was not run: automatic approval review rejected
authenticated provider execution because it could incur cost/external
effects and required explicit approval. No provider call ran; the
controlled-provider acceptance covers the causal restart and durable
scheduling behavior.
- Hosted acceptance and recovery of existing tasks require rollout. They
have not been claimed from the local provider fixture.
- Final-head review: Greptile 5/5 on `13c27c2c2f`, no outstanding
findings. Another contributor added the multiple-child history safety
repair in `13c27c2c2f` while CI was running. The delta was inspected and
the 54 focused recovery cases rerun successfully. Required CI passes on
that exact combined head:
https://github.com/paperclipai/paperclip/actions/runs/42241794918. All
55 successful checks include aggregate `ci / verify`, typecheck, build,
canary, all server and E2E shards, and Runner package/Rust/static
checks. The separate Paperclip Review service check remains queued
waiting for its assigned agent; no validated assessment has been
returned.

## Risks

A malformed preparation certificate or a false no-execution receipt
could permit duplicate work. The existing checks still reject provider
execution records, native coordinators, live controllers, unsettled
leases and unverified files. The pending continuation must survive a
restart without polling known admission blockers. A withheld retry
retains its original row and consumes its remaining failure budget.
Normal admission checks current ownership, approvals, pause and budgets.

## Model Used

OpenAI GPT-6 family via Codex. The exact model ID and context window are
unavailable in session metadata. Used reasoning, code editing and tool
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal task id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-06 09:36:46 -06:00
DottaandPaperclip 9ca9af544a Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at 6a4c193906.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 16:00:35 -05:00
DottaandPaperclip 3ee4c5b613 Make saved execution receipt recovery react to durable work (#18409)
Reviewed and verified at 8f306e0a6e.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 10:03:59 -05:00