Files
PaperClipAI/server/src
DottaandClaude Opus 5.5 77168d3367 fix(recovery): settle a dead controller's result lease for a run a pause cancelled (#20113)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop pauses the agent, which cancels its running work
and waits until every execution has provably stopped
> - The shared ownership reconciler (#19933) settles an accepted
result's executor lease when its controller is proven dead, so
finalization does not wait out the 20-minute lease
> - In the restart-chaos H1 wave, the pause cancelled a run 0.36 s after
its result was accepted, and a hard kill then took the controller
> - The settle only acted on runs that were still `running`, so the dead
lease held finalization, the agent stayed in `pausing`, and every later
task for it was stranded
> - This pull request settles that lease for a cancelled run too, and
records the budget pause with its reason
> - The benefit is that a pause crossed by a restart completes at once
and resume continues normally

## Linked Issues or Issue Description

Refs #19933, #20101

**What happened?**
Restart-chaos wave w-20261108T210118-s9802 (scenario H1). An agent-scope
budget hard stop paused agent "Chaos claude". The pause cancelled run
`dfc22298` at 21:02:27.818, 0.36 s after its result was accepted
(finalization phase `workspace_finalizing`, executor lease until
21:21:55, controller PID 78157). A K4 SIGKILL followed. After the
board's `raise_budget_and_resume`, the agent stayed `status=paused`,
`lifecycle_state=pausing`, `lifecycle_holds=[]`, with its lifecycle
operation at `attempts: 90`, `hostComplete: false` and no error. The
pause host step (`stopInvocationsForAgents` →
`agentExecutionsHaveStopped`) returns `pending` while any coordinator
holds an owner lease with an accepted result.
`settleNativeAcceptedResultLease` required `heartbeat_runs.status =
'running'`, so it returned `lease_no_longer_held` for the cancelled run.
The pause also wrote no activity entry.

**Expected behavior**
A proven-dead controller's accepted-result lease is settled whatever
stopped the run, so finalization finishes and the pause completes. The
budget pause is logged with its reason.

**Steps to reproduce**
Pause an agent right after its run's result is accepted, then SIGKILL
the server. After restart, the agent stays in `pausing` until the lease
expires (up to 20 minutes).

## What Changed

- `native-accepted-result-lease-settlement.ts`: settle the lease for a
`running` or `cancelled` run. All other guards are unchanged: exact
owner compare-and-set, owner proven gone by `proveOwnerGone`, remote
copyback excluded, another host never proof.
- `agent-lifecycle.ts` `applyAgentBudgetHoldInTransaction`: when the
budget hold is added or removed, records `agent.paused` /
`agent.resumed` with `reason: "budget"`, the lifecycle state and the
holds, in the transaction that applies the hold. Applying the same hold
again records nothing.
- `budgets-service.test.ts`: its activity-log mock now also stubs
`persistActivity`.

Not changed, by design: resume still waits for an incomplete pause to
finish before resuming. The pause's host step proves the agent's
executions stopped before new work starts. In H1 it was blocked only by
the dead lease, which is now settled at restart.

## Verification

- `heartbeat-native-restart-suspension.test.ts`: new "finalizes an
accepted result whose run a pause cancelled before a hard kill took its
controller". It fails on master (lease still 20 minutes out) and passes
with the fix (lease owner cleared, finalization past
`workspace_finalizing`). Whole file: 41 passed.
- New `agent-budget-hold-activity.test.ts`: the pause and release
entries, with the reason, written once each.
- `budgets-service.test.ts` (19), `budgets-history.test.ts`,
`company-portability-routes.test.ts`,
`agent-lifecycle/__tests__/lifecycle.test.ts`: passed.
- `server` typecheck (`scripts/typecheck-layers.mjs`, low-memory): no
errors.
- Not run: the H1 chaos rerun (Phase 3 schedules it after merge).

## Risks

- A cancelled run's accepted result is now finalized promptly after a
hard kill instead of after the lease term. The finalizer already handles
a run cancelled during finalization; the settle only shortens the wait.
- New activity entries on budget pause and release. Run-log and activity
only; no Telemetry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 17:06:13 -06:00
..
…
…
…
…
…