mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
e5abcbf7a9
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, native restart recovery claims an unfinished run
and prepares it again on the same heartbeat run.
> - The restart claim holds a lease on the run's native coordinator row.
For a remote claim with durable authority, the lease is 60 s.
> - When that preparation fails, the failure path first preserves the
remote workspace. Then it settles the run.
> - When preparation outlived the claim lease, preservation took the
run's own lapsed claim for another owner. It threw
`NativeWorkspaceFinalizationBusyError`, and the execution returned
without settling. The run stayed `running` with no event until a later
scan reclaimed it (4.5 min in one wave, never in another).
> - This pull request releases exactly that lapsed claim and asks
restart recovery to decide on the run right away. It also logs the cause
when the failure path defers.
> - The benefit is that a failed restart-claim preparation is reclaimed
or blocked visibly within seconds, instead of hanging.
## Linked Issues or Issue Description
Refs #20514 (the same restart-chaos investigation).
**What happened?**
In restart-chaos wave `w-20261110T101517-s11936` (Daytona R1, K2, ST1),
run `9d39550a` got a `bootstrap_incomplete` restart claim (generation
3). Its `heartbeat.prepare.run_output` step waited 141 s through an ST1
storage outage. The 60 s claim lease lapsed meanwhile, and the begin
failed with `run_log_writer_authority_lost` when storage came back. The
inner failure path then called `preserveNativeFailedWorkspace`. Its
`ownsClaim` check required an unexpired lease, so the run's own claim
counted as another owner and the call threw
`NativeWorkspaceFinalizationBusyError`.
`preserveNativeWorkspaceOnFailure` set `nativeOwnershipHeld` and the
execution returned before `runCompletion.failRun`. No failure was
recorded, and nothing was logged. The run stayed `running` until a later
scan reclaimed it 4.5 minutes later. In a second wave no such scan came,
and the run stayed `running` for 30 minutes.
**Expected behavior**
A failed preparation must settle or retry, or be claimed promptly by its
next owner. It must not stay `running` with no record.
**Steps to reproduce**
1. Claim a remote native run for restart recovery.
2. Keep its preparation busy past the 60 s claim lease, for example with
a storage outage during `runOutput.begin`.
3. Let preparation fail. The run stays `running` with no failure event.
**Paperclip version or commit**
`master` at 23fa37f5bb.
## What Changed
- `native-execution-lease.ts`: new `releaseLapsedNativeRestartClaim`. It
is one conditional UPDATE that releases the claim only when the run is
`running`, the coordinator is `observed` with no result, and it still
names this exact `leaseOwner`, `controllerGeneration` and `attempt` with
an expired lease. A successor claim, a result, or a live lease leaves
the row unchanged.
- `heartbeat.ts`: when failed-workspace preservation defers on a
restart-claim execution, the failure path tries that release. If it
succeeds, the execution schedules a restart-recovery pass for the run
(`scheduleNativeSessionResumeDispatch(..., "restart")`) after it leaves
`activeRunExecutions`. Recovery then reclaims the run, or blocks it with
a named reason, right away. The path also logs the redacted cause, the
claim kind and whether the claim lapsed. Before, this return was silent.
- `preserveNativeFailedWorkspace` is unchanged. It still refuses an
expired claim (`refuses a stale expired recovery claim`).
- New `heartbeat-restart-claim-lapse.test.ts`. A restart-claim execution
whose preparation fails after its claim lease lapsed, on a run with a
prepared remote workspace, gets an immediate new restart-recovery
decision. It failed before the change: the run stayed `running` with the
lapsed claim and recovery answered `controller_still_alive`.
Not in this PR: renewing the claim lease during preparation for
`bootstrap_incomplete` claims (recovery.ts). Another restart-chaos fix
owns that change. This PR makes sure a lapsed claim's failure is handed
back at once either way.
## Verification
- `cd server && npx vitest run
src/__tests__/heartbeat-restart-claim-lapse.test.ts`: passed. Without
the heartbeat change it times out.
- `cd server && npx vitest run
src/services/native-runtime/native-failed-workspace-recovery.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts
src/services/native-runtime/native-execution-lease.test.ts`: passed (195
tests across the four files).
- `cd server && pnpm typecheck`: no errors.
- Live Daytona proof uses `--native-trigger` from #20538. Results follow
in a comment.
## Risks
- Low risk. The release matches only the exact lapsed claim of a
running, result-less, `observed` run in one statement. A takeover
between the check and the release changes the owner or generation, so
the release does nothing.
- The run is not settled by this execution. Restart recovery makes the
decision under its own rules, which is the same path a later sweep would
take.
- Previous-release story: no persisted shape or identity changes. Rows
saved by the previous release are read and released the same way.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>