mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
DottaandClaude Opus 5.5
11c07c622c
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live or due run
after the control plane restarts.
> - #19616 added the `remote_runner_never_launched` proof. A claimed
remote run with no runner process request and no controller checkpoint
starts its bootstrap again.
> - A graceful shutdown closes the local runtime listener before its
drain. A remote run that is still starting then fails with
`local_runtime_api_listener_unavailable`. The executor settles it as a
same-run `bootstrap_retry` and releases its lease. #19596 treats this as
a restart detach only for local runners.
> - When the server restarted with that retry due, restart recovery
claimed the row. The proof accepted only rows that still held a lease,
so recovery blocked the task as ambiguous.
> - This pull request lets the same proof also admit that settled
`bootstrap_retry` row.
> - The benefit is that a graceful restart during Daytona startup
continues automatically.
## Linked Issues or Issue Description
**What happened?**
A chaos wave restarted the control plane gracefully (SIGTERM) while an
ACPX Claude run on Daytona was starting. At 2 s after `stage.workspace`,
the run failed with `local_runtime_api_listener_unavailable`, and the
executor persisted "same-run resume persisted" with `recoveryMode:
bootstrap_retry`. After the restart, restart recovery returned
`ambiguous_remote_recovery_evidence` with disposition `blocked`. The
task rested blocked.
Evidence in the saved database at that point:
- Coordinator `phase: retryable_failure`, `attempt: 1`,
`controllerGeneration: 1`, controller identity recorded, and no lease
owner.
- `failureDetail` recorded `recoveryMode: bootstrap_retry`,
`checkpointExists: false`, `providerEventsExist: false` and
`providerSessionEstablished: false`.
- The run had no runner checkpoint rows, no
`native.process_start_requested`, no process identity, and no
runner-sourced event. It also had no PID and no provider session.
**Expected behavior**
Restart recovery starts the bootstrap again, as it does for a claimed
run that holds a lease (#19616) and for a local run.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Send SIGTERM to the control plane after workspace staging and before
the runner is requested. The run settles as a same-run
`bootstrap_retry`.
3. Restart the control plane while that retry is due.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause: in
`server/src/services/native-runtime/native-restart-recovery.ts`, the
`remoteRunnerNeverLaunched` condition required `phase === "observed"`
and a lease owner. The settled `retryable_failure` / `bootstrap_retry`
row failed that check. The remote classifier then returned
`ambiguous_remote_recovery_evidence`.
Refs #19616, #19596, #19583
## What Changed
- `native-restart-recovery.ts`: `remoteRunnerNeverLaunched` now accepts
one of two coordinator shapes:
- an `observed` claim that holds a lease (unchanged);
- a `retryable_failure` with `recoveryMode: bootstrap_retry`, run status
`failed`, and no lease owner.
- All other proof conditions are unchanged. The run must still have no
process start, process identity, adapter, tool or runner-sourced event.
It must still have no process fields, no provider session, no provider
evidence and no checkpoint. The same three-attempt budget applies.
- An `exact_checkpoint_resume` retry keeps its own admission path.
- Admission: before claiming a settled retry, restart recovery applies
the same admission as the resumption reconciler
(`readNativeSessionResumptionAgent`): task pause hold, agent status,
dependencies, ownership, and competing runs. An ineligible retry is
deferred with `native_retry_admission_ineligible` and keeps waiting.
- `doc/run-log-events.md`: note the settled `bootstrap_retry` case.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "bootstrap retry"`. Before the fix, the proven case and the budget
case returned `blocked` / `ambiguous_remote_recovery_evidence`.
- New cases: the proven case gets `bootstrap_incomplete`. The
coordinator returns to `observed`, `nextAttemptAt` is cleared, the run
returns to `running`, and no recovery action is created. A requested
process start or a committed runner event keeps the hold. A spent budget
blocks with `execution_recovery_budget_exhausted`. An
`exact_checkpoint_resume` retry does not take this path.
- Regression: `native-runner-restart-recovery.integration`,
`heartbeat/recovery`, `native-restart-recovery`,
`native-bootstrap-retry.integration` and
`native-remote-retry-admission`: 337 passed, 14 skipped.
- Admission: a paused agent's due `bootstrap_retry` is deferred with
`native_retry_admission_ineligible`. It is not started (red before the
follow-up commit, green after).
- Live Daytona replay of `w-20261106T234719-s12/t02`, 3 runs, ACPX
Claude B1, using the recorded K4, K2 and K1 sequence. Image: the wave's
own `paperclip-daytona-runner@sha256:2767e91c…`, with runner inputs
pinned to its revision e482f0f632 locally. The replays ran the first
commit of this PR.
- Runs 1 and 3 passed R1. Run 2 passed R3: one Retry worked on the first
click.
- Runs 1 and 2 failed R6 because the wave-end sweep found a run lease
still in `pending_cleanup`. In run 1 the environment delete also timed
out. These failures are not in this change's path.
- The restarts landed in other startup windows, so none hit this exact
`bootstrap_retry` state. The integration tests cover that state with the
same evidence as the saved failing database.
- Run 2 also exposed a separate gap. A graceful drain that suspended a
remote run after the runner launched (`awaiting_runner_reattach`) was
blocked as ambiguous on the next boot. The Retry recovered it. That case
is outside this PR.
## Risks
- The proof still depends on the executor writing
`native.process_start_requested` before a backend may spawn a runner.
The fail-closed tests pin that gate.
- Restart recovery and the same-run resumption reconciler can both see a
due row. Both claim with a conditional update on phase and attempt, so
only one can win.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>