## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. The local control plane keeps the runner's controller state in
an encrypted Postgres checkpoint.
> - When the control plane restarts, restart recovery classifies each
live native run and decides to reattach, resume, bootstrap again, or
hold the task.
> - A restart chaos wave killed the control plane twice while a Daytona
runner was connecting. The run was claimed and the sandbox runner was
launched, but `session.open` never completed. No provider evidence
existed.
> - Recovery waited correctly for the old 60 s controller lease to
expire. Then the classifier had no rule for a claimed remote bootstrap.
It returned `ambiguous_remote_recovery_evidence` and blocked the task as
"Workspace repair required".
> - The controller writes each command to the Postgres checkpoint before
it sends it. After the lease expires, that checkpoint is the complete
record of what reached the sandbox runner.
> - This pull request lets recovery prove from that checkpoint that no
provider work exists, and then start the same-run bootstrap again.
> - The benefit is that a restart during Daytona runner startup
continues automatically instead of stopping the task.
## Linked Issues or Issue Description
**What happened?**
A restart during Daytona remote runner startup (native ACPX Claude, two
SIGKILLs about 20 s apart) left the task blocked. The run history showed
`remote_controller_lease_live` twice, then
`ambiguous_remote_recovery_evidence` with disposition `blocked`. The
task showed "Workspace repair required"
(`native_restart_recovery_blocked`) and did not continue.
**Expected behavior**
When the durable controller record proves that the runner received only
bootstrap commands, recovery continues the run with a fresh bootstrap.
Any evidence of provider work keeps the hold.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Kill the control plane after the sandbox runner connects and before
`session.open` completes.
3. Restart the control plane and wait for the old controller lease to
expire.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`
`classifyNativeRunnerRecoveryEvidence`, remote branch. It admitted only
an unclaimed bootstrap (`remote_execution_never_claimed`, #16349) or a
run with a provider checkpoint. A claimed bootstrap with no provider
evidence fell through to `ambiguous_remote_recovery_evidence`.
The decrypted checkpoint of the failed run held `connectionCount: 1`,
`run.prepare` delivered once and pending, `session.open` pending and
never delivered, no committed events, and `ackedSourceSeq: 0`.
Refs #16349, #19524
## What Changed
- `native-bootstrap-proof.ts`: move the incomplete-bootstrap predicates
from the executor into a small pure module. Restart recovery and the
executor now use the same proof. No behavior change.
- `native-runner-checkpoint.ts`: add
`nativeRunnerCheckpointProvesIncompleteBootstrap`. It reads only a
checkpoint with the bootstrap parts (header, cursor, `command:1`,
`command:2`), checks the run, runner instance, and session identity, and
applies the shared proof. A remote runner journal is not on this host,
so the proof uses the controller record alone.
- `native-restart-recovery.ts`: for a classic sandbox run whose owner
lease has expired, with a durable checkpoint, no session checkpoint, no
provider evidence, and no staged attachment, claim
`bootstrap_incomplete` with the reason
`remote_bootstrap_proven_incomplete` when the checkpoint proof holds. At
three provider attempts, block with
`execution_recovery_budget_exhausted`.
- `native-session-executor.ts`: add
`restoreRemoteBootstrapCheckpointForRecovery`. For a
`bootstrap_incomplete` recovery on a sandbox target, it restores the
Postgres checkpoint over the local external-state marker. The existing
incomplete-bootstrap proof then checks the restored state before it
quarantines the root and bootstraps again.
- `doc/run-log-events.md`: document the new disposition reason.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote bootstrap"`. Before the fix, the proven case and the budget
case returned `blocked` / `ambiguous_remote_recovery_evidence`.
- New cases: the proven claimed bootstrap gets `bootstrap_incomplete`
(attempt kept, generation +1, run still `running`, no recovery action).
The hold stays for a queued `turn.start`, an acknowledged event, adopted
state, no checkpoint, and a spent budget. The executor helper restores
the checkpoint only for a bootstrap recovery on a classic sandbox and
only over a marker for the same run.
- Regression set: `native-runner-restart-recovery.integration`,
`native-session-executor`, `native-restart-recovery`,
`native-bootstrap-retry.integration`, `heartbeat/recovery`: 1383 passed,
14 skipped.
- Live Daytona replay of the failing chaos task (double SIGKILL during
remote runner startup), 3 runs: REPLAY_RESULTS
## Risks
- A fresh bootstrap acquires a new sandbox. The old sandbox keeps its
lease until run settlement releases all run leases. The old runner gets
no turn command, because only the controller issues one and the new
controller never connects to it.
- If `session.open` reached the old runner, the old provider session can
exist without a turn. No provider work or tool call can happen in it.
- Any checkpoint shape other than the bootstrap-only shape keeps the
previous fail-closed hold.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
