mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
DottaandClaude Opus 5.5
15ee6a25a6
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live run after
the control plane restarts.
> - #19583 lets recovery start the same-run bootstrap again when the
Postgres controller checkpoint proves that the runner received only
bootstrap commands.
> - A live replay found an earlier window. The control plane died after
the executor claimed the run and staged the sandbox workspace, but
before it asked for a runner process. No runner checkpoint existed, so
#19583 did not apply.
> - Recovery classified that run as `ambiguous_remote_recovery_evidence`
and blocked the task. The task also showed "Workspace repair required",
which did not describe the real cause.
> - This pull request proves the "nothing reached a runner" case from
durable evidence and starts the bootstrap again. It also names the
restart in the settled hold note.
> - The benefit is that a restart at any point of Daytona startup before
runner launch continues automatically.
## Linked Issues or Issue Description
**What happened?**
A chaos replay killed the control plane about 2 s after a claimed ACPX
Claude run started on Daytona. The kill came after `stage.workspace` and
environment acquisition, before `native.process_start_requested`. The
restarted server took over because the previous controller process was
dead. It then returned `ambiguous_remote_recovery_evidence` with
disposition `blocked`. The task rested blocked as "Workspace repair
required".
Evidence in the saved database at that point: coordinator `attempt: 1`,
`controllerGeneration: 1`, previous controller identity recorded and
dead. Native workspace sync was `prepared`. There were no runner
checkpoint rows, no session checkpoint, no process PID or start time, no
provider session, and no execution-start or runner-sourced event. The
local session root was an empty directory.
**Expected behavior**
When durable evidence proves that no runner process was requested and no
controller checkpoint exists, recovery continues the run with a fresh
bootstrap. Any evidence that a runner may have started keeps the hold.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Kill the control plane after workspace staging and before the
executor requests the runner process.
3. Restart the control plane.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`,
function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only
no-checkpoint remote rule required an unclaimed coordinator (`attempt:
0`, no controller identity). A claimed run with no launch evidence fell
through to "ambiguous".
Refs #19583, #16349
## What Changed
- `native-restart-recovery.ts`: add the `remote_runner_never_launched`
proof, which leads to `bootstrap_incomplete`. It applies only to a
classic sandbox run with a prepared workspace and the preparation
fencing version. The run must have a real executor claim (controller
identity and lease owner) whose controller is gone (takeover allowed).
It needs no durable checkpoint, no session checkpoint, no process
fields, no provider session, and no provider evidence. The run must also
have no `native.process_start_requested`, process identity,
`adapter.invoke`, `tool.execution.started`, or runner-sourced event. The
same three-attempt budget applies. The shared event query moved into
`hasNativeExecutionStartEvidence`.
- `native-session-executor.ts`:
`restoreRemoteBootstrapCheckpointForRecovery` removes an empty session
root that preparation created. Without that step, the fresh bootstrap
read it as `durable_identity_unreadable`.
- `execution-recovery-resolution.ts`: the new
`executionRecoverySettlementNote` names a settled
`native_restart_recovery_blocked` hold as a stop after a server restart.
It no longer shows "Workspace repair required" because a later workspace
export was retried. Other causes keep their notes.
- `doc/run-log-events.md`: document the new reason.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "unlaunched|never requested|Postgres checkpoint"`. Before the fix,
the proven case (lease live with a dead owner, and lease expired) and
the budget case returned `blocked` /
`ambiguous_remote_recovery_evidence`. The empty-root executor case
failed.
- Fail-closed cases: these keep the hold:
`native.process_start_requested`, a process identity event,
`adapter.invoke`, a runner-sourced event, a recorded PID, a recorded
provider session, and a spent budget. A live claiming controller returns
`awaiting_evidence` / `controller_still_alive`. The existing
"contradictory evidence" cases (`claimed` with no controller identity,
`launch`, `process`, and others) stay blocked.
- Wording: `server/src/services/execution-recovery-resolution.test.ts`.
- Regression set (10 files, including `workspace-recovery`,
`native-safe-replacement`, `heartbeat/recovery`,
`native-session-executor`, and `native-restart-recovery`): 2035 passed,
14 skipped.
- Live Daytona replay of the failing chaos task, 3 runs, sequential,
ACPX Claude, double SIGKILL, current master plus this change. Image:
content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from
c6eecac1e0, with runner inputs pinned to it, because the image-only
build sat in the Actions queue. Result: **3/3 PASS** (R1 pass; R2, R4,
R5, R6 and T pass). Each run's faults landed after the runner launched.
Recovery waited on the live remote controller lease, then continued
without a Retry. Run 3 continued through `reattach_remote_runner`. The
saved logs for runs 1 and 2 do not name their claim kind. The earlier
pre-launch window (the subject of this PR) is not hit reliably by replay
timing. The integration tests cover that window with the same evidence
shape as the saved failing database.
## Risks
- The proof depends on the executor writing
`native.process_start_requested` before a backend may spawn a runner
(`onSessionAdmission` in `executeNativeSession`). A future change that
spawns a runner earlier would weaken this proof. The fail-closed tests
pin the event as the gate.
- A fresh bootstrap acquires a new sandbox. The staged sandbox from the
dead attempt keeps its lease until run settlement releases all run
leases. It never ran a runner.
- The note change affects only settled holds with the cause
`native_restart_recovery_blocked`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
21 lines
1.2 KiB
TypeScript
21 lines
1.2 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { executionRecoverySettlementNote } from "./execution-recovery-resolution.js";
|
|
|
|
describe("executionRecoverySettlementNote", () => {
|
|
it("names a restart-recovery hold instead of asking for a workspace repair", () => {
|
|
const note = executionRecoverySettlementNote({ current: true, cause: "native_restart_recovery_blocked",
|
|
workspaceRestoreFailed: true });
|
|
expect(note).not.toContain("Workspace repair required");
|
|
expect(note).toContain("after a server restart");
|
|
});
|
|
|
|
it("keeps the workspace repair note for other holds with a restore failure", () => {
|
|
expect(executionRecoverySettlementNote({ current: true, cause: "execution_finalization_deadline_exceeded",
|
|
workspaceRestoreFailed: true })).toContain("Workspace repair required");
|
|
expect(executionRecoverySettlementNote({ current: true, cause: "execution_finalization_deadline_exceeded",
|
|
workspaceRestoreFailed: false })).toContain("Automatic recovery stopped. Recorded work is preserved");
|
|
expect(executionRecoverySettlementNote({ current: false, cause: "native_restart_recovery_blocked",
|
|
workspaceRestoreFailed: true })).toContain("Recovery closed");
|
|
});
|
|
});
|