From 60b9147e7cb52c1166838a53bf3b8ce6f718a86f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 08:00:45 -0600 Subject: [PATCH] fix(native): accept harness diagnostics in a remote session-opened bootstrap proof (#20543) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) keep their controller state in a Postgres checkpoint. The controller writes every command to that checkpoint before it sends the command. > - After a restart, recovery uses that record to prove that a drained remote bootstrap reached no provider turn. If so, it bootstraps the run again on the same heartbeat run. > - A graceful restart seconds after native start often catches a runner that opened its session and emitted ACPX `harness.diagnostic` events, but was never sent a turn. > - The remote no-turn proof did not accept `harness.diagnostic`, so recovery blocked these runs as `ambiguous_remote_recovery_evidence` and the task needed a manual Retry. > - This pull request accepts `harness.diagnostic` in that proof. The proof's command check still requires that no turn command was ever issued. > - The benefit is that a restart just after native start re-bootstraps the run automatically. ## Linked Issues or Issue Description Refs #20538 (the harness trigger that reproduced it) **What happened?** Restart-chaos wave `w-20261110T091639-s11936` t02, run `c4fc7f03` (Daytona R1, K2): a K2 landed 13 s after native start. Recovery returned `ambiguous_remote_recovery_evidence`, and the run blocked with `native_restart_recovery_blocked` (R3 retry_needed). I reproduced it with a kept database in `w-20261110T132656-s11936` t02, run `ecd98d55`, using `--native-trigger native-start+5` from #20538. Decoded controller checkpoint: - commands: `run.prepare`, `session.open`, `session.goal.get` (no turn command); - committed events: `session.started`, `session.capabilities.updated`, `session.goal.snapshot`, `harness.diagnostic`, `harness.diagnostic`, `session.goal.snapshot`; - `ackedSourceSeq` 6, no `sessionCheckpoint`, no projected provider events. `nativeRunnerCheckpointRemoteBootstrapProof` returned `null` only because `harness.diagnostic` was not in `SESSION_OPEN_EVENT_TYPES`. **Expected behavior** Recovery classifies the run `bootstrap_incomplete` (`remote_session_opened_without_turn`) and bootstraps it again. **Steps to reproduce** 1. Run a Daytona R1 codex wave with `--faults K2 --native-trigger native-start+5`. 2. The suspended run is blocked with `ambiguous_remote_recovery_evidence`. **Paperclip version or commit** `master` at c874cfa4da. ## What Changed - `native-bootstrap-proof.ts`: add `harness.diagnostic` to `SESSION_OPEN_EVENT_TYPES`, which `runnerdStateHasNoProviderTurn` uses. The proof still requires every command to be `run.prepare`, `session.open` or runner maintenance (`session.goal.get` and similar), every committed event to be accounted for in order, and no adopted state or warm transition. A turn command or any turn event still fails it. The local opened-session proof already accepted `harness.diagnostic` through `PRE_TURN_EVENT_TYPES`. - `native-runner-restart-recovery.integration.test.ts`: a case built from the observed record (goal read plus two diagnostics). It is classified `bootstrap_incomplete` with `remote_session_opened_without_turn`, and the executor's restored-checkpoint proof agrees. A diagnostic followed by `turn.started` stays blocked. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 163 passed, 14 skipped. The new diagnostics case failed before the change with `{ kind: "blocked" }`. - Live: a Daytona R1 wave with `--native-trigger native-start+5` follows. Results will be in a comment. ## Risks - Low risk. `harness.diagnostic` is a harness report, not provider input. Provider work needs a turn command, which the controller writes to this record before sending, and the proof still rejects any turn command or turn event. - Previous-release story (upgrade-safe run state): no persisted shape changes. A controller checkpoint saved by the previous release with diagnostics and no turn is now re-bootstrapped instead of blocked. A record with a turn keeps the same named refusal. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../native-runtime/native-bootstrap-proof.ts | 1 + ...unner-restart-recovery.integration.test.ts | 36 +++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/server/src/services/native-runtime/native-bootstrap-proof.ts b/server/src/services/native-runtime/native-bootstrap-proof.ts index 0c6fde2767f..90d169be4cc 100644 --- a/server/src/services/native-runtime/native-bootstrap-proof.ts +++ b/server/src/services/native-runtime/native-bootstrap-proof.ts @@ -35,6 +35,7 @@ const SESSION_OPEN_EVENT_TYPES = new Set([ "session.updated", "session.capabilities.updated", "session.goal.snapshot", + "harness.diagnostic", ]); /** diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index b6af62a30fc..ba518e9e8b5 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -2787,6 +2787,42 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); } }); + it.each<[string, string[], string]>([ + ["harness diagnostics", ["session.started", "session.capabilities.updated", "session.goal.snapshot", "harness.diagnostic", + "harness.diagnostic", "session.goal.snapshot"], "bootstrap_incomplete"], + ["a turn event", ["session.started", "harness.diagnostic", "turn.started"], "blocked"], + ])("classifies a drained remote session that read its goal and committed %s", async (label, eventTypes, kind) => { + const fixture = await seedClaimedRemoteBootstrap(`SESSION-${label.replace(/\W+/g, "-")}`, { mutate: core => { + openSession(core, eventTypes); + core.queueCommand("session.goal.get", {}); + const goal = core.store.state.commands.at(-1)!; + goal.status = "completed"; + goal.result = { goal: null } as never; + } }); + const claims = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] }); + if (kind === "blocked") { + expect(claims).toEqual([{ kind: "blocked", runId: fixture.runId, reason: "ambiguous_remote_recovery_evidence" }]); + return; + } + expect(claims).toEqual([expect.objectContaining({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 })]); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ disposition: "bootstrap_incomplete", + reason: "remote_session_opened_without_turn" }); + const root = resolve(runtimeRoot, `diagnostics-${fixture.runId}`); + try { + const executor = await seedClaimedRemoteBootstrap(`SESSION-${label.replace(/\W+/g, "-")}-EXECUTOR`, { mutate: core => { + openSession(core, eventTypes); + } }); + expect(await restoreNativeRunnerCheckpoint(executor.db, { companyId, issueId: executor.issueId, runId: executor.runId }, + root)).not.toBeNull(); + expect(runnerdStateProvesNoRemoteProviderTurn(root)).toBe(true); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + it("lets the executor prove a drained remote session had no turn from its restored checkpoint", async () => { const fixture = await seedClaimedRemoteBootstrap("SESSION-OPENED-EXECUTOR", { mutate: core => openSession(core) }); const root = resolve(runtimeRoot, `opened-${fixture.runId}`);