mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(native): accept harness diagnostics in a remote session-opened bootstrap proof (#20543)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) keep their controller
state in a Postgres checkpoint. The controller writes every command to
that checkpoint before it sends the command.
> - After a restart, recovery uses that record to prove that a drained
remote bootstrap reached no provider turn. If so, it bootstraps the run
again on the same heartbeat run.
> - A graceful restart seconds after native start often catches a runner
that opened its session and emitted ACPX `harness.diagnostic` events,
but was never sent a turn.
> - The remote no-turn proof did not accept `harness.diagnostic`, so
recovery blocked these runs as `ambiguous_remote_recovery_evidence` and
the task needed a manual Retry.
> - This pull request accepts `harness.diagnostic` in that proof. The
proof's command check still requires that no turn command was ever
issued.
> - The benefit is that a restart just after native start re-bootstraps
the run automatically.
## Linked Issues or Issue Description
Refs #20538 (the harness trigger that reproduced it)
**What happened?**
Restart-chaos wave `w-20261110T091639-s11936` t02, run `c4fc7f03`
(Daytona R1, K2): a K2 landed 13 s after native start. Recovery returned
`ambiguous_remote_recovery_evidence`, and the run blocked with
`native_restart_recovery_blocked` (R3 retry_needed). I reproduced it
with a kept database in `w-20261110T132656-s11936` t02, run `ecd98d55`,
using `--native-trigger native-start+5` from #20538. Decoded controller
checkpoint:
- commands: `run.prepare`, `session.open`, `session.goal.get` (no turn
command);
- committed events: `session.started`, `session.capabilities.updated`,
`session.goal.snapshot`, `harness.diagnostic`, `harness.diagnostic`,
`session.goal.snapshot`;
- `ackedSourceSeq` 6, no `sessionCheckpoint`, no projected provider
events.
`nativeRunnerCheckpointRemoteBootstrapProof` returned `null` only
because `harness.diagnostic` was not in `SESSION_OPEN_EVENT_TYPES`.
**Expected behavior**
Recovery classifies the run `bootstrap_incomplete`
(`remote_session_opened_without_turn`) and bootstraps it again.
**Steps to reproduce**
1. Run a Daytona R1 codex wave with `--faults K2 --native-trigger
native-start+5`.
2. The suspended run is blocked with
`ambiguous_remote_recovery_evidence`.
**Paperclip version or commit**
`master` at c874cfa4da.
## What Changed
- `native-bootstrap-proof.ts`: add `harness.diagnostic` to
`SESSION_OPEN_EVENT_TYPES`, which `runnerdStateHasNoProviderTurn` uses.
The proof still requires every command to be `run.prepare`,
`session.open` or runner maintenance (`session.goal.get` and similar),
every committed event to be accounted for in order, and no adopted state
or warm transition. A turn command or any turn event still fails it. The
local opened-session proof already accepted `harness.diagnostic` through
`PRE_TURN_EVENT_TYPES`.
- `native-runner-restart-recovery.integration.test.ts`: a case built
from the observed record (goal read plus two diagnostics). It is
classified `bootstrap_incomplete` with
`remote_session_opened_without_turn`, and the executor's
restored-checkpoint proof agrees. A diagnostic followed by
`turn.started` stays blocked.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
163 passed, 14 skipped. The new diagnostics case failed before the
change with `{ kind: "blocked" }`.
- Live: a Daytona R1 wave with `--native-trigger native-start+5`
follows. Results will be in a comment.
## Risks
- Low risk. `harness.diagnostic` is a harness report, not provider
input. Provider work needs a turn command, which the controller writes
to this record before sending, and the proof still rejects any turn
command or turn event.
- Previous-release story (upgrade-safe run state): no persisted shape
changes. A controller checkpoint saved by the previous release with
diagnostics and no turn is now re-bootstrapped instead of blocked. A
record with a turn keeps the same named refusal.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7329eaf353
commit
60b9147e7c
2 files changed
+37
No files matched your search
@@ -35,6 +35,7 @@ const SESSION_OPEN_EVENT_TYPES = new Set([
|
||||
"session.updated",
|
||||
"session.capabilities.updated",
|
||||
"session.goal.snapshot",
|
||||
"harness.diagnostic",
|
||||
]);
|
||||
|
||||
/**
|
||||
|
||||
@@ -2787,6 +2787,42 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
}
|
||||
});
|
||||
|
||||
it.each<[string, string[], string]>([
|
||||
["harness diagnostics", ["session.started", "session.capabilities.updated", "session.goal.snapshot", "harness.diagnostic",
|
||||
"harness.diagnostic", "session.goal.snapshot"], "bootstrap_incomplete"],
|
||||
["a turn event", ["session.started", "harness.diagnostic", "turn.started"], "blocked"],
|
||||
])("classifies a drained remote session that read its goal and committed %s", async (label, eventTypes, kind) => {
|
||||
const fixture = await seedClaimedRemoteBootstrap(`SESSION-${label.replace(/\W+/g, "-")}`, { mutate: core => {
|
||||
openSession(core, eventTypes);
|
||||
core.queueCommand("session.goal.get", {});
|
||||
const goal = core.store.state.commands.at(-1)!;
|
||||
goal.status = "completed";
|
||||
goal.result = { goal: null } as never;
|
||||
} });
|
||||
const claims = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
|
||||
restartKind: "hard", runIds: [fixture.runId] });
|
||||
if (kind === "blocked") {
|
||||
expect(claims).toEqual([{ kind: "blocked", runId: fixture.runId, reason: "ambiguous_remote_recovery_evidence" }]);
|
||||
return;
|
||||
}
|
||||
expect(claims).toEqual([expect.objectContaining({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 })]);
|
||||
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ disposition: "bootstrap_incomplete",
|
||||
reason: "remote_session_opened_without_turn" });
|
||||
const root = resolve(runtimeRoot, `diagnostics-${fixture.runId}`);
|
||||
try {
|
||||
const executor = await seedClaimedRemoteBootstrap(`SESSION-${label.replace(/\W+/g, "-")}-EXECUTOR`, { mutate: core => {
|
||||
openSession(core, eventTypes);
|
||||
} });
|
||||
expect(await restoreNativeRunnerCheckpoint(executor.db, { companyId, issueId: executor.issueId, runId: executor.runId },
|
||||
root)).not.toBeNull();
|
||||
expect(runnerdStateProvesNoRemoteProviderTurn(root)).toBe(true);
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("lets the executor prove a drained remote session had no turn from its restored checkpoint", async () => {
|
||||
const fixture = await seedClaimedRemoteBootstrap("SESSION-OPENED-EXECUTOR", { mutate: core => openSession(core) });
|
||||
const root = resolve(runtimeRoot, `opened-${fixture.runId}`);
|
||||
|
||||
Reference in new issue
Block a user