Files
PaperClipAI/server/src/services/native-runtime/native-bootstrap-proof.ts
T
4cf04d64cb fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts a Runner in a remote sandbox for each
agent run. A bounded same-run retry (at most 3 startup attempts) handles
transient startup failures.
> - A server restart can stop a run during its first Runner bootstrap.
Restart recovery then claims `bootstrap_incomplete`. It stops and
quarantines the unfinished bootstrap and launches a fresh bootstrap on
the same run.
> - The executor builds its "unclaimed bootstrap" proof only when
`restartRecovery` is not set. Thus the fresh bootstrap of a
`bootstrap_incomplete` restart can never prove that a transient failure
is safe to retry.
> - A transient HTTP 502 from the Runner endpoint request in that fresh
bootstrap was then classified as `ambiguous_state`. The run stopped and
needed manual board recovery.
> - This pull request gives the unclaimed-bootstrap proof to a
`bootstrap_incomplete` restart as well as to a first attempt.
> - The benefit is that a transient endpoint failure after a restart
uses the bounded automatic retry and does not need a person to recover
the run.

## Linked Issues or Issue Description

Refs #20414 (related: it fixes executor admission of a
`bootstrap_incomplete` restart at state-root migration. It does not
change the retry evidence that this PR changes.)

**What happened?**
A server restart interrupted the first Runner bootstrap of a remote
native run. Restart recovery claimed `bootstrap_incomplete` and launched
a fresh bootstrap on the same run. The fresh bootstrap registered its
controller checkpoint. Then `runner.ingress.acquire` failed with HTTP
502, before any Runner connection or provider event. The run ended with
`nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3,
and the next action "resolve the ambiguous session state". The error was
`Runner connection endpoint could not be acquired (HTTP 502).`
(`runner_ingress_acquisition_failed`).

**What did you expect to happen?**
The transient 502 occurred before the Runner connected and before any
provider work. It should use the bounded same-run startup retry, as it
does on a first attempt.

**Steps to reproduce**
1. Start a native run on a remote sandbox target.
2. Restart the server during the first Runner bootstrap, so that restart
recovery claims `bootstrap_incomplete`.
3. Make the next `runner.ingress.acquire` call return HTTP 502.
4. See that the run stops as ambiguous state and does not schedule a
startup retry.

## What Changed

- Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`.
It returns true for a first attempt and for a `bootstrap_incomplete`
restart. It returns false for `reattach_existing_runner`,
`reattach_remote_runner` and `resume_dead_runner`, because these
continue an existing Runner or provider session.
- In `native-session-executor.ts`, use this helper in place of
`!input.restartRecovery` when the executor builds the
unclaimed-bootstrap evidence for a startup failure.
- Add `native-bootstrap-proof.test.ts` for the helper.
- Add a case to `native-startup-retry.integration.test.ts`. It runs the
real executor under a `bootstrap_incomplete` restart claim, and the
fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before
the Runner connects.

## Verification

- `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart
of an incomplete bootstrap" failed before the fix and passes after the
fix (5/5 tests pass, vitest 5.1.0).
- `native-startup-retry.integration.test.ts`, case "transient 502 after
a bootstrap_incomplete restart": the run must settle as
`retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1
and a set `nextAttemptAt`. Without the executor change, the executor
builds no bootstrap evidence and the run settles as `ambiguous_state`.
CI runs this suite; I could not run it locally.
- The other guards for a retry did not change. The checkpoint must be
unclaimed, there must be no provider events and no session checkpoint,
and the 3-attempt startup budget applies. If the earlier attempt opened
a provider session, provider events exist and the run still fails closed
as ambiguous.
- I could not run the full typecheck and the real-Runner integration
suites locally (no Rust toolchain in the sandbox). CI must run them.

## Risks

- Low risk. The change only affects a `bootstrap_incomplete` restart
whose fresh bootstrap fails during startup. Such a failure now gets the
same retry decision as a first attempt. All the other evidence checks
still apply, so a run with provider activity still stops for manual
recovery.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edit, and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:37:24 -08:00

226 lines
12 KiB
TypeScript

/**
* Pure proofs over durable runner state that a bootstrap never reached
* provider work. Restart recovery and the executor share them, so the claim
* and the executor's own check accept exactly the same shapes.
*/
export const RUNNERD_CONTROL_PLANE_STATE_SCHEMAS: ReadonlySet<string> = new Set([
"paperclip.runner.durable.control-plane-state.v1",
"paperclip.runner.durable.control-plane-state.v2",
]);
const RUNNERD_BOOTSTRAP_COMMAND_TYPES = new Set(["run.prepare", "session.open"]);
function record(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
/**
* Whether this attempt launches its own fresh Runner bootstrap, so that its
* unclaimed controller checkpoint can prove a startup failure retryable. A
* `bootstrap_incomplete` restart does: it stops and quarantines the unfinished
* bootstrap, then launches a new one on the same run. Other recoveries continue
* an existing Runner or provider session.
*/
export function attemptLaunchesFreshBootstrap(restartRecovery: { kind: string } | null | undefined): boolean {
return !restartRecovery || restartRecovery.kind === "bootstrap_incomplete";
}
/** The runner never connected, so every bootstrap command is still undelivered. */
export function runnerdStateIsUnconnectedBootstrap(state: Record<string, unknown>): boolean {
const commands = Array.isArray(state.commands) ? state.commands.map(record) : [];
const committedEvents = Array.isArray(state.committedEvents) ? state.committedEvents : [];
return RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema)) &&
state.connectionCount === 0 && committedEvents.length === 0 &&
commands.every(command => command.status === "pending" &&
RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)));
}
/** Events a runner commits while opening a provider session, before any turn. */
const SESSION_OPEN_EVENT_TYPES = new Set([
"harness.ready",
"session.started",
"session.resumed",
"session.updated",
"session.capabilities.updated",
"session.goal.snapshot",
"harness.diagnostic",
]);
/**
* A remote controller record whose runner opened its provider session but was
* never sent a turn. Only the controller issues turn commands, and it writes
* each one to this record before sending it, so a record holding just
* run.prepare and session.open, with every committed event a session-open
* event and every committed event accounted for, proves no provider turn,
* tool call or result exists. Use only where the controller record is the
* complete authority (a remote runner after its controller's lease expired).
*/
export function runnerdStateHasNoProviderTurn(state: Record<string, unknown>): boolean {
if (runnerdStateIsIncompleteBootstrap(state)) return true;
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
if (typeof state.connectionCount !== "number" || !Number.isSafeInteger(state.connectionCount) ||
state.connectionCount < 0) return false;
if (state.warmTransition != null || state.completedWarmTransition != null ||
state.stagedRunAttachment != null) return false;
if (record(state.bootstrapHistory).adopted === true) return false;
if (!Array.isArray(state.commands) || !Array.isArray(state.committedEvents)) return false;
const commands = state.commands.map(record);
if (!commands.every(command => RUNNER_MAINTENANCE_COMMAND_TYPES.has(String(command.type)) ||
(RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)) &&
(command.status === "pending" || command.status === "completed")))) return false;
const events = state.committedEvents.map(record);
// Every event since the first must still be here, in order, so none is hidden.
if (typeof state.ackedSourceSeq !== "number" || !Number.isSafeInteger(state.ackedSourceSeq) ||
state.ackedSourceSeq < 0 || state.ackedSourceSeq > events.length) return false;
return events.every((event, index) => event.sourceSeq === index + 1 &&
SESSION_OPEN_EVENT_TYPES.has(String(event.eventType)));
}
/**
* A provider turn can start only after session.open completes and the
* controller issues a turn command. A root whose controller journal stops
* before that point (run.prepare may be acknowledged, session.open never is,
* no event was committed or acknowledged) and whose runner journal emitted no
* event either holds no provider work, so restart recovery may quarantine it
* and bootstrap again. Any other shape stays fail-closed.
*
* `runnerJournal` is `undefined` when the runner journal is not on this host
* (never written locally, or a remote runner), `null` when it is unreadable.
*/
export function runnerdStateIsIncompleteBootstrap(
state: Record<string, unknown>,
runnerJournal?: Record<string, unknown> | null,
): boolean {
if (runnerdStateIsUnconnectedBootstrap(state)) return true;
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
if (typeof state.connectionCount !== "number" || !Number.isSafeInteger(state.connectionCount) ||
state.connectionCount < 0) return false;
if (!Array.isArray(state.committedEvents) || state.committedEvents.length !== 0) return false;
if (state.ackedSourceSeq !== 0) return false;
if (state.warmTransition != null || state.completedWarmTransition != null ||
state.stagedRunAttachment != null) return false;
if (record(state.bootstrapHistory).adopted === true) return false;
if (!Array.isArray(state.commands)) return false;
const commands = state.commands.map(record);
if (!commands.every(command => preOpenCommandIsSafe(command.type, command.status))) return false;
// The controller cannot see runner events that were emitted but never
// committed. When the runner journal exists it must agree that none were.
if (runnerJournal === undefined) return true;
if (runnerJournal === null) return false;
const processed = Object.values(record(runnerJournal.processedCommands)).map(record);
return Array.isArray(runnerJournal.outbox) && runnerJournal.outbox.length === 0 &&
runnerJournal.nextSourceSeq === 1 && runnerJournal.ackedSourceSeq === 0 &&
processed.every(command => preOpenCommandIsSafe(command.commandType, command.status));
}
/** Runner process lifecycle commands a graceful drain sends; none reaches the provider. */
const RUNNER_LIFECYCLE_COMMAND_TYPES = new Set(["runner.drain", "runner.suspend", "runner.shutdown"]);
/**
* Before any provider session exists: run.prepare may have completed,
* session.open may be pending or have failed (no session was handed back),
* and a drain may have sent runner lifecycle commands in any state.
*/
function preOpenCommandIsSafe(type: unknown, status: unknown): boolean {
if (type === "run.prepare") return status === "pending" || status === "completed";
if (type === "session.open") return status === "pending" || status === "failed";
return RUNNER_LIFECYCLE_COMMAND_TYPES.has(String(type));
}
/** Commands that manage the runner process or read session state; none sends provider input. */
const RUNNER_MAINTENANCE_COMMAND_TYPES = new Set([
"runner.drain", "runner.suspend", "runner.shutdown", "session.snapshot", "session.goal.get",
]);
/** Events a runner emits while opening a fresh session or managing its own process, before any turn. */
const PRE_TURN_EVENT_TYPES = new Set([
"session.starting", "session.started", "session.capabilities.updated", "session.goal.snapshot",
"harness.starting", "harness.ready", "harness.diagnostic",
"runner.connected", "runner.disconnected", "runner.reconnected", "runner.diagnostic", "runner.backpressure",
"runner.drain", "runner.draining", "runner.suspending", "runner.suspended", "runner.shutdown", "runner.stopped",
]);
/** Collects every recorded process and process-group id so each can be proven gone. */
function collectRecordedProcessIds(value: unknown, into: Array<{ id: unknown; group: boolean }>, depth = 0): boolean {
if (depth > 10) return false;
if (Array.isArray(value)) return value.every(entry => collectRecordedProcessIds(entry, into, depth + 1));
if (!value || typeof value !== "object") return true;
for (const [key, entry] of Object.entries(value)) {
// A null id records a launch that had not produced a process yet, so
// there is nothing to prove gone. Any other non-number fails the proof.
if (key === "processId" || key === "pid") { if (entry !== null) into.push({ id: entry, group: false }); }
else if (key === "processGroupId") { if (entry !== null) into.push({ id: entry, group: true }); }
else if (!collectRecordedProcessIds(entry, into, depth + 1)) return false;
}
return true;
}
function prpEventType(event: Record<string, unknown>): string {
return String(event.eventType ?? record(record(event.envelope).payload).eventType);
}
export type OpenedSessionProofOptions = {
/** True only when the runner and its provider processes ran on this host. */
localProcesses: boolean;
/** Whether a recorded process (or process-group) id has definitely exited. */
processGone: (id: unknown, group: boolean) => boolean;
};
/**
* A drain or restart can land after session.open started a fresh provider
* session but before the controller issued any turn. That root holds an
* empty provider session and no provider work, so recovery may quarantine it
* and bootstrap again when all of these hold:
*
* - controller commands are run.prepare and session.open (both completed, the
* session freshly started rather than a resumed conversation) plus runner
* maintenance only;
* - every committed event and every event left in the runner outbox is
* session setup or runner lifecycle, and compaction left no gap;
* - the runner journal is present and processed no other command; and
* - the session's provider process and every other recorded process id has
* exited on this host.
*
* Remote runners stay fail-closed here: their process ids belong to another host.
*/
export function runnerdStateIsOpenedSessionBeforeTurn(
state: Record<string, unknown>,
runnerJournal: Record<string, unknown> | null | undefined,
options: OpenedSessionProofOptions,
): boolean {
if (!options.localProcesses || !runnerJournal) return false;
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
if (state.warmTransition != null || state.completedWarmTransition != null ||
state.stagedRunAttachment != null) return false;
if (record(state.bootstrapHistory).adopted === true) return false;
if (!Array.isArray(state.commands) || !Array.isArray(state.committedEvents)) return false;
const commands = state.commands.map(record);
const prepares = commands.filter(command => command.type === "run.prepare");
const opens = commands.filter(command => command.type === "session.open");
if (prepares.length !== 1 || prepares[0]!.status !== "completed") return false;
if (opens.length !== 1 || opens[0]!.status !== "completed") return false;
if (!commands.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)) ||
RUNNER_MAINTENANCE_COMMAND_TYPES.has(String(command.type)))) return false;
const opened = record(record(opens[0]!.result).result);
if (opened.status !== "started" || typeof opened.processId !== "number") return false;
const committed = state.committedEvents.map(record);
if (state.ackedSourceSeq !== committed.length ||
!committed.every((event, index) => event.sourceSeq === index + 1)) return false;
if (!Array.isArray(runnerJournal.outbox)) return false;
const outbox = runnerJournal.outbox.map(record);
if (![...committed, ...outbox].every(event => PRE_TURN_EVENT_TYPES.has(prpEventType(event)))) return false;
const processed = Object.values(record(runnerJournal.processedCommands)).map(record);
if (!processed.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.commandType)) ||
RUNNER_MAINTENANCE_COMMAND_TYPES.has(String(command.commandType)))) return false;
const ids: Array<{ id: unknown; group: boolean }> = [{ id: opened.processId, group: false }];
if (!collectRecordedProcessIds([...committed, ...outbox, ...processed.map(command => command.result)], ids)) {
return false;
}
return ids.every(({ id, group }) => options.processGone(id, group));
}