mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime starts a Runner in a remote sandbox for each agent run. A bounded same-run retry (at most 3 startup attempts) handles transient startup failures. > - A server restart can stop a run during its first Runner bootstrap. Restart recovery then claims `bootstrap_incomplete`. It stops and quarantines the unfinished bootstrap and launches a fresh bootstrap on the same run. > - The executor builds its "unclaimed bootstrap" proof only when `restartRecovery` is not set. Thus the fresh bootstrap of a `bootstrap_incomplete` restart can never prove that a transient failure is safe to retry. > - A transient HTTP 502 from the Runner endpoint request in that fresh bootstrap was then classified as `ambiguous_state`. The run stopped and needed manual board recovery. > - This pull request gives the unclaimed-bootstrap proof to a `bootstrap_incomplete` restart as well as to a first attempt. > - The benefit is that a transient endpoint failure after a restart uses the bounded automatic retry and does not need a person to recover the run. ## Linked Issues or Issue Description Refs #20414 (related: it fixes executor admission of a `bootstrap_incomplete` restart at state-root migration. It does not change the retry evidence that this PR changes.) **What happened?** A server restart interrupted the first Runner bootstrap of a remote native run. Restart recovery claimed `bootstrap_incomplete` and launched a fresh bootstrap on the same run. The fresh bootstrap registered its controller checkpoint. Then `runner.ingress.acquire` failed with HTTP 502, before any Runner connection or provider event. The run ended with `nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3, and the next action "resolve the ambiguous session state". The error was `Runner connection endpoint could not be acquired (HTTP 502).` (`runner_ingress_acquisition_failed`). **What did you expect to happen?** The transient 502 occurred before the Runner connected and before any provider work. It should use the bounded same-run startup retry, as it does on a first attempt. **Steps to reproduce** 1. Start a native run on a remote sandbox target. 2. Restart the server during the first Runner bootstrap, so that restart recovery claims `bootstrap_incomplete`. 3. Make the next `runner.ingress.acquire` call return HTTP 502. 4. See that the run stops as ambiguous state and does not schedule a startup retry. ## What Changed - Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`. It returns true for a first attempt and for a `bootstrap_incomplete` restart. It returns false for `reattach_existing_runner`, `reattach_remote_runner` and `resume_dead_runner`, because these continue an existing Runner or provider session. - In `native-session-executor.ts`, use this helper in place of `!input.restartRecovery` when the executor builds the unclaimed-bootstrap evidence for a startup failure. - Add `native-bootstrap-proof.test.ts` for the helper. - Add a case to `native-startup-retry.integration.test.ts`. It runs the real executor under a `bootstrap_incomplete` restart claim, and the fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before the Runner connects. ## Verification - `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart of an incomplete bootstrap" failed before the fix and passes after the fix (5/5 tests pass, vitest 5.1.0). - `native-startup-retry.integration.test.ts`, case "transient 502 after a bootstrap_incomplete restart": the run must settle as `retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1 and a set `nextAttemptAt`. Without the executor change, the executor builds no bootstrap evidence and the run settles as `ambiguous_state`. CI runs this suite; I could not run it locally. - The other guards for a retry did not change. The checkpoint must be unclaimed, there must be no provider events and no session checkpoint, and the 3-attempt startup budget applies. If the earlier attempt opened a provider session, provider events exist and the run still fails closed as ambiguous. - I could not run the full typecheck and the real-Runner integration suites locally (no Rust toolchain in the sandbox). CI must run them. ## Risks - Low risk. The change only affects a `bootstrap_incomplete` restart whose fresh bootstrap fails during startup. Such a failure now gets the same retry decision as a first attempt. All the other evidence checks still apply, so a run with provider activity still stops for manual recovery. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with tool use (shell, file edit, and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
226 lines
12 KiB
TypeScript
226 lines
12 KiB
TypeScript
/**
|
|
* Pure proofs over durable runner state that a bootstrap never reached
|
|
* provider work. Restart recovery and the executor share them, so the claim
|
|
* and the executor's own check accept exactly the same shapes.
|
|
*/
|
|
|
|
export const RUNNERD_CONTROL_PLANE_STATE_SCHEMAS: ReadonlySet<string> = new Set([
|
|
"paperclip.runner.durable.control-plane-state.v1",
|
|
"paperclip.runner.durable.control-plane-state.v2",
|
|
]);
|
|
|
|
const RUNNERD_BOOTSTRAP_COMMAND_TYPES = new Set(["run.prepare", "session.open"]);
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return value && typeof value === "object" && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: {};
|
|
}
|
|
|
|
/**
|
|
* Whether this attempt launches its own fresh Runner bootstrap, so that its
|
|
* unclaimed controller checkpoint can prove a startup failure retryable. A
|
|
* `bootstrap_incomplete` restart does: it stops and quarantines the unfinished
|
|
* bootstrap, then launches a new one on the same run. Other recoveries continue
|
|
* an existing Runner or provider session.
|
|
*/
|
|
export function attemptLaunchesFreshBootstrap(restartRecovery: { kind: string } | null | undefined): boolean {
|
|
return !restartRecovery || restartRecovery.kind === "bootstrap_incomplete";
|
|
}
|
|
|
|
/** The runner never connected, so every bootstrap command is still undelivered. */
|
|
export function runnerdStateIsUnconnectedBootstrap(state: Record<string, unknown>): boolean {
|
|
const commands = Array.isArray(state.commands) ? state.commands.map(record) : [];
|
|
const committedEvents = Array.isArray(state.committedEvents) ? state.committedEvents : [];
|
|
return RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema)) &&
|
|
state.connectionCount === 0 && committedEvents.length === 0 &&
|
|
commands.every(command => command.status === "pending" &&
|
|
RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)));
|
|
}
|
|
|
|
/** Events a runner commits while opening a provider session, before any turn. */
|
|
const SESSION_OPEN_EVENT_TYPES = new Set([
|
|
"harness.ready",
|
|
"session.started",
|
|
"session.resumed",
|
|
"session.updated",
|
|
"session.capabilities.updated",
|
|
"session.goal.snapshot",
|
|
"harness.diagnostic",
|
|
]);
|
|
|
|
/**
|
|
* A remote controller record whose runner opened its provider session but was
|
|
* never sent a turn. Only the controller issues turn commands, and it writes
|
|
* each one to this record before sending it, so a record holding just
|
|
* run.prepare and session.open, with every committed event a session-open
|
|
* event and every committed event accounted for, proves no provider turn,
|
|
* tool call or result exists. Use only where the controller record is the
|
|
* complete authority (a remote runner after its controller's lease expired).
|
|
*/
|
|
export function runnerdStateHasNoProviderTurn(state: Record<string, unknown>): boolean {
|
|
if (runnerdStateIsIncompleteBootstrap(state)) return true;
|
|
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
|
|
if (typeof state.connectionCount !== "number" || !Number.isSafeInteger(state.connectionCount) ||
|
|
state.connectionCount < 0) return false;
|
|
if (state.warmTransition != null || state.completedWarmTransition != null ||
|
|
state.stagedRunAttachment != null) return false;
|
|
if (record(state.bootstrapHistory).adopted === true) return false;
|
|
if (!Array.isArray(state.commands) || !Array.isArray(state.committedEvents)) return false;
|
|
const commands = state.commands.map(record);
|
|
if (!commands.every(command => RUNNER_MAINTENANCE_COMMAND_TYPES.has(String(command.type)) ||
|
|
(RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)) &&
|
|
(command.status === "pending" || command.status === "completed")))) return false;
|
|
const events = state.committedEvents.map(record);
|
|
// Every event since the first must still be here, in order, so none is hidden.
|
|
if (typeof state.ackedSourceSeq !== "number" || !Number.isSafeInteger(state.ackedSourceSeq) ||
|
|
state.ackedSourceSeq < 0 || state.ackedSourceSeq > events.length) return false;
|
|
return events.every((event, index) => event.sourceSeq === index + 1 &&
|
|
SESSION_OPEN_EVENT_TYPES.has(String(event.eventType)));
|
|
}
|
|
|
|
/**
|
|
* A provider turn can start only after session.open completes and the
|
|
* controller issues a turn command. A root whose controller journal stops
|
|
* before that point (run.prepare may be acknowledged, session.open never is,
|
|
* no event was committed or acknowledged) and whose runner journal emitted no
|
|
* event either holds no provider work, so restart recovery may quarantine it
|
|
* and bootstrap again. Any other shape stays fail-closed.
|
|
*
|
|
* `runnerJournal` is `undefined` when the runner journal is not on this host
|
|
* (never written locally, or a remote runner), `null` when it is unreadable.
|
|
*/
|
|
export function runnerdStateIsIncompleteBootstrap(
|
|
state: Record<string, unknown>,
|
|
runnerJournal?: Record<string, unknown> | null,
|
|
): boolean {
|
|
if (runnerdStateIsUnconnectedBootstrap(state)) return true;
|
|
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
|
|
if (typeof state.connectionCount !== "number" || !Number.isSafeInteger(state.connectionCount) ||
|
|
state.connectionCount < 0) return false;
|
|
if (!Array.isArray(state.committedEvents) || state.committedEvents.length !== 0) return false;
|
|
if (state.ackedSourceSeq !== 0) return false;
|
|
if (state.warmTransition != null || state.completedWarmTransition != null ||
|
|
state.stagedRunAttachment != null) return false;
|
|
if (record(state.bootstrapHistory).adopted === true) return false;
|
|
if (!Array.isArray(state.commands)) return false;
|
|
const commands = state.commands.map(record);
|
|
if (!commands.every(command => preOpenCommandIsSafe(command.type, command.status))) return false;
|
|
// The controller cannot see runner events that were emitted but never
|
|
// committed. When the runner journal exists it must agree that none were.
|
|
if (runnerJournal === undefined) return true;
|
|
if (runnerJournal === null) return false;
|
|
const processed = Object.values(record(runnerJournal.processedCommands)).map(record);
|
|
return Array.isArray(runnerJournal.outbox) && runnerJournal.outbox.length === 0 &&
|
|
runnerJournal.nextSourceSeq === 1 && runnerJournal.ackedSourceSeq === 0 &&
|
|
processed.every(command => preOpenCommandIsSafe(command.commandType, command.status));
|
|
}
|
|
|
|
/** Runner process lifecycle commands a graceful drain sends; none reaches the provider. */
|
|
const RUNNER_LIFECYCLE_COMMAND_TYPES = new Set(["runner.drain", "runner.suspend", "runner.shutdown"]);
|
|
|
|
/**
|
|
* Before any provider session exists: run.prepare may have completed,
|
|
* session.open may be pending or have failed (no session was handed back),
|
|
* and a drain may have sent runner lifecycle commands in any state.
|
|
*/
|
|
function preOpenCommandIsSafe(type: unknown, status: unknown): boolean {
|
|
if (type === "run.prepare") return status === "pending" || status === "completed";
|
|
if (type === "session.open") return status === "pending" || status === "failed";
|
|
return RUNNER_LIFECYCLE_COMMAND_TYPES.has(String(type));
|
|
}
|
|
|
|
/** Commands that manage the runner process or read session state; none sends provider input. */
|
|
const RUNNER_MAINTENANCE_COMMAND_TYPES = new Set([
|
|
"runner.drain", "runner.suspend", "runner.shutdown", "session.snapshot", "session.goal.get",
|
|
]);
|
|
|
|
/** Events a runner emits while opening a fresh session or managing its own process, before any turn. */
|
|
const PRE_TURN_EVENT_TYPES = new Set([
|
|
"session.starting", "session.started", "session.capabilities.updated", "session.goal.snapshot",
|
|
"harness.starting", "harness.ready", "harness.diagnostic",
|
|
"runner.connected", "runner.disconnected", "runner.reconnected", "runner.diagnostic", "runner.backpressure",
|
|
"runner.drain", "runner.draining", "runner.suspending", "runner.suspended", "runner.shutdown", "runner.stopped",
|
|
]);
|
|
|
|
/** Collects every recorded process and process-group id so each can be proven gone. */
|
|
function collectRecordedProcessIds(value: unknown, into: Array<{ id: unknown; group: boolean }>, depth = 0): boolean {
|
|
if (depth > 10) return false;
|
|
if (Array.isArray(value)) return value.every(entry => collectRecordedProcessIds(entry, into, depth + 1));
|
|
if (!value || typeof value !== "object") return true;
|
|
for (const [key, entry] of Object.entries(value)) {
|
|
// A null id records a launch that had not produced a process yet, so
|
|
// there is nothing to prove gone. Any other non-number fails the proof.
|
|
if (key === "processId" || key === "pid") { if (entry !== null) into.push({ id: entry, group: false }); }
|
|
else if (key === "processGroupId") { if (entry !== null) into.push({ id: entry, group: true }); }
|
|
else if (!collectRecordedProcessIds(entry, into, depth + 1)) return false;
|
|
}
|
|
return true;
|
|
}
|
|
|
|
function prpEventType(event: Record<string, unknown>): string {
|
|
return String(event.eventType ?? record(record(event.envelope).payload).eventType);
|
|
}
|
|
|
|
export type OpenedSessionProofOptions = {
|
|
/** True only when the runner and its provider processes ran on this host. */
|
|
localProcesses: boolean;
|
|
/** Whether a recorded process (or process-group) id has definitely exited. */
|
|
processGone: (id: unknown, group: boolean) => boolean;
|
|
};
|
|
|
|
/**
|
|
* A drain or restart can land after session.open started a fresh provider
|
|
* session but before the controller issued any turn. That root holds an
|
|
* empty provider session and no provider work, so recovery may quarantine it
|
|
* and bootstrap again when all of these hold:
|
|
*
|
|
* - controller commands are run.prepare and session.open (both completed, the
|
|
* session freshly started rather than a resumed conversation) plus runner
|
|
* maintenance only;
|
|
* - every committed event and every event left in the runner outbox is
|
|
* session setup or runner lifecycle, and compaction left no gap;
|
|
* - the runner journal is present and processed no other command; and
|
|
* - the session's provider process and every other recorded process id has
|
|
* exited on this host.
|
|
*
|
|
* Remote runners stay fail-closed here: their process ids belong to another host.
|
|
*/
|
|
export function runnerdStateIsOpenedSessionBeforeTurn(
|
|
state: Record<string, unknown>,
|
|
runnerJournal: Record<string, unknown> | null | undefined,
|
|
options: OpenedSessionProofOptions,
|
|
): boolean {
|
|
if (!options.localProcesses || !runnerJournal) return false;
|
|
if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
|
|
if (state.warmTransition != null || state.completedWarmTransition != null ||
|
|
state.stagedRunAttachment != null) return false;
|
|
if (record(state.bootstrapHistory).adopted === true) return false;
|
|
if (!Array.isArray(state.commands) || !Array.isArray(state.committedEvents)) return false;
|
|
const commands = state.commands.map(record);
|
|
const prepares = commands.filter(command => command.type === "run.prepare");
|
|
const opens = commands.filter(command => command.type === "session.open");
|
|
if (prepares.length !== 1 || prepares[0]!.status !== "completed") return false;
|
|
if (opens.length !== 1 || opens[0]!.status !== "completed") return false;
|
|
if (!commands.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)) ||
|
|
RUNNER_MAINTENANCE_COMMAND_TYPES.has(String(command.type)))) return false;
|
|
const opened = record(record(opens[0]!.result).result);
|
|
if (opened.status !== "started" || typeof opened.processId !== "number") return false;
|
|
|
|
const committed = state.committedEvents.map(record);
|
|
if (state.ackedSourceSeq !== committed.length ||
|
|
!committed.every((event, index) => event.sourceSeq === index + 1)) return false;
|
|
if (!Array.isArray(runnerJournal.outbox)) return false;
|
|
const outbox = runnerJournal.outbox.map(record);
|
|
if (![...committed, ...outbox].every(event => PRE_TURN_EVENT_TYPES.has(prpEventType(event)))) return false;
|
|
const processed = Object.values(record(runnerJournal.processedCommands)).map(record);
|
|
if (!processed.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.commandType)) ||
|
|
RUNNER_MAINTENANCE_COMMAND_TYPES.has(String(command.commandType)))) return false;
|
|
|
|
const ids: Array<{ id: unknown; group: boolean }> = [{ id: opened.processId, group: false }];
|
|
if (!collectRecordedProcessIds([...committed, ...outbox, ...processed.map(command => command.result)], ids)) {
|
|
return false;
|
|
}
|
|
return ids.every(({ id, group }) => options.processGone(id, group));
|
|
}
|