mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.
## Linked Issues or Issue Description
**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.
**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.
**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.
**Paperclip version or commit**
Staging at 1d153f62b6. The code is the same on master at 704b3f4d63.
**Deployment mode**
Hosted staging (paperclip.staging.paperclip.app), remote sandbox
execution on Daytona.
**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).
## What Changed
- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapEvidence` returns the bootstrap
proof together with the runner instance that the checkpoint names.
`nativeRunnerCheckpointRemoteBootstrapProof` now wraps it and behaves as
before.
- `native-session-executor.ts`:
- State migration runs before this server registers its own controller
for the run. Under a `bootstrap_incomplete` claim for a remote sandbox,
it now reads the interrupted controller's checkpoint proof there
(`captureClaimTimeRemoteBootstrapProof`). A later read is not used,
because the new controller rewrites the checkpoint with its own
bootstrap commands.
- Host preparation: when this run's sandbox Runner is not suspended, it
runs the same-run restart proof. The proof accepts a Runner that never
connected (as before), or the claim-time proof for the exact runner
instance in the sandbox state. It then stops that exact Runner by its
identity marker and records the stop proof. A backup, a different run
id, another runner instance, or a failed stop still fails closed.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_connected_checkpoint_proven` case. The Runner is
`ready` and connected, and no stop proof exists yet. It fails before the
fix with `resumed_not_suspended` and passes after it.
`interrupted_bootstrap_unproven` still fails closed.
- `native-bootstrap-claim-time-proof.integration.test.ts` (embedded
Postgres, no mocks of the proof): an interrupted controller that had
connected registers its checkpoint, the claim-time proof is read, and
then a replacement controller registers its own checkpoint. A live read
after that returns `bootstrap_incomplete` even when the interrupted
controller had sent `turn.start`. The claim-time proof stays false for
that case and true for the bootstrap-only case, and it refuses another
runner instance.
## Verification
- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` fails on the new case.
- Green: the same command passes (65 tests after merging master).
- `pnpm exec vitest run
server/src/services/native-runtime/native-bootstrap-claim-time-proof.integration.test.ts`:
2 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
server/src/services/native-runtime/native-restart-recovery.test.ts`:
1474 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- Merged `origin/master` (including #21134) without conflicts and reran
the tests above.
## Risks
- Low. The new path runs only under a `bootstrap_incomplete` claim for
the same run id, with no failover backup, and only after the Postgres
controller checkpoint proves that no provider turn was sent. It stops a
Runner only when the identity marker and process fingerprint match,
which is the same check the reattach path uses.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #21134 (the never-launched branch of the same helper), #21079
(stops a leftover Runner at launch time; this change acts earlier, in
host preparation).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0bac3510db
commit
3fb35fcf13
4 files changed
+124
-13
No files matched your search
+58
@@ -0,0 +1,58 @@
|
|||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { eq } from "drizzle-orm";
|
||||||
|
import { expect, it } from "vitest";
|
||||||
|
import { nativeRunFinalizations } from "@paperclipai/db";
|
||||||
|
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
|
||||||
|
import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
|
||||||
|
import { nativeRunnerCheckpointRemoteBootstrapProof, registerNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||||
|
import { captureClaimTimeRemoteBootstrapProof, claimTimeRemoteBootstrapProofHolds } from "./native-session-executor.js";
|
||||||
|
|
||||||
|
describeEmbeddedPostgres("claim-time remote bootstrap proof", () => {
|
||||||
|
const environment = nativeRestartFixture();
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
{ label: "turn-sent", oldCommands: ["run.prepare", "session.open", "turn.start"], holds: false },
|
||||||
|
{ label: "bootstrap-only", oldCommands: ["run.prepare", "session.open"], holds: true },
|
||||||
|
])("keeps the interrupted controller's proof after the new controller registers: $label", async ({ label, oldCommands, holds }) => {
|
||||||
|
const { db, runId, issueId, native } = await environment.seedRun(`CLAIM-PROOF-${label}`);
|
||||||
|
const { companyId, successor, runtimeRoot } = environment.current;
|
||||||
|
const binding = { companyId, issueId, runId };
|
||||||
|
const identity = { runnerInstanceId: native.runnerInstanceId, environmentLeaseId: native.environmentLeaseId, runId,
|
||||||
|
normalizedSessionId: native.normalizedSessionId, turnId: native.turnId, itemId: native.itemId };
|
||||||
|
const controller = (directory: string) => new DurablePrpControlPlane({ stateDirectory: join(runtimeRoot, directory), identity,
|
||||||
|
expectedRunnerVersion: "fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
|
||||||
|
const own = async (generation: number) => db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId,
|
||||||
|
controllerGeneration: generation, leaseOwner: randomUUID(), leaseExpiresAt: new Date(Date.now() + 60_000) })
|
||||||
|
.where(eq(nativeRunFinalizations.runId, runId));
|
||||||
|
const execution = { binding, session: { normalizedSessionId: native.normalizedSessionId } } as never;
|
||||||
|
const claim = { kind: "bootstrap_incomplete" as const, runId, leaseOwner: "recovered", controllerGeneration: 2,
|
||||||
|
providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null };
|
||||||
|
const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace" } as never;
|
||||||
|
const interrupted = controller(`${runId}-interrupted`);
|
||||||
|
const replacement = controller(`${runId}-replacement`);
|
||||||
|
try {
|
||||||
|
await own(1);
|
||||||
|
for (const type of oldCommands) interrupted.queueCommand(type, {});
|
||||||
|
interrupted.store.state.connectionCount = 1;
|
||||||
|
await registerNativeRunnerCheckpoint({ db, authority: interrupted, bootId: successor.bootId, binding });
|
||||||
|
await interrupted.stop();
|
||||||
|
|
||||||
|
await own(2);
|
||||||
|
await captureClaimTimeRemoteBootstrapProof({ db, execution, restartRecovery: claim, runnerExecutionTarget: target });
|
||||||
|
expect(claimTimeRemoteBootstrapProofHolds(runId, native.runnerInstanceId)).toBe(holds);
|
||||||
|
|
||||||
|
replacement.queueCommand("run.prepare", {});
|
||||||
|
replacement.queueCommand("session.open", { reuse: "same_session" });
|
||||||
|
await registerNativeRunnerCheckpoint({ db, authority: replacement, bootId: successor.bootId, binding });
|
||||||
|
|
||||||
|
expect(await nativeRunnerCheckpointRemoteBootstrapProof(db, binding,
|
||||||
|
{ runnerInstanceId: native.runnerInstanceId, normalizedSessionId: native.normalizedSessionId })).toBe("bootstrap_incomplete");
|
||||||
|
expect(claimTimeRemoteBootstrapProofHolds(runId, native.runnerInstanceId)).toBe(holds);
|
||||||
|
expect(claimTimeRemoteBootstrapProofHolds(runId, "another-runner")).toBe(false);
|
||||||
|
} finally {
|
||||||
|
await interrupted.stop();
|
||||||
|
await replacement.stop();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -302,6 +302,17 @@ export type NativeRemoteBootstrapProof = "bootstrap_incomplete" | "session_opene
|
|||||||
export async function nativeRunnerCheckpointRemoteBootstrapProof(
|
export async function nativeRunnerCheckpointRemoteBootstrapProof(
|
||||||
db: Pick<Db, "select">, binding: Binding, expected: { runnerInstanceId: string; normalizedSessionId: string },
|
db: Pick<Db, "select">, binding: Binding, expected: { runnerInstanceId: string; normalizedSessionId: string },
|
||||||
): Promise<NativeRemoteBootstrapProof | null> {
|
): Promise<NativeRemoteBootstrapProof | null> {
|
||||||
|
const evidence = await nativeRunnerCheckpointRemoteBootstrapEvidence(db, binding, expected.normalizedSessionId);
|
||||||
|
return evidence?.runnerInstanceId === expected.runnerInstanceId ? evidence.proof : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The same proof, with the runner instance the checkpoint names. A caller that
|
||||||
|
* must hold the proof across a new controller's registration reads it first.
|
||||||
|
*/
|
||||||
|
export async function nativeRunnerCheckpointRemoteBootstrapEvidence(
|
||||||
|
db: Pick<Db, "select">, binding: Binding, normalizedSessionId: string,
|
||||||
|
): Promise<{ proof: NativeRemoteBootstrapProof; runnerInstanceId: string } | null> {
|
||||||
try {
|
try {
|
||||||
const limit = BOOTSTRAP_CHECKPOINT_PARTS.size + MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + MAX_BOOTSTRAP_CHECKPOINT_EVENTS;
|
const limit = BOOTSTRAP_CHECKPOINT_PARTS.size + MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + MAX_BOOTSTRAP_CHECKPOINT_EVENTS;
|
||||||
const parts = await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints).where(and(
|
const parts = await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints).where(and(
|
||||||
@@ -314,8 +325,8 @@ export async function nativeRunnerCheckpointRemoteBootstrapProof(
|
|||||||
|| parts.some(row => !BOOTSTRAP_CHECKPOINT_PARTS.has(row.part) && !/^(?:event|command):\d+$/.test(row.part))) return null;
|
|| parts.some(row => !BOOTSTRAP_CHECKPOINT_PARTS.has(row.part) && !/^(?:event|command):\d+$/.test(row.part))) return null;
|
||||||
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
|
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
|
||||||
if (!checkpoint || checkpoint.predecessor !== null
|
if (!checkpoint || checkpoint.predecessor !== null
|
||||||
|| checkpoint.recoveryEvidence.runnerInstanceId !== expected.runnerInstanceId
|
|| checkpoint.recoveryEvidence.normalizedSessionId !== normalizedSessionId) return null;
|
||||||
|| checkpoint.recoveryEvidence.normalizedSessionId !== expected.normalizedSessionId) return null;
|
const runnerInstanceId = checkpoint.recoveryEvidence.runnerInstanceId;
|
||||||
// The decoder above validated these canonical v2 records. Rebuild the
|
// The decoder above validated these canonical v2 records. Rebuild the
|
||||||
// controller state in memory without writing files under recovery locks.
|
// controller state in memory without writing files under recovery locks.
|
||||||
const read = (part: string) => {
|
const read = (part: string) => {
|
||||||
@@ -335,8 +346,8 @@ export async function nativeRunnerCheckpointRemoteBootstrapProof(
|
|||||||
.map(seq => read(`event:${seq}`)),
|
.map(seq => read(`event:${seq}`)),
|
||||||
};
|
};
|
||||||
if ((state.identity as { runId?: unknown } | undefined)?.runId !== binding.runId) return null;
|
if ((state.identity as { runId?: unknown } | undefined)?.runId !== binding.runId) return null;
|
||||||
if (runnerdStateIsIncompleteBootstrap(state)) return "bootstrap_incomplete";
|
if (runnerdStateIsIncompleteBootstrap(state)) return { proof: "bootstrap_incomplete", runnerInstanceId };
|
||||||
return runnerdStateHasNoProviderTurn(state) ? "session_opened_without_turn" : null;
|
return runnerdStateHasNoProviderTurn(state) ? { proof: "session_opened_without_turn", runnerInstanceId } : null;
|
||||||
} catch {
|
} catch {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21462,7 +21462,7 @@ describe("runnerd provider runtime wiring", () => {
|
|||||||
"interrupted_bootstrap_unopened", "interrupted_bootstrap_unopened_unproven", "interrupted_bootstrap_unopened_other_run",
|
"interrupted_bootstrap_unopened", "interrupted_bootstrap_unopened_unproven", "interrupted_bootstrap_unopened_other_run",
|
||||||
"interrupted_bootstrap_unopened_active_turn", "interrupted_bootstrap_unprepared", "interrupted_bootstrap_unprepared_unproven",
|
"interrupted_bootstrap_unopened_active_turn", "interrupted_bootstrap_unprepared", "interrupted_bootstrap_unprepared_unproven",
|
||||||
"interrupted_bootstrap_unprepared_other_run", "interrupted_bootstrap_unprepared_suspended", "interrupted_bootstrap_unprepared_backup",
|
"interrupted_bootstrap_unprepared_other_run", "interrupted_bootstrap_unprepared_suspended", "interrupted_bootstrap_unprepared_backup",
|
||||||
"interrupted_bootstrap_unprepared_unrecorded", "interrupted_bootstrap_unprepared_connected"]
|
"interrupted_bootstrap_unprepared_unrecorded", "interrupted_bootstrap_unprepared_connected", "interrupted_bootstrap_connected_checkpoint_proven"]
|
||||||
.map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })),
|
.map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })),
|
||||||
...["interrupted_bootstrap_unlaunched", "interrupted_bootstrap_unlaunched_marker", "interrupted_bootstrap_unlaunched_backup",
|
...["interrupted_bootstrap_unlaunched", "interrupted_bootstrap_unlaunched_marker", "interrupted_bootstrap_unlaunched_backup",
|
||||||
"interrupted_bootstrap_unlaunched_archived", "interrupted_bootstrap_unlaunched_nested_state"]
|
"interrupted_bootstrap_unlaunched_archived", "interrupted_bootstrap_unlaunched_nested_state"]
|
||||||
@@ -21544,10 +21544,12 @@ describe("runnerd provider runtime wiring", () => {
|
|||||||
exitCode: 1, timedOut: false, stdout: "", stderr: "" };
|
exitCode: 1, timedOut: false, stdout: "", stderr: "" };
|
||||||
if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unprepared")
|
if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unprepared")
|
||||||
&& command.args[1].includes("codex-provider-state.json")) return { exitCode: 1, timedOut: false, stdout: "", stderr: "" };
|
&& command.args[1].includes("codex-provider-state.json")) return { exitCode: 1, timedOut: false, stdout: "", stderr: "" };
|
||||||
if (command.args?.[2] === "paperclip-runner-recovery-identity" && scenario === "interrupted_bootstrap_unprepared_unrecorded") return {
|
if (command.args?.[2] === "paperclip-runner-recovery-identity" && ["interrupted_bootstrap_unprepared_unrecorded",
|
||||||
|
"interrupted_bootstrap_connected_checkpoint_proven"].includes(scenario)) return {
|
||||||
exitCode: 0, timedOut: false, stderr: "",
|
exitCode: 0, timedOut: false, stderr: "",
|
||||||
stdout: "exitednonce\n4242\n2026-11-12T00:32:10.725Z\nrunner-new-in-retained-workspace\nlinux:abcd-1234:100\n" };
|
stdout: "exitednonce\n4242\n2026-11-12T00:32:10.725Z\nrunner-new-in-retained-workspace\nlinux:abcd-1234:100\n" };
|
||||||
if (command.args?.[2] === "paperclip-runner-recovery-check" && scenario === "interrupted_bootstrap_unprepared_unrecorded") return {
|
if (command.args?.[2] === "paperclip-runner-recovery-check" && ["interrupted_bootstrap_unprepared_unrecorded",
|
||||||
|
"interrupted_bootstrap_connected_checkpoint_proven"].includes(scenario)) return {
|
||||||
exitCode: 45, timedOut: false, stdout: "", stderr: "" };
|
exitCode: 45, timedOut: false, stdout: "", stderr: "" };
|
||||||
if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unopened")
|
if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unopened")
|
||||||
&& command.args[1].includes("codex-provider-state.json") && !command.args[1].includes("runner-state.json")) return {
|
&& command.args[1].includes("codex-provider-state.json") && !command.args[1].includes("runner-state.json")) return {
|
||||||
@@ -21563,8 +21565,10 @@ describe("runnerd provider runtime wiring", () => {
|
|||||||
lifecycle: scenario.startsWith("interrupted_bootstrap_unopened") || scenario.endsWith("_suspended") ? "suspended"
|
lifecycle: scenario.startsWith("interrupted_bootstrap_unopened") || scenario.endsWith("_suspended") ? "suspended"
|
||||||
: scenario.startsWith("interrupted_bootstrap_unprepared") ? "connecting" : "ready",
|
: scenario.startsWith("interrupted_bootstrap_unprepared") ? "connecting" : "ready",
|
||||||
outbox: [], pendingTerminalDelivery: null,
|
outbox: [], pendingTerminalDelivery: null,
|
||||||
...(scenario.startsWith("interrupted_bootstrap_unprepared") ? { lastControllerCommandSeq: 0, reconnectCount: 0,
|
...(scenario.startsWith("interrupted_bootstrap_unprepared") || scenario === "interrupted_bootstrap_connected_checkpoint_proven"
|
||||||
processedCommands: {}, lastConnectionProtocolVersion: scenario.endsWith("_connected") ? 2 : null,
|
? { lastControllerCommandSeq: scenario === "interrupted_bootstrap_connected_checkpoint_proven" ? 2 : 0, reconnectCount: 0,
|
||||||
|
processedCommands: {}, lastConnectionProtocolVersion: scenario.endsWith("_connected")
|
||||||
|
|| scenario === "interrupted_bootstrap_connected_checkpoint_proven" ? 2 : null,
|
||||||
environmentLeaseId: "lease-resumed", turnId: `turn-${execution.binding.runId}`, itemId: `item-${execution.binding.runId}` } : {}) })).toString("base64"),
|
environmentLeaseId: "lease-resumed", turnId: `turn-${execution.binding.runId}`, itemId: `item-${execution.binding.runId}` } : {}) })).toString("base64"),
|
||||||
};
|
};
|
||||||
if (command.args?.[1]?.includes("base64")) return {
|
if (command.args?.[1]?.includes("base64")) return {
|
||||||
@@ -21572,6 +21576,10 @@ describe("runnerd provider runtime wiring", () => {
|
|||||||
};
|
};
|
||||||
return { exitCode: 0, timedOut: false, stdout: "", stderr: "" };
|
return { exitCode: 0, timedOut: false, stdout: "", stderr: "" };
|
||||||
});
|
});
|
||||||
|
const controllerProof = vi.spyOn(runnerCheckpoint, "nativeRunnerCheckpointRemoteBootstrapEvidence")
|
||||||
|
.mockResolvedValue(scenario === "interrupted_bootstrap_connected_checkpoint_proven"
|
||||||
|
? { proof: "session_opened_without_turn", runnerInstanceId: "runner-new-in-retained-workspace" } : null);
|
||||||
|
onTestFinished(() => controllerProof.mockRestore());
|
||||||
const runtimeContext = nativeRuntimeContextFixture();
|
const runtimeContext = nativeRuntimeContextFixture();
|
||||||
const executionWithContext = homeMode === "preconfigured"
|
const executionWithContext = homeMode === "preconfigured"
|
||||||
? { ...execution, schema: "paperclip.native-execution-input.v7" as const,
|
? { ...execution, schema: "paperclip.native-execution-input.v7" as const,
|
||||||
@@ -21686,7 +21694,7 @@ describe("runnerd provider runtime wiring", () => {
|
|||||||
noteRemoteBootstrapRestartProven(execution.binding.runId, "stopped");
|
noteRemoteBootstrapRestartProven(execution.binding.runId, "stopped");
|
||||||
}
|
}
|
||||||
if (scenario === "interrupted_bootstrap" || scenario === "interrupted_bootstrap_unopened" || scenario === "interrupted_bootstrap_unprepared"
|
if (scenario === "interrupted_bootstrap" || scenario === "interrupted_bootstrap_unopened" || scenario === "interrupted_bootstrap_unprepared"
|
||||||
|| scenario === "interrupted_bootstrap_unprepared_unrecorded") {
|
|| scenario === "interrupted_bootstrap_unprepared_unrecorded" || scenario === "interrupted_bootstrap_connected_checkpoint_proven") {
|
||||||
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
|
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
|
||||||
const siblings = await readdir(dirname(sessionRoot));
|
const siblings = await readdir(dirname(sessionRoot));
|
||||||
expect(siblings.filter(name => name.startsWith(`${basename(sessionRoot)}.bootstrap-restart-`))).toHaveLength(1);
|
expect(siblings.filter(name => name.startsWith(`${basename(sessionRoot)}.bootstrap-restart-`))).toHaveLength(1);
|
||||||
|
|||||||
@@ -161,7 +161,7 @@ import {
|
|||||||
import { lstat, readFile, readdir, readlink } from "node:fs/promises";
|
import { lstat, readFile, readdir, readlink } from "node:fs/promises";
|
||||||
import { hostname, tmpdir, userInfo } from "node:os";
|
import { hostname, tmpdir, userInfo } from "node:os";
|
||||||
import { DatabaseSync } from "node:sqlite";
|
import { DatabaseSync } from "node:sqlite";
|
||||||
import { NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, loadNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader, loadNativeUnclaimedBootstrapCheckpoint, nativeRunnerCheckpointPersistence, nativeRunnerNeverAuthenticated, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
import { NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, loadNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader, loadNativeUnclaimedBootstrapCheckpoint, nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRemoteBootstrapEvidence, nativeRunnerNeverAuthenticated, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||||
import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS, localRunnerProcessReconnectGraceMs } from "./native-runner-recovery-contract.js";
|
import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS, localRunnerProcessReconnectGraceMs } from "./native-runner-recovery-contract.js";
|
||||||
import type { NativeTaskSessionPublicationContext } from "./native-task-session-publication.js";
|
import type { NativeTaskSessionPublicationContext } from "./native-task-session-publication.js";
|
||||||
import { basename, dirname, join, posix, resolve } from "node:path";
|
import { basename, dirname, join, posix, resolve } from "node:path";
|
||||||
@@ -6283,6 +6283,7 @@ async function migrateRunnerdStateRootForExecution(input: {
|
|||||||
if (input.restartRecovery?.kind === "reattach_remote_runner" && !existsSync(scoped)) {
|
if (input.restartRecovery?.kind === "reattach_remote_runner" && !existsSync(scoped)) {
|
||||||
throw new Error("runner_state_identity_mismatch: remote_reattach_root_missing");
|
throw new Error("runner_state_identity_mismatch: remote_reattach_root_missing");
|
||||||
}
|
}
|
||||||
|
await captureClaimTimeRemoteBootstrapProof(input);
|
||||||
await restoreRemoteBootstrapCheckpointForRecovery({ ...input, root: scoped });
|
await restoreRemoteBootstrapCheckpointForRecovery({ ...input, root: scoped });
|
||||||
if (existsSync(scoped)) {
|
if (existsSync(scoped)) {
|
||||||
if (!isSafeNativeStateDirectory(scoped)) {
|
if (!isSafeNativeStateDirectory(scoped)) {
|
||||||
@@ -6926,6 +6927,34 @@ export function recordRemoteBootstrapRestartProof(runId: string,
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The controller checkpoint proof read before this server registers its own
|
||||||
|
* controller for the run. A new controller rewrites that checkpoint with its
|
||||||
|
* own bootstrap commands, so a later read would no longer describe what the
|
||||||
|
* interrupted runner received.
|
||||||
|
*/
|
||||||
|
const claimTimeRemoteBootstrapProofs = new Map<string, string>();
|
||||||
|
|
||||||
|
export async function captureClaimTimeRemoteBootstrapProof(input: {
|
||||||
|
db: Pick<Db, "select">;
|
||||||
|
execution: NativeExecutionInput;
|
||||||
|
restartRecovery?: NativeRestartRecoveryClaim;
|
||||||
|
runnerExecutionTarget?: AdapterExecutionTarget | null;
|
||||||
|
}): Promise<void> {
|
||||||
|
const runId = input.execution.binding.runId;
|
||||||
|
claimTimeRemoteBootstrapProofs.delete(runId);
|
||||||
|
const target = input.runnerExecutionTarget;
|
||||||
|
if (input.restartRecovery?.kind !== "bootstrap_incomplete" || input.restartRecovery.runId !== runId
|
||||||
|
|| target?.kind !== "remote" || target.transport !== "sandbox" || target.runnerTask) return;
|
||||||
|
const evidence = await nativeRunnerCheckpointRemoteBootstrapEvidence(input.db, input.execution.binding,
|
||||||
|
nativeSessionKey(input.execution));
|
||||||
|
if (evidence) claimTimeRemoteBootstrapProofs.set(runId, evidence.runnerInstanceId);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function claimTimeRemoteBootstrapProofHolds(runId: string, runnerInstanceId: unknown): boolean {
|
||||||
|
return typeof runnerInstanceId === "string" && claimTimeRemoteBootstrapProofs.get(runId) === runnerInstanceId;
|
||||||
|
}
|
||||||
|
|
||||||
export function hasRemoteBootstrapRestartProof(runId: string): boolean {
|
export function hasRemoteBootstrapRestartProof(runId: string): boolean {
|
||||||
return provenRemoteBootstrapRestarts.has(runId);
|
return provenRemoteBootstrapRestarts.has(runId);
|
||||||
}
|
}
|
||||||
@@ -16796,7 +16825,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
|||||||
if (!runnerState || runnerState.runnerInstanceId !== input.runnerInstanceId || runnerState.runId !== runId
|
if (!runnerState || runnerState.runnerInstanceId !== input.runnerInstanceId || runnerState.runId !== runId
|
||||||
|| runnerState.normalizedSessionId !== nativeSessionKey(input.execution)
|
|| runnerState.normalizedSessionId !== nativeSessionKey(input.execution)
|
||||||
|| typeof runnerState.lifecycle !== "string" || runnerState.lifecycle === "suspended") return null;
|
|| typeof runnerState.lifecycle !== "string" || runnerState.lifecycle === "suspended") return null;
|
||||||
if (!hasRemoteBootstrapRestartProof(runId) && !(remoteRunnerStateNeverConnected(runnerState)
|
if (!hasRemoteBootstrapRestartProof(runId) && !((remoteRunnerStateNeverConnected(runnerState)
|
||||||
|
|| claimTimeRemoteBootstrapProofHolds(runId, runnerState.runnerInstanceId))
|
||||||
&& recordRemoteBootstrapRestartProof(runId, await stopStaleRemoteRunnerBeforeBootstrap({
|
&& recordRemoteBootstrapRestartProof(runId, await stopStaleRemoteRunnerBeforeBootstrap({
|
||||||
execution: input.execution, runnerExecutionTarget: input.runnerExecutionTarget, identity: runnerState, onLog: input.onLog,
|
execution: input.execution, runnerExecutionTarget: input.runnerExecutionTarget, identity: runnerState, onLog: input.onLog,
|
||||||
})))) return null;
|
})))) return null;
|
||||||
@@ -16882,12 +16912,16 @@ async function createRunnerdBackendWithinSessionClaim(
|
|||||||
throw new Error("runner_harness_state_mismatch: runner_identity");
|
throw new Error("runner_harness_state_mismatch: runner_identity");
|
||||||
}
|
}
|
||||||
if (runnerState.lifecycle !== "suspended") {
|
if (runnerState.lifecycle !== "suspended") {
|
||||||
|
const notSuspendedRunId = typeof runnerState.runId === "string" ? runnerState.runId : null;
|
||||||
|
if (source === "host_preparation" && notSuspendedRunId === input.execution.binding.runId) {
|
||||||
|
await inspectOwnUnpreparedBootstrap();
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
complete: false,
|
complete: false,
|
||||||
runnerState: null,
|
runnerState: null,
|
||||||
providerSessionIdentity: null,
|
providerSessionIdentity: null,
|
||||||
incompleteReason: "not_suspended",
|
incompleteReason: "not_suspended",
|
||||||
notSuspendedRunId: typeof runnerState.runId === "string" ? runnerState.runId : null,
|
notSuspendedRunId,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
let providerState: Record<string, unknown>;
|
let providerState: Record<string, unknown>;
|
||||||
|
|||||||
Reference in new issue
Block a user