mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.
## Linked Issues or Issue Description
**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.
**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.
**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.
**Paperclip version or commit**
Staging at 1d153f62b6. The code is the same on master at 704b3f4d63.
**Deployment mode**
Hosted staging (paperclip.staging.paperclip.app), remote sandbox
execution on Daytona.
**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).
## What Changed
- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapEvidence` returns the bootstrap
proof together with the runner instance that the checkpoint names.
`nativeRunnerCheckpointRemoteBootstrapProof` now wraps it and behaves as
before.
- `native-session-executor.ts`:
- State migration runs before this server registers its own controller
for the run. Under a `bootstrap_incomplete` claim for a remote sandbox,
it now reads the interrupted controller's checkpoint proof there
(`captureClaimTimeRemoteBootstrapProof`). A later read is not used,
because the new controller rewrites the checkpoint with its own
bootstrap commands.
- Host preparation: when this run's sandbox Runner is not suspended, it
runs the same-run restart proof. The proof accepts a Runner that never
connected (as before), or the claim-time proof for the exact runner
instance in the sandbox state. It then stops that exact Runner by its
identity marker and records the stop proof. A backup, a different run
id, another runner instance, or a failed stop still fails closed.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_connected_checkpoint_proven` case. The Runner is
`ready` and connected, and no stop proof exists yet. It fails before the
fix with `resumed_not_suspended` and passes after it.
`interrupted_bootstrap_unproven` still fails closed.
- `native-bootstrap-claim-time-proof.integration.test.ts` (embedded
Postgres, no mocks of the proof): an interrupted controller that had
connected registers its checkpoint, the claim-time proof is read, and
then a replacement controller registers its own checkpoint. A live read
after that returns `bootstrap_incomplete` even when the interrupted
controller had sent `turn.start`. The claim-time proof stays false for
that case and true for the bootstrap-only case, and it refuses another
runner instance.
## Verification
- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` fails on the new case.
- Green: the same command passes (65 tests after merging master).
- `pnpm exec vitest run
server/src/services/native-runtime/native-bootstrap-claim-time-proof.integration.test.ts`:
2 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
server/src/services/native-runtime/native-restart-recovery.test.ts`:
1474 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- Merged `origin/master` (including #21134) without conflicts and reran
the tests above.
## Risks
- Low. The new path runs only under a `bootstrap_incomplete` claim for
the same run id, with no failover backup, and only after the Postgres
controller checkpoint proves that no provider turn was sent. It stops a
Runner only when the identity marker and process fingerprint match,
which is the same check the reattach path uses.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #21134 (the never-launched branch of the same helper), #21079
(stops a leftover Runner at launch time; this change acts earlier, in
host preparation).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0bac3510db
commit
3fb35fcf13
4 files changed
+124
-13
No files matched your search
+58
@@ -0,0 +1,58 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { expect, it } from "vitest";
|
||||
import { nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
|
||||
import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
|
||||
import { nativeRunnerCheckpointRemoteBootstrapProof, registerNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { captureClaimTimeRemoteBootstrapProof, claimTimeRemoteBootstrapProofHolds } from "./native-session-executor.js";
|
||||
|
||||
describeEmbeddedPostgres("claim-time remote bootstrap proof", () => {
|
||||
const environment = nativeRestartFixture();
|
||||
|
||||
it.each([
|
||||
{ label: "turn-sent", oldCommands: ["run.prepare", "session.open", "turn.start"], holds: false },
|
||||
{ label: "bootstrap-only", oldCommands: ["run.prepare", "session.open"], holds: true },
|
||||
])("keeps the interrupted controller's proof after the new controller registers: $label", async ({ label, oldCommands, holds }) => {
|
||||
const { db, runId, issueId, native } = await environment.seedRun(`CLAIM-PROOF-${label}`);
|
||||
const { companyId, successor, runtimeRoot } = environment.current;
|
||||
const binding = { companyId, issueId, runId };
|
||||
const identity = { runnerInstanceId: native.runnerInstanceId, environmentLeaseId: native.environmentLeaseId, runId,
|
||||
normalizedSessionId: native.normalizedSessionId, turnId: native.turnId, itemId: native.itemId };
|
||||
const controller = (directory: string) => new DurablePrpControlPlane({ stateDirectory: join(runtimeRoot, directory), identity,
|
||||
expectedRunnerVersion: "fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
|
||||
const own = async (generation: number) => db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId,
|
||||
controllerGeneration: generation, leaseOwner: randomUUID(), leaseExpiresAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(nativeRunFinalizations.runId, runId));
|
||||
const execution = { binding, session: { normalizedSessionId: native.normalizedSessionId } } as never;
|
||||
const claim = { kind: "bootstrap_incomplete" as const, runId, leaseOwner: "recovered", controllerGeneration: 2,
|
||||
providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null };
|
||||
const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace" } as never;
|
||||
const interrupted = controller(`${runId}-interrupted`);
|
||||
const replacement = controller(`${runId}-replacement`);
|
||||
try {
|
||||
await own(1);
|
||||
for (const type of oldCommands) interrupted.queueCommand(type, {});
|
||||
interrupted.store.state.connectionCount = 1;
|
||||
await registerNativeRunnerCheckpoint({ db, authority: interrupted, bootId: successor.bootId, binding });
|
||||
await interrupted.stop();
|
||||
|
||||
await own(2);
|
||||
await captureClaimTimeRemoteBootstrapProof({ db, execution, restartRecovery: claim, runnerExecutionTarget: target });
|
||||
expect(claimTimeRemoteBootstrapProofHolds(runId, native.runnerInstanceId)).toBe(holds);
|
||||
|
||||
replacement.queueCommand("run.prepare", {});
|
||||
replacement.queueCommand("session.open", { reuse: "same_session" });
|
||||
await registerNativeRunnerCheckpoint({ db, authority: replacement, bootId: successor.bootId, binding });
|
||||
|
||||
expect(await nativeRunnerCheckpointRemoteBootstrapProof(db, binding,
|
||||
{ runnerInstanceId: native.runnerInstanceId, normalizedSessionId: native.normalizedSessionId })).toBe("bootstrap_incomplete");
|
||||
expect(claimTimeRemoteBootstrapProofHolds(runId, native.runnerInstanceId)).toBe(holds);
|
||||
expect(claimTimeRemoteBootstrapProofHolds(runId, "another-runner")).toBe(false);
|
||||
} finally {
|
||||
await interrupted.stop();
|
||||
await replacement.stop();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -302,6 +302,17 @@ export type NativeRemoteBootstrapProof = "bootstrap_incomplete" | "session_opene
|
||||
export async function nativeRunnerCheckpointRemoteBootstrapProof(
|
||||
db: Pick<Db, "select">, binding: Binding, expected: { runnerInstanceId: string; normalizedSessionId: string },
|
||||
): Promise<NativeRemoteBootstrapProof | null> {
|
||||
const evidence = await nativeRunnerCheckpointRemoteBootstrapEvidence(db, binding, expected.normalizedSessionId);
|
||||
return evidence?.runnerInstanceId === expected.runnerInstanceId ? evidence.proof : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The same proof, with the runner instance the checkpoint names. A caller that
|
||||
* must hold the proof across a new controller's registration reads it first.
|
||||
*/
|
||||
export async function nativeRunnerCheckpointRemoteBootstrapEvidence(
|
||||
db: Pick<Db, "select">, binding: Binding, normalizedSessionId: string,
|
||||
): Promise<{ proof: NativeRemoteBootstrapProof; runnerInstanceId: string } | null> {
|
||||
try {
|
||||
const limit = BOOTSTRAP_CHECKPOINT_PARTS.size + MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + MAX_BOOTSTRAP_CHECKPOINT_EVENTS;
|
||||
const parts = await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints).where(and(
|
||||
@@ -314,8 +325,8 @@ export async function nativeRunnerCheckpointRemoteBootstrapProof(
|
||||
|| parts.some(row => !BOOTSTRAP_CHECKPOINT_PARTS.has(row.part) && !/^(?:event|command):\d+$/.test(row.part))) return null;
|
||||
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
|
||||
if (!checkpoint || checkpoint.predecessor !== null
|
||||
|| checkpoint.recoveryEvidence.runnerInstanceId !== expected.runnerInstanceId
|
||||
|| checkpoint.recoveryEvidence.normalizedSessionId !== expected.normalizedSessionId) return null;
|
||||
|| checkpoint.recoveryEvidence.normalizedSessionId !== normalizedSessionId) return null;
|
||||
const runnerInstanceId = checkpoint.recoveryEvidence.runnerInstanceId;
|
||||
// The decoder above validated these canonical v2 records. Rebuild the
|
||||
// controller state in memory without writing files under recovery locks.
|
||||
const read = (part: string) => {
|
||||
@@ -335,8 +346,8 @@ export async function nativeRunnerCheckpointRemoteBootstrapProof(
|
||||
.map(seq => read(`event:${seq}`)),
|
||||
};
|
||||
if ((state.identity as { runId?: unknown } | undefined)?.runId !== binding.runId) return null;
|
||||
if (runnerdStateIsIncompleteBootstrap(state)) return "bootstrap_incomplete";
|
||||
return runnerdStateHasNoProviderTurn(state) ? "session_opened_without_turn" : null;
|
||||
if (runnerdStateIsIncompleteBootstrap(state)) return { proof: "bootstrap_incomplete", runnerInstanceId };
|
||||
return runnerdStateHasNoProviderTurn(state) ? { proof: "session_opened_without_turn", runnerInstanceId } : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
@@ -21462,7 +21462,7 @@ describe("runnerd provider runtime wiring", () => {
|
||||
"interrupted_bootstrap_unopened", "interrupted_bootstrap_unopened_unproven", "interrupted_bootstrap_unopened_other_run",
|
||||
"interrupted_bootstrap_unopened_active_turn", "interrupted_bootstrap_unprepared", "interrupted_bootstrap_unprepared_unproven",
|
||||
"interrupted_bootstrap_unprepared_other_run", "interrupted_bootstrap_unprepared_suspended", "interrupted_bootstrap_unprepared_backup",
|
||||
"interrupted_bootstrap_unprepared_unrecorded", "interrupted_bootstrap_unprepared_connected"]
|
||||
"interrupted_bootstrap_unprepared_unrecorded", "interrupted_bootstrap_unprepared_connected", "interrupted_bootstrap_connected_checkpoint_proven"]
|
||||
.map(scenario => ({ outcome: "resumed", scenario, homeMode: "isolated" })),
|
||||
...["interrupted_bootstrap_unlaunched", "interrupted_bootstrap_unlaunched_marker", "interrupted_bootstrap_unlaunched_backup",
|
||||
"interrupted_bootstrap_unlaunched_archived", "interrupted_bootstrap_unlaunched_nested_state"]
|
||||
@@ -21544,10 +21544,12 @@ describe("runnerd provider runtime wiring", () => {
|
||||
exitCode: 1, timedOut: false, stdout: "", stderr: "" };
|
||||
if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unprepared")
|
||||
&& command.args[1].includes("codex-provider-state.json")) return { exitCode: 1, timedOut: false, stdout: "", stderr: "" };
|
||||
if (command.args?.[2] === "paperclip-runner-recovery-identity" && scenario === "interrupted_bootstrap_unprepared_unrecorded") return {
|
||||
if (command.args?.[2] === "paperclip-runner-recovery-identity" && ["interrupted_bootstrap_unprepared_unrecorded",
|
||||
"interrupted_bootstrap_connected_checkpoint_proven"].includes(scenario)) return {
|
||||
exitCode: 0, timedOut: false, stderr: "",
|
||||
stdout: "exitednonce\n4242\n2026-11-12T00:32:10.725Z\nrunner-new-in-retained-workspace\nlinux:abcd-1234:100\n" };
|
||||
if (command.args?.[2] === "paperclip-runner-recovery-check" && scenario === "interrupted_bootstrap_unprepared_unrecorded") return {
|
||||
if (command.args?.[2] === "paperclip-runner-recovery-check" && ["interrupted_bootstrap_unprepared_unrecorded",
|
||||
"interrupted_bootstrap_connected_checkpoint_proven"].includes(scenario)) return {
|
||||
exitCode: 45, timedOut: false, stdout: "", stderr: "" };
|
||||
if (command.args?.[1]?.includes("base64") && scenario.startsWith("interrupted_bootstrap_unopened")
|
||||
&& command.args[1].includes("codex-provider-state.json") && !command.args[1].includes("runner-state.json")) return {
|
||||
@@ -21563,8 +21565,10 @@ describe("runnerd provider runtime wiring", () => {
|
||||
lifecycle: scenario.startsWith("interrupted_bootstrap_unopened") || scenario.endsWith("_suspended") ? "suspended"
|
||||
: scenario.startsWith("interrupted_bootstrap_unprepared") ? "connecting" : "ready",
|
||||
outbox: [], pendingTerminalDelivery: null,
|
||||
...(scenario.startsWith("interrupted_bootstrap_unprepared") ? { lastControllerCommandSeq: 0, reconnectCount: 0,
|
||||
processedCommands: {}, lastConnectionProtocolVersion: scenario.endsWith("_connected") ? 2 : null,
|
||||
...(scenario.startsWith("interrupted_bootstrap_unprepared") || scenario === "interrupted_bootstrap_connected_checkpoint_proven"
|
||||
? { lastControllerCommandSeq: scenario === "interrupted_bootstrap_connected_checkpoint_proven" ? 2 : 0, reconnectCount: 0,
|
||||
processedCommands: {}, lastConnectionProtocolVersion: scenario.endsWith("_connected")
|
||||
|| scenario === "interrupted_bootstrap_connected_checkpoint_proven" ? 2 : null,
|
||||
environmentLeaseId: "lease-resumed", turnId: `turn-${execution.binding.runId}`, itemId: `item-${execution.binding.runId}` } : {}) })).toString("base64"),
|
||||
};
|
||||
if (command.args?.[1]?.includes("base64")) return {
|
||||
@@ -21572,6 +21576,10 @@ describe("runnerd provider runtime wiring", () => {
|
||||
};
|
||||
return { exitCode: 0, timedOut: false, stdout: "", stderr: "" };
|
||||
});
|
||||
const controllerProof = vi.spyOn(runnerCheckpoint, "nativeRunnerCheckpointRemoteBootstrapEvidence")
|
||||
.mockResolvedValue(scenario === "interrupted_bootstrap_connected_checkpoint_proven"
|
||||
? { proof: "session_opened_without_turn", runnerInstanceId: "runner-new-in-retained-workspace" } : null);
|
||||
onTestFinished(() => controllerProof.mockRestore());
|
||||
const runtimeContext = nativeRuntimeContextFixture();
|
||||
const executionWithContext = homeMode === "preconfigured"
|
||||
? { ...execution, schema: "paperclip.native-execution-input.v7" as const,
|
||||
@@ -21686,7 +21694,7 @@ describe("runnerd provider runtime wiring", () => {
|
||||
noteRemoteBootstrapRestartProven(execution.binding.runId, "stopped");
|
||||
}
|
||||
if (scenario === "interrupted_bootstrap" || scenario === "interrupted_bootstrap_unopened" || scenario === "interrupted_bootstrap_unprepared"
|
||||
|| scenario === "interrupted_bootstrap_unprepared_unrecorded") {
|
||||
|| scenario === "interrupted_bootstrap_unprepared_unrecorded" || scenario === "interrupted_bootstrap_connected_checkpoint_proven") {
|
||||
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
|
||||
const siblings = await readdir(dirname(sessionRoot));
|
||||
expect(siblings.filter(name => name.startsWith(`${basename(sessionRoot)}.bootstrap-restart-`))).toHaveLength(1);
|
||||
|
||||
@@ -161,7 +161,7 @@ import {
|
||||
import { lstat, readFile, readdir, readlink } from "node:fs/promises";
|
||||
import { hostname, tmpdir, userInfo } from "node:os";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, loadNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader, loadNativeUnclaimedBootstrapCheckpoint, nativeRunnerCheckpointPersistence, nativeRunnerNeverAuthenticated, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, loadNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader, loadNativeUnclaimedBootstrapCheckpoint, nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRemoteBootstrapEvidence, nativeRunnerNeverAuthenticated, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS, localRunnerProcessReconnectGraceMs } from "./native-runner-recovery-contract.js";
|
||||
import type { NativeTaskSessionPublicationContext } from "./native-task-session-publication.js";
|
||||
import { basename, dirname, join, posix, resolve } from "node:path";
|
||||
@@ -6283,6 +6283,7 @@ async function migrateRunnerdStateRootForExecution(input: {
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner" && !existsSync(scoped)) {
|
||||
throw new Error("runner_state_identity_mismatch: remote_reattach_root_missing");
|
||||
}
|
||||
await captureClaimTimeRemoteBootstrapProof(input);
|
||||
await restoreRemoteBootstrapCheckpointForRecovery({ ...input, root: scoped });
|
||||
if (existsSync(scoped)) {
|
||||
if (!isSafeNativeStateDirectory(scoped)) {
|
||||
@@ -6926,6 +6927,34 @@ export function recordRemoteBootstrapRestartProof(runId: string,
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* The controller checkpoint proof read before this server registers its own
|
||||
* controller for the run. A new controller rewrites that checkpoint with its
|
||||
* own bootstrap commands, so a later read would no longer describe what the
|
||||
* interrupted runner received.
|
||||
*/
|
||||
const claimTimeRemoteBootstrapProofs = new Map<string, string>();
|
||||
|
||||
export async function captureClaimTimeRemoteBootstrapProof(input: {
|
||||
db: Pick<Db, "select">;
|
||||
execution: NativeExecutionInput;
|
||||
restartRecovery?: NativeRestartRecoveryClaim;
|
||||
runnerExecutionTarget?: AdapterExecutionTarget | null;
|
||||
}): Promise<void> {
|
||||
const runId = input.execution.binding.runId;
|
||||
claimTimeRemoteBootstrapProofs.delete(runId);
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (input.restartRecovery?.kind !== "bootstrap_incomplete" || input.restartRecovery.runId !== runId
|
||||
|| target?.kind !== "remote" || target.transport !== "sandbox" || target.runnerTask) return;
|
||||
const evidence = await nativeRunnerCheckpointRemoteBootstrapEvidence(input.db, input.execution.binding,
|
||||
nativeSessionKey(input.execution));
|
||||
if (evidence) claimTimeRemoteBootstrapProofs.set(runId, evidence.runnerInstanceId);
|
||||
}
|
||||
|
||||
export function claimTimeRemoteBootstrapProofHolds(runId: string, runnerInstanceId: unknown): boolean {
|
||||
return typeof runnerInstanceId === "string" && claimTimeRemoteBootstrapProofs.get(runId) === runnerInstanceId;
|
||||
}
|
||||
|
||||
export function hasRemoteBootstrapRestartProof(runId: string): boolean {
|
||||
return provenRemoteBootstrapRestarts.has(runId);
|
||||
}
|
||||
@@ -16796,7 +16825,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
if (!runnerState || runnerState.runnerInstanceId !== input.runnerInstanceId || runnerState.runId !== runId
|
||||
|| runnerState.normalizedSessionId !== nativeSessionKey(input.execution)
|
||||
|| typeof runnerState.lifecycle !== "string" || runnerState.lifecycle === "suspended") return null;
|
||||
if (!hasRemoteBootstrapRestartProof(runId) && !(remoteRunnerStateNeverConnected(runnerState)
|
||||
if (!hasRemoteBootstrapRestartProof(runId) && !((remoteRunnerStateNeverConnected(runnerState)
|
||||
|| claimTimeRemoteBootstrapProofHolds(runId, runnerState.runnerInstanceId))
|
||||
&& recordRemoteBootstrapRestartProof(runId, await stopStaleRemoteRunnerBeforeBootstrap({
|
||||
execution: input.execution, runnerExecutionTarget: input.runnerExecutionTarget, identity: runnerState, onLog: input.onLog,
|
||||
})))) return null;
|
||||
@@ -16882,12 +16912,16 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
throw new Error("runner_harness_state_mismatch: runner_identity");
|
||||
}
|
||||
if (runnerState.lifecycle !== "suspended") {
|
||||
const notSuspendedRunId = typeof runnerState.runId === "string" ? runnerState.runId : null;
|
||||
if (source === "host_preparation" && notSuspendedRunId === input.execution.binding.runId) {
|
||||
await inspectOwnUnpreparedBootstrap();
|
||||
}
|
||||
return {
|
||||
complete: false,
|
||||
runnerState: null,
|
||||
providerSessionIdentity: null,
|
||||
incompleteReason: "not_suspended",
|
||||
notSuspendedRunId: typeof runnerState.runId === "string" ? runnerState.runId : null,
|
||||
notSuspendedRunId,
|
||||
};
|
||||
}
|
||||
let providerState: Record<string, unknown>;
|
||||
|
||||
Reference in new issue
Block a user