mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
fix(native): recover a run killed between runner launch and its identity record (#19694)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The server records the request to start it, then records its process identity after it spawns. > - If the server is killed between those two records, there is no process ID to check after the restart > - Restart recovery then restarted the bootstrap, but the executor failed with `durable_identity_unreadable` because the runner never wrote its state > - The user's Retry then waited forever with "Retry is waiting for Paperclip to verify that the previous execution stopped", because the stop check needs a process ID > - This pull request restarts that bootstrap and proves on the host that no runner for the run exists > - The benefit is that a restart in the runner launch window no longer loses the task or leaves Retry permanently stuck ## Linked Issues or Issue Description **What happened?** In restart-chaos testing, an ACPX Claude run was created and the server was SIGKILLed 1.3 s later, right after `native.process_start_requested` and before the runner identity was recorded. On restart, recovery claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`). The executor then threw `runner_state_identity_mismatch: durable_identity_unreadable`: the state root existed, but `control-plane/control-plane-state.json` had never been written. The run failed with `native_continuation_requires_reconciliation`. A Retry was accepted (HTTP 200), but no run was ever created. The issue stayed blocked with `retryWait.reason=process_identity_missing` for over 20 minutes and through another restart. **Expected behavior** When the runner never wrote state, no provider started, so the bootstrap can start again. A Retry should be admitted once the host proves that no runner for the run exists. **Paperclip version or commit** `master` at `fb3cf1a072`. **Steps to reproduce** Create a native local task and SIGKILL the server about 1 s after the run starts, after the runner launch is requested and before its identity is recorded. Restart the server, then click Retry. ## What Changed - `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`): for a `bootstrap_incomplete` restart on a local target, a root is quarantined (kept for inspection) and the bootstrap starts again only when both of these hold: - the root has no `control-plane-state.json`; - no live runner for the run exists on this host. A state file that exists but cannot be read, or a runner that started but has not connected yet, still fails closed with `durable_identity_unreadable`. - `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`) proves that a runner is absent when both of these hold: - no host process carries the run's `--run-id <runId>` argument (`paperclip-runnerd` is always started with it, detached and unreferenced); - no saved control-plane state for this run exists, in the canonical root or in quarantine. A provider starts only after its runner connects and writes that state. - `native.process_start_requested` now records the launching `hostname`. - `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`) applies the proof only when all of these hold: - the run is native; - the run has no process ID; - its latest launch event is `native.process_start_requested`, from this host; - all of its leases are local. A launch recorded without a host stays blocked. - The process listing runs asynchronously with a 5 s deadline. A failed or slow listing proves nothing. It then writes `native.local_process_stopped` with null process IDs and `basis: "unidentified_runner_absent"`. A later launch request invalidates it as usual. - The continuation and Retry gate (`explicit-native-continuation.ts`) uses this proof before returning `process_identity_missing`. A dry run checks the proof but records nothing. - `doc/run-log-events.md` documents the new receipt basis. ## Verification - `native-session-executor.test.ts`: - The bootstrap restart with absent state was red before the fix (`durable_identity_unreadable`) and is green after it (root quarantined, backend admitted). - Unreadable state, and a live runner that has not connected, still reject. - Proof cases: absent → true; a live process with `--run-id <runId>` → false; this run's saved state → false; another run ID → false. - `explicit-native-continuation.test.ts` (embedded Postgres, real admission): - The `absent` and `dry_run` cases were red before the fix (blocked) and are green after it (admitted). Only the real admission records the receipt. - The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch` cases stay blocked. - Broader local run after the review fixes: `explicit-native-continuation`, `native-session-executor`, `native-recovered-owner-followup`, `native-runner-restart-recovery.integration` and `task-retry` gave 1836 passed, 0 failed. - A chaos replay of the original task (`--replay w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after the runner identity was recorded (`resume_dead_runner`), not in the launch window, so the tests above are what reproduce that window. - `tsc --noEmit` for `server` passes. ## Risks - The proof depends on `ps -axww` listing every process on the launching host. If `ps` fails or times out, the proof fails closed (still blocked). Remote runners and other hosts are excluded. - A runner that was spawned but has not yet written its state could be mid-start when admission runs. It would still carry `--run-id` in its arguments, so the proof refuses. - Only the bootstrap path with no state file changes. Ownership, checkpoint and reattach checks are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
aca908a165
commit
30857db55e
6 files changed
+228
-5
No files matched your search
@@ -391,6 +391,15 @@ clears a local run's process identity, after it verifies that its PID and proces
|
||||
group are absent. The payload contains only those process IDs. Remote process
|
||||
IDs are never checked against the control-plane host.
|
||||
|
||||
A crash can land after `native.process_start_requested` and before the runner's
|
||||
process identity is recorded. Continuation admission then writes
|
||||
`native.local_process_stopped` with null process IDs and
|
||||
`basis: "unidentified_runner_absent"`. It does this only on the host named in
|
||||
the launch event's `hostname` payload, after it checks two things. First, no
|
||||
process carries the run's `--run-id` runner argument. Second, no
|
||||
saved runner control-plane state (canonical or quarantined) belongs to the run;
|
||||
a provider starts only after its runner connects and writes that state.
|
||||
|
||||
The server writes `native.process_start_requested` before a backend can spawn,
|
||||
and `native.process_identity_recorded` when it stores a new native process
|
||||
identity. Either invalidates an earlier local stop receipt, including a crash
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { legacyControllerBootId } from "./legacy-controller-lease.js";
|
||||
import { execFile } from "node:child_process";
|
||||
import { execFile, spawn } from "node:child_process";
|
||||
import { once } from "node:events";
|
||||
import os from "node:os";
|
||||
import { createSystemRecovery, failSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
@@ -1120,6 +1121,63 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
|
||||
} finally { verify.mockRestore(); }
|
||||
});
|
||||
|
||||
it.each(["absent", "live_runner", "saved_state", "dry_run", "foreign_host", "unbound_launch"] as const)(
|
||||
"admits a Retry after a launch whose runner identity was never recorded only when no runner exists (%s)", async kind => {
|
||||
// RC-42: the server was killed after requesting the runner launch and
|
||||
// before recording its process identity, so there is no PID to verify.
|
||||
const f = await seed();
|
||||
const stateBase = await mkdtemp(join(tmpdir(), "unidentified-runner-"));
|
||||
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase;
|
||||
let child: ReturnType<typeof spawn> | undefined;
|
||||
try {
|
||||
const nativeSessionId = randomUUID();
|
||||
const execution = {
|
||||
schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
||||
binding: { companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.sourceRunId, executionWorkspaceId: "workspace" },
|
||||
task: { identifier: "TEST", title: "Continue", description: null, prompt: "Continue", workMode: "standard" },
|
||||
workspace: { cwd: stateBase, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: nativeSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Continue" }] } },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, nativeSessionId: null, runnerInstanceId: null,
|
||||
errorCode: "adapter_failed", runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
||||
// Only the launching host can inspect its process table.
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId, eventType: PROCESS_START_REQUESTED,
|
||||
...(kind === "unbound_launch" ? {} : { payload: { hostname: kind === "foreign_host" ? `other-${os.hostname()}` : os.hostname() } }) });
|
||||
if (kind === "live_runner") {
|
||||
child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", f.sourceRunId], { stdio: "ignore" });
|
||||
await once(child, "spawn");
|
||||
}
|
||||
if (kind === "saved_state") {
|
||||
const canonical = (value: unknown): string => value && typeof value === "object" && !Array.isArray(value)
|
||||
? `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`).join(",")}}`
|
||||
: JSON.stringify(value);
|
||||
const root = join(stateBase, createHash("sha256").update(canonical({
|
||||
schema: "paperclip.native-session-scope.v2", companyId: f.companyId, agentId: f.agentId,
|
||||
workspace: { kind: "managed", executionWorkspaceId: "workspace" },
|
||||
provider: { driverKind: "codex_app_server", identity: { kind: "codex" } }, normalizedSessionId: nativeSessionId,
|
||||
})).digest("hex"));
|
||||
await mkdir(join(root, "control-plane"), { recursive: true });
|
||||
await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify({
|
||||
schema: "paperclip.runner.durable.control-plane-state.v1", identity: { runId: f.sourceRunId } }));
|
||||
}
|
||||
const admitted = await admit(f, kind === "dry_run");
|
||||
if (kind === "absent" || kind === "dry_run") expect(admitted).toMatchObject({ previousRunId: f.sourceRunId });
|
||||
else expect(admitted).toBeNull();
|
||||
// Only a real admission records the stop; a later launch invalidates it.
|
||||
expect(await hasNativeLocalProcessStop(db, f.companyId, f.sourceRunId)).toBe(kind === "absent");
|
||||
} finally {
|
||||
if (child) { const exited = once(child, "exit"); child.kill("SIGKILL"); await exited; }
|
||||
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
||||
await rm(stateBase, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["suspended", "ready", "wrong_run", "wrong_thread", "active_provider", "pending_tool", "pending_output", "missing_state", "new_launch"])(
|
||||
"recovers a historical run without process metadata only from exact suspended state (%s)", async kind => {
|
||||
const f = await seed();
|
||||
|
||||
@@ -16,7 +16,7 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation.js";
|
||||
import { readQueuedInteractionResponse } from "./queued-interaction-response.js";
|
||||
import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js";
|
||||
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "./native-local-process-stop.js";
|
||||
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped, recordUnidentifiedLocalRunnerStop } from "./native-local-process-stop.js";
|
||||
import { completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
||||
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
|
||||
import { z } from "zod";
|
||||
@@ -485,7 +485,9 @@ export async function admitExplicitNativeContinuation(input: {
|
||||
// A missing process identity is not evidence that a provider exited.
|
||||
if (!run.processPid && !run.processGroupId &&
|
||||
!await hasNativeLocalProcessStop(db, companyId, run.id) &&
|
||||
!await hasHistoricalSuspendedNativeSession(db, run)) return blocked("process_identity_missing", "The previous run has no verified stop record. Paperclip cannot start this message yet.");
|
||||
!await hasHistoricalSuspendedNativeSession(db, run) &&
|
||||
!(run.runtimeMode === "native" &&
|
||||
await recordUnidentifiedLocalRunnerStop(db, run, { record: !input.dryRun }))) return blocked("process_identity_missing", "The previous run has no verified stop record. Paperclip cannot start this message yet.");
|
||||
if (run.processPid && !processStopped(run.processPid)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
|
||||
if (run.processGroupId && !processStopped(-run.processGroupId)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
|
||||
}
|
||||
|
||||
@@ -97,6 +97,43 @@ export async function hasNativeLocalProcessStop(db: Db, companyId: string, runId
|
||||
return event?.eventType === LOCAL_PROCESS_STOPPED;
|
||||
}
|
||||
|
||||
/**
|
||||
* A local runner launch can be requested and the server killed before the
|
||||
* runner's process identity is recorded. With no PID to check, prove on this
|
||||
* host that no runner for the run exists and that it never wrote runner state
|
||||
* (no provider can start before that), then record the stop like a verified
|
||||
* PID. A later launch request invalidates the receipt as usual.
|
||||
*/
|
||||
export async function recordUnidentifiedLocalRunnerStop(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "companyId" | "agentId" | "runtimeMode" | "processPid" | "processGroupId" | "runnerProfileJson">,
|
||||
options: { record: boolean } = { record: true }): Promise<boolean> {
|
||||
if (run.runtimeMode !== "native" || run.processPid || run.processGroupId) return false;
|
||||
const [latest] = await db.select({ eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
|
||||
eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
|
||||
isNull(heartbeatRunEvents.sourceEventId),
|
||||
inArray(heartbeatRunEvents.eventType, [PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, LOCAL_PROCESS_STOPPED]),
|
||||
)).orderBy(desc(heartbeatRunEvents.seq)).limit(1);
|
||||
// Only the host that requested the launch can inspect its process table.
|
||||
if (latest?.eventType !== PROCESS_START_REQUESTED || latest.payload?.hostname !== os.hostname()) return false;
|
||||
const leases = await db.select({ provider: environmentLeases.provider }).from(environmentLeases)
|
||||
.where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id)));
|
||||
if (leases.some(lease => lease.provider !== "local")) return false;
|
||||
const { unidentifiedLocalRunnerProvenAbsent } = await import("./native-runtime/native-session-executor.js");
|
||||
if (!await unidentifiedLocalRunnerProvenAbsent({ runId: run.id, execution: run.runnerProfileJson?.nativeExecutionInput })) return false;
|
||||
if (!options.record) return true;
|
||||
await appendHeartbeatRunEvent(db, {
|
||||
companyId: run.companyId,
|
||||
runId: run.id,
|
||||
agentId: run.agentId,
|
||||
eventType: LOCAL_PROCESS_STOPPED,
|
||||
stream: "system",
|
||||
level: "info",
|
||||
message: "Verified that no runner for this run exists on this host and none saved runner state.",
|
||||
payload: { processPid: null, processGroupId: null, processStartedAt: null, basis: "unidentified_runner_absent" },
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Pre-receipt native runs can retain an exact suspended session after their
|
||||
* mutable process fields were cleared. This is admission evidence for a new
|
||||
* user turn only, never permission to replay the old run or infer its outcomes.
|
||||
|
||||
@@ -17,6 +17,7 @@ import { configuredEnvironment, configuredEnvironmentProjection } from "../../ve
|
||||
import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
|
||||
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { once } from "node:events";
|
||||
import { PgDialect } from "drizzle-orm/pg-core";
|
||||
import type { SQL } from "drizzle-orm";
|
||||
import {
|
||||
@@ -392,6 +393,7 @@ import {
|
||||
createGovernedWaitEventObservation,
|
||||
createRemoteRunnerProcessLauncher,
|
||||
createRunnerdBackend,
|
||||
unidentifiedLocalRunnerProvenAbsent,
|
||||
executePaperclipNativeSession,
|
||||
detachNativeSessionsForRestart,
|
||||
NativeControllerDetachedForRestartError,
|
||||
@@ -13862,6 +13864,60 @@ describe("runnerd provider runtime wiring", () => {
|
||||
return state.createTransport.mock.calls[0]![0].stateDirectory!;
|
||||
}
|
||||
|
||||
it.each(["absent", "unreadable", "live_runner"] as const)("restarts a local bootstrap whose runner state is %s only when no state was written", async mode => {
|
||||
// RC-42: the server was killed after creating the root and before the
|
||||
// runner's first state write. Nothing connected, so no provider started.
|
||||
const current = scopedNativeExecution(`bootstrap-state-${mode}`);
|
||||
const root = await initializeScopedRunnerRoot(current, "runner");
|
||||
await mkdir(join(root, "diagnostics"), { recursive: true });
|
||||
if (mode === "unreadable") {
|
||||
await mkdir(join(root, "control-plane"), { recursive: true });
|
||||
await writeFile(join(root, "control-plane/control-plane-state.json"), "{");
|
||||
}
|
||||
const quarantine = join(dirname(root), "quarantine");
|
||||
const before = existsSync(quarantine) ? (await readdir(quarantine)).length : 0;
|
||||
// A runner that started but has not connected yet has written no state.
|
||||
const runner = mode === "live_runner"
|
||||
? spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", current.binding.runId], { stdio: "ignore" })
|
||||
: undefined;
|
||||
if (runner) await once(runner, "spawn");
|
||||
try {
|
||||
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: "runner",
|
||||
restartRecovery: { kind: "bootstrap_incomplete" as const, runId: current.binding.runId, leaseOwner: "controller",
|
||||
controllerGeneration: 2, providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null } });
|
||||
if (mode === "absent") {
|
||||
await expect(backend).resolves.toBeDefined();
|
||||
expect((await readdir(quarantine)).length).toBe(before + 1);
|
||||
} else {
|
||||
// Written but unreadable state, or a live unconnected runner, keeps failing closed.
|
||||
await expect(backend).rejects.toThrow("durable_identity_unreadable");
|
||||
}
|
||||
} finally {
|
||||
if (runner) { const exited = once(runner, "exit"); runner.kill("SIGKILL"); await exited; }
|
||||
}
|
||||
});
|
||||
|
||||
it("proves an unidentified local runner absent only without a runner process or saved state", async () => {
|
||||
const current = scopedNativeExecution("unidentified-runner");
|
||||
const runId = current.binding.runId;
|
||||
const root = await initializeScopedRunnerRoot(current, "runner");
|
||||
expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(true);
|
||||
// Another run's binding is not this run's proof.
|
||||
expect(await unidentifiedLocalRunnerProvenAbsent({ runId: "another-run", execution: current })).toBe(false);
|
||||
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", runId], { stdio: "ignore" });
|
||||
try {
|
||||
await vi.waitFor(async () => expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(false));
|
||||
} finally {
|
||||
const exited = once(child, "exit"); child.kill("SIGKILL"); await exited;
|
||||
}
|
||||
expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(true);
|
||||
await mkdir(join(root, "control-plane"), { recursive: true });
|
||||
await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify(durableControlPlaneState({
|
||||
runId, normalizedSessionId: current.session.normalizedSessionId!, runnerInstanceId: "runner",
|
||||
environmentLeaseId: "lease", turnId: "turn", itemId: "item" })));
|
||||
expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps physical task cleanup binding after the startup context finishes", async () => {
|
||||
const current = scopedNativeExecution("physical-task-cleanup");
|
||||
const context = new NativeExecutionContext(current, 1_000);
|
||||
|
||||
@@ -115,7 +115,7 @@ import {
|
||||
EXECUTION_CONTROL_DEADLINE_MS,
|
||||
} from "../execution-control-deadline.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { execFile, execFileSync } from "node:child_process";
|
||||
import {
|
||||
chmodSync,
|
||||
copyFileSync,
|
||||
@@ -137,7 +137,7 @@ import {
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir, userInfo } from "node:os";
|
||||
import { hostname, tmpdir, userInfo } from "node:os";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, loadNativeRunnerCheckpointHeader, loadNativeUnclaimedBootstrapCheckpoint, nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js";
|
||||
@@ -2119,6 +2119,54 @@ function runnerdStateBase(): string {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* A runner launch requested without a recorded process identity leaves no PID
|
||||
* to verify. On this host it is proven absent when no process carries the
|
||||
* run's runner arguments and no saved runner state belongs to the run: a
|
||||
* provider starts only after its runner connects and writes that state.
|
||||
*/
|
||||
export async function unidentifiedLocalRunnerProvenAbsent(input: { runId: string; execution: unknown }): Promise<boolean> {
|
||||
let execution: NativeExecutionInput;
|
||||
try {
|
||||
execution = parseNativeExecutionInput(input.execution);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (execution.binding.runId !== input.runId) return false;
|
||||
if (!(await noLocalRunnerProcessForRun(input.runId))) return false;
|
||||
const root = scopedRunnerdStateRoot(execution);
|
||||
const candidates = [root];
|
||||
const quarantine = resolve(runnerdStateBase(), "quarantine");
|
||||
try {
|
||||
if (existsSync(quarantine)) {
|
||||
for (const name of readdirSync(quarantine)) {
|
||||
if (name.startsWith(`${basename(root)}.`)) candidates.push(resolve(quarantine, name));
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
for (const candidate of candidates) {
|
||||
if (!existsSync(resolve(candidate, "control-plane", "control-plane-state.json"))) continue;
|
||||
try {
|
||||
if (record(readRunnerdControlPlaneState(candidate).identity).runId === input.runId) return false;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/** `paperclip-runnerd` is always launched with `--run-id <runId>`. A failed or
|
||||
* slow process listing proves nothing, so it keeps the caller blocked. */
|
||||
async function noLocalRunnerProcessForRun(runId: string): Promise<boolean> {
|
||||
const listing = await new Promise<string | null>(resolve => {
|
||||
execFile("ps", ["-axww", "-o", "command="], { encoding: "utf8", timeout: 5_000, maxBuffer: 64 * 1024 * 1024 },
|
||||
(error, stdout) => resolve(error ? null : stdout));
|
||||
});
|
||||
return listing !== null && !listing.split("\n").some(line => line.includes(`--run-id ${runId}`));
|
||||
}
|
||||
|
||||
export function scopedRunnerdStateRoot(execution: NativeExecutionInput): string {
|
||||
// Antigravity's first release owns a fresh per-turn daemon. Its native
|
||||
// conversation is stored separately and may load under each new run's grant.
|
||||
@@ -5563,6 +5611,17 @@ async function migrateRunnerdStateRootForExecution(input: {
|
||||
}
|
||||
await assertCleanupActivationCommitted(input.db, scoped, input.execution);
|
||||
const identity = readRunnerdDurableIdentity(scoped);
|
||||
if (!identity && input.restartRecovery?.kind === "bootstrap_incomplete" &&
|
||||
(!input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local") &&
|
||||
!existsSync(resolve(scoped, "control-plane", "control-plane-state.json")) &&
|
||||
await noLocalRunnerProcessForRun(input.execution.binding.runId)) {
|
||||
// The server was killed between creating the root and the local runner's
|
||||
// first state write. No runner for the run is alive on this host and none
|
||||
// connected, so no provider started and no command was acknowledged.
|
||||
// Keep the partial root for inspection and start the bootstrap again.
|
||||
quarantineRunnerdStateRoot(scoped, "identity_indeterminate");
|
||||
return;
|
||||
}
|
||||
if (!identity) {
|
||||
if (input.restartRecovery?.kind !== "reattach_existing_runner" &&
|
||||
input.restartRecovery?.kind !== "reattach_remote_runner") {
|
||||
@@ -10415,6 +10474,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
stream: "system",
|
||||
level: "info",
|
||||
message: "Native execution requested; prior local stop evidence no longer applies.",
|
||||
// Binds a later identity-less stop proof to the host that launched.
|
||||
payload: { hostname: hostname() },
|
||||
});
|
||||
},
|
||||
input: runnerExecution,
|
||||
|
||||
Reference in new issue
Block a user