fix(native): recover a run killed between runner launch and its identity record (#19694)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a local `paperclip-runnerd` process. The server
records the request to start it, then records its process identity after
it spawns.
> - If the server is killed between those two records, there is no
process ID to check after the restart
> - Restart recovery then restarted the bootstrap, but the executor
failed with `durable_identity_unreadable` because the runner never wrote
its state
> - The user's Retry then waited forever with "Retry is waiting for
Paperclip to verify that the previous execution stopped", because the
stop check needs a process ID
> - This pull request restarts that bootstrap and proves on the host
that no runner for the run exists
> - The benefit is that a restart in the runner launch window no longer
loses the task or leaves Retry permanently stuck

## Linked Issues or Issue Description

**What happened?**
In restart-chaos testing, an ACPX Claude run was created and the server
was SIGKILLed 1.3 s later, right after `native.process_start_requested`
and before the runner identity was recorded. On restart, recovery
claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`).
The executor then threw `runner_state_identity_mismatch:
durable_identity_unreadable`: the state root existed, but
`control-plane/control-plane-state.json` had never been written. The run
failed with `native_continuation_requires_reconciliation`. A Retry was
accepted (HTTP 200), but no run was ever created. The issue stayed
blocked with `retryWait.reason=process_identity_missing` for over 20
minutes and through another restart.

**Expected behavior**
When the runner never wrote state, no provider started, so the bootstrap
can start again. A Retry should be admitted once the host proves that no
runner for the run exists.

**Paperclip version or commit**
`master` at `fb3cf1a072`.

**Steps to reproduce**
Create a native local task and SIGKILL the server about 1 s after the
run starts, after the runner launch is requested and before its identity
is recorded. Restart the server, then click Retry.

## What Changed

- `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`):
for a `bootstrap_incomplete` restart on a local target, a root is
quarantined (kept for inspection) and the bootstrap starts again only
when both of these hold:
  - the root has no `control-plane-state.json`;
  - no live runner for the run exists on this host.

A state file that exists but cannot be read, or a runner that started
but has not connected yet, still fails closed with
`durable_identity_unreadable`.
- `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`)
proves that a runner is absent when both of these hold:
- no host process carries the run's `--run-id <runId>` argument
(`paperclip-runnerd` is always started with it, detached and
unreferenced);
- no saved control-plane state for this run exists, in the canonical
root or in quarantine.

A provider starts only after its runner connects and writes that state.
- `native.process_start_requested` now records the launching `hostname`.
- `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`)
applies the proof only when all of these hold:
  - the run is native;
  - the run has no process ID;
- its latest launch event is `native.process_start_requested`, from this
host;
  - all of its leases are local.

  A launch recorded without a host stays blocked.
- The process listing runs asynchronously with a 5 s deadline. A failed
or slow listing proves nothing.

It then writes `native.local_process_stopped` with null process IDs and
`basis: "unidentified_runner_absent"`. A later launch request
invalidates it as usual.
- The continuation and Retry gate (`explicit-native-continuation.ts`)
uses this proof before returning `process_identity_missing`. A dry run
checks the proof but records nothing.
- `doc/run-log-events.md` documents the new receipt basis.

## Verification

- `native-session-executor.test.ts`:
- The bootstrap restart with absent state was red before the fix
(`durable_identity_unreadable`) and is green after it (root quarantined,
backend admitted).
- Unreadable state, and a live runner that has not connected, still
reject.
- Proof cases: absent → true; a live process with `--run-id <runId>` →
false; this run's saved state → false; another run ID → false.
- `explicit-native-continuation.test.ts` (embedded Postgres, real
admission):
- The `absent` and `dry_run` cases were red before the fix (blocked) and
are green after it (admitted). Only the real admission records the
receipt.
- The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch`
cases stay blocked.
- Broader local run after the review fixes:
`explicit-native-continuation`, `native-session-executor`,
`native-recovered-owner-followup`,
`native-runner-restart-recovery.integration` and `task-retry` gave 1836
passed, 0 failed.
- A chaos replay of the original task (`--replay
w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after
the runner identity was recorded (`resume_dead_runner`), not in the
launch window, so the tests above are what reproduce that window.
- `tsc --noEmit` for `server` passes.

## Risks

- The proof depends on `ps -axww` listing every process on the launching
host. If `ps` fails or times out, the proof fails closed (still
blocked). Remote runners and other hosts are excluded.
- A runner that was spawned but has not yet written its state could be
mid-start when admission runs. It would still carry `--run-id` in its
arguments, so the proof refuses.
- Only the bootstrap path with no state file changes. Ownership,
checkpoint and reattach checks are unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-06 20:06:36 -06:00
1 parent aca908a165
commit 30857db55e
6 files changed
+228 -5

No files matched your search

+9
View File
@@ -391,6 +391,15 @@ clears a local run's process identity, after it verifies that its PID and proces
group are absent. The payload contains only those process IDs. Remote process
IDs are never checked against the control-plane host.
A crash can land after `native.process_start_requested` and before the runner's
process identity is recorded. Continuation admission then writes
`native.local_process_stopped` with null process IDs and
`basis: "unidentified_runner_absent"`. It does this only on the host named in
the launch event's `hostname` payload, after it checks two things. First, no
process carries the run's `--run-id` runner argument. Second, no
saved runner control-plane state (canonical or quarantined) belongs to the run;
a provider starts only after its runner connects and writes that state.
The server writes `native.process_start_requested` before a backend can spawn,
and `native.process_identity_recorded` when it stores a new native process
identity. Either invalidates an earlier local stop receipt, including a crash
@@ -1,5 +1,6 @@
import { legacyControllerBootId } from "./legacy-controller-lease.js";
import { execFile } from "node:child_process";
import { execFile, spawn } from "node:child_process";
import { once } from "node:events";
import os from "node:os";
import { createSystemRecovery, failSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
import { instanceSettingsService } from "./instance-settings.js";
@@ -1120,6 +1121,63 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
} finally { verify.mockRestore(); }
});
it.each(["absent", "live_runner", "saved_state", "dry_run", "foreign_host", "unbound_launch"] as const)(
"admits a Retry after a launch whose runner identity was never recorded only when no runner exists (%s)", async kind => {
// RC-42: the server was killed after requesting the runner launch and
// before recording its process identity, so there is no PID to verify.
const f = await seed();
const stateBase = await mkdtemp(join(tmpdir(), "unidentified-runner-"));
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase;
let child: ReturnType<typeof spawn> | undefined;
try {
const nativeSessionId = randomUUID();
const execution = {
schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
binding: { companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.sourceRunId, executionWorkspaceId: "workspace" },
task: { identifier: "TEST", title: "Continue", description: null, prompt: "Continue", workMode: "standard" },
workspace: { cwd: stateBase, repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: nativeSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Continue" }] } },
interactionResponses: [], credentialBindings: [],
};
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, nativeSessionId: null, runnerInstanceId: null,
errorCode: "adapter_failed", runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, f.sourceRunId));
// Only the launching host can inspect its process table.
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId, eventType: PROCESS_START_REQUESTED,
...(kind === "unbound_launch" ? {} : { payload: { hostname: kind === "foreign_host" ? `other-${os.hostname()}` : os.hostname() } }) });
if (kind === "live_runner") {
child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", f.sourceRunId], { stdio: "ignore" });
await once(child, "spawn");
}
if (kind === "saved_state") {
const canonical = (value: unknown): string => value && typeof value === "object" && !Array.isArray(value)
? `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`).join(",")}}`
: JSON.stringify(value);
const root = join(stateBase, createHash("sha256").update(canonical({
schema: "paperclip.native-session-scope.v2", companyId: f.companyId, agentId: f.agentId,
workspace: { kind: "managed", executionWorkspaceId: "workspace" },
provider: { driverKind: "codex_app_server", identity: { kind: "codex" } }, normalizedSessionId: nativeSessionId,
})).digest("hex"));
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify({
schema: "paperclip.runner.durable.control-plane-state.v1", identity: { runId: f.sourceRunId } }));
}
const admitted = await admit(f, kind === "dry_run");
if (kind === "absent" || kind === "dry_run") expect(admitted).toMatchObject({ previousRunId: f.sourceRunId });
else expect(admitted).toBeNull();
// Only a real admission records the stop; a later launch invalidates it.
expect(await hasNativeLocalProcessStop(db, f.companyId, f.sourceRunId)).toBe(kind === "absent");
} finally {
if (child) { const exited = once(child, "exit"); child.kill("SIGKILL"); await exited; }
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
await rm(stateBase, { recursive: true, force: true });
}
},
);
it.each(["suspended", "ready", "wrong_run", "wrong_thread", "active_provider", "pending_tool", "pending_output", "missing_state", "new_launch"])(
"recovers a historical run without process metadata only from exact suspended state (%s)", async kind => {
const f = await seed();
@@ -16,7 +16,7 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation.js";
import { readQueuedInteractionResponse } from "./queued-interaction-response.js";
import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js";
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "./native-local-process-stop.js";
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped, recordUnidentifiedLocalRunnerStop } from "./native-local-process-stop.js";
import { completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
import { z } from "zod";
@@ -485,7 +485,9 @@ export async function admitExplicitNativeContinuation(input: {
// A missing process identity is not evidence that a provider exited.
if (!run.processPid && !run.processGroupId &&
!await hasNativeLocalProcessStop(db, companyId, run.id) &&
!await hasHistoricalSuspendedNativeSession(db, run)) return blocked("process_identity_missing", "The previous run has no verified stop record. Paperclip cannot start this message yet.");
!await hasHistoricalSuspendedNativeSession(db, run) &&
!(run.runtimeMode === "native" &&
await recordUnidentifiedLocalRunnerStop(db, run, { record: !input.dryRun }))) return blocked("process_identity_missing", "The previous run has no verified stop record. Paperclip cannot start this message yet.");
if (run.processPid && !processStopped(run.processPid)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
if (run.processGroupId && !processStopped(-run.processGroupId)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
}
@@ -97,6 +97,43 @@ export async function hasNativeLocalProcessStop(db: Db, companyId: string, runId
return event?.eventType === LOCAL_PROCESS_STOPPED;
}
/**
* A local runner launch can be requested and the server killed before the
* runner's process identity is recorded. With no PID to check, prove on this
* host that no runner for the run exists and that it never wrote runner state
* (no provider can start before that), then record the stop like a verified
* PID. A later launch request invalidates the receipt as usual.
*/
export async function recordUnidentifiedLocalRunnerStop(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "runtimeMode" | "processPid" | "processGroupId" | "runnerProfileJson">,
options: { record: boolean } = { record: true }): Promise<boolean> {
if (run.runtimeMode !== "native" || run.processPid || run.processGroupId) return false;
const [latest] = await db.select({ eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
isNull(heartbeatRunEvents.sourceEventId),
inArray(heartbeatRunEvents.eventType, [PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, LOCAL_PROCESS_STOPPED]),
)).orderBy(desc(heartbeatRunEvents.seq)).limit(1);
// Only the host that requested the launch can inspect its process table.
if (latest?.eventType !== PROCESS_START_REQUESTED || latest.payload?.hostname !== os.hostname()) return false;
const leases = await db.select({ provider: environmentLeases.provider }).from(environmentLeases)
.where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id)));
if (leases.some(lease => lease.provider !== "local")) return false;
const { unidentifiedLocalRunnerProvenAbsent } = await import("./native-runtime/native-session-executor.js");
if (!await unidentifiedLocalRunnerProvenAbsent({ runId: run.id, execution: run.runnerProfileJson?.nativeExecutionInput })) return false;
if (!options.record) return true;
await appendHeartbeatRunEvent(db, {
companyId: run.companyId,
runId: run.id,
agentId: run.agentId,
eventType: LOCAL_PROCESS_STOPPED,
stream: "system",
level: "info",
message: "Verified that no runner for this run exists on this host and none saved runner state.",
payload: { processPid: null, processGroupId: null, processStartedAt: null, basis: "unidentified_runner_absent" },
});
return true;
}
/** Pre-receipt native runs can retain an exact suspended session after their
* mutable process fields were cleared. This is admission evidence for a new
* user turn only, never permission to replay the old run or infer its outcomes.
@@ -17,6 +17,7 @@ import { configuredEnvironment, configuredEnvironmentProjection } from "../../ve
import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
import { readFileSync } from "node:fs";
import { once } from "node:events";
import { PgDialect } from "drizzle-orm/pg-core";
import type { SQL } from "drizzle-orm";
import {
@@ -392,6 +393,7 @@ import {
createGovernedWaitEventObservation,
createRemoteRunnerProcessLauncher,
createRunnerdBackend,
unidentifiedLocalRunnerProvenAbsent,
executePaperclipNativeSession,
detachNativeSessionsForRestart,
NativeControllerDetachedForRestartError,
@@ -13862,6 +13864,60 @@ describe("runnerd provider runtime wiring", () => {
return state.createTransport.mock.calls[0]![0].stateDirectory!;
}
it.each(["absent", "unreadable", "live_runner"] as const)("restarts a local bootstrap whose runner state is %s only when no state was written", async mode => {
// RC-42: the server was killed after creating the root and before the
// runner's first state write. Nothing connected, so no provider started.
const current = scopedNativeExecution(`bootstrap-state-${mode}`);
const root = await initializeScopedRunnerRoot(current, "runner");
await mkdir(join(root, "diagnostics"), { recursive: true });
if (mode === "unreadable") {
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane/control-plane-state.json"), "{");
}
const quarantine = join(dirname(root), "quarantine");
const before = existsSync(quarantine) ? (await readdir(quarantine)).length : 0;
// A runner that started but has not connected yet has written no state.
const runner = mode === "live_runner"
? spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", current.binding.runId], { stdio: "ignore" })
: undefined;
if (runner) await once(runner, "spawn");
try {
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: "runner",
restartRecovery: { kind: "bootstrap_incomplete" as const, runId: current.binding.runId, leaseOwner: "controller",
controllerGeneration: 2, providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null } });
if (mode === "absent") {
await expect(backend).resolves.toBeDefined();
expect((await readdir(quarantine)).length).toBe(before + 1);
} else {
// Written but unreadable state, or a live unconnected runner, keeps failing closed.
await expect(backend).rejects.toThrow("durable_identity_unreadable");
}
} finally {
if (runner) { const exited = once(runner, "exit"); runner.kill("SIGKILL"); await exited; }
}
});
it("proves an unidentified local runner absent only without a runner process or saved state", async () => {
const current = scopedNativeExecution("unidentified-runner");
const runId = current.binding.runId;
const root = await initializeScopedRunnerRoot(current, "runner");
expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(true);
// Another run's binding is not this run's proof.
expect(await unidentifiedLocalRunnerProvenAbsent({ runId: "another-run", execution: current })).toBe(false);
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", runId], { stdio: "ignore" });
try {
await vi.waitFor(async () => expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(false));
} finally {
const exited = once(child, "exit"); child.kill("SIGKILL"); await exited;
}
expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(true);
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify(durableControlPlaneState({
runId, normalizedSessionId: current.session.normalizedSessionId!, runnerInstanceId: "runner",
environmentLeaseId: "lease", turnId: "turn", itemId: "item" })));
expect(await unidentifiedLocalRunnerProvenAbsent({ runId, execution: current })).toBe(false);
});
it("keeps physical task cleanup binding after the startup context finishes", async () => {
const current = scopedNativeExecution("physical-task-cleanup");
const context = new NativeExecutionContext(current, 1_000);
@@ -115,7 +115,7 @@ import {
EXECUTION_CONTROL_DEADLINE_MS,
} from "../execution-control-deadline.js";
import { createHash, randomUUID } from "node:crypto";
import { execFileSync } from "node:child_process";
import { execFile, execFileSync } from "node:child_process";
import {
chmodSync,
copyFileSync,
@@ -137,7 +137,7 @@ import {
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir, userInfo } from "node:os";
import { hostname, tmpdir, userInfo } from "node:os";
import { DatabaseSync } from "node:sqlite";
import { NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, loadNativeRunnerCheckpointHeader, loadNativeUnclaimedBootstrapCheckpoint, nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js";
@@ -2119,6 +2119,54 @@ function runnerdStateBase(): string {
);
}
/**
* A runner launch requested without a recorded process identity leaves no PID
* to verify. On this host it is proven absent when no process carries the
* run's runner arguments and no saved runner state belongs to the run: a
* provider starts only after its runner connects and writes that state.
*/
export async function unidentifiedLocalRunnerProvenAbsent(input: { runId: string; execution: unknown }): Promise<boolean> {
let execution: NativeExecutionInput;
try {
execution = parseNativeExecutionInput(input.execution);
} catch {
return false;
}
if (execution.binding.runId !== input.runId) return false;
if (!(await noLocalRunnerProcessForRun(input.runId))) return false;
const root = scopedRunnerdStateRoot(execution);
const candidates = [root];
const quarantine = resolve(runnerdStateBase(), "quarantine");
try {
if (existsSync(quarantine)) {
for (const name of readdirSync(quarantine)) {
if (name.startsWith(`${basename(root)}.`)) candidates.push(resolve(quarantine, name));
}
}
} catch {
return false;
}
for (const candidate of candidates) {
if (!existsSync(resolve(candidate, "control-plane", "control-plane-state.json"))) continue;
try {
if (record(readRunnerdControlPlaneState(candidate).identity).runId === input.runId) return false;
} catch {
return false;
}
}
return true;
}
/** `paperclip-runnerd` is always launched with `--run-id <runId>`. A failed or
* slow process listing proves nothing, so it keeps the caller blocked. */
async function noLocalRunnerProcessForRun(runId: string): Promise<boolean> {
const listing = await new Promise<string | null>(resolve => {
execFile("ps", ["-axww", "-o", "command="], { encoding: "utf8", timeout: 5_000, maxBuffer: 64 * 1024 * 1024 },
(error, stdout) => resolve(error ? null : stdout));
});
return listing !== null && !listing.split("\n").some(line => line.includes(`--run-id ${runId}`));
}
export function scopedRunnerdStateRoot(execution: NativeExecutionInput): string {
// Antigravity's first release owns a fresh per-turn daemon. Its native
// conversation is stored separately and may load under each new run's grant.
@@ -5563,6 +5611,17 @@ async function migrateRunnerdStateRootForExecution(input: {
}
await assertCleanupActivationCommitted(input.db, scoped, input.execution);
const identity = readRunnerdDurableIdentity(scoped);
if (!identity && input.restartRecovery?.kind === "bootstrap_incomplete" &&
(!input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local") &&
!existsSync(resolve(scoped, "control-plane", "control-plane-state.json")) &&
await noLocalRunnerProcessForRun(input.execution.binding.runId)) {
// The server was killed between creating the root and the local runner's
// first state write. No runner for the run is alive on this host and none
// connected, so no provider started and no command was acknowledged.
// Keep the partial root for inspection and start the bootstrap again.
quarantineRunnerdStateRoot(scoped, "identity_indeterminate");
return;
}
if (!identity) {
if (input.restartRecovery?.kind !== "reattach_existing_runner" &&
input.restartRecovery?.kind !== "reattach_remote_runner") {
@@ -10415,6 +10474,8 @@ async function executePaperclipNativeSessionWithinScope(
stream: "system",
level: "info",
message: "Native execution requested; prior local stop evidence no longer applies.",
// Binds a later identity-less stop proof to the host that launched.
payload: { hostname: hostname() },
});
},
input: runnerExecution,