fix(recovery): start a Retry after a restart cut off a relaunch whose earlier Runners were proven stopped (#20537)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails, a person presses Retry and expects a new run to
start
> - Before a native Retry starts, Paperclip must prove that the previous
run's Runner and provider stopped, so two providers never work on the
same task
> - When the server is killed after it asked for a Runner launch but
before it recorded the Runner's process, there is no PID to check.
Paperclip then proves the stop from two facts: no Runner process for the
run exists on this host, and the run never saved Runner state
> - A run that already crossed restarts did save Runner state, from its
earlier Runners. Those Runners were proven stopped, with stop receipts.
The proof still refused, because it only asked whether any state
belonged to the run
> - The Retry returned 202, but the task then waited for a board
operator to confirm the stop (`confirm_run_stopped`,
`process_identity_missing`), until the task timed out
> - This pull request lets the proof accept Runner state that only the
earlier, proven-stopped Runners could have written
> - The benefit is that a Retry after a restart-interrupted relaunch
starts right away, with no operator step

## Linked Issues or Issue Description

**Subsystem affected**
Server native-run stop proof for a Retry or a new message
(`server/src/services/native-local-process-stop.ts`,
`unidentifiedLocalRunnerProvenAbsent` in
`server/src/services/native-runtime/native-session-executor.ts`).

**Problem or motivation**
Restart-chaos wave `w-20261110T111245-s701`, task t03 (local, codex, C1,
K1/K2, `--storage s3 --storage-faults ST7`): run `f1ffeed3` crossed
three restarts. Its event history is:

1. Launch request, process identity, stop receipt (first Runner, killed
by K1)
2. Launch request, process identity, stop receipt (second Runner, killed
by K1)
3. Launch request at 11:13:54.999, then a K1 kill before the Runner's
identity was recorded
4. A fourth controller resumed the run, a K2 drain interrupted its
preparation, and the run failed (`adapter_failed`, fixed separately by
#20514)

The person's Retry at 11:14:37 returned 202. Admission reached
`recordUnidentifiedLocalRunnerStop`. No Runner process for the run
existed, but the Runner state root held `control-plane-state.json` with
this run's id, written by Runners 1 and 2. The proof refused, the issue
was saved as `waiting_for_user` / `confirm_run_stopped` /
`process_identity_missing`, and the wake stayed deferred until the task
timed out about 15 minutes later. The harness graded
`R3:retry_no_new_run`.

**Proposed solution**
A Runner persists its command journal (`runner/runner-state.json`)
before any command takes effect, and a provider starts only from a
command. Its `runner/process-lineage/` writer marker is written before
any supervised process starts. So a launch whose Runner never wrote
either file after the launch request cannot own a provider.

- The launch request event now records `requestedAt`, taken on the
launching host just before any backend may spawn.
- `recordUnidentifiedLocalRunnerStop` passes that time on only when the
latest process event before the request is a stop receipt (every earlier
Runner of the run was proven stopped) or there is none.
- `unidentifiedLocalRunnerProvenAbsent` then accepts saved state of this
run only when `runner/runner-state.json` (if present) and every
`runner/process-lineage/` entry were last modified before `requestedAt`.
A file time that falls on a whole second (a coarse-timestamp filesystem)
must predate the request by 2 s. The existing checks stay: no `--run-id`
Runner process on this host, the launching host matches, and no remote
lease.

**Previous-release story**
A launch request saved by the previous release has no `requestedAt`. It
keeps today's proof: it is accepted only without saved Runner state of
the run, otherwise the board shows the operator confirmation. Nothing is
migrated or rewritten.

**Alternatives considered**
- Treating a proven-gone controller as proof that its Runner stopped.
Local Runners outlive a controller restart (graceful drains reattach
them), so a gone controller says nothing about its Runner or provider.
- Running `paperclip-runnerd reap-lineage` for each writer marker. It
needs a Runner PID and binary on the hot Retry path, and a Runner whose
journal write failed has no marker to find.

## What Changed

- `native-session-executor.ts`: the `native.process_start_requested`
payload adds `requestedAt`. `unidentifiedLocalRunnerProvenAbsent` takes
`earlierRunnersStoppedBefore` and accepts this run's saved state only
when the Runner journal and lineage entries predate it
(`runnerStateUnchangedSince`).
- `native-local-process-stop.ts`: `recordUnidentifiedLocalRunnerStop`
reads the last two process events and passes the request time only after
a stop receipt (or no earlier launch).
- `explicit-native-continuation.test.ts`: new embedded-Postgres case. A
native run with a proven-stopped Runner, saved Runner state, and a later
unidentified relaunch: the Retry is admitted and the stop is recorded.
It stays blocked (`process_identity_missing`) when the Runner journal or
a lineage entry changed after the request, when the request has no
`requestedAt` (previous release), when the earlier Runner has no stop
receipt, when a whole-second file time is within 2 s of the request, or
when a `--run-id` Runner process is alive. A whole-second time 3.5 s
before the request is admitted.

## Verification

- Red/green: `cd server && npx vitest run
src/services/explicit-native-continuation.test.ts -t "proven stopped"`.
On master the `proven` case fails (admission returns null); with the fix
all 8 cases pass.
- Related suites: `explicit-native-continuation.test.ts`,
`issue-confirm-run-stopped-routes.test.ts`, and the unidentified-Runner
and stop cases of `native-session-executor.test.ts`.
- Live, the s701 seed with the original wave's flags (`--env local
--adapters codex,claude --workloads A1,C1 --faults K1,K2 --triggers
time,result --concurrency 4 --tasks 4 --seed 701 --storage s3
--storage-faults ST7 --keep`):
- `w-20261110T131243-s701` (final head) and `w-20261110T125259-s701`,
master plus this fix plus #20514 merged locally: 4/4 pass each, every
task crossed a restart, no R3 signature, 0 speed breaches.
- `w-20261110T125616-s701`, master plus this fix without #20514: t01 and
t02 pass. t03 and t04 hit the drain-during-preparation failure that
#20514 fixes (`R3:retry_needed`). The Retry started a new run each time
and both tasks finished (`retry_no_new_run` is gone). New launch
requests carry `requestedAt`.
- The restart timings of these replays did not land in the exact gap
between a launch request and its process identity. The red/green test
covers that window.

## Risks

- Low. The change only widens one fail-closed proof, and only for launch
requests saved by this release, after a stop receipt, with Runner files
older than the request. The proof compares file times with a time taken
on the same host. A backwards wall-clock step between the request and a
Runner's first write could hide that write. The `--run-id` process check
still has to pass, so that Runner must also be gone.
- The Retry path adds one `lstat` per file in a small directory. It adds
no wait and no work before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/explicit-native-continuation.test.ts

@@ -10,10 +10,10 @@ import { issueTreeControlService } from "./issue-tree-control.js";
 import { createReleaseIssueExecution } from "../modules/wake-queue/application/use-cases.js";
 import * as nativeExecutor from "./native-runtime/native-session-executor.js";
 import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
-import { recordNativeLocalProcessStop, hasNativeLocalProcessStop, PRIOR_OWNER_FENCE_REFUSED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
+import { recordNativeLocalProcessStop, hasNativeLocalProcessStop, PRIOR_OWNER_FENCE_REFUSED, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
 import { remoteTerminationReceipt } from "./remote-execution-termination.js";
 import { createHash, randomBytes, randomUUID } from "node:crypto";
-import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";
+import { mkdtemp, mkdir, writeFile, rm, utimes } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { and, eq, sql } from "drizzle-orm";
@@ -1273,6 +1273,87 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
     },
   );
 
+  it.each(["proven", "coarse_before_request", "coarse_same_second", "state_after_request", "lineage_after_request", "previous_release",
+    "earlier_runner_unproven", "live_runner"] as const)(
+    "starts a Retry after restarts cut off a relaunch whose earlier runners were proven stopped (%s)", async kind => {
+      const f = await seed();
+      const stateBase = await mkdtemp(join(tmpdir(), "relaunched-runner-"));
+      const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
+      process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase;
+      let child: ReturnType<typeof spawn> | undefined;
+      try {
+        const nativeSessionId = randomUUID();
+        const execution = {
+          schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
+          binding: { companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.sourceRunId, executionWorkspaceId: "workspace" },
+          task: { identifier: "TEST", title: "Continue", description: null, prompt: "Continue", workMode: "standard" },
+          workspace: { cwd: stateBase, repoUrl: null, repoRef: null, branchName: null },
+          session: { normalizedSessionId: nativeSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
+          completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
+            contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Continue" }] } },
+          interactionResponses: [], credentialBindings: [],
+        };
+        await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, nativeSessionId: null, runnerInstanceId: null,
+          errorCode: "adapter_failed", runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, f.sourceRunId));
+        const event = (eventType: string, payload: Record<string, unknown>) => appendHeartbeatRunEvent(db, {
+          companyId: f.companyId, runId: f.sourceRunId, agentId: f.agentId, eventType, payload });
+        const host = os.hostname(), deadPid = 999999998;
+        await event(PROCESS_START_REQUESTED, { hostname: host, providerAttempt: 1 });
+        await event(PROCESS_IDENTITY_RECORDED, { hostname: host, targetKind: "local", processPid: deadPid, processGroupId: deadPid,
+          processStartedAt: "2026-11-10T11:13:34.000Z" });
+        if (kind !== "earlier_runner_unproven") await event("native.local_process_stopped",
+          { processPid: deadPid, processGroupId: deadPid, processStartedAt: "2026-11-10T11:13:34.000Z" });
+        const canonical = (value: unknown): string => value && typeof value === "object" && !Array.isArray(value)
+          ? `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`).join(",")}}`
+          : JSON.stringify(value);
+        const root = join(stateBase, createHash("sha256").update(canonical({
+          schema: "paperclip.native-session-scope.v2", companyId: f.companyId, agentId: f.agentId,
+          workspace: { kind: "managed", executionWorkspaceId: "workspace" },
+          provider: { driverKind: "codex_app_server", identity: { kind: "codex" } }, normalizedSessionId: nativeSessionId,
+        })).digest("hex"));
+        await mkdir(join(root, "control-plane"), { recursive: true });
+        await mkdir(join(root, "runner"), { recursive: true });
+        await mkdir(join(root, "runner/process-lineage"), { recursive: true });
+        await writeFile(join(root, "control-plane/control-plane-state.json"), JSON.stringify({
+          schema: "paperclip.runner.durable.control-plane-state.v1", identity: { runId: f.sourceRunId } }));
+        await writeFile(join(root, "runner/runner-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.state.v1" }));
+        await writeFile(join(root, `runner/process-lineage/writer-${deadPid}.json`), JSON.stringify({ schema: "paperclip.runner.process-lineage-writer.v1" }));
+        const requestedAt = new Date(Math.floor(Date.now() / 1000) * 1000 - 59_500);
+        const earlier = new Date(requestedAt.getTime() - 30_000), later = new Date(requestedAt.getTime() + 30_250);
+        const stateTimes: Partial<Record<typeof kind, Date>> = { state_after_request: later,
+          coarse_same_second: new Date(requestedAt.getTime() - 500), coarse_before_request: new Date(requestedAt.getTime() - 3_500) };
+        const stateTime = stateTimes[kind] ?? earlier;
+        await utimes(join(root, "runner/runner-state.json"), earlier, stateTime);
+        await utimes(join(root, `runner/process-lineage/writer-${deadPid}.json`), earlier, kind === "lineage_after_request" ? later : earlier);
+        await event(PROCESS_START_REQUESTED, { hostname: host, providerAttempt: 1,
+          ...(kind === "previous_release" ? {} : { requestedAt: requestedAt.toISOString() }) });
+        if (kind === "live_runner") {
+          child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", "--", "--run-id", f.sourceRunId], { stdio: "ignore" });
+          await once(child, "spawn");
+        }
+        const blocked: string[] = [];
+        const admitted = await db.transaction(async tx => {
+          await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
+          return admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, reason: "retry_failed_run",
+            failedRunId: f.sourceRunId, onBlocked: reason => blocked.push(reason) });
+        });
+        if (kind === "proven" || kind === "coarse_before_request") {
+          expect(admitted).toMatchObject({ previousRunId: f.sourceRunId, failedRunId: f.sourceRunId });
+          expect(blocked).toEqual([]);
+        } else {
+          expect(admitted).toBeNull();
+          expect(blocked).toEqual(["process_identity_missing"]);
+        }
+        expect(await hasNativeLocalProcessStop(db, f.companyId, f.sourceRunId)).toBe(kind === "proven" || kind === "coarse_before_request");
+      } finally {
+        if (child) { const exited = once(child, "exit"); child.kill("SIGKILL"); await exited; }
+        if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
+        else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
+        await rm(stateBase, { recursive: true, force: true });
+      }
+    },
+  );
+
   it.each(["suspended", "ready", "wrong_run", "wrong_thread", "active_provider", "pending_tool", "pending_output", "missing_state", "new_launch"])(
     "recovers a historical run without process metadata only from exact suspended state (%s)", async kind => {
       const f = await seed();

server/src/services/native-local-process-stop.ts

@@ -112,26 +112,30 @@ export async function hasNativeLocalProcessStop(db: Db, companyId: string, runId
 /**
  * A local runner launch can be requested and the server killed before the
  * runner's process identity is recorded. With no PID to check, prove on this
- * host that no runner for the run exists and that it never wrote runner state
- * (no provider can start before that), then record the stop like a verified
- * PID. A later launch request invalidates the receipt as usual.
+ * host that no runner for the run exists and that this launch wrote no runner
+ * state (no provider can start before that), then record the stop like a
+ * verified PID. A later launch request invalidates the receipt as usual.
  */
 export async function recordUnidentifiedLocalRunnerStop(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
   "id" | "companyId" | "agentId" | "runtimeMode" | "processPid" | "processGroupId" | "runnerProfileJson">,
   options: { record: boolean } = { record: true }): Promise<boolean> {
   if (run.runtimeMode !== "native" || run.processPid || run.processGroupId) return false;
-  const [latest] = await db.select({ eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
+  const [latest, previous] = await db.select({ eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
     eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
     isNull(heartbeatRunEvents.sourceEventId),
     inArray(heartbeatRunEvents.eventType, [PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, LOCAL_PROCESS_STOPPED]),
-  )).orderBy(desc(heartbeatRunEvents.seq)).limit(1);
+  )).orderBy(desc(heartbeatRunEvents.seq)).limit(2);
   // Only the host that requested the launch can inspect its process table.
   if (latest?.eventType !== PROCESS_START_REQUESTED || latest.payload?.hostname !== os.hostname()) return false;
   const leases = await db.select({ provider: environmentLeases.provider }).from(environmentLeases)
     .where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id)));
   if (leases.some(lease => lease.provider !== "local")) return false;
+  const requestedAt = typeof latest.payload?.requestedAt === "string" ? Date.parse(latest.payload.requestedAt) : Number.NaN;
+  const launchRequestedAt = (!previous || previous.eventType === LOCAL_PROCESS_STOPPED) && Number.isFinite(requestedAt)
+    ? requestedAt : null;
   const { unidentifiedLocalRunnerProvenAbsent } = await import("./native-runtime/native-session-executor.js");
-  if (!await unidentifiedLocalRunnerProvenAbsent({ runId: run.id, execution: run.runnerProfileJson?.nativeExecutionInput })) return false;
+  if (!await unidentifiedLocalRunnerProvenAbsent({ runId: run.id, execution: run.runnerProfileJson?.nativeExecutionInput,
+    launchRequestedAt })) return false;
   if (!options.record) return true;
   await appendHeartbeatRunEvent(db, {
     companyId: run.companyId,
@@ -140,7 +144,7 @@ export async function recordUnidentifiedLocalRunnerStop(db: Db, run: Pick<typeof
     eventType: LOCAL_PROCESS_STOPPED,
     stream: "system",
     level: "info",
-    message: "Verified that no runner for this run exists on this host and none saved runner state.",
+    message: "Verified that no runner for this run exists on this host and none saved runner state after the launch request.",
     payload: { processPid: null, processGroupId: null, processStartedAt: null, basis: "unidentified_runner_absent" },
   });
   return true;

server/src/services/native-runtime/native-session-executor.ts

@@ -2192,10 +2192,15 @@ function runnerdStateBase(): string {
 /**
  * A runner launch requested without a recorded process identity leaves no PID
  * to verify. On this host it is proven absent when no process carries the
- * run's runner arguments and no saved runner state belongs to the run: a
- * provider starts only after its runner connects and writes that state.
+ * run's runner arguments and no saved runner state belongs to the run, or,
+ * when the caller passes `launchRequestedAt` because every earlier runner of
+ * the run was proven stopped, when that state's command journal and lineage
+ * files all predate the launch request: a provider starts only from a command
+ * the runner journals first.
  */
-export async function unidentifiedLocalRunnerProvenAbsent(input: { runId: string; execution: unknown }): Promise<boolean> {
+export async function unidentifiedLocalRunnerProvenAbsent(input: {
+  runId: string; execution: unknown; launchRequestedAt?: number | null;
+}): Promise<boolean> {
   let execution: NativeExecutionInput;
   try {
     execution = parseNativeExecutionInput(input.execution);
@@ -2219,7 +2224,8 @@ export async function unidentifiedLocalRunnerProvenAbsent(input: { runId: string
   for (const candidate of candidates) {
     if (!existsSync(resolve(candidate, "control-plane", "control-plane-state.json"))) continue;
     try {
-      if (record(readRunnerdControlPlaneState(candidate).identity).runId === input.runId) return false;
+      if (record(readRunnerdControlPlaneState(candidate).identity).runId === input.runId &&
+          !runnerStateUnchangedSince(candidate, input.launchRequestedAt)) return false;
     } catch {
       return false;
     }
@@ -2227,6 +2233,22 @@ export async function unidentifiedLocalRunnerProvenAbsent(input: { runId: string
   return true;
 }
 
+const COARSE_FILE_TIMESTAMP_MARGIN_MS = 2_000;
+
+function runnerStateUnchangedSince(root: string, requestedAt: number | null | undefined): boolean {
+  if (typeof requestedAt !== "number" || !Number.isFinite(requestedAt)) return false;
+  const stateFile = resolve(root, "runner", "runner-state.json");
+  const lineage = resolve(root, "runner", "process-lineage");
+  const paths = [stateFile, ...existsSync(lineage) ? readdirSync(lineage).map(name => resolve(lineage, name)) : []];
+  return paths.every(path => {
+    const stats = lstatSync(path, { throwIfNoEntry: false });
+    if (path === stateFile && !stats) return true;
+    if (!stats?.isFile()) return false;
+    const coarseTimestamp = stats.mtimeMs % 1000 === 0;
+    return stats.mtimeMs + (coarseTimestamp ? COARSE_FILE_TIMESTAMP_MARGIN_MS : 0) < requestedAt;
+  });
+}
+
 /** `paperclip-runnerd` is always launched with `--run-id <runId>`. A failed or
  * slow process listing proves nothing, so it keeps the caller blocked. */
 export async function noLocalRunnerProcessForRun(runId: string): Promise<boolean> {
@@ -11243,7 +11265,7 @@ async function executePaperclipNativeSessionAttempt(
                 // Binds a later identity-less stop proof to the host that launched.
                 // The provider attempt opens that attempt's outcome window for
                 // the work-based budget (provider-attempt-outcome.ts).
-                payload: { hostname: hostname(), providerAttempt: attempt },
+                payload: { hostname: hostname(), providerAttempt: attempt, requestedAt: new Date().toISOString() },
               });
               crashPoint("native.process_start.after_requested_before_spawn");
             },