fix(native): a same-boot stall never leaves a run waiting on itself (#20721)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep an execution lease. Restart recovery takes over a
run only when it can prove that the recorded controller stopped.
> - A stall (SIGSTOP) longer than the lease, with no restart, makes the
executor lose its own lease. The executor then detaches for "the next
controller" and exits.
> - The next controller is this same process. Recovery asks the PID
probe if the recorded controller is alive. The recorded controller is
this process, so the answer is always "alive".
> - The task then waits forever: `in_progress`, no Retry, an active run,
and an active environment lease.
> - A second gap: after back-to-back stalls mid-tool, the executor
returns and nothing in this process ever looks at the run again. No
terminal state, no recovery and no Retry.
> - This pull request lets recovery recognise its own process. When this
process no longer drives the run and its own lease lapsed, the prior
owner is gone. The existing evidence checks then decide, as for any
stopped controller.
> - It also makes heartbeat schedule that recovery as soon as such an
executor exits.
> - The benefit is that a same-boot stall never needs a manual Retry.
When provider work may have started, the run fails with a working Retry
instead of a wait with no end.

## Linked Issues or Issue Description

Refs #20694 (the same class of fix for legacy `preparing` runs), #20529,
#20691.

**What happened?**
Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault
TM1). The server process was frozen with SIGSTOP for about 84 s while
run `89a3922b` prepared. After SIGCONT, `native.process_start_requested`
was recorded and `native.session.execute` failed after 118 ms: the lease
renewal found the lease lapsed, and the executor detached and exited.
From then on, recovery recorded `native.recovery.transition` "waiting
for safe ownership evidence" with reason `controller_still_alive` every
30 s for more than 20 minutes. The run stayed `running`, the issue
stayed `in_progress`, no Retry was shown, and the environment lease
stayed `active`.

Chaos wave `w-20261111T040439-s9331` (Daytona, codex/claude/opencode,
T1, TM1). Three stalls of about 90 s came back to back, with less than 1
s between them. Each run was in a tool call. After the second thaw,
`native.session.execute` failed for all three runs (lease lost, executor
detached). After that, no run had any event: no terminal state, no
`native.recovery.transition` and no Retry, for 20 minutes. The
in-sandbox turns kept going. A single 90 s stall (wave s9340) stays
green, because a later recovery signal happens to find the run. With
stacked stalls, that signal can land while the executor is still tearing
down, so it skips the run and sets no deadline.

**Expected behavior**
When this process no longer drives a run that it recorded as its own,
recovery takes the run back. No turn reached the runner here, so the
bootstrap restarts on the same run. When a turn or runner work may have
started, the run fails and shows a working Retry.

**Steps to reproduce**
1. Start a native remote (Daytona) run.
2. Send SIGSTOP to the server during preparation, and hold it longer
than the execution lease.
3. Send SIGCONT. The executor loses its lease, detaches and exits.
4. See that every recovery scan waits with `controller_still_alive`, and
the run never settles.

The new tests in `native-runner-restart-recovery.integration.test.ts`
("same-boot controller whose own lease lapsed") reproduce this at the
recovery layer.

## What Changed

- `claimNativeRestartRecoveries`: when the recorded controller is this
exact process (same boot ID and PID):
- If this process still drives the run (heartbeat execution or native
executor registered), recovery leaves the run to it
(`current_controller_preparing`).
- If neither drives it and the self lease lapsed, the takeover is
allowed (`current_controller_released`). The existing evidence checks
then choose the outcome: bootstrap again on the same run when a durable
record proves no command reached the runner, or fail with a Retry
(`blocked`) when runner or provider work may have started.
- An unexpired self lease still waits. A claim that has not reached its
executor holds it, and it lapses by itself.
- A run that this process handed to the next server (graceful drain
`restart_suspended` record, or the in-process hot-restart handoff
marker) stays with the next server.
- `heartbeat.ts`: when the executor ends with
`NativeControllerDetachedForRestartError`, there was no shutdown or
hot-restart handoff, and this process's own lease lapsed
(`ownNativeExecutionLeaseLapsed`), heartbeat schedules restart recovery
for the run when the executor exits. Recovery then reattaches the remote
runner whose turn is still running, restarts an unstarted bootstrap, or
fails the run with a working Retry.
- `claimNativeRestartRecoveries`: a self-owned lease that is still live,
with no driver here, now returns a `retryAt` at its expiry, so the wait
is bounded.
- `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`.
Heartbeat recovery passes it to the claim.
- `suspendedByCleanDrain` now uses the shared
`handedToNextControllerSinceClaim` helper. Behavior is the same.
- Heartbeat test: an executor detaches after its own lease lapsed, with
no restart. The same process must take the run back. Red before (timed
out, the run stayed `running`), green after.
- Tests: five cases for the same-boot lapsed lease: bootstrap again with
no runner command; fail with a working Retry when a runner identity was
recorded; leave the run to a live in-process executor; leave a
handed-off run to the next server; still wait for a different live
controller.

## Verification

- Red before the fix: the three same-boot cases returned
`awaiting_evidence` / `controller_still_alive`.
- Green after the fix:
- `pnpm --dir server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`
- Daytona wave `w-20261111T050504-s9331` from this branch (same seed and
shape as s9331: 3 tasks, concurrency 3, three 90 s TM1 stalls with about
1 s between them). All 3 tasks reached `done` with no Retry. Run
`632f0564` (claude) shows the exact s9331 sequence:
`native.session.execute` failed after 216 s, then
`reattach_remote_runner` 0.4 s later (before the third stall), then a
second reattach after it, and the turn completed. Two tasks are graded
FAIL only on the T1 crossing check: their own fault landed after their
tool loop ended (for example loop ended 05:11:26, fault 05:11:44). Their
status is `done`, with no Retry.
- `tsc --noEmit -p server` is clean.

## Risks

- A same-process recovery scan can now claim a run that this process
recorded as its own. The guard needs all of these: the self lease
lapsed, no in-process execution of the run, and no handoff to the next
server. A claim that is still on its way to its executor holds a live
lease, so it is not taken twice.
- Upgrade safety: the decision uses only the coordinator row and
in-process state. Old saved rows are judged by the same rules. A new
process never matches an old boot ID, so deploys keep the existing proof
path.
- No new waits or file operations run before `turn.start`.
- There is still no ceiling on `awaiting_evidence` for a genuinely
different live controller (case c). This PR does not change that policy.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use
(shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 23:55:10 -06:00
1 parent fcf4e39b41
commit 16024ffa2a
8 files changed
+202 -9

No files matched your search

@@ -384,6 +384,55 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
expect(reached).toHaveBeenCalled();
}, 60_000);
// Daytona waves s9330 and s9331 (TM1): stalls of the same server outlived
// the execution lease. Renewal found the lease lapsed and detached the
// executor for "the next controller" with no shutdown handoff, and the
// executor returned with the run still running. The next controller is this
// same process; nothing may leave the run running with no recovery.
it("recovers a run whose executor detached after its own lease lapsed, with no restart", async () => {
const f = await seedRecoverableRun("SELF-DETACHED-LAPSED");
let opened = 0;
const reachedAgain = vi.fn();
const backend = vi.fn((): NativeSessionBackend => {
opened += 1;
const first = opened === 1;
const open = async () => {
if (first) {
// What the remote renewal's loss does: the lease lapsed during the
// stall, and the executor is detached for the next controller.
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) })
.where(eq(nativeRunFinalizations.runId, f.runId));
void detachNativeSessionsForRestart([f.runId]);
throw new Error("runner stream lost while the controller was stopped");
}
reachedAgain();
throw new Error("stop at provider dispatch");
};
return {
async descriptor() {
return { kind: "mock", name: "self-detached", version: "1", capabilities: {
resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true },
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } };
},
openSession: open,
recoverSession: open,
} as unknown as NativeSessionBackend;
});
const heartbeat = heartbeatService(f.db, { nativeSessionBackendFactory: backend });
try {
const { claims } = await heartbeat.recoverNativeRunsAfterRestart(new Date(), [f.runId]);
expect(claims).toEqual([expect.objectContaining({ runId: f.runId })]);
await heartbeat.drainActiveRunExecutions();
// The same process takes the run back on its own, without a restart.
await waitForCondition("the same process to take the run back", () => reachedAgain.mock.calls.length > 0, 15_000);
} finally { await heartbeat.drainActiveRunExecutions(); }
expect(reachedAgain).toHaveBeenCalled();
const transitions = await f.db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId),
eq(heartbeatRunEvents.eventType, "native.recovery.transition")));
expect(transitions.map(event => (event.payload as Record<string, unknown>).reason))
.not.toContain("controller_still_alive");
}, 60_000);
async function shutdownDispositions(db: Awaited<ReturnType<typeof seedRecoverableRun>>["db"], runId: string) {
const events = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId),
@@ -832,7 +881,9 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
.where(eq(nativeRunFinalizations.runId, f.runId));
const onClaim = vi.fn();
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim }))
.toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_still_alive" }]);
.toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_still_alive",
// This process's own live lease lapses by itself; recovery looks again then.
retryAt: expect.any(Date) }]);
expect(onClaim).not.toHaveBeenCalled();
}, 60_000);
+9 -2
View File
@@ -55,7 +55,7 @@ import { prepareHeartbeatTaskContext } from "./heartbeat/task-context.js";
import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js";
import { reconcileMuseNativeTerminalRun } from "./native-runtime/muse-terminal-reconciliation.js";
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
import { nativeRecoveryClaimRelinquished, releaseLapsedNativeRestartClaim } from "./native-runtime/native-execution-lease.js";
import { nativeRecoveryClaimRelinquished, ownNativeExecutionLeaseLapsed, releaseLapsedNativeRestartClaim } from "./native-runtime/native-execution-lease.js";
import { executeHeartbeatRuntime, NativeSessionResumeScheduledError, NativeWorkspaceFinalizeScheduledError } from "./heartbeat/runtime-execution.js";
import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
import { nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
@@ -2041,6 +2041,11 @@ export function heartbeatService(
Parameters<typeof cleanupGitHubOperationLaunchers>[0] | null = null;
let nativeSessionResumeScheduled = false;
let restartClaimLapsed = false;
// The executor detached for "the next controller" because its own lease
// lapsed (for example after a stall), with no shutdown or hot-restart
// handoff. The next controller is this process: recover the run here once
// the executor has exited. A shutdown detach leaves it to the next server.
let detachedWithoutHandoff = false;
const shutdownHandedOff = watchShutdownHandoff(run.id);
let nativeOwnershipHeld = false;
let nativeStartupCleanupScheduled = false;
@@ -3707,6 +3712,8 @@ export function heartbeatService(
}
if (err instanceof NativeControllerDetachedForRestartError) {
nativeSessionResumeScheduled = true;
detachedWithoutHandoff = !shutdownHandedOff() && await ownNativeExecutionLeaseLapsed(db, run)
.catch(() => false);
return;
}
if (err instanceof NativeRunnerOwnershipUnverifiedError) {
@@ -4168,7 +4175,7 @@ export function heartbeatService(
} finally {
controllerLease.stop();
activeRunExecutions.delete(run.id);
if (restartClaimLapsed) scheduleNativeSessionResumeDispatch(run.id, new Date(), "restart");
if (restartClaimLapsed || detachedWithoutHandoff) scheduleNativeSessionResumeDispatch(run.id, new Date(), "restart");
// A failed owned Stop remains visible until this exact executor settles,
// including a graceful exit result arriving after the cancellation error.
// It is never retained beyond the active execution's cleanup.
+2 -1
View File
@@ -1,5 +1,5 @@
import { settleOwnableKind } from "../recovery/ownership-invariant-report.js";
import { markRunHandedOffForShutdown } from "./shutdown-handoff.js";
import { isRunHandedOffForShutdown, markRunHandedOffForShutdown } from "./shutdown-handoff.js";
import { ORPHANED_RUN_LEASE_FAILURE_REASON } from "../environment-run-lease-settlement.js";
import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../environment-lease-controller.js";
import { readNativeStartupCleanup, readNativeStartupContext } from "../native-runtime/native-startup-cleanup.js";
@@ -1009,6 +1009,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
},
db,
liveRunExecutions: activeRunExecutions,
handedOffForShutdown: isRunHandedOffForShutdown,
runIds,
restartKind,
recoveryRequestId: intent?.recoveryRequestId ?? null,
@@ -11,3 +11,8 @@ export function watchShutdownHandoff(runId: string): () => boolean {
return current !== undefined && current !== atStart;
};
}
/** This process handed the run to the next server (graceful drain or hot restart). */
export function isRunHandedOffForShutdown(runId: string): boolean {
return runsHandedOffForShutdown.has(runId);
}
@@ -220,6 +220,23 @@ export async function nativeRecoveryClaimRelinquished(input: {
&& row.lastDisposition === "restart_suspended";
}
/**
* The run's controller is this process and its execution lease has lapsed
* (or was released): the executor that held it detached because of that loss,
* not because a shutdown or hot restart handed the run on.
*/
export async function ownNativeExecutionLeaseLapsed(db: Db, input: { companyId: string; id: string }): Promise<boolean> {
const controller = await currentNativeControllerIdentity();
const [row] = await db.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
.where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.id),
eq(heartbeatRuns.status, "running"), isNull(nativeRunFinalizations.resultId),
eq(nativeRunFinalizations.controllerBootId, controller.bootId), eq(nativeRunFinalizations.controllerPid, controller.pid),
or(isNull(nativeRunFinalizations.leaseExpiresAt), sql`${nativeRunFinalizations.leaseExpiresAt} <= clock_timestamp()`)))
.limit(1);
return Boolean(row);
}
export async function releaseLapsedNativeRestartClaim(input: {
db: Db; companyId: string; runId: string; leaseOwner: string; controllerGeneration: number; providerAttempt: number;
}): Promise<boolean> {
@@ -87,7 +87,9 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
expect(await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.runId] })).toEqual([
{ kind: kind === "owned" ? "awaiting_evidence" : "blocked", runId: f.runId,
reason: kind === "owned" ? "controller_still_alive" : "provider_checkpoint_permanently_failed" },
reason: kind === "owned" ? "controller_still_alive" : "provider_checkpoint_permanently_failed",
// This process's own live lease lapses by itself; recovery looks again then.
...(kind === "owned" ? { retryAt: expect.any(Date) } : {}) },
]);
const current = await readRun(f.runId);
expect(current.status).toBe(kind === "owned" ? "running" : "failed");
@@ -164,9 +164,13 @@ export function suspendedByCleanDrain(coordinator: {
}): boolean {
// Attempt 0 means no execution has claimed the run yet. Its first provider
// start must still be counted, whatever interrupted preparation.
if (coordinator.attempt <= 0) return false;
for (let index = coordinator.recoveryHistory.length - 1; index >= 0; index -= 1) {
const disposition = coordinator.recoveryHistory[index]?.disposition;
return coordinator.attempt > 0 && handedToNextControllerSinceClaim(coordinator.recoveryHistory);
}
/** The latest record since the last claim hands the run to the next server. */
function handedToNextControllerSinceClaim(history: ReadonlyArray<{ disposition?: unknown }>): boolean {
for (let index = history.length - 1; index >= 0; index -= 1) {
const disposition = history[index]?.disposition;
if (disposition === "restart_suspended") return true;
if (RECOVERY_CLAIM_DISPOSITIONS.has(disposition)) return false;
}
@@ -1029,6 +1033,8 @@ export async function claimNativeRestartRecoveries(input: {
recoveryRequestId?: string | null;
runIds?: string[];
liveRunExecutions?: Pick<ReadonlySet<string>, "has">;
/** Runs this process handed to the next server (graceful drain or hot restart). */
handedOffForShutdown?: (runId: string) => boolean;
now?: Date;
limit?: number;
/** Called after each committed claim, before scanning the next candidate. */
@@ -1306,8 +1312,27 @@ export async function claimNativeRestartRecoveries(input: {
&& (controllerHostHash === "unknown" || row.coordinator.controllerHostHash !== controllerHostHash);
const nonExecutionOwner = row.coordinator.leaseOwner?.startsWith("native-finalizer:")
|| row.coordinator.leaseOwner?.startsWith("native-cleanup:");
// The recorded controller is this exact process (same boot and PID), so
// a PID probe can only ever answer "alive" and the wait would never end.
// This process knows whether it still drives the run: its executor is
// registered for the whole execution. While it is, leave the run to it.
// Once it has exited (for example after a stall outlived its lease and
// renewal detached it for "the next controller"), the prior owner is
// provably gone, and the evidence checks below decide as for any
// stopped controller. An unexpired self lease still waits: a claim that
// has not reached its executor yet holds it, and it lapses on its own.
// A run this process handed to the next server (graceful drain or hot
// restart) stays with that server: this process is about to exit.
const ownControllerProcess = !nonExecutionOwner && isCurrentNativeControllerProcess(row.coordinator);
if (ownControllerProcess && activeNativeExecutionRunIds.has(row.run.id)) {
return { kind: "awaiting_evidence", runId: row.run.id, reason: "current_controller_preparing" } as const;
}
const ownLeaseLapsed = ownControllerProcess && row.coordinator.leaseExpiresAt !== null
&& row.coordinator.leaseExpiresAt <= now && !input.handedOffForShutdown?.(row.run.id)
&& !handedToNextControllerSinceClaim(row.coordinator.recoveryHistory);
const evaluatedTakeover = nonExecutionOwner
? { allowed: false, reason: "native_finalization_or_cleanup_owner" }
: ownLeaseLapsed ? { allowed: true, reason: "current_controller_released" }
: await evaluateNativeControllerTakeover({
owner: row.coordinator,
now,
@@ -1379,6 +1404,10 @@ export async function claimNativeRestartRecoveries(input: {
runId: row.run.id,
reason: takeover.reason,
...(durableAuthority && row.coordinator.leaseExpiresAt ? { retryAt: row.coordinator.leaseExpiresAt }
// This process's own unexpired lease with no driver here lapses by
// itself; look again then instead of waiting for another signal.
: ownControllerProcess && row.coordinator.leaseExpiresAt && row.coordinator.leaseExpiresAt > now
&& !input.handedOffForShutdown?.(row.run.id) ? { retryAt: row.coordinator.leaseExpiresAt }
: ["controller_lease_held", "controller_lease_grace"].includes(takeover.reason)
? { retryAt: new Date(now.getTime() + CONTROLLER_LEASE_RENEW_MS) } : {}),
} as const;
@@ -38,7 +38,7 @@ import {
nativeRunnerCheckpoints,
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
} from "@paperclipai/db";
import { beforeAll, expect, it, vi } from "vitest";
import { beforeAll, describe, expect, it, vi } from "vitest";
import { heartbeatService } from "../heartbeat.js";
import {
@@ -79,6 +79,7 @@ import {
restoreNativeRunnerCheckpoint,
} from "./native-runner-checkpoint.js";
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
import { setNativeExecutionActive } from "./native-active-runs.js";
import { RUNNER_BRIDGE_READ_OPERATIONS, RUNNER_BRIDGE_SCHEMAS } from "./runner-bridge-tools.js";
import { HttpError } from "../../errors.js";
import { legacyControllerBootId } from "../legacy-controller-lease.js";
@@ -3297,6 +3298,86 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ runId: fixture.runId, ...expected })]);
});
// Daytona wave s9330 t02 (TM1): SIGSTOP held this same server past its
// execution lease while the executor was starting the native process. On
// SIGCONT the lease renewal found the lease lapsed, the executor detached for
// "the next controller" and exited, and every recovery scan then waited on
// the recorded controller: this same live PID and boot. Nothing else can hold
// a lease that names this process, and nothing in it drives the run any more.
describe("same-boot controller whose own lease lapsed", () => {
it("restarts the bootstrap instead of waiting on itself when no turn reached the runner", async () => {
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-UNCOMMANDED",
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_start_requested" } });
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() });
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 });
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" });
});
it("fails with a Retry instead of waiting on itself when the runner may have started", async () => {
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-IDENTITY",
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_identity_recorded" } });
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() }))
.toEqual([expect.objectContaining({ kind: "blocked", runId: fixture.runId, reason: "ambiguous_remote_recovery_evidence" })]);
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(coordinator).toMatchObject({ phase: "terminal_failure", leaseOwner: null, failureCode: "native_restart_recovery_blocked" });
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(run.status).toBe("failed");
const [issue] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId));
expect(await taskInterruptionPresentation(fixture.db, issue)).toMatchObject({
state: "failed", canRetry: true, observedRunId: fixture.runId });
const personId = randomUUID();
await fixture.db.insert(authUsers).values({ id: personId, name: "Retry person", email: `${personId}@fixture.invalid`,
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
await fixture.db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: personId,
status: "active", membershipRole: "member" });
expect(await requestTaskRetry({ db: fixture.db, companyId, issueId: fixture.issueId, actorId: personId,
actorSource: "session", idempotencyKey: `self-lapsed-${fixture.runId}`, observedRunId: fixture.runId }))
.toMatchObject({ accepted: true });
});
it("leaves the run to this process's executor while it still drives the run", async () => {
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-EXECUTING",
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_start_requested" } });
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set([fixture.runId]) }))
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "current_controller_preparing" })]);
setNativeExecutionActive(fixture.runId, true);
try {
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() }))
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "current_controller_preparing" })]);
} finally {
setNativeExecutionActive(fixture.runId, false);
}
});
it("leaves a run this process handed to the next server for that server", async () => {
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-HANDED-OFF",
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_start_requested" } });
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set(),
handedOffForShutdown: runId => runId === fixture.runId }))
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "controller_still_alive" })]);
});
it("keeps waiting for a different live controller in this process space", async () => {
const fixture = await seedUnlaunchedRemoteClaim("OTHER-LIVE-LAPSED",
{ leaseExpired: true, event: { eventType: "native.process_start_requested" } });
// Another live process (the test runner's parent) recorded as controller.
const parentStartedAt = new Date((await readProcessStartedAt(process.ppid))!);
await fixture.db.update(nativeRunFinalizations).set({ controllerPid: process.ppid,
controllerProcessStartedAt: parentStartedAt }).where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() }))
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "controller_still_alive" })]);
});
});
it.each([
{ attempt: 2, driverKind: "codex_app_server" },
{ attempt: 3, driverKind: "codex_app_server" },