mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path - Restart recovery should be fast and fault-proof; "restarts are product failures". Remote (Daytona) runs must recover as quickly as local ones once it is safe. - Phase 3's Daytona K13 storm (`w-20261109T100435-s9503`) showed Codex `restart_recovery_control_plane` p50 34.9 s, p95 61.1 s. That's Paperclip's own share, before any provider load. - The run logs show why. After a hard restart, each slow run sat in "Native restart recovery is waiting for safe ownership evidence" with reason `remote_controller_lease_live` for about 51 s, until the dead controller's 60 s remote-controller lease expired. - On the remote path, `evaluateNativeControllerTakeover` deliberately skips every process proof and waits for the lease. That's correct when the previous controller might live in another container or on another host. When it recorded this exact process space, the shared prover (pid, process start, boot id) can prove it gone, which is the same proof the local path already trusts. ## Linked Issues or Issue Description **What happened?** After a hard control-plane restart, a remote (Daytona) native run could not be recovered until the dead controller's 60 s remote lease expired. Recovery took about 35–60 s, even though the old controller process was provably gone. **Steps to reproduce** 1. Run a Codex task in a Daytona environment. 2. Hard-kill the control plane (SIGKILL) while the run is active, then restart it. 3. The run logs `native.recovery.transition` "waiting for safe ownership evidence" (`remote_controller_lease_live`) and resumes only when the lease expires. **Expected behavior** When the previous controller ran in this same process space and is proven gone, recovery takes over the remote runner immediately. It still waits for the lease whenever the previous controller cannot be proven gone. ## What Changed - `server/src/services/native-runtime/native-restart-recovery.ts`, `evaluateNativeControllerTakeover` with `remoteLeaseFenced`: - An expired or released lease is allowed, as before. - A live lease whose holder recorded this exact process space (`controllerHostHash`, neither missing nor unknown) and a controller PID goes through the shared owner-gone prover, with `requireStoppedController`. A positive proof allows the takeover, with reason `remote_lease_<proof reason>`, for example `remote_lease_controller_process_dead`. - The one-release null-host fallback is never accepted for a remote runner. - A holder in another process space is never probed (the existing "must not probe another container's PID" test still holds). It waits for the lease, as before. - `native-restart-recovery.test.ts`: a new test, red on master, covers: - takeover when this-process-space controller is dead; - waiting while it is alive; - waiting without a probe when the process space differs. ## Verification - The 7 server test files that cover the takeover decision pass (351 tests): `native-restart-recovery`, `native-controller-host-recovery`, `native-remote-retry-admission`, `native-recovered-owner-followup`, `native-runner-restart-diskless.integration`, `heartbeat/recovery` and `controller-instance-lease`. - Evidence: kept-instance DB copy from the Daytona K13 wave. In run `ba194ae7`, the `awaiting_evidence` transition `remote_controller_lease_live` lasted from 04:26:06.8 to 04:26:57.9, then `bootstrap_incomplete` restart. - A Daytona K13 rerun will measure the before and after. ## Risks - **Takeover soundness** rests on the shared prover, which the local path already uses for the same controllers: same process space, PID dead or definitely restarted, and boot-id proof where needed. If the old controller is alive, the decision still waits for the lease. - **Multi-host or containerized controllers** record a different process space, so they are unaffected and still fenced by the lease. - **The remote runner** keeps its own reconnect grace (7 min). The new controller reattaches through the existing durable-authority path; only the wait before that changes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
537 lines
22 KiB
TypeScript
537 lines
22 KiB
TypeScript
import { currentControllerBootId, currentProcessSpaceHash } from "../controller-process-identity.js";
|
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
|
|
|
import {
|
|
classifyNativeRunnerRecoveryEvidence,
|
|
evaluateNativeControllerTakeover,
|
|
evaluateNativeProviderProcesses,
|
|
nextNativeProviderAttempt,
|
|
proveNativeLocalRunnerStopped,
|
|
} from "./native-restart-recovery.js";
|
|
|
|
describe("local runner stop proof", () => {
|
|
const startedAt = new Date("2026-11-06T17:58:54.000Z");
|
|
const base = { processPid: 72454, processGroupId: 72454, processStartedAt: startedAt };
|
|
const probe = (alive: number[], starts: Record<number, Date | null>, groups: number[] = []) => ({
|
|
isProcessAlive: (pid: number) => alive.includes(pid),
|
|
isProcessGroupAlive: (group: number) => groups.includes(group),
|
|
readProcessStartedAt: async (pid: number) => starts[pid] ?? null,
|
|
});
|
|
|
|
it.each([
|
|
["the runner and its group exited", base, probe([], {}), { stopped: true }],
|
|
["the runner PID now belongs to another process", base,
|
|
probe([72454], { 72454: new Date(startedAt.getTime() + 60_000) }), { stopped: true }],
|
|
["the recorded runner is alive", base, probe([72454], { 72454: startedAt }),
|
|
{ stopped: false, reason: "runner_process_alive" }],
|
|
["the live PID start time is unknown", base, probe([72454], {}),
|
|
{ stopped: false, reason: "runner_process_identity_unverifiable" }],
|
|
["the runner group is alive", base, probe([], {}, [72454]),
|
|
{ stopped: false, reason: "runner_process_group_alive" }],
|
|
["no runner PID was recorded", { ...base, processPid: null }, probe([], {}),
|
|
{ stopped: false, reason: "runner_process_identity_missing" }],
|
|
["no runner start time was recorded", { ...base, processStartedAt: null }, probe([], {}),
|
|
{ stopped: false, reason: "runner_process_identity_missing" }],
|
|
["no runner process group was recorded", { ...base, processGroupId: null }, probe([], {}),
|
|
{ stopped: false, reason: "runner_process_identity_missing" }],
|
|
["a stop receipt proved the runner absent", { ...base, processStartedAt: null, runnerStopRecorded: true },
|
|
probe([], {}), { stopped: true }],
|
|
] as const)("decides when %s", async (_case, identity, checks, expected) => {
|
|
await expect(proveNativeLocalRunnerStopped({ ...identity, ...checks })).resolves.toEqual(expected);
|
|
});
|
|
|
|
it("refuses while a recorded provider process is alive", async () => {
|
|
const providerStartedAt = new Date("2026-11-06T17:58:56.000Z");
|
|
await expect(proveNativeLocalRunnerStopped({ ...base,
|
|
providerProcesses: [{ pid: 72460, processStartedAt: providerStartedAt }],
|
|
...probe([72460], { 72460: providerStartedAt }) })).resolves.toEqual({ stopped: false, reason: "provider_process_alive" });
|
|
});
|
|
});
|
|
|
|
describe("native restart recovery classification", () => {
|
|
it.each([false, true])("recognizes verified staged remote evidence with target snapshot=%s", hasCheckpoint => {
|
|
const evidence = { remoteSandbox: true, runnerPidAlive: false, runnerGroupAlive: false,
|
|
processStartMatches: false, hasCheckpoint, hasProviderEvidence: false };
|
|
expect(classifyNativeRunnerRecoveryEvidence(evidence).claimKind).toBeNull();
|
|
expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true })).toEqual({
|
|
claimKind: "reattach_remote_runner", reason: "staged_remote_attachment_requires_sandbox_verification",
|
|
});
|
|
expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true,
|
|
checkpointFailed: true }).claimKind).toBeNull();
|
|
expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, remoteSandbox: false,
|
|
stagedAttachmentVerified: true }).claimKind).toBe(hasCheckpoint ? null : "bootstrap_incomplete");
|
|
});
|
|
it.each([true, false])("uses the remote fenced lease instead of the old container PID (live=%s)", async live => {
|
|
const now = new Date();
|
|
const probe = vi.fn(() => { throw new Error("must not probe another container's PID"); });
|
|
const result = await evaluateNativeControllerTakeover({ now, remoteLeaseFenced: true,
|
|
owner: { leaseOwner: "old-controller", leaseExpiresAt: new Date(now.getTime() + (live ? 1000 : -1000)),
|
|
controllerPid: process.pid, controllerProcessStartedAt: now }, isProcessAlive: probe });
|
|
expect(result.allowed).toBe(!live);
|
|
expect(probe).not.toHaveBeenCalled();
|
|
});
|
|
it("takes over a live remote lease at once when its controller is proven gone in this process space", async () => {
|
|
const now = new Date();
|
|
const startedAt = new Date(now.getTime() - 60_000);
|
|
const owner = { leaseOwner: "old-controller", leaseExpiresAt: new Date(now.getTime() + 50_000),
|
|
controllerPid: 424242, controllerProcessStartedAt: startedAt,
|
|
controllerHostHash: currentProcessSpaceHash(), controllerBootId: currentControllerBootId() };
|
|
// A hard restart: the previous controller's PID is gone.
|
|
await expect(evaluateNativeControllerTakeover({ now, remoteLeaseFenced: true, owner, isProcessAlive: () => false }))
|
|
.resolves.toMatchObject({ allowed: true, reason: "remote_lease_controller_process_dead" });
|
|
// The previous controller is still running: keep waiting for its lease.
|
|
await expect(evaluateNativeControllerTakeover({ now, remoteLeaseFenced: true, owner,
|
|
isProcessAlive: () => true, readProcessStartedAt: async () => startedAt }))
|
|
.resolves.toEqual({ allowed: false, reason: "remote_controller_lease_live" });
|
|
// Another process space (a different container or host): never probed, keep waiting.
|
|
const probe = vi.fn(() => false);
|
|
await expect(evaluateNativeControllerTakeover({ now, remoteLeaseFenced: true,
|
|
owner: { ...owner, controllerHostHash: "sha256:another-process-space" }, isProcessAlive: probe }))
|
|
.resolves.toEqual({ allowed: false, reason: "remote_controller_lease_live" });
|
|
expect(probe).not.toHaveBeenCalled();
|
|
});
|
|
it("does not reopen a failed checkpoint or reset an exhausted provider budget", () => {
|
|
const evidence = {
|
|
runnerPidAlive: false,
|
|
runnerGroupAlive: false,
|
|
processStartMatches: false,
|
|
hasCheckpoint: true,
|
|
hasProviderEvidence: true,
|
|
};
|
|
expect(
|
|
classifyNativeRunnerRecoveryEvidence({
|
|
...evidence,
|
|
checkpointFailed: true,
|
|
}),
|
|
).toMatchObject({
|
|
claimKind: null,
|
|
reason: "provider_checkpoint_permanently_failed",
|
|
});
|
|
expect(
|
|
classifyNativeRunnerRecoveryEvidence({ ...evidence, providerAttempt: 3 }),
|
|
).toMatchObject({
|
|
claimKind: null,
|
|
reason: "execution_recovery_budget_exhausted",
|
|
});
|
|
});
|
|
it("keeps controller-only recovery out of the provider retry budget", () => {
|
|
expect(nextNativeProviderAttempt(2, "reattach_existing_runner")).toBe(2);
|
|
expect(nextNativeProviderAttempt(2, "reattach_remote_runner")).toBe(2);
|
|
expect(nextNativeProviderAttempt(2, "bootstrap_incomplete")).toBe(3);
|
|
expect(nextNativeProviderAttempt(2, "resume_dead_runner")).toBe(3);
|
|
});
|
|
|
|
it("verifies remote ownership in the sandbox instead of interpreting its PIDs locally", () => {
|
|
const evidence = {
|
|
remoteSandbox: true,
|
|
runnerPidAlive: true,
|
|
runnerGroupAlive: true,
|
|
processStartMatches: false,
|
|
knownProviderProcessAlive: true,
|
|
knownProviderProcessIdentityAmbiguous: true,
|
|
hasCheckpoint: true,
|
|
checkpointIdentityMatches: true,
|
|
hasProviderEvidence: true,
|
|
providerAttempt: 3,
|
|
};
|
|
expect(classifyNativeRunnerRecoveryEvidence(evidence).claimKind).toBe(
|
|
"reattach_remote_runner",
|
|
);
|
|
expect(
|
|
classifyNativeRunnerRecoveryEvidence({
|
|
...evidence,
|
|
checkpointIdentityMatches: false,
|
|
}).claimKind,
|
|
).toBeNull();
|
|
expect(
|
|
classifyNativeRunnerRecoveryEvidence({
|
|
...evidence,
|
|
checkpointFailed: true,
|
|
}).claimKind,
|
|
).toBeNull();
|
|
expect(
|
|
classifyNativeRunnerRecoveryEvidence({
|
|
...evidence,
|
|
hasCheckpoint: false,
|
|
}).claimKind,
|
|
).toBeNull();
|
|
});
|
|
|
|
it.each([
|
|
{
|
|
name: "reattaches an exact live runner",
|
|
evidence: {
|
|
runnerPidAlive: true,
|
|
runnerGroupAlive: true,
|
|
processStartMatches: true,
|
|
hasCheckpoint: true,
|
|
hasProviderEvidence: true,
|
|
},
|
|
expected: "reattach_existing_runner",
|
|
},
|
|
{
|
|
name: "resumes a dead checkpointed runner",
|
|
evidence: {
|
|
runnerPidAlive: false,
|
|
runnerGroupAlive: false,
|
|
processStartMatches: false,
|
|
hasCheckpoint: true,
|
|
hasProviderEvidence: true,
|
|
},
|
|
expected: "resume_dead_runner",
|
|
},
|
|
{
|
|
name: "retries an incomplete bootstrap on the same run",
|
|
evidence: {
|
|
runnerPidAlive: false,
|
|
runnerGroupAlive: false,
|
|
processStartMatches: false,
|
|
hasCheckpoint: false,
|
|
hasProviderEvidence: false,
|
|
},
|
|
expected: "bootstrap_incomplete",
|
|
},
|
|
{
|
|
name: "blocks a recycled live PID",
|
|
evidence: {
|
|
runnerPidAlive: true,
|
|
runnerGroupAlive: true,
|
|
processStartMatches: false,
|
|
hasCheckpoint: true,
|
|
hasProviderEvidence: true,
|
|
},
|
|
expected: null,
|
|
},
|
|
{
|
|
name: "blocks ambiguous checkpoint evidence",
|
|
evidence: {
|
|
runnerPidAlive: false,
|
|
runnerGroupAlive: false,
|
|
processStartMatches: false,
|
|
hasCheckpoint: true,
|
|
hasProviderEvidence: false,
|
|
},
|
|
expected: null,
|
|
},
|
|
{
|
|
name: "blocks a checkpoint bound to a different native identity",
|
|
evidence: {
|
|
runnerPidAlive: false,
|
|
runnerGroupAlive: false,
|
|
processStartMatches: false,
|
|
hasCheckpoint: true,
|
|
checkpointIdentityMatches: false,
|
|
hasProviderEvidence: true,
|
|
},
|
|
expected: null,
|
|
},
|
|
{
|
|
name: "blocks while a known provider process outlives its runner",
|
|
evidence: {
|
|
runnerPidAlive: false,
|
|
runnerGroupAlive: false,
|
|
processStartMatches: false,
|
|
knownProviderProcessAlive: true,
|
|
hasCheckpoint: true,
|
|
hasProviderEvidence: true,
|
|
},
|
|
expected: null,
|
|
},
|
|
{
|
|
name: "blocks while a live provider PID has no durable fingerprint",
|
|
evidence: {
|
|
runnerPidAlive: false,
|
|
runnerGroupAlive: false,
|
|
processStartMatches: false,
|
|
knownProviderProcessIdentityAmbiguous: true,
|
|
hasCheckpoint: true,
|
|
hasProviderEvidence: true,
|
|
},
|
|
expected: null,
|
|
},
|
|
])("$name", ({ evidence, expected }) => {
|
|
expect(classifyNativeRunnerRecoveryEvidence(evidence).claimKind).toBe(
|
|
expected,
|
|
);
|
|
});
|
|
});
|
|
|
|
describe("native controller takeover fencing", () => {
|
|
const now = new Date("2026-09-04T12:00:00.000Z");
|
|
const recordedStart = new Date("2026-09-04T11:00:00.000Z");
|
|
|
|
function owner(
|
|
overrides: Partial<{
|
|
leaseOwner: string | null;
|
|
leaseExpiresAt: Date | null;
|
|
controllerPid: number | null;
|
|
controllerProcessStartedAt: Date | null;
|
|
}> = {},
|
|
) {
|
|
return {
|
|
leaseOwner: "old-controller",
|
|
leaseExpiresAt: new Date("2026-09-04T12:20:00.000Z"),
|
|
controllerPid: 123,
|
|
controllerProcessStartedAt: recordedStart,
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
it("takes over immediately when the exact prior controller process died", async () => {
|
|
await expect(
|
|
evaluateNativeControllerTakeover({
|
|
owner: owner(),
|
|
now,
|
|
isProcessAlive: () => false,
|
|
}),
|
|
).resolves.toEqual({
|
|
allowed: true,
|
|
reason: "controller_process_dead",
|
|
});
|
|
});
|
|
|
|
// Phase 2: the local branch is the shared prover (proveOwnerGone).
|
|
it("accepts a reused PID only when the live process started definitely later", async () => {
|
|
const live = { isProcessAlive: () => true };
|
|
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
|
readProcessStartedAt: async () => new Date(recordedStart.getTime() + 5_000) }))
|
|
.resolves.toEqual({ allowed: true, reason: "controller_pid_recycled" });
|
|
// An observed start before the recorded one is an inconsistent stamp,
|
|
// which the previous private proof accepted as recycled.
|
|
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
|
readProcessStartedAt: async () => new Date(recordedStart.getTime() - 5_000) }))
|
|
.resolves.toEqual({ allowed: false, reason: "controller_pid_start_inconsistent" });
|
|
// Within the 1 s precision window it is the same live controller.
|
|
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
|
readProcessStartedAt: async () => new Date(recordedStart.getTime() + 500) }))
|
|
.resolves.toEqual({ allowed: false, reason: "controller_still_alive" });
|
|
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
|
readProcessStartedAt: async () => null }))
|
|
.resolves.toEqual({ allowed: false, reason: "controller_still_alive" });
|
|
});
|
|
|
|
it("judges a controller recorded in this PID space, and keeps the operator attestation", async () => {
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: currentProcessSpaceHash() },
|
|
now, isProcessAlive: () => false })).resolves.toEqual({ allowed: true, reason: "controller_process_dead" });
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: "operator_attested" },
|
|
now, isProcessAlive: () => true })).resolves.toEqual({ allowed: true, reason: "operator_attested_controller_stopped" });
|
|
});
|
|
|
|
it("preserves same-space process proof when boot metadata is present", async () => {
|
|
const local = { ...owner(), controllerBootId: currentControllerBootId(), controllerHostHash: currentProcessSpaceHash() };
|
|
expect(await evaluateNativeControllerTakeover({ owner: local, now, isProcessAlive: () => false }))
|
|
.toEqual({ allowed: true, reason: "controller_process_dead" });
|
|
expect(await evaluateNativeControllerTakeover({ owner: local, now, isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => recordedStart })).toEqual({ allowed: false, reason: "controller_still_alive" });
|
|
});
|
|
|
|
it.each([
|
|
["another host", "another-host-pid-space"],
|
|
["an unreadable PID space", "unknown"],
|
|
])("never treats a dead or recycled local PID as proof for a controller on %s", async (_name, controllerHostHash) => {
|
|
for (const probe of [{ isProcessAlive: () => false }, { isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => new Date("2026-09-04T11:30:00.000Z") }]) {
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash }, now, ...probe }))
|
|
.resolves.toEqual({ allowed: false, reason: "controller_host_unprovable" });
|
|
}
|
|
});
|
|
|
|
describe("rows saved before the controller host was recorded", () => {
|
|
afterEach(() => vi.unstubAllEnvs());
|
|
|
|
it("keep the local dead-PID proof on a single-host server and flag the fallback", async () => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "");
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: null }, now,
|
|
isProcessAlive: () => false })).resolves.toEqual({ allowed: true, reason: "controller_process_dead", legacyHostFallback: true });
|
|
});
|
|
|
|
it("are unprovable on an ephemeral server, whatever a local probe says", async () => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: null }, now,
|
|
isProcessAlive: () => false })).resolves.toEqual({ allowed: false, reason: "controller_host_unrecorded" });
|
|
});
|
|
|
|
it("stay unprovable on another host in either mode", async () => {
|
|
for (const mode of ["", "true"]) {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", mode);
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: "another-host-pid-space" }, now,
|
|
isProcessAlive: () => false })).resolves.toEqual({ allowed: false, reason: "controller_host_unprovable" });
|
|
}
|
|
});
|
|
|
|
it("accept a board confirmation that the exact controller stopped", async () => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: "operator_attested" }, now,
|
|
isProcessAlive: () => true })).resolves.toEqual({ allowed: true, reason: "operator_attested_controller_stopped" });
|
|
});
|
|
|
|
it("leave receipt owners that never stored a host unchanged", async () => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
|
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, isProcessAlive: () => false }))
|
|
.resolves.toEqual({ allowed: true, reason: "controller_process_dead" });
|
|
});
|
|
});
|
|
|
|
it("proves a controller gone only in its own recorded PID space", async () => {
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: currentProcessSpaceHash() },
|
|
now, isProcessAlive: () => false })).resolves.toEqual({ allowed: true, reason: "controller_process_dead" });
|
|
// A row saved before the PID space was recorded keeps the previous local proof for one release.
|
|
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: null },
|
|
now, isProcessAlive: () => false })).resolves.toEqual({ allowed: true, reason: "controller_process_dead", legacyHostFallback: true });
|
|
});
|
|
|
|
it.each([true, false])("requires a stopped idle controller even after its run lease was released: alive=%s", async alive => {
|
|
await expect(evaluateNativeControllerTakeover({
|
|
owner: owner({ leaseOwner: null, leaseExpiresAt: null }), now, requireStoppedController: true,
|
|
isProcessAlive: () => alive, readProcessStartedAt: async () => recordedStart,
|
|
})).resolves.toEqual({ allowed: !alive, reason: alive ? "controller_still_alive" : "controller_process_dead" });
|
|
});
|
|
|
|
it("takes over a recycled controller PID", async () => {
|
|
await expect(
|
|
evaluateNativeControllerTakeover({
|
|
owner: owner(),
|
|
now,
|
|
isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => new Date("2026-09-04T11:30:00.000Z"),
|
|
}),
|
|
).resolves.toEqual({
|
|
allowed: true,
|
|
reason: "controller_pid_recycled",
|
|
});
|
|
});
|
|
|
|
it("does not treat a sub-second process-start precision difference as PID reuse", async () => {
|
|
await expect(
|
|
evaluateNativeControllerTakeover({
|
|
owner: owner({
|
|
controllerProcessStartedAt: new Date("2026-09-04T11:00:00.456Z"),
|
|
}),
|
|
now,
|
|
isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => new Date("2026-09-04T11:00:00.000Z"),
|
|
}),
|
|
).resolves.toEqual({
|
|
allowed: false,
|
|
reason: "controller_still_alive",
|
|
});
|
|
});
|
|
|
|
it("does not steal a live controller lease", async () => {
|
|
const readStartedAt = vi.fn(async () => recordedStart);
|
|
await expect(
|
|
evaluateNativeControllerTakeover({
|
|
owner: owner(),
|
|
now,
|
|
isProcessAlive: () => true,
|
|
readProcessStartedAt: readStartedAt,
|
|
}),
|
|
).resolves.toEqual({
|
|
allowed: false,
|
|
reason: "controller_still_alive",
|
|
});
|
|
expect(readStartedAt).toHaveBeenCalledWith(123);
|
|
});
|
|
|
|
it("does not let lease expiry bypass a live controller fence", async () => {
|
|
const isProcessAlive = vi.fn(() => true);
|
|
await expect(
|
|
evaluateNativeControllerTakeover({
|
|
owner: owner({
|
|
leaseExpiresAt: new Date("2026-09-04T11:59:59.000Z"),
|
|
}),
|
|
now,
|
|
isProcessAlive,
|
|
readProcessStartedAt: async () => recordedStart,
|
|
}),
|
|
).resolves.toEqual({ allowed: false, reason: "controller_still_alive" });
|
|
expect(isProcessAlive).toHaveBeenCalledWith(123);
|
|
});
|
|
|
|
it("takes over an expired lease after proving the controller died", async () => {
|
|
await expect(
|
|
evaluateNativeControllerTakeover({
|
|
owner: owner({
|
|
leaseExpiresAt: new Date("2026-09-04T11:59:59.000Z"),
|
|
}),
|
|
now,
|
|
isProcessAlive: () => false,
|
|
}),
|
|
).resolves.toEqual({
|
|
allowed: true,
|
|
reason: "expired_lease_controller_process_dead",
|
|
});
|
|
});
|
|
|
|
it("keeps a coordinated previous controller fenced until that exact process exits", async () => {
|
|
await expect(
|
|
evaluateNativeControllerTakeover({
|
|
owner: owner(),
|
|
now,
|
|
coordinatedPreviousController: {
|
|
pid: 123,
|
|
processStartedAt: recordedStart,
|
|
},
|
|
isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => recordedStart,
|
|
}),
|
|
).resolves.toEqual({
|
|
allowed: false,
|
|
reason: "coordinated_controller_still_alive",
|
|
});
|
|
});
|
|
});
|
|
|
|
describe("native provider process fencing", () => {
|
|
const recordedStart = new Date("2026-09-04T11:00:00.000Z");
|
|
|
|
it("does not mistake a recycled provider PID for the old live provider", async () => {
|
|
await expect(
|
|
evaluateNativeProviderProcesses({
|
|
identities: [{ pid: 456, processStartedAt: recordedStart }],
|
|
isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => new Date("2026-09-04T11:30:00.000Z"),
|
|
}),
|
|
).resolves.toEqual({
|
|
knownPids: [456],
|
|
livePids: [],
|
|
ambiguousLivePids: [],
|
|
recycledPids: [456],
|
|
});
|
|
});
|
|
|
|
it("fails closed when a live provider has no comparable fingerprint", async () => {
|
|
await expect(
|
|
evaluateNativeProviderProcesses({
|
|
identities: [{ pid: 456, processStartedAt: null }],
|
|
isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => recordedStart,
|
|
}),
|
|
).resolves.toMatchObject({
|
|
livePids: [],
|
|
ambiguousLivePids: [456],
|
|
recycledPids: [],
|
|
});
|
|
});
|
|
|
|
it("fails closed on a sub-second provider process-start precision difference", async () => {
|
|
await expect(
|
|
evaluateNativeProviderProcesses({
|
|
identities: [
|
|
{
|
|
pid: 456,
|
|
processStartedAt: new Date("2026-09-04T11:00:00.456Z"),
|
|
},
|
|
],
|
|
isProcessAlive: () => true,
|
|
readProcessStartedAt: async () => new Date("2026-09-04T11:00:00.000Z"),
|
|
}),
|
|
).resolves.toEqual({
|
|
knownPids: [456],
|
|
livePids: [],
|
|
ambiguousLivePids: [456],
|
|
recycledPids: [],
|
|
});
|
|
});
|
|
});
|