mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
refactor(recovery): judge native owner processes with the shared prover (#19997)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs record the server and adapter processes that own them, and recovery must prove an owner gone before it takes over or releases its work > - Phase 2 of the durable ownership plan gave the repository one prover, `proveOwnerGone`. The native runtime still had four private proofs of the same fact > - The private proofs disagreed with the shared rules in edge cases: some judged a PID from another PID space, and one accepted an inconsistent start time as proof of reuse > - This pull request moves those four proofs onto `proveOwnerGone` and deletes the private liveness code > - The benefit is one rule for "the owner is gone" across native recovery, with the stricter shared behaviour where they differed ## Linked Issues or Issue Description Refs #19933, #19950, #19953 (the shared ownership path and earlier Phase 2 kinds). **What is the problem?** Native recovery proved dead owners with four private checks instead of the shared prover. The checks used different rules for PID spaces, reused PIDs and unreadable start times. **What should happen?** Every native owner proof uses `proveOwnerGone`. Only a `gone` verdict releases or takes over. The remote fence and the one-release legacy fallback stay as they are. **Why does it matter?** Recovery decisions about the same process must agree. A dead PID in another PID space must never be used as proof. ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: the local-process branch now calls `proveOwnerGone`. The remote fenced branch (#19830), the `operator_attested` sentinel and the null-host fallback (`processSpaceUnrecorded` with `legacyHostFallback`, recorded by callers as `legacy_host_unknown_dead_pid_proof`) are kept. Reasons are unchanged, plus a new `controller_pid_start_inconsistent`. Owners built from receipts with no PID space are judged in this PID space: their callers check the recorded host first. - `native-failed-run-owner-release.ts`: `recordedProcessMayBeLive` is deleted. `runProcessMayBeLive` and `controllerMayBeLive` release only on a `gone` verdict. Each use of the legacy fallback is recorded as `legacy_host_unknown_dead_pid_proof` when the release commits. - `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` is replaced by `recoveryProcessMayBeAlive`, which is false only when the prover proves the runner gone. Signalling a recovered runner requires a proven same process. - `native-local-process-stop.ts` `recordedLocalExecutionStopped`: the bound identity (launch event, host, PID, group, start time) is judged by the prover. It uses `process_host_hash` when the row has one. - New `native-runtime/native-owner-stamps.ts`: controller and adapter-process stamp builders, shared with the ownership registry. Not changed: `proveLocalWorkspaceOwnerStopped` (workspaces), `evaluateNativeProviderProcesses`, the stop observation in `recordNativeLocalProcessStop`, and `runProcessMayBeLive` in `native-workspace-sync-retention.ts`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-restart-recovery.test.ts src/services/controller-process-identity.test.ts`: 51 passed. - `cd server && npx vitest run src/services/native-runtime/native-failed-run-owner-release.test.ts`: 14 passed. - `cd server && npx vitest run src/services/native-runtime/native-controller-host-recovery.test.ts`: 2 passed. - `cd server && npx vitest run src/services/quarantined-task-retry.test.ts -t "keeps quarantine"`: 18 passed. `explicit-native-continuation.test.ts`: 414 passed. `ownership-reconciliation-report.test.ts`: 23 passed. - Red with the master versions of the two modules: 3 new tests fail ("accepts a reused PID only when the live process started definitely later", "never releases on a dead PID recorded in another PID space", "keeps the one-release fallback for a run process saved before its PID space, and records it"). The tests were written after the code. - Two fixtures changed to match the shared rules: - `controller-process-identity.test.ts`: the "recycled" case recorded a start after the live process, which is an inconsistent stamp. It now records an earlier start (a real reuse), and the inconsistent case is asserted as alive. - `quarantined-task-retry.test.ts` `worker_alive`: the live worker now records this process's real start time. The old fixture was a reused PID, which the prover correctly proves gone. - `cd server && npx vitest run src/services/native-local-process-stop.test.ts`: 5 passed (review round 1). With the master module, 2 fail: the other-PID-space case and the reused-PID case. - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks Behaviour changes, all from the shared rules: - A live PID observed to start before the recorded controller is no longer accepted as recycled (`controller_pid_start_inconsistent`; the run waits for evidence). - A failed-run owner whose recorded adapter process or controller is in another PID space, or is a remote sandbox PID, is no longer released by a local probe. It keeps the lease path. - An unreadable start time is no longer exit evidence for a recovered local runner. - A local execution whose PID was reused by a later process now counts as stopped. Before, any live PID blocked the proof. - A local execution whose row records another PID space is no longer counted as stopped by a dead local PID. - A PID space that cannot be read here makes a hostless receipt owner unprovable. No persisted state changes. Rows saved by the previous release keep the one-release null-host fallback, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
8475826d2d
commit
a4dbde6181
11 files changed
+291
-112
No files matched your search
@@ -18,6 +18,7 @@ import {
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { agentExecutionsHaveStopped } from "../agent-execution-stop.js";
|
||||
import { readProcessStartedAt } from "../hot-restart.js";
|
||||
import { currentProcessSpaceHash } from "../controller-process-identity.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import { environmentService } from "../environments.js";
|
||||
import { executionControlSchedule } from "../execution-control-work.js";
|
||||
@@ -254,6 +255,32 @@ describe("failed native run owner release", () => {
|
||||
}
|
||||
});
|
||||
|
||||
// Phase 2: run processes and controllers are judged by the shared prover.
|
||||
it("never releases on a dead PID recorded in another PID space", async () => {
|
||||
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
||||
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
||||
await db.update(heartbeatRuns).set({ processPid: 2_000_000_001, processGroupId: null,
|
||||
processStartedAt: new Date(Date.now() - 60_000), processHostHash: "another-pid-space" })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(0);
|
||||
expect(await coordinator(runId)).toMatchObject({ phase: "observed" });
|
||||
await db.update(heartbeatRuns).set({ processHostHash: currentProcessSpaceHash() }).where(eq(heartbeatRuns.id, runId));
|
||||
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(1);
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, runId),
|
||||
eq(activityLog.action, "legacy_host_unknown_dead_pid_proof")))).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the one-release fallback for a run process saved before its PID space, and records it", async () => {
|
||||
const { runId } = await nativeRun({ status: "failed", errorCode: "adapter_failed",
|
||||
coordinator: { leaseOwner: `default:resume:${randomUUID()}`, leaseExpiresAt: new Date(0) } });
|
||||
await db.update(heartbeatRuns).set({ processPid: 2_000_000_001, processGroupId: null,
|
||||
processStartedAt: new Date(Date.now() - 60_000), processHostHash: null }).where(eq(heartbeatRuns.id, runId));
|
||||
await expect(releaseAbandonedNativeOwners(db)).resolves.toBe(1);
|
||||
const proofs = await db.select().from(activityLog).where(and(eq(activityLog.runId, runId),
|
||||
eq(activityLog.action, "legacy_host_unknown_dead_pid_proof")));
|
||||
expect(proofs.map(proof => proof.details)).toEqual([{ reason: "pid_dead", source: "native_failed_run_process" }]);
|
||||
});
|
||||
|
||||
// `ps -o lstart` on macOS reports whole seconds, while a recorded start can
|
||||
// carry milliseconds. A sub-second gap must not be read as a recycled PID.
|
||||
it.each(["controller", "run process"])("keeps the owner when the live %s start time differs by under a second", async (holder) => {
|
||||
|
||||
@@ -2,15 +2,15 @@ import { and, asc, eq, gt, inArray, isNotNull, isNull, lte, or, sql } from "driz
|
||||
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import { adapterExecutionControls } from "../adapter-execution-control.js";
|
||||
import { processExists, processStartsDefinitelyDiffer } from "../controller-process-identity.js";
|
||||
import { proveOwnerGone, type LocalOwnerStamp } from "../owner-proof.js";
|
||||
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { readProcessStartedAt } from "../hot-restart.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { LEGACY_WORKSPACE_RECOVERY_KEY } from "../workspace-restore-recovery-state.js";
|
||||
import { activeNativeExecutionRunIds } from "./native-active-runs.js";
|
||||
import { isCurrentNativeControllerProcess } from "./native-restart-recovery.js";
|
||||
import { isCurrentNativeControllerProcess, recordLegacyHostFallbackProof } from "./native-restart-recovery.js";
|
||||
import { nativeControllerOwnerStamp, runProcessOwnerStamp } from "./native-owner-stamps.js";
|
||||
|
||||
/**
|
||||
* A native run can fail outside the native session executor: environment lease
|
||||
@@ -85,51 +85,48 @@ export async function nextAbandonedNativeOwnerReleaseAt(db: Db): Promise<Date |
|
||||
return row?.at ?? null;
|
||||
}
|
||||
|
||||
function processAbsent(pid: number): boolean {
|
||||
return !processExists(pid);
|
||||
}
|
||||
/** A proof that used the one-release fallback for a row saved before its PID
|
||||
* space was recorded; each is logged once the release commits. */
|
||||
type FallbackUse = { reason: string; source: string };
|
||||
|
||||
/** True unless the recorded process is gone or provably a different process. */
|
||||
async function recordedProcessMayBeLive(pid: number | null, startedAt: Date | null): Promise<boolean> {
|
||||
if (!pid || processAbsent(pid)) return false;
|
||||
if (!startedAt) return true;
|
||||
const observed = await readProcessStartedAt(pid).catch(() => null);
|
||||
return !processStartsDefinitelyDiffer(startedAt, observed);
|
||||
/** True unless the shared prover proves the stamped owner gone. Only a `gone`
|
||||
* verdict releases: a live, unprovable or foreign-space owner may be live. */
|
||||
async function ownerMayBeLive(stamp: LocalOwnerStamp, source: string, fallbacks: FallbackUse[]): Promise<boolean> {
|
||||
const proof = await proveOwnerGone(stamp);
|
||||
if (proof.verdict !== "gone") return true;
|
||||
if (proof.legacyHostFallback) fallbacks.push({ reason: proof.reason, source });
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* The run-level process evidence `agentExecutionsHaveStopped` requires. This
|
||||
* process answers for itself from its in-memory execution registries.
|
||||
* process answers for itself from its in-memory execution registries; a
|
||||
* recorded adapter process (PID and group) is judged by the shared prover.
|
||||
*/
|
||||
async function runProcessMayBeLive(run: Run, now: Date): Promise<boolean> {
|
||||
async function runProcessMayBeLive(run: Run, now: Date, fallbacks: FallbackUse[]): Promise<boolean> {
|
||||
if (adapterExecutionControls.has(run.id) || activeNativeExecutionRunIds.has(run.id)) return true;
|
||||
if (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > now) return true;
|
||||
// Same process identity rule as `agentExecutionsHaveStopped`: a recorded PID
|
||||
// that now has another start time was recycled, and so was its own group.
|
||||
let processAlive = Boolean(run.processPid && !processAbsent(run.processPid));
|
||||
let groupAlive = Boolean(run.processGroupId && !processAbsent(-run.processGroupId));
|
||||
if (processAlive && run.processStartedAt) {
|
||||
const observed = await readProcessStartedAt(run.processPid!).catch(() => null);
|
||||
if (processStartsDefinitelyDiffer(run.processStartedAt, observed)) {
|
||||
processAlive = false;
|
||||
if (run.processGroupId === run.processPid) groupAlive = false;
|
||||
}
|
||||
}
|
||||
return processAlive || groupAlive;
|
||||
if (!run.processPid && !run.processGroupId) return false;
|
||||
return ownerMayBeLive(runProcessOwnerStamp(run, "native_failed_run_process", { legacyFallback: true }),
|
||||
"native_failed_run_process", fallbacks);
|
||||
}
|
||||
|
||||
/** An expired owner's recorded controller, unless this process is that controller. */
|
||||
async function controllerMayBeLive(coordinator: Coordinator): Promise<boolean> {
|
||||
async function controllerMayBeLive(coordinator: Coordinator, fallbacks: FallbackUse[]): Promise<boolean> {
|
||||
if (isCurrentNativeControllerProcess(coordinator)) return false;
|
||||
// A previous boot of this process can share its PID, but cannot be its controller.
|
||||
if (coordinator.controllerPid === process.pid) return false;
|
||||
return recordedProcessMayBeLive(coordinator.controllerPid, coordinator.controllerProcessStartedAt);
|
||||
if (!coordinator.controllerPid) return false;
|
||||
// Not this boot (checked above), so the prover must not treat its boot id as live.
|
||||
return ownerMayBeLive({ ...nativeControllerOwnerStamp(coordinator, "native_failed_run_controller"), bootId: null },
|
||||
"native_failed_run_controller", fallbacks);
|
||||
}
|
||||
|
||||
async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promise<boolean> {
|
||||
const publications: ActivityPublication[] = [];
|
||||
const settled = await db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
const fallbacks: FallbackUse[] = [];
|
||||
await tx.execute(sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`);
|
||||
const [binding] = await tx.select({ companyId: nativeRunFinalizations.companyId, issueId: nativeRunFinalizations.issueId })
|
||||
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, release.runId));
|
||||
@@ -151,8 +148,9 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis
|
||||
|| run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null) return false;
|
||||
const [heldElsewhere] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.id, run.id), or(unsettledRunEnvironmentLease, pendingNativeStop))).limit(1);
|
||||
if (heldElsewhere || await runProcessMayBeLive(run, now) || await controllerMayBeLive(coordinator)) return false;
|
||||
} else if (await runProcessMayBeLive(run, now)) {
|
||||
if (heldElsewhere || await runProcessMayBeLive(run, now, fallbacks)
|
||||
|| await controllerMayBeLive(coordinator, fallbacks)) return false;
|
||||
} else if (await runProcessMayBeLive(run, now, fallbacks)) {
|
||||
// The claim never reached the executor, but the run may still own a local process.
|
||||
return false;
|
||||
}
|
||||
@@ -194,6 +192,9 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis
|
||||
if (!updated) return false;
|
||||
await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: now, updatedAt: now })
|
||||
.where(eq(heartbeatRuns.id, run.id));
|
||||
for (const use of fallbacks) {
|
||||
await recordLegacyHostFallbackProof(tx, run, { allowed: true, reason: use.reason, legacyHostFallback: true }, use.source);
|
||||
}
|
||||
await logActivity(tx, {
|
||||
companyId: run.companyId, actorType: "system", actorId: ACTOR_ID,
|
||||
action: "native.failed_run_owner_released", entityType: "heartbeat_run", entityId: run.id,
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
import type { heartbeatRuns, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import type { LocalOwnerStamp } from "../owner-proof.js";
|
||||
|
||||
/**
|
||||
* Owner stamps for the native records that `proveOwnerGone` judges (plan
|
||||
* Phase 2). The ownership registry and the native recovery paths build their
|
||||
* stamps here, so one record has one proof.
|
||||
*/
|
||||
|
||||
type ControllerRow = Pick<typeof nativeRunFinalizations.$inferSelect, "controllerBootId" | "controllerPid"
|
||||
| "controllerProcessStartedAt"> & {
|
||||
controllerHostHash?: string | null;
|
||||
controllerGeneration?: number | null;
|
||||
};
|
||||
|
||||
/** NRF records the controller's boot, PID, start time and, since #19868, its
|
||||
* PID space (`controller_host_hash`). Rows saved before that keep the
|
||||
* one-release local proof on single-host servers (`processSpaceUnrecorded`,
|
||||
* recorded as `legacy_host_unknown_dead_pid_proof`, #19883). */
|
||||
export function nativeControllerOwnerStamp(row: ControllerRow, scope: string): LocalOwnerStamp {
|
||||
return {
|
||||
kind: "local", scope, generation: row.controllerGeneration ?? null, host: null,
|
||||
processSpace: row.controllerHostHash ?? null,
|
||||
processSpaceUnrecorded: row.controllerHostHash == null,
|
||||
bootId: row.controllerBootId, pid: row.controllerPid,
|
||||
processStartedAt: row.controllerProcessStartedAt?.toISOString() ?? null, processGroupId: null,
|
||||
};
|
||||
}
|
||||
|
||||
type RunProcessRow = Pick<typeof heartbeatRuns.$inferSelect, "processPid" | "processGroupId" | "processStartedAt"> & {
|
||||
processHostHash?: string | null;
|
||||
};
|
||||
|
||||
/** heartbeat_runs records the adapter child's PID, group, start time and,
|
||||
* since #19871, its PID space (`process_host_hash`, or "remote" for a sandbox
|
||||
* PID, which a local probe never judges). With `legacyFallback`, a row saved
|
||||
* before that column keeps the one-release local proof; without it, such a row
|
||||
* is unprovable. */
|
||||
export function runProcessOwnerStamp(row: RunProcessRow, scope: string,
|
||||
options: { legacyFallback?: boolean } = {}): LocalOwnerStamp {
|
||||
return {
|
||||
kind: "local", scope, generation: null, host: null, processSpace: row.processHostHash ?? null,
|
||||
...(options.legacyFallback ? { processSpaceUnrecorded: row.processHostHash == null } : {}),
|
||||
bootId: null, pid: row.processPid, processStartedAt: row.processStartedAt?.toISOString() ?? null,
|
||||
processGroupId: row.processGroupId,
|
||||
};
|
||||
}
|
||||
@@ -270,6 +270,33 @@ describe("native controller takeover fencing", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// Phase 2: the local branch is the shared prover (proveOwnerGone).
|
||||
it("accepts a reused PID only when the live process started definitely later", async () => {
|
||||
const live = { isProcessAlive: () => true };
|
||||
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
||||
readProcessStartedAt: async () => new Date(recordedStart.getTime() + 5_000) }))
|
||||
.resolves.toEqual({ allowed: true, reason: "controller_pid_recycled" });
|
||||
// An observed start before the recorded one is an inconsistent stamp,
|
||||
// which the previous private proof accepted as recycled.
|
||||
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
||||
readProcessStartedAt: async () => new Date(recordedStart.getTime() - 5_000) }))
|
||||
.resolves.toEqual({ allowed: false, reason: "controller_pid_start_inconsistent" });
|
||||
// Within the 1 s precision window it is the same live controller.
|
||||
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
||||
readProcessStartedAt: async () => new Date(recordedStart.getTime() + 500) }))
|
||||
.resolves.toEqual({ allowed: false, reason: "controller_still_alive" });
|
||||
await expect(evaluateNativeControllerTakeover({ owner: owner(), now, ...live,
|
||||
readProcessStartedAt: async () => null }))
|
||||
.resolves.toEqual({ allowed: false, reason: "controller_still_alive" });
|
||||
});
|
||||
|
||||
it("judges a controller recorded in this PID space, and keeps the operator attestation", async () => {
|
||||
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: currentProcessSpaceHash() },
|
||||
now, isProcessAlive: () => false })).resolves.toEqual({ allowed: true, reason: "controller_process_dead" });
|
||||
await expect(evaluateNativeControllerTakeover({ owner: { ...owner(), controllerHostHash: "operator_attested" },
|
||||
now, isProcessAlive: () => true })).resolves.toEqual({ allowed: true, reason: "operator_attested_controller_stopped" });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["another host", "another-host-pid-space"],
|
||||
["an unreadable PID space", "unknown"],
|
||||
|
||||
@@ -36,10 +36,10 @@ import {
|
||||
import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js";
|
||||
import { PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT, providerAttemptChargesBudget, providerAttemptWindowStart, readProviderAttemptOutcome } from "./provider-attempt-outcome.js";
|
||||
import { readProcessStartedAt } from "../hot-restart.js";
|
||||
import { ephemeralServerEnabled } from "../ephemeral-server-mode.js";
|
||||
import { currentProcessProbe, proveOwnerGone, type ProcessProbe } from "../owner-proof.js";
|
||||
import { persistActivity } from "../activity-log.js";
|
||||
import { OPERATOR_ATTESTED_CONTROLLER_HOST, requestNativeControllerStopConfirmation } from "./native-controller-host-recovery.js";
|
||||
import { currentControllerBootId, currentProcessSpaceHash, currentProcessStartedAtMs, processStartsDefinitelyDiffer, sameProcessSpace } from "../controller-process-identity.js";
|
||||
import { currentControllerBootId, currentProcessSpaceHash, currentProcessStartedAtMs, processStartsDefinitelyDiffer } from "../controller-process-identity.js";
|
||||
import { getServerProcessStartedAt } from "../../server-info.js";
|
||||
import { redactSensitiveText } from "../../redaction.js";
|
||||
import { isUniqueViolation } from "../../db-errors.js";
|
||||
@@ -418,8 +418,6 @@ export async function evaluateNativeControllerTakeover(input: {
|
||||
? { allowed: false, reason: "remote_controller_lease_live" }
|
||||
: { allowed: true, reason: "remote_controller_lease_released_or_expired" };
|
||||
}
|
||||
const isAlive = input.isProcessAlive ?? processIsAlive;
|
||||
const readStartedAt = input.readProcessStartedAt ?? observedProcessStart;
|
||||
if ((!owner.leaseOwner || !owner.leaseExpiresAt) && !input.requireStoppedController) {
|
||||
return { allowed: true, reason: "unowned" };
|
||||
}
|
||||
@@ -440,21 +438,32 @@ export async function evaluateNativeControllerTakeover(input: {
|
||||
if (recordsHost && owner.controllerHostHash === OPERATOR_ATTESTED_CONTROLLER_HOST) {
|
||||
return { allowed: true, reason: "operator_attested_controller_stopped" };
|
||||
}
|
||||
// A PID can only be judged in the PID space that recorded it. A controller
|
||||
// on another host or namespace (or an unreadable local space) is never
|
||||
// proven gone by a local probe; its lease or fence decides instead.
|
||||
if (owner.controllerHostHash && !sameProcessSpace(owner.controllerHostHash)) {
|
||||
return { allowed: false, reason: "controller_host_unprovable" };
|
||||
}
|
||||
// A row saved before the host column existed. An ephemeral (multi-host)
|
||||
// server can never assume it was written here; a board operator decides.
|
||||
if (recordsHost && owner.controllerHostHash == null && ephemeralServerEnabled()) {
|
||||
return { allowed: false, reason: "controller_host_unrecorded" };
|
||||
}
|
||||
// Single-host servers keep the previous local proof for one release, and
|
||||
// every use is recorded so the release that removes it can check for rows.
|
||||
const fallback = recordsHost && owner.controllerHostHash == null ? { legacyHostFallback: true as const } : {};
|
||||
if (!isAlive(priorPid)) {
|
||||
// The local branch is the shared prover (plan Phase 2): a PID is judged
|
||||
// only in the PID space that recorded it, a reused PID must have started
|
||||
// definitely later, and EPERM is alive. A coordinator row saved before the
|
||||
// host column keeps the one-release local proof on single-host servers
|
||||
// (`processSpaceUnrecorded`; each use is recorded by the caller as
|
||||
// LEGACY_HOST_FALLBACK_EVENT), and an ephemeral server never judges it.
|
||||
// Owners built from receipts that never stored a PID space are judged in
|
||||
// this PID space: their callers checked the recorded host first.
|
||||
const base = currentProcessProbe();
|
||||
const probe: ProcessProbe = {
|
||||
...base,
|
||||
// No platform-deployment shortcut: this stamp records no worker.
|
||||
deployment: null,
|
||||
pidExists: input.isProcessAlive ?? base.pidExists,
|
||||
readStartedAt: input.readProcessStartedAt ?? base.readStartedAt,
|
||||
};
|
||||
const proof = await proveOwnerGone({
|
||||
kind: "local", scope: "native_controller", generation: null, host: null,
|
||||
processSpace: recordsHost ? owner.controllerHostHash ?? null : probe.processSpace,
|
||||
processSpaceUnrecorded: recordsHost && owner.controllerHostHash == null,
|
||||
bootId: null, pid: priorPid, processStartedAt: owner.controllerProcessStartedAt?.toISOString() ?? null,
|
||||
processGroupId: null,
|
||||
}, probe);
|
||||
const fallback = proof.legacyHostFallback ? { legacyHostFallback: true as const } : {};
|
||||
if (proof.verdict === "gone") {
|
||||
if (proof.reason === "pid_reused") return { allowed: true, reason: "controller_pid_recycled", ...fallback };
|
||||
return {
|
||||
allowed: true,
|
||||
reason:
|
||||
@@ -464,18 +473,14 @@ export async function evaluateNativeControllerTakeover(input: {
|
||||
...fallback,
|
||||
};
|
||||
}
|
||||
|
||||
const observedStartedAt = await readStartedAt(priorPid);
|
||||
if (
|
||||
owner.controllerProcessStartedAt &&
|
||||
observedStartedAt &&
|
||||
definitivelyDifferentProcessStart(
|
||||
owner.controllerProcessStartedAt,
|
||||
observedStartedAt,
|
||||
)
|
||||
) {
|
||||
return { allowed: true, reason: "controller_pid_recycled", ...fallback };
|
||||
if (proof.reason === "process_space_unrecorded") return { allowed: false, reason: "controller_host_unrecorded" };
|
||||
if (proof.reason === "different_process_space" || proof.reason === "unknown_process_space"
|
||||
|| proof.reason === "different_host") {
|
||||
return { allowed: false, reason: "controller_host_unprovable" };
|
||||
}
|
||||
// A live PID observed to start before the recorded controller is an
|
||||
// inconsistent stamp, never proof of reuse.
|
||||
if (proof.reason === "pid_start_inconsistent") return { allowed: false, reason: "controller_pid_start_inconsistent" };
|
||||
|
||||
const coordinated = input.coordinatedPreviousController;
|
||||
if (
|
||||
|
||||
@@ -302,8 +302,8 @@ import { noteNativeRunnerEvidence } from "./native-runner-evidence-signal.js";
|
||||
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
|
||||
import { registerLiveRunnerGoalController } from "../runner-goal-control-broker.js";
|
||||
import { applyRunnerGoalPrpEvent } from "../runner-goals.js";
|
||||
import { readProcessStartedAt } from "../hot-restart.js";
|
||||
import { processExists, processStartsDefinitelyDiffer } from "../controller-process-identity.js";
|
||||
import { currentProcessSpaceHash } from "../controller-process-identity.js";
|
||||
import { proveOwnerGone } from "../owner-proof.js";
|
||||
import {
|
||||
NativeRunnerOwnershipUnverifiedError,
|
||||
isNativeRunnerOwnershipHeld,
|
||||
@@ -591,32 +591,33 @@ const nativeRuntimeRequestResolutions = new Map<
|
||||
NativeRuntimeRequestResolution
|
||||
>();
|
||||
|
||||
/** The shared prover's verdict on a recovered local runner, recorded on this
|
||||
* host by an earlier boot (no boot id, so this process is never assumed). */
|
||||
function proveRecoveryProcess(input: { pid: number; startedAt: string }) {
|
||||
return proveOwnerGone({ kind: "local", scope: "native_recovery_runner", generation: null, host: null,
|
||||
processSpace: currentProcessSpaceHash(), bootId: null, pid: input.pid, processStartedAt: input.startedAt,
|
||||
processGroupId: null });
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the recorded runner still owns its PID. A start time within the
|
||||
* shared precision window is the same process: `ps -o lstart` on macOS has
|
||||
* whole-second precision, so an exact match would call a live runner gone.
|
||||
* False only when the shared prover proves the recorded runner gone (its PID
|
||||
* is dead or reused by a later process). The runner treats false as exit
|
||||
* evidence, so an unreadable or inconsistent start time stays alive.
|
||||
*/
|
||||
export async function verifiedRecoveryProcessIsAlive(input: {
|
||||
export async function recoveryProcessMayBeAlive(input: {
|
||||
pid: number;
|
||||
startedAt: string;
|
||||
}): Promise<boolean> {
|
||||
if (!processExists(input.pid)) return false;
|
||||
try {
|
||||
const observed = await readProcessStartedAt(input.pid);
|
||||
return (
|
||||
observed !== null &&
|
||||
!processStartsDefinitelyDiffer(input.startedAt, observed)
|
||||
);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
return (await proveRecoveryProcess(input)).verdict !== "gone";
|
||||
}
|
||||
|
||||
async function signalVerifiedRecoveryProcess(
|
||||
input: { pid: number; startedAt: string },
|
||||
signal: NodeJS.Signals,
|
||||
): Promise<boolean> {
|
||||
if (!(await verifiedRecoveryProcessIsAlive(input))) return false;
|
||||
// Signal only the proven same process (start time within the shared
|
||||
// precision window), never an ambiguous or reused PID.
|
||||
if ((await proveRecoveryProcess(input)).verdict !== "alive") return false;
|
||||
try {
|
||||
return process.kill(input.pid, signal);
|
||||
} catch {
|
||||
@@ -5467,7 +5468,7 @@ const RETIREMENT_REASON_NAMES = {
|
||||
recoveryState: new Set(["blocked", "resuming_session", "awaiting_evidence", "awaiting_runner_reattach"]),
|
||||
controllerReason: new Set(["controller_still_alive", "coordinated_controller_still_alive", "remote_controller_lease_live",
|
||||
"expired_lease_without_controller_identity", "live_legacy_lease_without_controller_identity",
|
||||
"controller_host_unprovable", "controller_host_unrecorded"]),
|
||||
"controller_host_unprovable", "controller_host_unrecorded", "controller_pid_start_inconsistent"]),
|
||||
sandboxOutcome: new Set(["created", "resumed", "replacement"]),
|
||||
} as const;
|
||||
|
||||
@@ -14664,7 +14665,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
target, binding: input.execution.binding, trace: input.trace, onLog: input.onLog,
|
||||
}).controlPlaneRegistration,
|
||||
adoptExistingRunner: remoteDot?.adoptExistingRunner ?? (recoveredProcess ? { ...recoveredProcess,
|
||||
isAlive: () => verifiedRecoveryProcessIsAlive(recoveredProcess),
|
||||
isAlive: () => recoveryProcessMayBeAlive(recoveredProcess),
|
||||
signal: (signal: NodeJS.Signals = "SIGTERM") => signalVerifiedRecoveryProcess(recoveredProcess, signal) } : undefined),
|
||||
};
|
||||
const backendOptions = {
|
||||
@@ -16012,7 +16013,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
: target.kind === "local" && localRecoveryProcess
|
||||
? {
|
||||
...localRecoveryProcess,
|
||||
isAlive: () => verifiedRecoveryProcessIsAlive(localRecoveryProcess),
|
||||
isAlive: () => recoveryProcessMayBeAlive(localRecoveryProcess),
|
||||
signal: (signal: NodeJS.Signals) =>
|
||||
signalVerifiedRecoveryProcess(localRecoveryProcess, signal),
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user