mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A release swap runs two servers for a short overlap. The old server drains, and the new server takes over the runs. > - Each server proves that it is alive with a controller-instance lease. Recovery uses that lease to decide whether a run's controller is gone. > - In local swap cells, a healthy old server exited during the overlap, because its lease renewal waited for a connection on the busy main pool. Every run it drove was interrupted. > - When a controller crashed, its runs waited for another server boot, or for a timer that restarted on every look. Tasks stood still for 21 to 28 minutes. > - This pull request renews the lease on its own connection, and lets the live server end a crashed controller's runs within a bounded time. > - The benefit is that a swap does not kill a healthy server, and a crashed server's runs end in about 2 minutes, so their tasks continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. During a two-server release swap (local cells, load 14 to 34), the outgoing server exited with "controller instance lease lost while holding work". Five lease renewals in a row timed out after 10 s. The 10 s timeout included the wait for a connection on the shared main pool, which the traffic switch and the replacement's startup pass kept busy. 2. When the old server was killed in the middle of the overlap, its runs stayed in progress for 21 to 28 minutes. The live server did not re-check a crashed controller's runs: only a new boot did. The unverified-controller wait started when a server first looked at the run, not when the controller left, and the successor's own wait record counted as the dead owner's sign of life. **Expected behavior** A healthy server keeps its lease during a swap. When a controller is gone (its lease lapsed and it is fenced or stopped), the live server ends that controller's runs within a bounded time, without another server boot, so restart recovery continues the tasks. **Steps to reproduce** 1. Run two servers against one database with a release swap and a 200 s overlap. 2. Kill the old server 40 s into the overlap (SIGKILL). 3. See that its runs stay in progress for more than 20 minutes, until another server starts. ## What Changed - Lease renewal on its own connection (`packages/db/src/client.ts`, `packages/db/src/index.ts`, `server/src/services/controller-instance-lease.ts`): - New `controllerInstanceLeaseDb(db)`: one private 1-connection pool (`application_name` `paperclip-controller-instance-lease`), made the same way as the existing `controllerLeaseDb` and closed by `closeLivenessDb`. - The renewal is one statement on that connection, as before. - The connection sends `statement_timeout` = 8 s (below the 10 s client timeout) as a startup parameter (new `DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The server cancels a slow renewal, so it never holds the single connection that the next renewal needs, and there is no extra round trip per renewal. (An earlier revision set it per transaction; that added three round trips to every renewal and broke the idle-server receipt test.) - If a connection pooler refuses the startup parameter ("unsupported startup parameter"), the lease logs a warning and renews on a replacement connection without it. - A crashed controller's runs (`server/src/services/recovery/ownership-registry.ts`, `server/src/index.ts`, `server/src/services/native-runtime/native-restart-recovery.ts`): - The ownership reconciler on the live server hands runs whose controller is proven gone to restart recovery. It does not wait for another server boot. - The unverified-controller wait is bounded from the moment the controller's lease lapsed, not from the first look. - When the controller boot is `stopped` or `fenced`, the run ends at once. - A successor's wait record no longer refreshes the dead owner's sign of life. Safety of ending a run when its controller boot is fenced or stopped: - Only `stopped` (written by the boot's own graceful stop) and `fenced` (a newer epoch fenced it after its lease expired plus 30 s grace) count. `live` and `grace` never count, so a controller that is alive and renewing is never treated as gone. `expired` alone does not count either. - A boot that cannot renew stops itself at its self-fence window (about its lease expiry), and a boot that finds itself fenced exits at once. So when another server can fence it, the old process has already stopped driving runs. The lease change above removes the main cause of healthy boots failing to renew. - If the old process is still alive (partition or stall) and its Runner continues: the old run is fenced. Its agent token is refused for writes (`agent_run_superseded`), its remote workspace epoch is abandoned or the continuation gets a new workspace, and the native coordinator records `replacementDenied`. There are never two writers in one workspace. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/controller-instance-lease.test.ts` "own connection": red (10004 ms timeout while the main pool is held) to green. Full file green (26). - `server/src/__tests__/controller-instance-lease.test.ts` "has the server cancel a renewal blocked on a row lock, and renews on the same connection next time": red without the connection-level timeout (the renewal runs to the 10 s client timeout and stays active), green with it (cancelled at about 8 s, no active lease statement left, the next renewal succeeds in under 2 s on the same backend). - `packages/db/src/client-options.test.ts`: the statement timeout is sent as a startup parameter. - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`: "lapsed longer ago" / "end of the bound" (wait counted from the lease lapse), the live-server re-check of a crashed controller, the fenced/stopped boot ending the run at once, and the successor's wait record. Full file green. Changed existing expectation (it asserted the old wait for a stopped or fenced controller): - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` "keeps foreign local Runner and provider ownership after its controller is stopped/fenced": the disposition is now `blocked` (`controller_process_space_unverified`) instead of `awaiting_evidence`, the run is `failed` with `native_restart_recovery_blocked`, and the finalization is `terminal_failure` with `executorStopEvidence: "unverified"`. The safety checks stay: no local PID probe, no claim, no stamp, no local stop proof, the run keeps its process identity and session checkpoint, and no result is written. Local commands, on this branch based on current master: - `cd server && npx vitest run src/__tests__/controller-instance-lease.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140 tests passed. - `npx vitest run src/__tests__/controller-instance-lease.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 225 passed, 14 skipped (real-process cases that need CI). - `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts -t "releases an ordinary held task Retry"`: passed (idle receipt: one lease statement after the work ends). Local swap cells (two servers, embedded Postgres, real agents, old server SIGKILLed 40 s into a 200 s overlap): - Before: runs of the killed server stayed in progress 21 to 28 minutes. - N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the successor's wait write still moved the timer; fixed by the last commit). - R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4 hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run end 3 min 39 s. - N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min 01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the lease TTL (90 s) plus the fence grace (30 s). - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds this fix and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lease and crashed-controller fixes were visibly working in it. ## Risks - A pooled database URL whose pooler refuses `statement_timeout` as a startup parameter: the first renewal fails over to a connection without it (logged), so the lease keeps working without the server-side cancel. - One extra database connection per server, held while the server runs. It is idle except for one renewal every few seconds. - A run whose controller boot is fenced or stopped now ends at once instead of after the bound. See the safety notes above: only a boot that is not alive can be in these states. - No migration. No schema change. Nothing changes before `turn.start` on the normal path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
52817dff55
commit
500bc694b5
11 files changed
+348
-29
No files matched your search
@@ -93,6 +93,12 @@ describe("databaseClientOptionsFromEnv", () => {
|
||||
connection: { application_name: "paperclip-web" },
|
||||
});
|
||||
});
|
||||
|
||||
it("sends a dedicated pool's statement timeout as a startup parameter", () => {
|
||||
expect(postgresJsOptions({ applicationName: "paperclip-controller-instance-lease", statementTimeoutMs: 8_000 })).toEqual({
|
||||
connection: { application_name: "paperclip-controller-instance-lease", statement_timeout: "8000" },
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveDatabaseClientOptions", () => {
|
||||
|
||||
@@ -163,6 +163,11 @@ export interface DatabaseClientOptions {
|
||||
* `postgres.js`).
|
||||
*/
|
||||
applicationName?: string;
|
||||
/**
|
||||
* Connection-level `statement_timeout` in milliseconds, sent as a startup
|
||||
* parameter (no extra round trip). For dedicated single-purpose pools only.
|
||||
*/
|
||||
statementTimeoutMs?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -256,9 +261,10 @@ export function postgresJsOptions(options: DatabaseClientOptions): Record<string
|
||||
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
|
||||
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
|
||||
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
|
||||
if (options.applicationName !== undefined) {
|
||||
driverOptions.connection = { application_name: options.applicationName };
|
||||
}
|
||||
const connection: Record<string, string> = {};
|
||||
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
|
||||
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
|
||||
if (Object.keys(connection).length) driverOptions.connection = connection;
|
||||
return driverOptions;
|
||||
}
|
||||
|
||||
@@ -324,11 +330,42 @@ export function controllerLeaseDb(db: Db): Db {
|
||||
return created;
|
||||
}
|
||||
|
||||
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
|
||||
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
|
||||
|
||||
/** One private connection for the server's controller-instance lease, so its
|
||||
* renewal never waits behind request or worker traffic on the main pool.
|
||||
* `statementTimeoutMs` (first call wins) is a connection-level
|
||||
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
|
||||
* single connection that the next renewal needs, at no extra round trip. */
|
||||
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
|
||||
const existing = controllerInstanceLeaseDbs.get(db);
|
||||
if (existing) return existing;
|
||||
const factory = controllerInstanceLeaseDbFactories.get(db);
|
||||
if (!factory) return db;
|
||||
const created = factory(options.statementTimeoutMs ?? null);
|
||||
controllerInstanceLeaseDbs.set(db, created);
|
||||
return created;
|
||||
}
|
||||
|
||||
/** Replaces the lease connection with one that sends no `statement_timeout`
|
||||
* startup parameter, for a connection pooler that refuses it. */
|
||||
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
|
||||
const factory = controllerInstanceLeaseDbFactories.get(db);
|
||||
if (!factory) return db;
|
||||
const previous = controllerInstanceLeaseDbs.get(db);
|
||||
const created = factory(null);
|
||||
controllerInstanceLeaseDbs.set(db, created);
|
||||
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
|
||||
return created;
|
||||
}
|
||||
|
||||
/** Ends the liveness pool if one was created. The ended pool stays cached, so
|
||||
* a late renewal fails like a query on the ended primary pool. */
|
||||
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
|
||||
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
||||
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
|
||||
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
||||
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
|
||||
}
|
||||
|
||||
// Control checkpoints must not occupy the request or lease-renewal pools
|
||||
@@ -406,6 +443,10 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob
|
||||
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
|
||||
...resolved, maxConnections: 1, applicationName: "paperclip-legacy-controller-lease",
|
||||
}, db));
|
||||
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
|
||||
...resolved, maxConnections: 1, applicationName: "paperclip-controller-instance-lease",
|
||||
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
|
||||
}, db));
|
||||
checkpointDbFactories.set(db, () => createDbClient(url, {
|
||||
...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint",
|
||||
}, db));
|
||||
|
||||
@@ -5,6 +5,8 @@ export {
|
||||
type OwnedDatabaseListener,
|
||||
livenessDb,
|
||||
controllerLeaseDb,
|
||||
controllerInstanceLeaseDb,
|
||||
controllerInstanceLeaseDbWithoutStatementTimeout,
|
||||
closeLivenessDb,
|
||||
checkpointDb,
|
||||
closeCheckpointDb,
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, controllerInstances, createDb, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
|
||||
import { agents, companies, controllerInstanceLeaseDb, controllerInstances, createDb, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
controllerInstanceLeaseEnabled, proveControllerInstance, registerControllerInstanceDb, registerControllerInstanceLeaseWork,
|
||||
RENEW_STATEMENT_TIMEOUT_MS, controllerInstanceLeaseEnabled, proveControllerInstance, registerControllerInstanceDb, registerControllerInstanceLeaseWork,
|
||||
resetControllerInstanceLeaseForTests, stopControllerInstanceLease, tickControllerInstanceLease,
|
||||
} from "../services/controller-instance-lease.js";
|
||||
import { currentControllerBootId, currentControllerProcessStamp, currentProcessSpaceHash } from "../services/controller-process-identity.js";
|
||||
@@ -224,6 +224,91 @@ suite("controller instance lease: a replaced container's records are provable",
|
||||
} finally { vi.useRealTimers(); }
|
||||
});
|
||||
|
||||
it("renews on its own connection while the main pool is saturated (deploy overlap traffic switch)", async () => {
|
||||
// One main-pool connection, held by a slow statement: before the fix the renewal
|
||||
// waited behind it, timed out, and five such timeouts made a healthy server exit.
|
||||
const busy = createDb(temporary!.connectionString, { maxConnections: 1 });
|
||||
try {
|
||||
registerControllerInstanceDb(busy);
|
||||
const lost = vi.fn();
|
||||
registerControllerInstanceLeaseWork(busy, coordinator, { onLost: lost });
|
||||
currentControllerProcessStamp();
|
||||
await tickControllerInstanceLease();
|
||||
// Drizzle queries are lazy: attach a continuation so the statement starts now.
|
||||
const hold = busy.execute(sql`select pg_sleep(12)`).then(() => undefined);
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
const [before] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
|
||||
.where(eq(controllerInstances.bootId, currentControllerBootId()));
|
||||
currentControllerProcessStamp();
|
||||
const started = Date.now();
|
||||
await tickControllerInstanceLease();
|
||||
expect(Date.now() - started).toBeLessThan(5_000);
|
||||
const [after] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
|
||||
.where(eq(controllerInstances.bootId, currentControllerBootId()));
|
||||
expect(after.renewedAt!.getTime()).toBeGreaterThan(before.renewedAt!.getTime());
|
||||
expect(lost).not.toHaveBeenCalled();
|
||||
await hold;
|
||||
} finally {
|
||||
await stopControllerInstanceLease(busy);
|
||||
await busy.$client.end({ timeout: 5 });
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("has the server cancel a renewal blocked on a row lock, and renews on the same connection next time", async () => {
|
||||
// Five renewals failed 15 s apart: a client-timed-out statement kept running on the
|
||||
// single lease connection and every next renewal queued behind it.
|
||||
const lockDb = createDb(temporary!.connectionString, { maxConnections: 1 });
|
||||
// Resolving twice is harmless, so the normal path and the cleanup both release.
|
||||
let release: () => void = () => {};
|
||||
const released = new Promise<void>(resolve => { release = resolve; });
|
||||
let holder: Promise<unknown> = Promise.resolve();
|
||||
try {
|
||||
registerControllerInstanceDb(db);
|
||||
registerControllerInstanceLeaseWork(db, coordinator, { onLost: vi.fn() });
|
||||
currentControllerProcessStamp();
|
||||
await tickControllerInstanceLease();
|
||||
const lease = controllerInstanceLeaseDb(db);
|
||||
const backend = async () => (await lease.execute(sql`select pg_backend_pid() as pid`) as unknown as Array<{ pid: number }>)[0]!.pid;
|
||||
const pidBefore = await backend();
|
||||
const [before] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
|
||||
.where(eq(controllerInstances.bootId, currentControllerBootId()));
|
||||
let locked!: () => void;
|
||||
const lockTaken = new Promise<void>(resolve => { locked = resolve; });
|
||||
holder = lockDb.transaction(async tx => {
|
||||
await tx.execute(sql`select 1 from controller_instances where boot_id = ${currentControllerBootId()} for update`);
|
||||
locked();
|
||||
await released;
|
||||
});
|
||||
await lockTaken;
|
||||
currentControllerProcessStamp();
|
||||
const started = Date.now();
|
||||
await tickControllerInstanceLease();
|
||||
const elapsed = Date.now() - started;
|
||||
expect(elapsed).toBeGreaterThanOrEqual(RENEW_STATEMENT_TIMEOUT_MS - 500);
|
||||
expect(elapsed).toBeLessThan(RENEW_STATEMENT_TIMEOUT_MS + 1_500);
|
||||
// Cancelled by the server, not left running behind the client timeout.
|
||||
const running = await db.execute(sql`select count(*)::int as count from pg_stat_activity
|
||||
where application_name = 'paperclip-controller-instance-lease' and state = 'active'`) as unknown as Array<{ count: number }>;
|
||||
expect(running[0]!.count).toBe(0);
|
||||
release();
|
||||
await holder;
|
||||
currentControllerProcessStamp();
|
||||
const next = Date.now();
|
||||
await tickControllerInstanceLease();
|
||||
expect(Date.now() - next).toBeLessThan(2_000);
|
||||
const [after] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
|
||||
.where(eq(controllerInstances.bootId, currentControllerBootId()));
|
||||
expect(after.renewedAt!.getTime()).toBeGreaterThan(before.renewedAt!.getTime());
|
||||
expect(await backend()).toBe(pidBefore);
|
||||
} finally {
|
||||
// Free the controller row before the stop record writes it.
|
||||
release();
|
||||
await holder.catch(() => undefined);
|
||||
await stopControllerInstanceLease(db);
|
||||
await lockDb.$client.end({ timeout: 5 });
|
||||
}
|
||||
}, 40_000);
|
||||
|
||||
it("proves a cleanly stopped boot gone at once", async () => {
|
||||
const stopped = await otherBoot({ liveForMs: 60_000 });
|
||||
await db.update(controllerInstances).set({ stoppedAt: sql`now()` }).where(eq(controllerInstances.bootId, stopped));
|
||||
|
||||
@@ -37,7 +37,7 @@ import { currentControllerBootId, currentProcessSpaceHash } from "../services/co
|
||||
import { createOwnershipReportState, runOwnershipReconciliationPass, settleOwnableKind, type OwnershipReport } from "../services/recovery/ownership-invariant-report.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { reconcileNativeFinalizations, registerOwnershipReconciliationWork } from "../services/native-runtime/native-finalization-reconciler.js";
|
||||
import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
|
||||
import { OWNABLE_KINDS, registerNativeRestartRecheck } from "../services/recovery/ownership-registry.js";
|
||||
import { currentProcessProbe, proveOwnerGone } from "../services/owner-proof.js";
|
||||
import { createRunLogWriterAuthority } from "../services/heartbeat/run-log-authority.js";
|
||||
import { resolveHotRestartIntentPath, writeHotRestartIntent } from "../services/hot-restart.js";
|
||||
@@ -948,6 +948,82 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
|
||||
} finally { probe.restore(); unknown?.mockRestore(); }
|
||||
}, 60_000);
|
||||
|
||||
it("ends an unverifiable run on the first look once its controller's lease lapsed longer ago than the bound", async () => {
|
||||
// A successor that boots (or first scans) minutes after the old server left
|
||||
// must not restart the 2-minute wait from its own first observation.
|
||||
const f = await seedClaim("CLAIM-SPACE-LAPSED-LONG-AGO", "expired", "foreign-process-space");
|
||||
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 3 * 60_000) })
|
||||
.where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const probe = observeOwnerPid(), onClaim = vi.fn();
|
||||
try {
|
||||
const dispositions = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim });
|
||||
expect(dispositions).toEqual([{ kind: "blocked", runId: f.runId, reason: "controller_process_space_unverified" }]);
|
||||
expect(probe.observed).toEqual([]);
|
||||
expect(onClaim).not.toHaveBeenCalled();
|
||||
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
|
||||
.toMatchObject({ status: "failed", errorCode: "native_restart_recovery_blocked" });
|
||||
} finally { probe.restore(); }
|
||||
}, 60_000);
|
||||
|
||||
it("ends at once a run whose controller boot is fenced, even with its 20-minute claim lease unexpired (N1)", async () => {
|
||||
// N1 cell: old server SIGKILLed mid-overlap at 22:09:31; its boot was fenced at 22:13:02, and the
|
||||
// run still waited a further 2 minutes from that first look. A fenced boot already outlived its lease.
|
||||
const f = await seedClaim("CLAIM-SPACE-FENCED-BOOT", "live", "foreign-process-space");
|
||||
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
await f.db.insert(controllerInstances).values({ bootId: owner!.controllerBootId!, processSpace: "foreign-process-space",
|
||||
pid: priorPid, leaseExpiresAt: new Date(Date.now() - 3 * 60_000), fencedAt: new Date(Date.now() - 10_000) });
|
||||
registerControllerInstanceDb(f.db);
|
||||
const probe = observeOwnerPid();
|
||||
try {
|
||||
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() }))
|
||||
.toEqual([{ kind: "blocked", runId: f.runId, reason: "controller_process_space_unverified" }]);
|
||||
} finally { probe.restore(); resetControllerInstanceLeaseForTests(); }
|
||||
}, 60_000);
|
||||
|
||||
it("keeps waiting for a controller whose boot is still live, whatever the run's own lease says", async () => {
|
||||
const f = await seedClaim("CLAIM-SPACE-LIVE-BOOT", "expired", "foreign-process-space");
|
||||
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
await f.db.insert(controllerInstances).values({ bootId: owner!.controllerBootId!, processSpace: "foreign-process-space",
|
||||
pid: priorPid, leaseExpiresAt: new Date(Date.now() + 60_000) });
|
||||
registerControllerInstanceDb(f.db);
|
||||
const probe = observeOwnerPid();
|
||||
try {
|
||||
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() }))
|
||||
.toMatchObject([{ kind: "awaiting_evidence", runId: f.runId }]);
|
||||
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running");
|
||||
} finally { probe.restore(); resetControllerInstanceLeaseForTests(); }
|
||||
}, 60_000);
|
||||
|
||||
it("never moves the owner's last sign of life when a successor records its wait (fence time unchanged)", async () => {
|
||||
// N1: the successor's awaiting-evidence write bumped updated_at, which the prover reads as the dead
|
||||
// owner's last write, and the fence slipped from about 2 minutes to 3.5 after the crash.
|
||||
const f = await seedClaim("CLAIM-SPACE-OWNER-TOUCH", "expired", "foreign-process-space");
|
||||
const ownerLastWrite = new Date(Date.now() - 5 * 60_000);
|
||||
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 60_000), updatedAt: ownerLastWrite })
|
||||
.where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const probe = observeOwnerPid();
|
||||
try {
|
||||
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() }))
|
||||
.toMatchObject([{ kind: "awaiting_evidence", reason: "controller_process_space_unverified" }]);
|
||||
const [after] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
expect(after!.recoveryState).toBe("awaiting_evidence");
|
||||
expect(after!.updatedAt.getTime()).toBe(ownerLastWrite.getTime());
|
||||
} finally { probe.restore(); }
|
||||
}, 60_000);
|
||||
|
||||
it("schedules the recheck for the end of the bound measured from the lease lapse", async () => {
|
||||
const f = await seedClaim("CLAIM-SPACE-LAPSED-RECENTLY", "expired", "foreign-process-space");
|
||||
const lapsedAt = Date.now() - 30_000;
|
||||
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(lapsedAt) }).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const probe = observeOwnerPid();
|
||||
try {
|
||||
const [disposition] = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() });
|
||||
expect(disposition).toMatchObject({ kind: "awaiting_evidence", reason: "controller_process_space_unverified" });
|
||||
const retryAt = (disposition as { retryAt: Date }).retryAt.getTime();
|
||||
expect(Math.abs(retryAt - (lapsedAt + 2 * 60_000))).toBeLessThan(5_000);
|
||||
} finally { probe.restore(); }
|
||||
}, 60_000);
|
||||
|
||||
it("keeps a real controller-only attestation from authorizing an in-flight claim", async () => {
|
||||
const f = await seedClaim("CLAIM-CONTROLLER-ONLY", "expired", "foreign-process-space");
|
||||
const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
@@ -1686,6 +1762,35 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
|
||||
expect(second.violations.filter(violation => violation.kind === "native_accepted_result_lease")).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
it("hands a run awaiting evidence to restart recovery once the reconciler proves its controller gone (crash mid-overlap)", async () => {
|
||||
// A crashed old server's run was classified by the new server's startup pass while the
|
||||
// old server lived, so no restart re-check was scheduled for it. Only the reconciler
|
||||
// sees the controller die; it must hand the run to restart recovery on the live server.
|
||||
const f = await seedRecoverableRun("AWAITING-CRASHED-CONTROLLER");
|
||||
await f.db.update(nativeRunFinalizations).set({ recoveryState: "awaiting_evidence",
|
||||
leaseOwner: `old-controller:${randomUUID()}`, leaseExpiresAt: new Date(Date.now() + 15 * 60_000),
|
||||
controllerBootId: randomUUID(), controllerPid: 2_000_000_001, controllerHostHash: currentProcessSpaceHash(),
|
||||
controllerProcessStartedAt: new Date(Date.now() - 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const kind = OWNABLE_KINDS.find(candidate => candidate.id === "native_awaiting_evidence")!;
|
||||
const kinds = [{ ...kind, listInProgress: async (db: typeof f.db) => ({ records: await kind.listForRuns!(db, [f.runId]), truncated: false }) }];
|
||||
const recheck = vi.fn(async () => undefined);
|
||||
const unregister = registerNativeRestartRecheck(recheck);
|
||||
const original = process.kill.bind(process);
|
||||
const kill = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
|
||||
if (pid === 2_000_000_001 && signal === 0) throw Object.assign(new Error("gone"), { code: "ESRCH" });
|
||||
return original(pid, signal);
|
||||
});
|
||||
try {
|
||||
const state = createOwnershipReportState();
|
||||
const first = await runOwnershipReconciliationPass(f.db, { kinds, state });
|
||||
expect(first.kinds[0]).toMatchObject({ verdicts: { gone: 1 } });
|
||||
expect(recheck).toHaveBeenCalledWith([f.runId]);
|
||||
// The next 30 s pass does not hand it over again; restart recovery schedules its own re-checks.
|
||||
await runOwnershipReconciliationPass(f.db, { kinds, state });
|
||||
expect(recheck).toHaveBeenCalledTimes(1);
|
||||
} finally { kill.mockRestore(); unregister(); }
|
||||
}, 60_000);
|
||||
|
||||
it("keeps a live earlier-boot controller's result lease in the reconciler pass", async () => {
|
||||
const f = await seedAcceptedResult("ACCEPTED-RECONCILER-LIVE", (runner) => `${runner}:${randomUUID()}`);
|
||||
const self = await currentNativeControllerIdentity();
|
||||
|
||||
@@ -161,7 +161,7 @@ describe("report-only ownership reconciliation", () => {
|
||||
expect(new Set(names).size).toBe(names.length);
|
||||
expect(names.length).toBeGreaterThanOrEqual(40);
|
||||
expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
|
||||
.toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "remote_workspace_unsaved_files", "remote_snapshot_ledger", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "native_completed_turn_cleanup_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
|
||||
.toEqual(["native_executor_claim", "native_restart_recovery_claim", "native_awaiting_evidence", "native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "remote_workspace_unsaved_files", "remote_snapshot_ledger", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "native_completed_turn_cleanup_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
|
||||
expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
|
||||
});
|
||||
|
||||
|
||||
@@ -92,6 +92,7 @@ import { registerNativeCleanupReleaseWake } from "./services/native-runtime/nati
|
||||
import { logNativeCleanupQuarantines } from "./services/native-runtime/native-cleanup-quarantine-log.js";
|
||||
import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js";
|
||||
import { deliverExecutionStatuses } from "./services/execution-status-delivery.js";
|
||||
import { registerNativeRestartRecheck } from "./services/recovery/ownership-registry.js";
|
||||
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js";
|
||||
import { deliverPendingPauseResumes } from "./services/issue-tree-resume.js";
|
||||
import { deliverBudgetResumeContinuations, type BudgetResumeDelivery } from "./services/budget-resume-continuation.js";
|
||||
@@ -962,6 +963,9 @@ async function startServerWithDatabaseTeardown(
|
||||
? heartbeatService(db as any, { pluginWorkerManager })
|
||||
: null;
|
||||
const accountingHeartbeat = heartbeat ?? heartbeatService(db as any, { pluginWorkerManager });
|
||||
// A native run whose controller the ownership reconciler proves gone is re-checked here,
|
||||
// so a crashed old server's runs end or resume without another server boot.
|
||||
if (heartbeat) registerNativeRestartRecheck(runIds => heartbeat.recoverNativeRunsAfterRestart(new Date(), runIds));
|
||||
const decisionServiceOptions = {
|
||||
wakeOriginAgent: createDecisionWakeOriginAgent(heartbeat?.wakeup ?? null),
|
||||
};
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { and, eq, isNull, sql } from "drizzle-orm";
|
||||
import { controllerInstances, type Db } from "@paperclipai/db";
|
||||
import { controllerInstanceLeaseDb, controllerInstanceLeaseDbWithoutStatementTimeout, controllerInstances, type Db } from "@paperclipai/db";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import {
|
||||
currentControllerBootId, currentProcessSpaceHash, onControllerStamp,
|
||||
@@ -20,6 +20,8 @@ const STAMP_COMMIT_GRACE_MS = CONTROLLER_LEASE_TTL_MS;
|
||||
const SETTLE_RECHECK_MIN_MS = 5_000;
|
||||
/** A renewal that has not answered within this bound counts as failed. */
|
||||
const RENEW_TIMEOUT_MS = 10_000;
|
||||
/** Server-side cancel for a renewal statement, below the client timeout. */
|
||||
export const RENEW_STATEMENT_TIMEOUT_MS = 8_000;
|
||||
const RETRY_MS = 5_000;
|
||||
|
||||
export type ControllerInstanceState = "live" | "grace" | "expired" | "fenced" | "stopped";
|
||||
@@ -95,14 +97,29 @@ function renewInDatabase(db: Db): (stampedRecently: boolean) => Promise<RenewRes
|
||||
const records = holdsStampedWorkSql(bootId);
|
||||
const holds = sql`(${stampedRecently} or ${records})`;
|
||||
const expires = sql`now() + ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval`;
|
||||
const rows = await db.execute(sql`
|
||||
// Its own connection: a busy main pool (request bursts at a traffic switch,
|
||||
// a replacement's startup pass) must not delay the proof that this boot is
|
||||
// alive. One statement per renewal, so an idle boot's last check stays a
|
||||
// single round trip. The connection's own statement_timeout (below the
|
||||
// client timeout) makes the server cancel a slow renewal, so it never holds
|
||||
// the connection the next renewal needs.
|
||||
const renew = (lease: Db) => lease.execute(sql`
|
||||
insert into controller_instances (boot_id, process_space, pid, lease_expires_at)
|
||||
values (${bootId}, ${currentProcessSpaceHash()}, ${process.pid}, ${expires})
|
||||
on conflict (boot_id) do update set
|
||||
lease_expires_at = case when ${holds} then ${expires} else controller_instances.lease_expires_at end,
|
||||
renewed_at = case when ${holds} then now() else controller_instances.renewed_at end
|
||||
where controller_instances.fenced_at is null and controller_instances.stopped_at is null
|
||||
returning epoch, ${holds} as holds, ${records} as records`) as unknown as Array<{ epoch: string | number; holds: boolean; records: boolean }>;
|
||||
returning epoch, ${holds} as holds, ${records} as records`) as unknown as Promise<Array<{ epoch: string | number; holds: boolean; records: boolean }>>;
|
||||
let rows: Awaited<ReturnType<typeof renew>>;
|
||||
try {
|
||||
rows = await renew(controllerInstanceLeaseDb(db, { statementTimeoutMs: RENEW_STATEMENT_TIMEOUT_MS }));
|
||||
} catch (error) {
|
||||
// A connection pooler may refuse the startup parameter: renew without it.
|
||||
if (!/unsupported startup parameter/i.test(error instanceof Error ? error.message : String(error))) throw error;
|
||||
logger.warn({ err: error }, "controller instance lease connection refused statement_timeout; renewing without it");
|
||||
rows = await renew(controllerInstanceLeaseDbWithoutStatementTimeout(db));
|
||||
}
|
||||
const row = rows[0];
|
||||
if (!row) return "fenced";
|
||||
if (state) state.epoch = Number(row.epoch);
|
||||
|
||||
@@ -1433,11 +1433,29 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
// old run's writes revoked and a new workspace root.
|
||||
// The same holds when the previous controller's host was never recorded
|
||||
// or cannot be proven from here: confirming its stop is optional.
|
||||
// The run's own server boot is proven stopped or fenced (controller
|
||||
// instance record): its lease is no sign of life, however long it was
|
||||
// granted. The claim lease is 20 minutes for a local controller.
|
||||
const ownerControllerStopped = evaluatedTakeover.allowed
|
||||
&& ["controller_stopped", "controller_fenced"].includes(evaluatedTakeover.reason);
|
||||
const leaseLapsedAt = row.coordinator.leaseExpiresAt === null ? null
|
||||
: row.coordinator.leaseExpiresAt <= now ? row.coordinator.leaseExpiresAt.getTime() : null;
|
||||
const unverifiableLapsedController = !takeover.allowed && !nonExecutionOwner
|
||||
&& (row.coordinator.leaseExpiresAt === null || row.coordinator.leaseExpiresAt <= now)
|
||||
&& (row.coordinator.leaseExpiresAt === null || leaseLapsedAt !== null || ownerControllerStopped)
|
||||
&& (namespaceAttentionNeeded || ["controller_host_unrecorded", "controller_host_unprovable"].includes(takeover.reason));
|
||||
let unverifiedWaitEndsAt: number | null = null;
|
||||
if (unverifiableLapsedController) {
|
||||
const waitingSince = namespaceWaitStartedAt(row.coordinator.recoveryHistory, takeover.reason);
|
||||
// The bounded wait starts when the run's controller went away (its lease
|
||||
// lapsed), not when a later server first looked: a successor that boots
|
||||
// minutes later, or re-records the wait, must not restart the clock.
|
||||
const observedSince = namespaceWaitStartedAt(row.coordinator.recoveryHistory, takeover.reason);
|
||||
const starts = [observedSince, leaseLapsedAt].filter((at): at is number => at !== null);
|
||||
// A controller boot proven stopped or fenced already outlived its lease
|
||||
// (90 s) and the fence grace (30 s): waiting longer proves nothing more
|
||||
// about a Runner in another process space, so the run ends now.
|
||||
const waitingSince = ownerControllerStopped ? now.getTime() - NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS
|
||||
: starts.length ? Math.min(...starts) : null;
|
||||
unverifiedWaitEndsAt = (waitingSince ?? now.getTime()) + NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS;
|
||||
if (waitingSince !== null && now.getTime() - waitingSince >= NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS) {
|
||||
await endNamespaceUnverifiedRun(tx as unknown as Db, row, now, takeover.reason, historyEntry({
|
||||
now, restartKind: input.restartKind, controller, generation: row.coordinator.controllerGeneration,
|
||||
@@ -1480,8 +1498,12 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
// Repeated scans of one ownership wait record it once, so the
|
||||
// bounded history keeps the drain record the next claim reads.
|
||||
...(waitAlreadyRecorded ? {} : { recoveryHistory: appendBoundedRecoveryHistory(event) }),
|
||||
updatedAt: waitAlreadyRecorded && row.coordinator.recoveryState === "awaiting_evidence"
|
||||
&& row.coordinator.recoveryRequestId === (input.recoveryRequestId ?? null) ? row.coordinator.updatedAt : now,
|
||||
// updated_at is the owner's last sign of life: the ownership prover
|
||||
// fences a dead boot only once it has been quiet past that write plus
|
||||
// the lease and grace, and the fallback reads it as the owner's
|
||||
// progress. A successor recording its own wait never moves it.
|
||||
updatedAt: !ownControllerProcess || (waitAlreadyRecorded && row.coordinator.recoveryState === "awaiting_evidence"
|
||||
&& row.coordinator.recoveryRequestId === (input.recoveryRequestId ?? null)) ? row.coordinator.updatedAt : now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
@@ -1502,7 +1524,8 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
: ["controller_lease_held", "controller_lease_grace"].includes(takeover.reason)
|
||||
? { retryAt: new Date(now.getTime() + CONTROLLER_LEASE_RENEW_MS) }
|
||||
// The bounded wait for an unverifiable controller is checked again when it ends.
|
||||
: unverifiableLapsedController ? { retryAt: new Date(now.getTime() + NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS) } : {}),
|
||||
: unverifiableLapsedController ? { retryAt: new Date(Math.max(now.getTime() + 1_000,
|
||||
unverifiedWaitEndsAt ?? now.getTime() + NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS)) } : {}),
|
||||
} as const;
|
||||
}
|
||||
|
||||
|
||||
+12
-7
@@ -2478,18 +2478,23 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
const stopped = await hasNativeLocalProcessStop(fixture.db, companyId, fixture.runId);
|
||||
expect(probes).toEqual([]);
|
||||
// The namespace wait is bounded: it is checked again when the wait ends.
|
||||
expect(dispositions).toEqual([{ kind: "awaiting_evidence", runId: fixture.runId,
|
||||
reason: "controller_process_space_unverified", retryAt: expect.any(Date) }]);
|
||||
// A stopped or fenced controller boot already outlived its lease and the
|
||||
// fence grace, so the namespace wait ends on the first look: the run ends
|
||||
// as unverified (never as a proven stop) and the task continues in a fresh
|
||||
// run. Nothing local is probed, claimed or recorded as stopped.
|
||||
expect(dispositions).toEqual([{ kind: "blocked", runId: fixture.runId,
|
||||
reason: "controller_process_space_unverified" }]);
|
||||
expect(claimed).not.toHaveBeenCalled();
|
||||
expect(stamped).not.toHaveBeenCalled();
|
||||
expect(stopped).toBe(false);
|
||||
expect(after).toMatchObject({ phase: before.phase, attempt: before.attempt,
|
||||
expect(after).toMatchObject({ phase: "terminal_failure", attempt: before.attempt,
|
||||
controllerGeneration: before.controllerGeneration, controllerBootId: before.controllerBootId,
|
||||
controllerPid: before.controllerPid, controllerHostHash: before.controllerHostHash,
|
||||
leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, resultId: null });
|
||||
expect(run).toMatchObject({ status: "running", processPid: runnerPid, processGroupId: runnerGroup,
|
||||
processStartedAt: startedAt, processHostHash: foreignSpace });
|
||||
failureCode: "native_restart_recovery_blocked",
|
||||
failureDetail: expect.objectContaining({ reason: "controller_process_space_unverified", executorStopEvidence: "unverified" }),
|
||||
resultId: null });
|
||||
expect(run).toMatchObject({ status: "failed", errorCode: "native_restart_recovery_blocked",
|
||||
processPid: runnerPid, processGroupId: runnerGroup, processStartedAt: startedAt, processHostHash: foreignSpace });
|
||||
expect(run.runnerProfileJson).toEqual({ ...fixture.preparedProfile, sessionCheckpoint: {
|
||||
identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId,
|
||||
sessionId: fixture.native.normalizedSessionId },
|
||||
|
||||
@@ -275,6 +275,37 @@ function nrfKind(id: string, title: string, obligation: Obligation, where: SQL |
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A native run whose controller died after another server's startup pass
|
||||
* (a crash mid-overlap, not a graceful drain) has no server re-checking it:
|
||||
* restart recovery only schedules re-checks for runs it saw. When the
|
||||
* reconciler proves such a run's owner gone, this server runs the same restart
|
||||
* recovery for that run now; its bounded wait and re-check schedule then end
|
||||
* or resume the run without waiting for another server to boot.
|
||||
*/
|
||||
let nativeRestartRecheck: ((runIds: string[]) => Promise<unknown>) | null = null;
|
||||
export function registerNativeRestartRecheck(recheck: (runIds: string[]) => Promise<unknown>) {
|
||||
nativeRestartRecheck = recheck;
|
||||
return () => { if (nativeRestartRecheck === recheck) nativeRestartRecheck = null; };
|
||||
}
|
||||
/** The reconciler re-checks every 30 s; restart recovery schedules its own
|
||||
* follow-ups, so one hand-off per run per minute is enough. */
|
||||
const NATIVE_RESTART_RECHECK_SPACING_MS = 60_000;
|
||||
const lastNativeRestartRecheck = new Map<string, number>();
|
||||
const recheckNativeRunOwnership: SettleRecord = async (_db, record, context) => {
|
||||
if (!nativeRestartRecheck || !record.runId) return { settled: false, reason: "restart_recheck_unavailable" };
|
||||
const last = lastNativeRestartRecheck.get(record.runId);
|
||||
if (last !== undefined && context.now.getTime() - last < NATIVE_RESTART_RECHECK_SPACING_MS)
|
||||
return { settled: false, reason: "restart_recovery_recheck_recent" };
|
||||
if (lastNativeRestartRecheck.size > 10_000) lastNativeRestartRecheck.clear();
|
||||
lastNativeRestartRecheck.set(record.runId, context.now.getTime());
|
||||
await nativeRestartRecheck([record.runId]);
|
||||
return { settled: false, reason: "restart_recovery_rechecked" };
|
||||
};
|
||||
function withNativeRestartRecheck(kind: OwnableKind): OwnableKind {
|
||||
return { ...kind, settle: recheckNativeRunOwnership };
|
||||
}
|
||||
|
||||
const notFinalizerOrCleanup = and(notLike(nativeRunFinalizations.leaseOwner, "native-finalizer:%"),
|
||||
notLike(nativeRunFinalizations.leaseOwner, "native-cleanup:%"));
|
||||
|
||||
@@ -682,13 +713,13 @@ const acceptedNativeResultLease = nrfKind("native_accepted_result_lease", "Accep
|
||||
|
||||
export const OWNABLE_KINDS: readonly OwnableKind[] = [
|
||||
// Native runtime and Runner
|
||||
nrfKind("native_executor_claim", "Executor execution claim (fresh run)", "ownership",
|
||||
withNativeRestartRecheck(nrfKind("native_executor_claim", "Executor execution claim (fresh run)", "ownership",
|
||||
and(eq(nativeRunFinalizations.phase, "observed"), isNull(nativeRunFinalizations.resultId),
|
||||
isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup, isNull(nativeRunFinalizations.recoveryState))),
|
||||
nrfKind("native_restart_recovery_claim", "Restart-recovery claim", "ownership",
|
||||
inArray(nativeRunFinalizations.recoveryState, ["awaiting_runner_reattach", "resuming_session", "bootstrap_incomplete"])),
|
||||
nrfKind("native_awaiting_evidence", "\"Awaiting evidence\" wait", "ownership",
|
||||
eq(nativeRunFinalizations.recoveryState, "awaiting_evidence"), undefined, [1, 2, 7]),
|
||||
isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup, isNull(nativeRunFinalizations.recoveryState)))),
|
||||
withNativeRestartRecheck(nrfKind("native_restart_recovery_claim", "Restart-recovery claim", "ownership",
|
||||
inArray(nativeRunFinalizations.recoveryState, ["awaiting_runner_reattach", "resuming_session", "bootstrap_incomplete"]))),
|
||||
withNativeRestartRecheck(nrfKind("native_awaiting_evidence", "\"Awaiting evidence\" wait", "ownership",
|
||||
eq(nativeRunFinalizations.recoveryState, "awaiting_evidence"), undefined, [1, 2, 7])),
|
||||
{
|
||||
// Shared path since Phase 2: a hard kill after `run.result` is accepted
|
||||
// leaves the result leased to the dead controller; settle expires that
|
||||
|
||||
Reference in new issue
Block a user