fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap runs two servers for a short overlap. The old server
drains, and the new server takes over the runs.
> - Each server proves that it is alive with a controller-instance
lease. Recovery uses that lease to decide whether a run's controller is
gone.
> - In local swap cells, a healthy old server exited during the overlap,
because its lease renewal waited for a connection on the busy main pool.
Every run it drove was interrupted.
> - When a controller crashed, its runs waited for another server boot,
or for a timer that restarted on every look. Tasks stood still for 21 to
28 minutes.
> - This pull request renews the lease on its own connection, and lets
the live server end a crashed controller's runs within a bounded time.
> - The benefit is that a swap does not kill a healthy server, and a
crashed server's runs end in about 2 minutes, so their tasks continue by
themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. During a two-server release swap (local cells, load 14 to 34), the
outgoing server exited with "controller instance lease lost while
holding work". Five lease renewals in a row timed out after 10 s. The 10
s timeout included the wait for a connection on the shared main pool,
which the traffic switch and the replacement's startup pass kept busy.
2. When the old server was killed in the middle of the overlap, its runs
stayed in progress for 21 to 28 minutes. The live server did not
re-check a crashed controller's runs: only a new boot did. The
unverified-controller wait started when a server first looked at the
run, not when the controller left, and the successor's own wait record
counted as the dead owner's sign of life.

**Expected behavior**
A healthy server keeps its lease during a swap. When a controller is
gone (its lease lapsed and it is fenced or stopped), the live server
ends that controller's runs within a bounded time, without another
server boot, so restart recovery continues the tasks.

**Steps to reproduce**
1. Run two servers against one database with a release swap and a 200 s
overlap.
2. Kill the old server 40 s into the overlap (SIGKILL).
3. See that its runs stay in progress for more than 20 minutes, until
another server starts.

## What Changed

- Lease renewal on its own connection (`packages/db/src/client.ts`,
`packages/db/src/index.ts`,
`server/src/services/controller-instance-lease.ts`):
- New `controllerInstanceLeaseDb(db)`: one private 1-connection pool
(`application_name` `paperclip-controller-instance-lease`), made the
same way as the existing `controllerLeaseDb` and closed by
`closeLivenessDb`.
  - The renewal is one statement on that connection, as before.
- The connection sends `statement_timeout` = 8 s (below the 10 s client
timeout) as a startup parameter (new
`DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The
server cancels a slow renewal, so it never holds the single connection
that the next renewal needs, and there is no extra round trip per
renewal. (An earlier revision set it per transaction; that added three
round trips to every renewal and broke the idle-server receipt test.)
- If a connection pooler refuses the startup parameter ("unsupported
startup parameter"), the lease logs a warning and renews on a
replacement connection without it.
- A crashed controller's runs
(`server/src/services/recovery/ownership-registry.ts`,
`server/src/index.ts`,
`server/src/services/native-runtime/native-restart-recovery.ts`):
- The ownership reconciler on the live server hands runs whose
controller is proven gone to restart recovery. It does not wait for
another server boot.
- The unverified-controller wait is bounded from the moment the
controller's lease lapsed, not from the first look.
- When the controller boot is `stopped` or `fenced`, the run ends at
once.
- A successor's wait record no longer refreshes the dead owner's sign of
life.

Safety of ending a run when its controller boot is fenced or stopped:
- Only `stopped` (written by the boot's own graceful stop) and `fenced`
(a newer epoch fenced it after its lease expired plus 30 s grace) count.
`live` and `grace` never count, so a controller that is alive and
renewing is never treated as gone. `expired` alone does not count
either.
- A boot that cannot renew stops itself at its self-fence window (about
its lease expiry), and a boot that finds itself fenced exits at once. So
when another server can fence it, the old process has already stopped
driving runs. The lease change above removes the main cause of healthy
boots failing to renew.
- If the old process is still alive (partition or stall) and its Runner
continues: the old run is fenced. Its agent token is refused for writes
(`agent_run_superseded`), its remote workspace epoch is abandoned or the
continuation gets a new workspace, and the native coordinator records
`replacementDenied`. There are never two writers in one workspace.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/controller-instance-lease.test.ts` "own
connection": red (10004 ms timeout while the main pool is held) to
green. Full file green (26).
- `server/src/__tests__/controller-instance-lease.test.ts` "has the
server cancel a renewal blocked on a row lock, and renews on the same
connection next time": red without the connection-level timeout (the
renewal runs to the 10 s client timeout and stays active), green with it
(cancelled at about 8 s, no active lease statement left, the next
renewal succeeds in under 2 s on the same backend).
- `packages/db/src/client-options.test.ts`: the statement timeout is
sent as a startup parameter.
- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`:
"lapsed longer ago" / "end of the bound" (wait counted from the lease
lapse), the live-server re-check of a crashed controller, the
fenced/stopped boot ending the run at once, and the successor's wait
record. Full file green.

Changed existing expectation (it asserted the old wait for a stopped or
fenced controller):
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
"keeps foreign local Runner and provider ownership after its controller
is stopped/fenced": the disposition is now `blocked`
(`controller_process_space_unverified`) instead of `awaiting_evidence`,
the run is `failed` with `native_restart_recovery_blocked`, and the
finalization is `terminal_failure` with `executorStopEvidence:
"unverified"`. The safety checks stay: no local PID probe, no claim, no
stamp, no local stop proof, the run keeps its process identity and
session checkpoint, and no result is written.

Local commands, on this branch based on current master:
- `cd server && npx vitest run
src/__tests__/controller-instance-lease.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140
tests passed.
- `npx vitest run src/__tests__/controller-instance-lease.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
225 passed, 14 skipped (real-process cases that need CI).
- `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts
-t "releases an ordinary held task Retry"`: passed (idle receipt: one
lease statement after the work ends).

Local swap cells (two servers, embedded Postgres, real agents, old
server SIGKILLed 40 s into a 200 s overlap):
- Before: runs of the killed server stayed in progress 21 to 28 minutes.
- N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the
successor's wait write still moved the timer; fixed by the last commit).
- R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run
end 3 min 39 s.
- N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min
01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the
lease TTL (90 s) plus the fence grace (30 s).
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds this fix and the related recovery fixes: 11/11 tasks finished
hands-off (master: 9/11). The lease and crashed-controller fixes were
visibly working in it.

## Risks

- A pooled database URL whose pooler refuses `statement_timeout` as a
startup parameter: the first renewal fails over to a connection without
it (logged), so the lease keeps working without the server-side cancel.
- One extra database connection per server, held while the server runs.
It is idle except for one renewal every few seconds.
- A run whose controller boot is fenced or stopped now ends at once
instead of after the bound. See the safety notes above: only a boot that
is not alive can be in these states.
- No migration. No schema change. Nothing changes before `turn.start` on
the normal path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 20:11:27 -06:00
1 parent 52817dff55
commit 500bc694b5
11 files changed
+348 -29

No files matched your search

+6
View File
@@ -93,6 +93,12 @@ describe("databaseClientOptionsFromEnv", () => {
connection: { application_name: "paperclip-web" },
});
});
it("sends a dedicated pool's statement timeout as a startup parameter", () => {
expect(postgresJsOptions({ applicationName: "paperclip-controller-instance-lease", statementTimeoutMs: 8_000 })).toEqual({
connection: { application_name: "paperclip-controller-instance-lease", statement_timeout: "8000" },
});
});
});
describe("resolveDatabaseClientOptions", () => {
+45 -4
View File
@@ -163,6 +163,11 @@ export interface DatabaseClientOptions {
* `postgres.js`).
*/
applicationName?: string;
/**
* Connection-level `statement_timeout` in milliseconds, sent as a startup
* parameter (no extra round trip). For dedicated single-purpose pools only.
*/
statementTimeoutMs?: number;
}
/**
@@ -256,9 +261,10 @@ export function postgresJsOptions(options: DatabaseClientOptions): Record<string
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
if (options.applicationName !== undefined) {
driverOptions.connection = { application_name: options.applicationName };
}
const connection: Record<string, string> = {};
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
if (Object.keys(connection).length) driverOptions.connection = connection;
return driverOptions;
}
@@ -324,11 +330,42 @@ export function controllerLeaseDb(db: Db): Db {
return created;
}
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
/** One private connection for the server's controller-instance lease, so its
* renewal never waits behind request or worker traffic on the main pool.
* `statementTimeoutMs` (first call wins) is a connection-level
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
* single connection that the next renewal needs, at no extra round trip. */
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
const existing = controllerInstanceLeaseDbs.get(db);
if (existing) return existing;
const factory = controllerInstanceLeaseDbFactories.get(db);
if (!factory) return db;
const created = factory(options.statementTimeoutMs ?? null);
controllerInstanceLeaseDbs.set(db, created);
return created;
}
/** Replaces the lease connection with one that sends no `statement_timeout`
* startup parameter, for a connection pooler that refuses it. */
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
const factory = controllerInstanceLeaseDbFactories.get(db);
if (!factory) return db;
const previous = controllerInstanceLeaseDbs.get(db);
const created = factory(null);
controllerInstanceLeaseDbs.set(db, created);
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
return created;
}
/** Ends the liveness pool if one was created. The ended pool stays cached, so
* a late renewal fails like a query on the ended primary pool. */
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
}
// Control checkpoints must not occupy the request or lease-renewal pools
@@ -406,6 +443,10 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: 1, applicationName: "paperclip-legacy-controller-lease",
}, db));
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
...resolved, maxConnections: 1, applicationName: "paperclip-controller-instance-lease",
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
}, db));
checkpointDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint",
}, db));
+2
View File
@@ -5,6 +5,8 @@ export {
type OwnedDatabaseListener,
livenessDb,
controllerLeaseDb,
controllerInstanceLeaseDb,
controllerInstanceLeaseDbWithoutStatementTimeout,
closeLivenessDb,
checkpointDb,
closeCheckpointDb,
@@ -1,10 +1,10 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, controllerInstances, createDb, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
import { agents, companies, controllerInstanceLeaseDb, controllerInstances, createDb, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import {
controllerInstanceLeaseEnabled, proveControllerInstance, registerControllerInstanceDb, registerControllerInstanceLeaseWork,
RENEW_STATEMENT_TIMEOUT_MS, controllerInstanceLeaseEnabled, proveControllerInstance, registerControllerInstanceDb, registerControllerInstanceLeaseWork,
resetControllerInstanceLeaseForTests, stopControllerInstanceLease, tickControllerInstanceLease,
} from "../services/controller-instance-lease.js";
import { currentControllerBootId, currentControllerProcessStamp, currentProcessSpaceHash } from "../services/controller-process-identity.js";
@@ -224,6 +224,91 @@ suite("controller instance lease: a replaced container's records are provable",
} finally { vi.useRealTimers(); }
});
it("renews on its own connection while the main pool is saturated (deploy overlap traffic switch)", async () => {
// One main-pool connection, held by a slow statement: before the fix the renewal
// waited behind it, timed out, and five such timeouts made a healthy server exit.
const busy = createDb(temporary!.connectionString, { maxConnections: 1 });
try {
registerControllerInstanceDb(busy);
const lost = vi.fn();
registerControllerInstanceLeaseWork(busy, coordinator, { onLost: lost });
currentControllerProcessStamp();
await tickControllerInstanceLease();
// Drizzle queries are lazy: attach a continuation so the statement starts now.
const hold = busy.execute(sql`select pg_sleep(12)`).then(() => undefined);
await new Promise(resolve => setTimeout(resolve, 200));
const [before] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
.where(eq(controllerInstances.bootId, currentControllerBootId()));
currentControllerProcessStamp();
const started = Date.now();
await tickControllerInstanceLease();
expect(Date.now() - started).toBeLessThan(5_000);
const [after] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
.where(eq(controllerInstances.bootId, currentControllerBootId()));
expect(after.renewedAt!.getTime()).toBeGreaterThan(before.renewedAt!.getTime());
expect(lost).not.toHaveBeenCalled();
await hold;
} finally {
await stopControllerInstanceLease(busy);
await busy.$client.end({ timeout: 5 });
}
}, 30_000);
it("has the server cancel a renewal blocked on a row lock, and renews on the same connection next time", async () => {
// Five renewals failed 15 s apart: a client-timed-out statement kept running on the
// single lease connection and every next renewal queued behind it.
const lockDb = createDb(temporary!.connectionString, { maxConnections: 1 });
// Resolving twice is harmless, so the normal path and the cleanup both release.
let release: () => void = () => {};
const released = new Promise<void>(resolve => { release = resolve; });
let holder: Promise<unknown> = Promise.resolve();
try {
registerControllerInstanceDb(db);
registerControllerInstanceLeaseWork(db, coordinator, { onLost: vi.fn() });
currentControllerProcessStamp();
await tickControllerInstanceLease();
const lease = controllerInstanceLeaseDb(db);
const backend = async () => (await lease.execute(sql`select pg_backend_pid() as pid`) as unknown as Array<{ pid: number }>)[0]!.pid;
const pidBefore = await backend();
const [before] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
.where(eq(controllerInstances.bootId, currentControllerBootId()));
let locked!: () => void;
const lockTaken = new Promise<void>(resolve => { locked = resolve; });
holder = lockDb.transaction(async tx => {
await tx.execute(sql`select 1 from controller_instances where boot_id = ${currentControllerBootId()} for update`);
locked();
await released;
});
await lockTaken;
currentControllerProcessStamp();
const started = Date.now();
await tickControllerInstanceLease();
const elapsed = Date.now() - started;
expect(elapsed).toBeGreaterThanOrEqual(RENEW_STATEMENT_TIMEOUT_MS - 500);
expect(elapsed).toBeLessThan(RENEW_STATEMENT_TIMEOUT_MS + 1_500);
// Cancelled by the server, not left running behind the client timeout.
const running = await db.execute(sql`select count(*)::int as count from pg_stat_activity
where application_name = 'paperclip-controller-instance-lease' and state = 'active'`) as unknown as Array<{ count: number }>;
expect(running[0]!.count).toBe(0);
release();
await holder;
currentControllerProcessStamp();
const next = Date.now();
await tickControllerInstanceLease();
expect(Date.now() - next).toBeLessThan(2_000);
const [after] = await db.select({ renewedAt: controllerInstances.renewedAt }).from(controllerInstances)
.where(eq(controllerInstances.bootId, currentControllerBootId()));
expect(after.renewedAt!.getTime()).toBeGreaterThan(before.renewedAt!.getTime());
expect(await backend()).toBe(pidBefore);
} finally {
// Free the controller row before the stop record writes it.
release();
await holder.catch(() => undefined);
await stopControllerInstanceLease(db);
await lockDb.$client.end({ timeout: 5 });
}
}, 40_000);
it("proves a cleanly stopped boot gone at once", async () => {
const stopped = await otherBoot({ liveForMs: 60_000 });
await db.update(controllerInstances).set({ stoppedAt: sql`now()` }).where(eq(controllerInstances.bootId, stopped));
@@ -37,7 +37,7 @@ import { currentControllerBootId, currentProcessSpaceHash } from "../services/co
import { createOwnershipReportState, runOwnershipReconciliationPass, settleOwnableKind, type OwnershipReport } from "../services/recovery/ownership-invariant-report.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { reconcileNativeFinalizations, registerOwnershipReconciliationWork } from "../services/native-runtime/native-finalization-reconciler.js";
import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
import { OWNABLE_KINDS, registerNativeRestartRecheck } from "../services/recovery/ownership-registry.js";
import { currentProcessProbe, proveOwnerGone } from "../services/owner-proof.js";
import { createRunLogWriterAuthority } from "../services/heartbeat/run-log-authority.js";
import { resolveHotRestartIntentPath, writeHotRestartIntent } from "../services/hot-restart.js";
@@ -948,6 +948,82 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
} finally { probe.restore(); unknown?.mockRestore(); }
}, 60_000);
it("ends an unverifiable run on the first look once its controller's lease lapsed longer ago than the bound", async () => {
// A successor that boots (or first scans) minutes after the old server left
// must not restart the 2-minute wait from its own first observation.
const f = await seedClaim("CLAIM-SPACE-LAPSED-LONG-AGO", "expired", "foreign-process-space");
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 3 * 60_000) })
.where(eq(nativeRunFinalizations.runId, f.runId));
const probe = observeOwnerPid(), onClaim = vi.fn();
try {
const dispositions = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim });
expect(dispositions).toEqual([{ kind: "blocked", runId: f.runId, reason: "controller_process_space_unverified" }]);
expect(probe.observed).toEqual([]);
expect(onClaim).not.toHaveBeenCalled();
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
.toMatchObject({ status: "failed", errorCode: "native_restart_recovery_blocked" });
} finally { probe.restore(); }
}, 60_000);
it("ends at once a run whose controller boot is fenced, even with its 20-minute claim lease unexpired (N1)", async () => {
// N1 cell: old server SIGKILLed mid-overlap at 22:09:31; its boot was fenced at 22:13:02, and the
// run still waited a further 2 minutes from that first look. A fenced boot already outlived its lease.
const f = await seedClaim("CLAIM-SPACE-FENCED-BOOT", "live", "foreign-process-space");
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
await f.db.insert(controllerInstances).values({ bootId: owner!.controllerBootId!, processSpace: "foreign-process-space",
pid: priorPid, leaseExpiresAt: new Date(Date.now() - 3 * 60_000), fencedAt: new Date(Date.now() - 10_000) });
registerControllerInstanceDb(f.db);
const probe = observeOwnerPid();
try {
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() }))
.toEqual([{ kind: "blocked", runId: f.runId, reason: "controller_process_space_unverified" }]);
} finally { probe.restore(); resetControllerInstanceLeaseForTests(); }
}, 60_000);
it("keeps waiting for a controller whose boot is still live, whatever the run's own lease says", async () => {
const f = await seedClaim("CLAIM-SPACE-LIVE-BOOT", "expired", "foreign-process-space");
const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
await f.db.insert(controllerInstances).values({ bootId: owner!.controllerBootId!, processSpace: "foreign-process-space",
pid: priorPid, leaseExpiresAt: new Date(Date.now() + 60_000) });
registerControllerInstanceDb(f.db);
const probe = observeOwnerPid();
try {
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() }))
.toMatchObject([{ kind: "awaiting_evidence", runId: f.runId }]);
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running");
} finally { probe.restore(); resetControllerInstanceLeaseForTests(); }
}, 60_000);
it("never moves the owner's last sign of life when a successor records its wait (fence time unchanged)", async () => {
// N1: the successor's awaiting-evidence write bumped updated_at, which the prover reads as the dead
// owner's last write, and the fence slipped from about 2 minutes to 3.5 after the crash.
const f = await seedClaim("CLAIM-SPACE-OWNER-TOUCH", "expired", "foreign-process-space");
const ownerLastWrite = new Date(Date.now() - 5 * 60_000);
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 60_000), updatedAt: ownerLastWrite })
.where(eq(nativeRunFinalizations.runId, f.runId));
const probe = observeOwnerPid();
try {
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() }))
.toMatchObject([{ kind: "awaiting_evidence", reason: "controller_process_space_unverified" }]);
const [after] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
expect(after!.recoveryState).toBe("awaiting_evidence");
expect(after!.updatedAt.getTime()).toBe(ownerLastWrite.getTime());
} finally { probe.restore(); }
}, 60_000);
it("schedules the recheck for the end of the bound measured from the lease lapse", async () => {
const f = await seedClaim("CLAIM-SPACE-LAPSED-RECENTLY", "expired", "foreign-process-space");
const lapsedAt = Date.now() - 30_000;
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(lapsedAt) }).where(eq(nativeRunFinalizations.runId, f.runId));
const probe = observeOwnerPid();
try {
const [disposition] = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim: vi.fn() });
expect(disposition).toMatchObject({ kind: "awaiting_evidence", reason: "controller_process_space_unverified" });
const retryAt = (disposition as { retryAt: Date }).retryAt.getTime();
expect(Math.abs(retryAt - (lapsedAt + 2 * 60_000))).toBeLessThan(5_000);
} finally { probe.restore(); }
}, 60_000);
it("keeps a real controller-only attestation from authorizing an in-flight claim", async () => {
const f = await seedClaim("CLAIM-CONTROLLER-ONLY", "expired", "foreign-process-space");
const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
@@ -1686,6 +1762,35 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
expect(second.violations.filter(violation => violation.kind === "native_accepted_result_lease")).toEqual([]);
}, 60_000);
it("hands a run awaiting evidence to restart recovery once the reconciler proves its controller gone (crash mid-overlap)", async () => {
// A crashed old server's run was classified by the new server's startup pass while the
// old server lived, so no restart re-check was scheduled for it. Only the reconciler
// sees the controller die; it must hand the run to restart recovery on the live server.
const f = await seedRecoverableRun("AWAITING-CRASHED-CONTROLLER");
await f.db.update(nativeRunFinalizations).set({ recoveryState: "awaiting_evidence",
leaseOwner: `old-controller:${randomUUID()}`, leaseExpiresAt: new Date(Date.now() + 15 * 60_000),
controllerBootId: randomUUID(), controllerPid: 2_000_000_001, controllerHostHash: currentProcessSpaceHash(),
controllerProcessStartedAt: new Date(Date.now() - 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
const kind = OWNABLE_KINDS.find(candidate => candidate.id === "native_awaiting_evidence")!;
const kinds = [{ ...kind, listInProgress: async (db: typeof f.db) => ({ records: await kind.listForRuns!(db, [f.runId]), truncated: false }) }];
const recheck = vi.fn(async () => undefined);
const unregister = registerNativeRestartRecheck(recheck);
const original = process.kill.bind(process);
const kill = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
if (pid === 2_000_000_001 && signal === 0) throw Object.assign(new Error("gone"), { code: "ESRCH" });
return original(pid, signal);
});
try {
const state = createOwnershipReportState();
const first = await runOwnershipReconciliationPass(f.db, { kinds, state });
expect(first.kinds[0]).toMatchObject({ verdicts: { gone: 1 } });
expect(recheck).toHaveBeenCalledWith([f.runId]);
// The next 30 s pass does not hand it over again; restart recovery schedules its own re-checks.
await runOwnershipReconciliationPass(f.db, { kinds, state });
expect(recheck).toHaveBeenCalledTimes(1);
} finally { kill.mockRestore(); unregister(); }
}, 60_000);
it("keeps a live earlier-boot controller's result lease in the reconciler pass", async () => {
const f = await seedAcceptedResult("ACCEPTED-RECONCILER-LIVE", (runner) => `${runner}:${randomUUID()}`);
const self = await currentNativeControllerIdentity();
@@ -161,7 +161,7 @@ describe("report-only ownership reconciliation", () => {
expect(new Set(names).size).toBe(names.length);
expect(names.length).toBeGreaterThanOrEqual(40);
expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
.toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "remote_workspace_unsaved_files", "remote_snapshot_ledger", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "native_completed_turn_cleanup_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
.toEqual(["native_executor_claim", "native_restart_recovery_claim", "native_awaiting_evidence", "native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "remote_workspace_unsaved_files", "remote_snapshot_ledger", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "native_completed_turn_cleanup_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
});
+4
View File
@@ -92,6 +92,7 @@ import { registerNativeCleanupReleaseWake } from "./services/native-runtime/nati
import { logNativeCleanupQuarantines } from "./services/native-runtime/native-cleanup-quarantine-log.js";
import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js";
import { deliverExecutionStatuses } from "./services/execution-status-delivery.js";
import { registerNativeRestartRecheck } from "./services/recovery/ownership-registry.js";
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js";
import { deliverPendingPauseResumes } from "./services/issue-tree-resume.js";
import { deliverBudgetResumeContinuations, type BudgetResumeDelivery } from "./services/budget-resume-continuation.js";
@@ -962,6 +963,9 @@ async function startServerWithDatabaseTeardown(
? heartbeatService(db as any, { pluginWorkerManager })
: null;
const accountingHeartbeat = heartbeat ?? heartbeatService(db as any, { pluginWorkerManager });
// A native run whose controller the ownership reconciler proves gone is re-checked here,
// so a crashed old server's runs end or resume without another server boot.
if (heartbeat) registerNativeRestartRecheck(runIds => heartbeat.recoverNativeRunsAfterRestart(new Date(), runIds));
const decisionServiceOptions = {
wakeOriginAgent: createDecisionWakeOriginAgent(heartbeat?.wakeup ?? null),
};
@@ -1,5 +1,5 @@
import { and, eq, isNull, sql } from "drizzle-orm";
import { controllerInstances, type Db } from "@paperclipai/db";
import { controllerInstanceLeaseDb, controllerInstanceLeaseDbWithoutStatementTimeout, controllerInstances, type Db } from "@paperclipai/db";
import { logger } from "../middleware/logger.js";
import {
currentControllerBootId, currentProcessSpaceHash, onControllerStamp,
@@ -20,6 +20,8 @@ const STAMP_COMMIT_GRACE_MS = CONTROLLER_LEASE_TTL_MS;
const SETTLE_RECHECK_MIN_MS = 5_000;
/** A renewal that has not answered within this bound counts as failed. */
const RENEW_TIMEOUT_MS = 10_000;
/** Server-side cancel for a renewal statement, below the client timeout. */
export const RENEW_STATEMENT_TIMEOUT_MS = 8_000;
const RETRY_MS = 5_000;
export type ControllerInstanceState = "live" | "grace" | "expired" | "fenced" | "stopped";
@@ -95,14 +97,29 @@ function renewInDatabase(db: Db): (stampedRecently: boolean) => Promise<RenewRes
const records = holdsStampedWorkSql(bootId);
const holds = sql`(${stampedRecently} or ${records})`;
const expires = sql`now() + ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval`;
const rows = await db.execute(sql`
// Its own connection: a busy main pool (request bursts at a traffic switch,
// a replacement's startup pass) must not delay the proof that this boot is
// alive. One statement per renewal, so an idle boot's last check stays a
// single round trip. The connection's own statement_timeout (below the
// client timeout) makes the server cancel a slow renewal, so it never holds
// the connection the next renewal needs.
const renew = (lease: Db) => lease.execute(sql`
insert into controller_instances (boot_id, process_space, pid, lease_expires_at)
values (${bootId}, ${currentProcessSpaceHash()}, ${process.pid}, ${expires})
on conflict (boot_id) do update set
lease_expires_at = case when ${holds} then ${expires} else controller_instances.lease_expires_at end,
renewed_at = case when ${holds} then now() else controller_instances.renewed_at end
where controller_instances.fenced_at is null and controller_instances.stopped_at is null
returning epoch, ${holds} as holds, ${records} as records`) as unknown as Array<{ epoch: string | number; holds: boolean; records: boolean }>;
returning epoch, ${holds} as holds, ${records} as records`) as unknown as Promise<Array<{ epoch: string | number; holds: boolean; records: boolean }>>;
let rows: Awaited<ReturnType<typeof renew>>;
try {
rows = await renew(controllerInstanceLeaseDb(db, { statementTimeoutMs: RENEW_STATEMENT_TIMEOUT_MS }));
} catch (error) {
// A connection pooler may refuse the startup parameter: renew without it.
if (!/unsupported startup parameter/i.test(error instanceof Error ? error.message : String(error))) throw error;
logger.warn({ err: error }, "controller instance lease connection refused statement_timeout; renewing without it");
rows = await renew(controllerInstanceLeaseDbWithoutStatementTimeout(db));
}
const row = rows[0];
if (!row) return "fenced";
if (state) state.epoch = Number(row.epoch);
@@ -1433,11 +1433,29 @@ export async function claimNativeRestartRecoveries(input: {
// old run's writes revoked and a new workspace root.
// The same holds when the previous controller's host was never recorded
// or cannot be proven from here: confirming its stop is optional.
// The run's own server boot is proven stopped or fenced (controller
// instance record): its lease is no sign of life, however long it was
// granted. The claim lease is 20 minutes for a local controller.
const ownerControllerStopped = evaluatedTakeover.allowed
&& ["controller_stopped", "controller_fenced"].includes(evaluatedTakeover.reason);
const leaseLapsedAt = row.coordinator.leaseExpiresAt === null ? null
: row.coordinator.leaseExpiresAt <= now ? row.coordinator.leaseExpiresAt.getTime() : null;
const unverifiableLapsedController = !takeover.allowed && !nonExecutionOwner
&& (row.coordinator.leaseExpiresAt === null || row.coordinator.leaseExpiresAt <= now)
&& (row.coordinator.leaseExpiresAt === null || leaseLapsedAt !== null || ownerControllerStopped)
&& (namespaceAttentionNeeded || ["controller_host_unrecorded", "controller_host_unprovable"].includes(takeover.reason));
let unverifiedWaitEndsAt: number | null = null;
if (unverifiableLapsedController) {
const waitingSince = namespaceWaitStartedAt(row.coordinator.recoveryHistory, takeover.reason);
// The bounded wait starts when the run's controller went away (its lease
// lapsed), not when a later server first looked: a successor that boots
// minutes later, or re-records the wait, must not restart the clock.
const observedSince = namespaceWaitStartedAt(row.coordinator.recoveryHistory, takeover.reason);
const starts = [observedSince, leaseLapsedAt].filter((at): at is number => at !== null);
// A controller boot proven stopped or fenced already outlived its lease
// (90 s) and the fence grace (30 s): waiting longer proves nothing more
// about a Runner in another process space, so the run ends now.
const waitingSince = ownerControllerStopped ? now.getTime() - NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS
: starts.length ? Math.min(...starts) : null;
unverifiedWaitEndsAt = (waitingSince ?? now.getTime()) + NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS;
if (waitingSince !== null && now.getTime() - waitingSince >= NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS) {
await endNamespaceUnverifiedRun(tx as unknown as Db, row, now, takeover.reason, historyEntry({
now, restartKind: input.restartKind, controller, generation: row.coordinator.controllerGeneration,
@@ -1480,8 +1498,12 @@ export async function claimNativeRestartRecoveries(input: {
// Repeated scans of one ownership wait record it once, so the
// bounded history keeps the drain record the next claim reads.
...(waitAlreadyRecorded ? {} : { recoveryHistory: appendBoundedRecoveryHistory(event) }),
updatedAt: waitAlreadyRecorded && row.coordinator.recoveryState === "awaiting_evidence"
&& row.coordinator.recoveryRequestId === (input.recoveryRequestId ?? null) ? row.coordinator.updatedAt : now,
// updated_at is the owner's last sign of life: the ownership prover
// fences a dead boot only once it has been quiet past that write plus
// the lease and grace, and the fallback reads it as the owner's
// progress. A successor recording its own wait never moves it.
updatedAt: !ownControllerProcess || (waitAlreadyRecorded && row.coordinator.recoveryState === "awaiting_evidence"
&& row.coordinator.recoveryRequestId === (input.recoveryRequestId ?? null)) ? row.coordinator.updatedAt : now,
})
.where(
and(
@@ -1502,7 +1524,8 @@ export async function claimNativeRestartRecoveries(input: {
: ["controller_lease_held", "controller_lease_grace"].includes(takeover.reason)
? { retryAt: new Date(now.getTime() + CONTROLLER_LEASE_RENEW_MS) }
// The bounded wait for an unverifiable controller is checked again when it ends.
: unverifiableLapsedController ? { retryAt: new Date(now.getTime() + NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS) } : {}),
: unverifiableLapsedController ? { retryAt: new Date(Math.max(now.getTime() + 1_000,
unverifiedWaitEndsAt ?? now.getTime() + NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS)) } : {}),
} as const;
}
@@ -2478,18 +2478,23 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
const stopped = await hasNativeLocalProcessStop(fixture.db, companyId, fixture.runId);
expect(probes).toEqual([]);
// The namespace wait is bounded: it is checked again when the wait ends.
expect(dispositions).toEqual([{ kind: "awaiting_evidence", runId: fixture.runId,
reason: "controller_process_space_unverified", retryAt: expect.any(Date) }]);
// A stopped or fenced controller boot already outlived its lease and the
// fence grace, so the namespace wait ends on the first look: the run ends
// as unverified (never as a proven stop) and the task continues in a fresh
// run. Nothing local is probed, claimed or recorded as stopped.
expect(dispositions).toEqual([{ kind: "blocked", runId: fixture.runId,
reason: "controller_process_space_unverified" }]);
expect(claimed).not.toHaveBeenCalled();
expect(stamped).not.toHaveBeenCalled();
expect(stopped).toBe(false);
expect(after).toMatchObject({ phase: before.phase, attempt: before.attempt,
expect(after).toMatchObject({ phase: "terminal_failure", attempt: before.attempt,
controllerGeneration: before.controllerGeneration, controllerBootId: before.controllerBootId,
controllerPid: before.controllerPid, controllerHostHash: before.controllerHostHash,
leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, resultId: null });
expect(run).toMatchObject({ status: "running", processPid: runnerPid, processGroupId: runnerGroup,
processStartedAt: startedAt, processHostHash: foreignSpace });
failureCode: "native_restart_recovery_blocked",
failureDetail: expect.objectContaining({ reason: "controller_process_space_unverified", executorStopEvidence: "unverified" }),
resultId: null });
expect(run).toMatchObject({ status: "failed", errorCode: "native_restart_recovery_blocked",
processPid: runnerPid, processGroupId: runnerGroup, processStartedAt: startedAt, processHostHash: foreignSpace });
expect(run.runnerProfileJson).toEqual({ ...fixture.preparedProfile, sessionCheckpoint: {
identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId,
sessionId: fixture.native.normalizedSessionId },
@@ -275,6 +275,37 @@ function nrfKind(id: string, title: string, obligation: Obligation, where: SQL |
};
}
/**
* A native run whose controller died after another server's startup pass
* (a crash mid-overlap, not a graceful drain) has no server re-checking it:
* restart recovery only schedules re-checks for runs it saw. When the
* reconciler proves such a run's owner gone, this server runs the same restart
* recovery for that run now; its bounded wait and re-check schedule then end
* or resume the run without waiting for another server to boot.
*/
let nativeRestartRecheck: ((runIds: string[]) => Promise<unknown>) | null = null;
export function registerNativeRestartRecheck(recheck: (runIds: string[]) => Promise<unknown>) {
nativeRestartRecheck = recheck;
return () => { if (nativeRestartRecheck === recheck) nativeRestartRecheck = null; };
}
/** The reconciler re-checks every 30 s; restart recovery schedules its own
* follow-ups, so one hand-off per run per minute is enough. */
const NATIVE_RESTART_RECHECK_SPACING_MS = 60_000;
const lastNativeRestartRecheck = new Map<string, number>();
const recheckNativeRunOwnership: SettleRecord = async (_db, record, context) => {
if (!nativeRestartRecheck || !record.runId) return { settled: false, reason: "restart_recheck_unavailable" };
const last = lastNativeRestartRecheck.get(record.runId);
if (last !== undefined && context.now.getTime() - last < NATIVE_RESTART_RECHECK_SPACING_MS)
return { settled: false, reason: "restart_recovery_recheck_recent" };
if (lastNativeRestartRecheck.size > 10_000) lastNativeRestartRecheck.clear();
lastNativeRestartRecheck.set(record.runId, context.now.getTime());
await nativeRestartRecheck([record.runId]);
return { settled: false, reason: "restart_recovery_rechecked" };
};
function withNativeRestartRecheck(kind: OwnableKind): OwnableKind {
return { ...kind, settle: recheckNativeRunOwnership };
}
const notFinalizerOrCleanup = and(notLike(nativeRunFinalizations.leaseOwner, "native-finalizer:%"),
notLike(nativeRunFinalizations.leaseOwner, "native-cleanup:%"));
@@ -682,13 +713,13 @@ const acceptedNativeResultLease = nrfKind("native_accepted_result_lease", "Accep
export const OWNABLE_KINDS: readonly OwnableKind[] = [
// Native runtime and Runner
nrfKind("native_executor_claim", "Executor execution claim (fresh run)", "ownership",
withNativeRestartRecheck(nrfKind("native_executor_claim", "Executor execution claim (fresh run)", "ownership",
and(eq(nativeRunFinalizations.phase, "observed"), isNull(nativeRunFinalizations.resultId),
isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup, isNull(nativeRunFinalizations.recoveryState))),
nrfKind("native_restart_recovery_claim", "Restart-recovery claim", "ownership",
inArray(nativeRunFinalizations.recoveryState, ["awaiting_runner_reattach", "resuming_session", "bootstrap_incomplete"])),
nrfKind("native_awaiting_evidence", "\"Awaiting evidence\" wait", "ownership",
eq(nativeRunFinalizations.recoveryState, "awaiting_evidence"), undefined, [1, 2, 7]),
isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup, isNull(nativeRunFinalizations.recoveryState)))),
withNativeRestartRecheck(nrfKind("native_restart_recovery_claim", "Restart-recovery claim", "ownership",
inArray(nativeRunFinalizations.recoveryState, ["awaiting_runner_reattach", "resuming_session", "bootstrap_incomplete"]))),
withNativeRestartRecheck(nrfKind("native_awaiting_evidence", "\"Awaiting evidence\" wait", "ownership",
eq(nativeRunFinalizations.recoveryState, "awaiting_evidence"), undefined, [1, 2, 7])),
{
// Shared path since Phase 2: a hard kill after `run.result` is accepted
// leaves the result leased to the dead controller; settle expires that