Files
PaperClipAI/server/src/services/native-runtime/native-runner-checkpoint.ts
T
DottaandClaude Opus 5.5 df9914dbd9 fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - Since #21774, a follow-up that the incoming server claims during the
overlap waits (`native_controller_recovery_waiting_for_owner`) and
checks again every 5 s.
> - About 1 s after the traffic switch (#21627), the outgoing server
tries to save and close its idle warm session. The checkpoint fence
refuses that save with `native_runner_checkpoint_owner_lost`. The task's
current run is the parked follow-up. That run has its own finalization
row under another controller boot, and the fence does not accept that
kind of successor.
> - The close retry from #21782 fails the same way each time. So the
follow-up waits until the outgoing server exits, or until the
predecessor's finish time plus the idle window plus 2 min. In the CHA-5
case that was about 225 s, and the worst case is about 425 s. Speed is a
requirement.
> - This pull request records which owner a parked follow-up waits on,
and lets the fence accept the outgoing owner's final save when the
follow-up is parked on exactly that owner and has not started.
> - The benefit is that the outgoing server saves and closes the session
on its first attempt, and the follow-up continues at its next 5 s check.

## Linked Issues or Issue Description

Refs #21774, #21782, #21697, #21627, #21279

**What happened?**

During a release-swap overlap, a warm follow-up was claimed by the
incoming server while the outgoing server still owned the idle warm
session. The follow-up waited correctly. The outgoing server's idle
close failed with `native_runner_checkpoint_owner_lost` on every
attempt. The wait ended only when the outgoing server exited, about 225
s after the follow-up was created.

**Expected behavior**

The outgoing server saves and closes the session while the follow-up
waits. The follow-up sees the saved session and continues within one 5 s
check.

**Steps to reproduce**

1. Run a warm native session on a remote sandbox and let a run finish on
server A.
2. Start server B with an overlap and switch traffic to it.
3. Post a follow-up comment so that B claims the follow-up while A is
still alive.
4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and
the follow-up waits until A exits.

**Paperclip version or commit**

master `d14a3b88b1`.

## What Changed

- `NativeControllerRecoveryWaitingError` carries the owner that the
follow-up waits on: the predecessor run and its controller boot.
`waitForLiveWarmSessionOwner` and the live-owner wait in
`inspectRemoteWarmSessionRecovery` set it.
- `deferNativeControllerRecovery` stores that owner in the finalization
`failureDetail.waitingForOwner`. The executor passes it through.
- `validateNativeRunnerCheckpointOwner` accepts one more successor kind:
a parked follow-up waiting on this exact owner. All of these must be
true, or the fence still refuses:
- The checkpoint holds an accepted result and has no new work (unchanged
rule).
- The follow-up finalization is `retryable_failure` with
`native_controller_recovery_waiting_for_owner`, `attempt = 0`, no
result, and no lease owner (no claim is running it).
  - `waitingForOwner` names this owner's run and controller boot.
- The follow-up run is a running native run for the same company and
task, on the same native session as the owner's run.
- The follow-up has no process fields, no own process-start events, and
no checkpoint rows.
- The fence locks the follow-up's finalization row (`for share`), so a
resumption claim and the save cannot interleave.
- Tests in `native-controller-recovery.integration.test.ts` (real
Postgres):
- The outgoing server saves and closes its idle warm session while the
incoming follow-up waits, and the follow-up resumes within one recheck.
- Negative cases: the save is still refused when the follow-up has its
own process, its own process-start event, its own checkpoint, or a live
resumption claim, or when it waits on a different owner boot or run.

## Verification

- Red on master: with the `native-runner-checkpoint.ts` change removed,
the scenario test fails with `native_runner_checkpoint_owner_lost` at
the outgoing server's first close attempt. The follow-up keeps waiting.
Green with the change.
- The negative cases pass with and without the change.
- Measured in the test scenario (warm idle window 600 s):
- Before: the follow-up waits until the outgoing server exits, or about
720 s (finish + 600 s idle window + 120 s margin).
- After: the outgoing save and close succeed on the first attempt. The
follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after
it parked.
- `vitest run` of `native-controller-recovery.integration.test.ts`,
`native-runner-restart-recovery.integration.test.ts` and
`native-harness-backup-storage.integration.test.ts`: 349 passed, 14
skipped.
- `vitest run` of `native-session-executor.test.ts`,
`remote-warm-session-recovery.test.ts`,
`native-runner-checkpoint-save.test.ts`,
`native-runner-checkpoint-retirement.test.ts` and
`native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7
skipped.
- `tsc --noEmit` for `server` passes.
- `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies
the broad coverage.
- Not run: a release-swap wave. The fix is not yet measured on a live
overlap.

## Risks

- The fence is wider. It accepts only a final save with an accepted
result and no new work, and only while the current run is parked on this
exact owner and has not started. A follow-up that has a claim, a
process, process-start events or checkpoint rows still causes a refusal.
- A follow-up that a server parked before this change has no
`waitingForOwner`. The fence refuses its owner's save as before. At its
next recheck, the follow-up parks again with the owner recorded.
- While a resumption claim runs the follow-up's recheck (a short window
every 5 s), the save is refused. The close retry from #21782 then tries
again.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:02:31 -06:00

608 lines
38 KiB
TypeScript

import { readNativeCheckpointRetirement } from "./native-runner-checkpoint-retirement.js";
import { NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { createHash } from "node:crypto";
import { and, eq, exists, gt, isNull, lte, inArray, notExists, or, sql, type SQL } from "drizzle-orm";
import { checkpointDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, primeDatabaseWorkTransactionId, type Db } from "@paperclipai/db";
import { inspectDurablePrpCheckpointHeader, inspectUnauthenticatedBootstrapCheckpoint, inspectUnclaimedBootstrapCheckpoint, decodeDurablePrpCheckpoint, type DurablePrpCheckpoint, type DurablePrpControlPlane, type DurableRecoveryIdentity } from "../../vendor/paperclip-runner/index.js";
import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from "../../secrets/local-encrypted-provider.js";
import { legacyControllerBootId } from "../legacy-controller-lease.js";
import { assertNativeSourceCheckpointConsumer, lockSourceRecoveryCoordinators } from "./native-source-recovery-ownership.js";
import { runnerdStateHasNoProviderTurn, runnerdStateIsIncompleteBootstrap } from "./native-bootstrap-proof.js";
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js";
import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js";
type Binding = { companyId: string; issueId: string; runId: string };
export const NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES = 1024 * 1024;
export const NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES = 8 * 1024 * 1024;
export type NativeRunnerCheckpointOwner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string };
export type NativeRunnerCheckpointFenceOwner = Omit<NativeRunnerCheckpointOwner, "leaseOwner"> & { leaseOwner: string | null };
function encryptParts(binding: Binding, records: ReadonlyMap<string, string>) {
if (!records.size) return [];
const cipher = createLocalEncryptedCipher("write");
return [...records].map(([part, contents]) => ({
companyId: binding.companyId, issueId: binding.issueId, runId: binding.runId, part,
material: cipher.encrypt(JSON.stringify({ companyId: binding.companyId, issueId: binding.issueId, runId: binding.runId, part, contents })),
}));
}
export async function insertMigratedNativeRunnerCheckpoint(tx: CheckpointTransaction, binding: Binding, checkpoint: DurablePrpCheckpoint) {
if (checkpoint.runId !== binding.runId) throw new Error("native_runner_checkpoint_binding_mismatch");
let batch: ReturnType<typeof encryptParts> = [];
let batchBytes = 0;
for (const part of encryptParts(binding, checkpoint.records)) {
const bytes = Buffer.byteLength(JSON.stringify(part));
if (batch.length && (batch.length >= 64 || batchBytes + bytes > 512 * 1024)) {
await tx.insert(nativeRunnerCheckpoints).values(batch);
batch = [];
batchBytes = 0;
}
batch.push(part);
batchBytes += bytes;
}
if (batch.length) await tx.insert(nativeRunnerCheckpoints).values(batch);
}
function decryptPart(cipher: ReturnType<typeof createLocalEncryptedCipher>, binding: Binding,
part: string, material: Record<string, unknown>): string {
const envelope = JSON.parse(cipher.decrypt(material));
if (envelope.companyId !== binding.companyId || envelope.issueId !== binding.issueId
|| envelope.runId !== binding.runId || envelope.part !== part || typeof envelope.contents !== "string") {
throw new Error("native_runner_checkpoint_binding_mismatch");
}
return envelope.contents;
}
type CheckpointTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
/** A dependent run-lock query can consume this CTE to preserve coordinator-first
* lock order without adding a second database round trip. */
export function nativeRunnerCheckpointOwnerLock(tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner) {
return tx.$with("checkpoint_owner").as(tx.select({
leaseOwner: nativeRunFinalizations.leaseOwner,
leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
resultId: nativeRunFinalizations.resultId,
phase: nativeRunFinalizations.phase,
}).from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, owner.companyId), eq(nativeRunFinalizations.issueId, owner.issueId),
eq(nativeRunFinalizations.runId, owner.authorityRunId), eq(nativeRunFinalizations.controllerBootId, owner.bootId),
eq(nativeRunFinalizations.controllerGeneration, owner.generation))).for("update"));
}
/** Locks the coordinator before callers lock a run. A caller that subsequently
* waits for another lock must recheck the fence before writing. */
export async function lockNativeRunnerCheckpointOwner(
tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner,
) {
// A locking CTE acquires ownership before the outer projection samples
// wall time. Sampling in the locking SELECT can precede its lock wait.
// Keep both steps in one statement without another database round trip.
const locked = nativeRunnerCheckpointOwnerLock(tx, owner);
const [row] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner,
leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, phase: locked.phase, now: sql<Date>`clock_timestamp()`,
xid: sql<string>`pg_current_xact_id()::text` }).from(locked);
if (!row) throw new Error("native_runner_checkpoint_owner_lost");
const { xid, ...owned } = row;
primeDatabaseWorkTransactionId(tx, xid);
return owned;
}
export async function validateNativeRunnerCheckpointOwner(
tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner,
owned: Omit<Awaited<ReturnType<typeof lockNativeRunnerCheckpointOwner>>, "phase">, newWork: boolean,
): Promise<void> {
const current = owned;
const activeOwner = owner.leaseOwner !== null && current.leaseOwner === owner.leaseOwner
&& current.leaseExpiresAt !== null && current.leaseExpiresAt > new Date(owned.now);
if (!activeOwner) {
await assertNativeSourceCheckpointConsumer(tx, owner);
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
.where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId)));
const [successor] = task?.executionRunId && task.executionRunId !== owner.runId
? await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, task.executionRunId), eq(nativeRunFinalizations.companyId, owner.companyId),
eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.controllerBootId, owner.bootId),
gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))) : [];
const [preparingSuccessor] = !successor && task?.executionRunId && task.executionRunId !== owner.runId
? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId),
eq(heartbeatRuns.issueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "legacy"),
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.executionStage, "preparing"),
eq(heartbeatRuns.controllerBootId, legacyControllerBootId),
gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : [];
const [waitingSuccessor] = !successor && !preparingSuccessor && task?.executionRunId && task.executionRunId !== owner.runId
? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId),
eq(heartbeatRuns.issueId, owner.issueId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"]),
notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : [];
const [parkedSuccessor] = !successor && !preparingSuccessor && !waitingSuccessor && task?.executionRunId && task.executionRunId !== owner.runId
? await parkedSuccessorWaitingForOwner(tx, owner, task.executionRunId) : [];
if (!current.resultId || newWork || !task
|| (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor && !waitingSuccessor && !parkedSuccessor)) {
throw new Error("native_runner_checkpoint_owner_lost");
}
}
}
function parkedSuccessorWaitingForOwner(tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner, successorRunId: string) {
return tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)).where(and(
eq(nativeRunFinalizations.runId, successorRunId), eq(nativeRunFinalizations.companyId, owner.companyId),
eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.phase, "retryable_failure"),
eq(nativeRunFinalizations.failureCode, NATIVE_RETAINED_OWNER_WAIT), eq(nativeRunFinalizations.attempt, 0),
isNull(nativeRunFinalizations.resultId), isNull(nativeRunFinalizations.leaseOwner),
sql`${nativeRunFinalizations.failureDetail}->'waitingForOwner'->>'runId' = ${owner.runId}`,
sql`${nativeRunFinalizations.failureDetail}->'waitingForOwner'->>'controllerBootId' = ${owner.bootId}`,
eq(heartbeatRuns.companyId, owner.companyId), eq(heartbeatRuns.nativeIssueId, owner.issueId),
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "running"),
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt),
sql`${heartbeatRuns.nativeSessionId} = (select source.native_session_id from heartbeat_runs source
where source.id = ${owner.runId} and source.company_id = ${owner.companyId})`,
notExists(tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, successorRunId)))),
notExists(tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, owner.companyId), eq(heartbeatRunEvents.runId, successorRunId),
isNull(heartbeatRunEvents.sourceEventId), inArray(heartbeatRunEvents.eventType, [...OWN_PROCESS_START_EVENT_TYPES])))),
)).for("share", { of: nativeRunFinalizations });
}
export async function assertNativeRunnerCheckpointOwner(tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, newWork: boolean) {
const locked = await lockNativeRunnerCheckpointOwner(tx, owner);
await validateNativeRunnerCheckpointOwner(tx, owner, locked, newWork);
}
const ROUTINE_CHECKPOINT_SAVE_SQL = `
with locked_owner as (
select lease_expires_at from native_run_finalizations
where company_id = $1::uuid and issue_id = $2::uuid and run_id = $3::uuid and controller_boot_id = $4::text
and controller_generation = $5::integer and lease_owner = $6::text
for update
), owner as (
select 1 from locked_owner where lease_expires_at > clock_timestamp()
), written as (
insert into "native_runner_checkpoints" (company_id, issue_id, run_id, part, material)
select $1::uuid, $2::uuid, $7::uuid, part.value ->> 'part', part.value -> 'material'
from owner cross join jsonb_array_elements($8::jsonb) as part(value)
on conflict (run_id, part) do update set material = excluded.material, updated_at = now()
returning 1
)
select (select count(*)::int from owner) as owned, (select count(*)::int from written) as written`;
export function nativeRunnerCheckpointWriter(db: Db, owner: NativeRunnerCheckpointOwner) {
const persistenceDb = checkpointDb(db);
let previousWork = new Set<string>();
return async (checkpoint: DurablePrpCheckpoint): Promise<void> => {
if (checkpoint.runId !== owner.runId) throw new Error("native_runner_checkpoint_binding_mismatch");
const changed = encryptParts(owner, checkpoint.changedRecords);
if (changed.length > 0 && !checkpoint.reset && checkpoint.removedRecords.length === 0 && !checkpoint.changedRecords.has("header")
&& !checkpoint.predecessor && owner.runId === owner.authorityRunId) {
const [saved] = await persistenceDb.$client.unsafe<{ owned: number; written: number }[]>(ROUTINE_CHECKPOINT_SAVE_SQL, [
owner.companyId, owner.issueId, owner.authorityRunId, owner.bootId, String(owner.generation), owner.leaseOwner, owner.runId,
JSON.stringify(changed.map(part => ({ part: part.part, material: part.material })))]);
if (saved?.owned === 1 && saved.written === changed.length) {
previousWork = new Set(checkpoint.workCommandIds);
return;
}
}
await persistenceDb.transaction(async tx => {
if (owner.runId !== owner.authorityRunId) {
await lockSourceRecoveryCoordinators(tx, owner.companyId, [owner.runId, owner.authorityRunId]);
await assertNativeSourceCheckpointConsumer(tx, owner, checkpoint);
}
await assertNativeRunnerCheckpointOwner(tx, owner, checkpoint.workCommandIds.some(id => !previousWork.has(id)));
if (checkpoint.reset || checkpoint.changedRecords.has("header") || checkpoint.removedRecords.includes("header")) {
const [header] = await tx.select({ material: nativeRunnerCheckpoints.material }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, owner.runId),
eq(nativeRunnerCheckpoints.part, "header"))).for("share").limit(1);
if (header && readNativeCheckpointRetirement(header.material, owner)) throw new Error("native_runner_checkpoint_retired");
}
if (checkpoint.predecessor) {
const [prior] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, checkpoint.predecessor.runId), eq(heartbeatRuns.companyId, owner.companyId),
eq(heartbeatRuns.nativeIssueId, owner.issueId),
eq(heartbeatRuns.nativeSessionId, checkpoint.predecessor.normalizedSessionId)));
if (!prior) throw new Error("native_runner_checkpoint_binding_mismatch");
}
await notifyLiveRunRecovery(tx, { companyId: owner.companyId, runId: owner.runId });
if (checkpoint.reset) await tx.delete(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, owner.runId)));
else if (checkpoint.removedRecords.length) await tx.delete(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, owner.runId),
inArray(nativeRunnerCheckpoints.part, [...checkpoint.removedRecords])));
if (changed.length) await tx.insert(nativeRunnerCheckpoints).values(changed)
.onConflictDoUpdate({ target: [nativeRunnerCheckpoints.runId, nativeRunnerCheckpoints.part],
set: { material: sql`excluded.material`, updatedAt: sql`now()` } });
});
previousWork = new Set(checkpoint.workCommandIds);
};
}
const authorities = new WeakMap<DurablePrpControlPlane, Map<string, { fence: string; write: ReturnType<typeof nativeRunnerCheckpointWriter> }>>();
export function nativeRunnerCheckpointPersistence(db: Db, bootId: string, authorityRunId: string | (() => string),
detach: (runIds: readonly string[], leaseOwner?: string) => Promise<unknown>) {
return { db, bootId, authorityRunId, onFailure: async (runId: string, leaseOwner: string) => {
try { await detach([runId], leaseOwner); }
finally {
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: sql`clock_timestamp()` })
.where(and(eq(nativeRunFinalizations.runId, runId), eq(nativeRunFinalizations.controllerBootId, bootId), eq(nativeRunFinalizations.leaseOwner, leaseOwner),
isNull(nativeRunFinalizations.resultId)));
}
} };
}
/** Attachment identity owns the stored records; current business authority owns
* the write lease. They differ when a completed warm session is recovered. */
export function nativeRunnerCheckpointRegistration(input: Omit<Parameters<typeof registerNativeRunnerCheckpoint>[0], "authority">) {
return (authority: DurablePrpControlPlane, identity?: { runId: string }) => registerNativeRunnerCheckpoint({
...input, authority, binding: { ...input.binding, runId: identity?.runId ?? input.binding.runId },
});
}
export async function registerNativeRunnerCheckpoint(input: {
db: Db; binding: Binding; bootId: string; authority: DurablePrpControlPlane;
authorityRunId?: string | (() => string);
onFailure?: (runId: string, leaseOwner: string) => Promise<void>;
}): Promise<void> {
const authorityRunId = typeof input.authorityRunId === "function"
? input.authorityRunId() : input.authorityRunId ?? input.binding.runId;
const [owner] = await input.db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.binding.companyId),
eq(nativeRunFinalizations.issueId, input.binding.issueId),
eq(nativeRunFinalizations.runId, authorityRunId),
eq(nativeRunFinalizations.controllerBootId, input.bootId),
gt(nativeRunFinalizations.leaseExpiresAt, sql`now()`)));
if (!owner?.leaseOwner) throw new Error("native_runner_checkpoint_owner_lost");
let writers = authorities.get(input.authority);
if (!writers) {
writers = new Map();
const boundWriters = writers;
input.authority.setCheckpointPersistence(async snapshot => {
const writer = boundWriters.get(snapshot.runId);
if (!writer) throw new Error("native_runner_checkpoint_owner_lost");
await writer.write(snapshot);
});
authorities.set(input.authority, writers);
}
const fence = JSON.stringify([authorityRunId, input.bootId, owner.controllerGeneration, owner.leaseOwner]);
if (writers.get(input.binding.runId)?.fence !== fence) {
const write = nativeRunnerCheckpointWriter(input.db, {
...input.binding, authorityRunId, bootId: input.bootId, generation: owner.controllerGeneration, leaseOwner: owner.leaseOwner,
});
writers.set(input.binding.runId, { fence, write: async checkpoint => {
try { await write(checkpoint); }
catch (error) {
void input.onFailure?.(authorityRunId, owner.leaseOwner!).catch(() => undefined);
throw error;
}
} });
}
await input.authority.flushCheckpoint();
}
/** Exact-run bounded metadata read; never loads command/event payload records. */
export async function loadNativeRunnerCheckpointHeader(db: Pick<Db, "select">, binding: Binding): Promise<string | null> {
const [row] = await db.select({ material: nativeRunnerCheckpoints.material }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId), eq(nativeRunnerCheckpoints.part, "header"))).limit(1);
if (!row) return null;
if (readNativeCheckpointRetirement(row.material, binding)) throw new Error("native_runner_checkpoint_retired");
try { return decryptPart(createLocalEncryptedCipher("read"), binding, "header", row.material); }
catch { return null; }
}
export async function loadNativeRunnerCheckpoint(db: Pick<Db, "select">, binding: Binding): Promise<DurablePrpCheckpoint | null> {
const rows = await db.select().from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId)));
if (!rows.length) return null;
const header = rows.find(row => row.part === "header");
if (header && readNativeCheckpointRetirement(header.material, binding)) throw new Error("native_runner_checkpoint_retired");
const cipher = createLocalEncryptedCipher("read");
const parts = new Map(rows.map(row =>
[row.part, decryptPart(cipher, binding, row.part, row.material)] as const));
const checkpoint = decodeDurablePrpCheckpoint(parts);
if (checkpoint.runId !== binding.runId) throw new Error("native_runner_checkpoint_binding_mismatch");
return checkpoint;
}
export async function loadNativeUnclaimedBootstrapCheckpoint(
db: Pick<Db, "select">, binding: Binding, identity: DurableRecoveryIdentity,
) {
try {
if (binding.runId !== identity.runId) return null;
const rows = await db.select().from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(5);
if (rows.length !== 4 || rows.reduce((bytes, row) => bytes + Buffer.byteLength(JSON.stringify(row.material)), 0)
> 2 * NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES) return null;
const cipher = createLocalEncryptedCipher("read");
const records = new Map(rows.map(row => [row.part, decryptPart(cipher, binding, row.part, row.material)] as const));
return inspectUnclaimedBootstrapCheckpoint(records, identity);
} catch { return null; }
}
export async function nativeRunnerNeverAuthenticated(
db: Pick<Db, "select">, binding: Binding,
identity: Pick<DurableRecoveryIdentity, "runId" | "normalizedSessionId" | "runnerInstanceId">,
): Promise<boolean> {
try {
if (binding.runId !== identity.runId) return false;
const rows = await db.select().from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + 3);
if (rows.length < 4 || rows.length > MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + 2
|| rows.reduce((bytes, row) => bytes + Buffer.byteLength(JSON.stringify(row.material)), 0)
> 2 * NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES) return false;
const cipher = createLocalEncryptedCipher("read");
const records = new Map(rows.map(row => [row.part, decryptPart(cipher, binding, row.part, row.material)] as const));
return inspectUnauthenticatedBootstrapCheckpoint(records, identity);
} catch { return false; }
}
const BOOTSTRAP_CHECKPOINT_PARTS = new Set(["header", "cursor"]);
const MAX_BOOTSTRAP_CHECKPOINT_COMMANDS = 8;
/** Session-open events are few; a longer journal is never a bootstrap. */
const MAX_BOOTSTRAP_CHECKPOINT_EVENTS = 16;
export type NativeRemoteBootstrapProof = "bootstrap_incomplete" | "session_opened_without_turn";
/**
* A remote runner's controller state lives only in this checkpoint, and every
* command is written here before the controller sends it. Once the previous
* controller's lease has expired, the checkpoint is therefore a complete record
* of what reached the sandbox runner. Returns which proof holds:
* `bootstrap_incomplete` when it holds only run.prepare and an unfinished
* session.open with no committed or acknowledged event, or
* `session_opened_without_turn` when session.open completed and every committed
* event is a session-open event. Either way no provider turn can exist. The
* sandbox runner journal is not on this host; the proof uses the controller
* record alone, as for a local root whose runner never journaled.
*/
export async function nativeRunnerCheckpointRemoteBootstrapProof(
db: Pick<Db, "select">, binding: Binding, expected: { runnerInstanceId: string; normalizedSessionId: string },
): Promise<NativeRemoteBootstrapProof | null> {
const evidence = await nativeRunnerCheckpointRemoteBootstrapEvidence(db, binding, expected.normalizedSessionId);
return evidence?.runnerInstanceId === expected.runnerInstanceId ? evidence.proof : null;
}
/**
* The same proof, with the runner instance the checkpoint names. A caller that
* must hold the proof across a new controller's registration reads it first.
*/
export async function nativeRunnerCheckpointRemoteBootstrapEvidence(
db: Pick<Db, "select">, binding: Binding, normalizedSessionId: string,
): Promise<{ proof: NativeRemoteBootstrapProof; runnerInstanceId: string } | null> {
try {
const limit = BOOTSTRAP_CHECKPOINT_PARTS.size + MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + MAX_BOOTSTRAP_CHECKPOINT_EVENTS;
const parts = await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(limit + 1);
const eventParts = parts.map(row => row.part).filter(part => /^event:\d+$/.test(part));
const commandParts = parts.map(row => row.part).filter(part => /^command:\d+$/.test(part));
if (!parts.length || parts.length > limit || eventParts.length > MAX_BOOTSTRAP_CHECKPOINT_EVENTS
|| commandParts.length > MAX_BOOTSTRAP_CHECKPOINT_COMMANDS
|| parts.some(row => !BOOTSTRAP_CHECKPOINT_PARTS.has(row.part) && !/^(?:event|command):\d+$/.test(row.part))) return null;
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
if (!checkpoint || checkpoint.predecessor !== null
|| checkpoint.recoveryEvidence.normalizedSessionId !== normalizedSessionId) return null;
const runnerInstanceId = checkpoint.recoveryEvidence.runnerInstanceId;
// The decoder above validated these canonical v2 records. Rebuild the
// controller state in memory without writing files under recovery locks.
const read = (part: string) => {
const value = checkpoint.records.get(part);
return value === undefined ? undefined : JSON.parse(value) as Record<string, unknown>;
};
const header = read("header");
const cursor = read("cursor");
if (header?.schema !== "paperclip.runner.controller-checkpoint.v2" || !cursor
|| cursor.commandCount !== commandParts.length) return null;
const state: Record<string, unknown> = {
...(header.state as Record<string, unknown>),
ackedSourceSeq: cursor.ackedSourceSeq,
commands: commandParts.map(part => Number(part.slice("command:".length))).sort((a, b) => a - b)
.map(seq => read(`command:${seq}`)),
committedEvents: eventParts.map(part => Number(part.slice("event:".length))).sort((a, b) => a - b)
.map(seq => read(`event:${seq}`)),
};
if ((state.identity as { runId?: unknown } | undefined)?.runId !== binding.runId) return null;
if (runnerdStateIsIncompleteBootstrap(state)) return { proof: "bootstrap_incomplete", runnerInstanceId };
return runnerdStateHasNoProviderTurn(state) ? { proof: "session_opened_without_turn", runnerInstanceId } : null;
} catch {
return null;
}
}
export async function restoreNativeRunnerCheckpoint(db: Db, binding: Binding, root: string, normalizedSessionId?: string) {
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
if (!checkpoint) return null;
if (normalizedSessionId !== undefined && checkpoint.recoveryEvidence.normalizedSessionId !== normalizedSessionId) {
throw new Error("native_runner_checkpoint_binding_mismatch");
}
checkpoint.restoreSessionDirectory(root);
return checkpoint;
}
export async function hasNativeRunnerCheckpoint(db: Pick<Db, "select">, binding: Binding): Promise<boolean> {
const [row] = await db.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(1);
return row !== undefined;
}
/** Recovery history dispositions of a restart claim that took a run over. */
const NATIVE_RESTART_CLAIM_DISPOSITIONS = ["reattach_existing_runner", "reattach_remote_runner",
"resume_dead_runner", "bootstrap_incomplete"] as const;
export async function listExpiredNativeRunnerAuthorities(db: Db, now: Date, runIds: string[]): Promise<string[]> {
if (!runIds.length) return [];
// The caller supplies the finite expired active-run inventory. Return all of
// it: a blocked owner's unchanged row must not starve later recoverable runs.
const rows = await db.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
.where(and(inArray(nativeRunFinalizations.runId, runIds),
eq(heartbeatRuns.status, "running"), eq(nativeRunFinalizations.phase, "observed"),
isNull(nativeRunFinalizations.resultId),
or(isNull(nativeRunFinalizations.leaseExpiresAt), lte(nativeRunFinalizations.leaseExpiresAt, now)),
or(exists(db.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.runId, nativeRunFinalizations.runId),
eq(nativeRunnerCheckpoints.issueId, nativeRunFinalizations.issueId),
eq(nativeRunnerCheckpoints.companyId, nativeRunFinalizations.companyId)))),
and(sql`${heartbeatRuns.runnerProfileJson}->'nativePreparationFencingVersion' = ${JSON.stringify(NATIVE_PREPARATION_FENCING_VERSION)}::jsonb`,
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt)),
// A graceful shutdown handed this run to the next controller. That
// handoff exists before any checkpoint (for example, a run stopped
// just after its claim), and a follow-up may carry its predecessor's
// Runner identity. Without it nothing but a later server boot would
// look at the run again. The restart claim still proves the takeover.
// It stays eligible through ownership waits (the departing server may
// still be alive at the first look) until a claim follows the handoff.
and(inArray(nativeRunFinalizations.recoveryState, ["awaiting_runner_reattach", "awaiting_evidence"]),
sql`exists (select 1 from jsonb_array_elements(coalesce(${nativeRunFinalizations.recoveryHistory}, '[]'::jsonb))
with ordinality as handoff(item, ordinal)
where handoff.item ->> 'disposition' = 'restart_suspended'
and not exists (select 1 from jsonb_array_elements(${nativeRunFinalizations.recoveryHistory})
with ordinality as later(item, ordinal)
where later.ordinal > handoff.ordinal and later.item ->> 'disposition' in (${sql.join(
NATIVE_RESTART_CLAIM_DISPOSITIONS.map(disposition => sql`${disposition}`), sql`, `)})))`))));
return rows.map(row => row.runId);
}
/** Only a header that decrypted has a repeatable answer: a key or material failure may not recur. */
function inspectUpgradeHeaderMaterial(binding: Binding, material: Record<string, unknown>) {
let contents: string;
try { contents = decryptPart(createLocalEncryptedCipher("read"), binding, "header", material); }
catch { return { decrypted: false as const, identity: null }; }
return { decrypted: true as const, identity: inspectDurablePrpCheckpointHeader(contents) };
}
/** Inspect only the bounded encrypted header. Protocol interpretation stays in Runner. */
export function inspectNativeRunnerUpgradeHeader(binding: Binding, material: Record<string, unknown>) {
return inspectUpgradeHeaderMaterial(binding, material).identity;
}
/** The only header fields that rollout evidence compares, and that a memo keeps. */
export type NativeRunnerUpgradeHeaderIdentity = Pick<DurableRecoveryIdentity,
"runId" | "runnerInstanceId" | "normalizedSessionId" | "environmentLeaseId">;
const upgradeHeaderIdentity = (identity: DurableRecoveryIdentity | null): NativeRunnerUpgradeHeaderIdentity | null => identity && {
runId: identity.runId, runnerInstanceId: identity.runnerInstanceId,
normalizedSessionId: identity.normalizedSessionId, environmentLeaseId: identity.environmentLeaseId };
/** One read of header evidence. `known` is a JSON object from binding to the row
* revision this read answers without the material. Call `finish` after a completed read. */
export type NativeRunnerUpgradeHeaderRead = {
readonly known: string;
inspect(binding: Binding, revision: string, material: Record<string, unknown> | null): NativeRunnerUpgradeHeaderIdentity | null;
finish(): void;
};
/** PostgreSQL writes a new row version, with a new xmin or ctid, for every change to
* a row, so an unchanged revision means unchanged material. The write time and stored
* size are included so that no single field carries that guarantee. */
export function nativeRunnerCheckpointRevisionSql(checkpoint: SQL) {
return sql`${checkpoint}.xmin::text || ':' || ${checkpoint}.ctid::text
|| ':' || (extract(epoch from ${checkpoint}.updated_at) * 1000000)::bigint::text
|| ':' || pg_column_size(${checkpoint}.material)::text`;
}
/** `material` must already be null for a row the reader will not inspect. The statement
* that reads a run sends its header's material only for a revision the read does not know. */
export function nativeRunnerUpgradeHeaderSql(read: Pick<NativeRunnerUpgradeHeaderRead, "known">,
row: { companyId: SQL; issueId: SQL; runId: SQL; revision: SQL; material: SQL }) {
return {
join: sql`left join jsonb_each_text(${read.known}::jsonb) as known_header(binding, revision)
on known_header.binding = ${row.companyId}::text || '/' || ${row.issueId}::text || '/' || ${row.runId}::text`,
revision: sql`case when ${row.material} is not null then ${row.revision} end`,
material: sql`case when ${row.revision} is distinct from known_header.revision then ${row.material} end`,
};
}
/** A read that remembers nothing: every inspected header is sent and decrypted. */
export function directNativeRunnerUpgradeHeaderRead(): NativeRunnerUpgradeHeaderRead {
return { known: "{}", finish() {},
inspect: (binding, _revision, material) => material ? upgradeHeaderIdentity(inspectNativeRunnerUpgradeHeader(binding, material)) : null };
}
/** A digest of a value derived from the secrets key for this purpose alone. It opens nothing. */
function upgradeHeaderKeyIdentity(): string | null {
try {
return createHash("sha256").update(deriveLocalEncryptedInstanceSecret("native-runner-upgrade-header-memo",
process.env, { allowCreate: false })).digest("hex");
} catch { return null; }
}
/** Remembers the answer for a header that decrypted, so an unchanged header is not
* sent or decrypted again. An answer is reused only for the same binding, row revision
* and secrets key; a header that did not decrypt is read and tried on every read.
* An entry holds four identifiers, entries are bounded in number and bytes, and a
* read forgets every header it did not inspect, such as that of a run that ended. */
export function createNativeRunnerUpgradeHeaderMemo(limits: { maxEntries?: number; maxBytes?: number } = {}) {
const maxEntries = limits.maxEntries ?? 1_200, maxBytes = limits.maxBytes ?? 1024 * 1024;
type Answer = { revision: string; identity: NativeRunnerUpgradeHeaderIdentity | null };
const entries = new Map<string, Answer & { keyIdentity: string; bytes: number }>();
let bytes = 0;
const forget = (key: string) => {
const entry = entries.get(key);
if (entry && entries.delete(key)) bytes -= entry.bytes;
};
return {
get size() { return entries.size; },
get bytes() { return bytes; },
read(): NativeRunnerUpgradeHeaderRead {
let keyIdentity = entries.size ? upgradeHeaderKeyIdentity() : undefined;
const usable = new Map([...entries].filter(([, entry]) => entry.keyIdentity === keyIdentity));
const reused = new Set<string>(), decrypted = new Map<string, Answer>();
let keyChanged = false;
// A direct read loads the key for each header, after the query. An answer from memory
// is as conservative only if the key is loaded and unchanged when the answer is used.
const sameKey = () => {
const now = upgradeHeaderKeyIdentity();
if (keyIdentity === undefined) keyIdentity = now;
const same = now !== null && now === keyIdentity;
if (!same) keyChanged = true;
return same;
};
return {
known: JSON.stringify(Object.fromEntries([...usable].map(([key, entry]) => [key, entry.revision]))),
inspect(binding, revision, material) {
const key = `${binding.companyId}/${binding.issueId}/${binding.runId}`;
if (!material) {
// The query withholds material only for a revision this read offered.
const entry = usable.get(key);
if (entry?.revision !== revision || !sameKey()) return null;
reused.add(key);
return entry.identity;
}
const before = sameKey();
const result = inspectUpgradeHeaderMaterial(binding, material);
const identity = upgradeHeaderIdentity(result.identity);
if (result.decrypted && before && sameKey()) decrypted.set(key, { revision, identity });
else decrypted.delete(key);
return identity;
},
finish() {
// A read that saw the key missing or changed keeps and remembers nothing.
const current = keyChanged || !keyIdentity ? null : keyIdentity;
for (const [key, entry] of entries) {
if (!reused.has(key) || entry.keyIdentity !== current) forget(key);
}
for (const [key, answer] of current ? decrypted : []) {
const size = 2 * (key.length + answer.revision.length + current!.length
+ Object.values(answer.identity ?? {}).reduce((sum, value) => sum + value.length, 0));
forget(key);
entries.set(key, { ...answer, keyIdentity: current!, bytes: size });
bytes += size;
}
for (const key of entries.keys()) {
if (entries.size <= maxEntries && bytes <= maxBytes) break;
forget(key);
}
},
};
},
};
}