Files
PaperClipAI/server/src/services/native-runtime/native-runner-checkpoint.ts
T
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00

552 lines
33 KiB
TypeScript

import { readNativeCheckpointRetirement } from "./native-runner-checkpoint-retirement.js";
import { NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { createHash } from "node:crypto";
import { and, eq, exists, gt, isNull, lte, inArray, notExists, or, sql, type SQL } from "drizzle-orm";
import { checkpointDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
import { inspectDurablePrpCheckpointHeader, inspectUnauthenticatedBootstrapCheckpoint, inspectUnclaimedBootstrapCheckpoint, decodeDurablePrpCheckpoint, type DurablePrpCheckpoint, type DurablePrpControlPlane, type DurableRecoveryIdentity } from "../../vendor/paperclip-runner/index.js";
import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from "../../secrets/local-encrypted-provider.js";
import { legacyControllerBootId } from "../legacy-controller-lease.js";
import { assertNativeSourceCheckpointConsumer, lockSourceRecoveryCoordinators } from "./native-source-recovery-ownership.js";
import { runnerdStateHasNoProviderTurn, runnerdStateIsIncompleteBootstrap } from "./native-bootstrap-proof.js";
type Binding = { companyId: string; issueId: string; runId: string };
export const NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES = 1024 * 1024;
export const NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES = 8 * 1024 * 1024;
export type NativeRunnerCheckpointOwner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string };
export type NativeRunnerCheckpointFenceOwner = Omit<NativeRunnerCheckpointOwner, "leaseOwner"> & { leaseOwner: string | null };
function encryptParts(binding: Binding, records: ReadonlyMap<string, string>) {
if (!records.size) return [];
const cipher = createLocalEncryptedCipher("write");
return [...records].map(([part, contents]) => ({
companyId: binding.companyId, issueId: binding.issueId, runId: binding.runId, part,
material: cipher.encrypt(JSON.stringify({ companyId: binding.companyId, issueId: binding.issueId, runId: binding.runId, part, contents })),
}));
}
export async function insertMigratedNativeRunnerCheckpoint(tx: CheckpointTransaction, binding: Binding, checkpoint: DurablePrpCheckpoint) {
if (checkpoint.runId !== binding.runId) throw new Error("native_runner_checkpoint_binding_mismatch");
let batch: ReturnType<typeof encryptParts> = [];
let batchBytes = 0;
for (const part of encryptParts(binding, checkpoint.records)) {
const bytes = Buffer.byteLength(JSON.stringify(part));
if (batch.length && (batch.length >= 64 || batchBytes + bytes > 512 * 1024)) {
await tx.insert(nativeRunnerCheckpoints).values(batch);
batch = [];
batchBytes = 0;
}
batch.push(part);
batchBytes += bytes;
}
if (batch.length) await tx.insert(nativeRunnerCheckpoints).values(batch);
}
function decryptPart(cipher: ReturnType<typeof createLocalEncryptedCipher>, binding: Binding,
part: string, material: Record<string, unknown>): string {
const envelope = JSON.parse(cipher.decrypt(material));
if (envelope.companyId !== binding.companyId || envelope.issueId !== binding.issueId
|| envelope.runId !== binding.runId || envelope.part !== part || typeof envelope.contents !== "string") {
throw new Error("native_runner_checkpoint_binding_mismatch");
}
return envelope.contents;
}
type CheckpointTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
/** A dependent run-lock query can consume this CTE to preserve coordinator-first
* lock order without adding a second database round trip. */
export function nativeRunnerCheckpointOwnerLock(tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner) {
return tx.$with("checkpoint_owner").as(tx.select({
leaseOwner: nativeRunFinalizations.leaseOwner,
leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
resultId: nativeRunFinalizations.resultId,
phase: nativeRunFinalizations.phase,
}).from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, owner.companyId), eq(nativeRunFinalizations.issueId, owner.issueId),
eq(nativeRunFinalizations.runId, owner.authorityRunId), eq(nativeRunFinalizations.controllerBootId, owner.bootId),
eq(nativeRunFinalizations.controllerGeneration, owner.generation))).for("update"));
}
/** Locks the coordinator before callers lock a run. A caller that subsequently
* waits for another lock must recheck the fence before writing. */
export async function lockNativeRunnerCheckpointOwner(
tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner,
) {
// A locking CTE acquires ownership before the outer projection samples
// wall time. Sampling in the locking SELECT can precede its lock wait.
// Keep both steps in one statement without another database round trip.
const locked = nativeRunnerCheckpointOwnerLock(tx, owner);
const [owned] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner,
leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, phase: locked.phase, now: sql<Date>`clock_timestamp()` }).from(locked);
if (!owned) throw new Error("native_runner_checkpoint_owner_lost");
return owned;
}
export async function validateNativeRunnerCheckpointOwner(
tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner,
owned: Omit<Awaited<ReturnType<typeof lockNativeRunnerCheckpointOwner>>, "phase">, newWork: boolean,
): Promise<void> {
const current = owned;
const activeOwner = owner.leaseOwner !== null && current.leaseOwner === owner.leaseOwner
&& current.leaseExpiresAt !== null && current.leaseExpiresAt > new Date(owned.now);
if (!activeOwner) {
await assertNativeSourceCheckpointConsumer(tx, owner);
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
.where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId)));
const [successor] = task?.executionRunId && task.executionRunId !== owner.runId
? await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, task.executionRunId), eq(nativeRunFinalizations.companyId, owner.companyId),
eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.controllerBootId, owner.bootId),
gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))) : [];
const [preparingSuccessor] = !successor && task?.executionRunId && task.executionRunId !== owner.runId
? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId),
eq(heartbeatRuns.issueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "legacy"),
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.executionStage, "preparing"),
eq(heartbeatRuns.controllerBootId, legacyControllerBootId),
gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : [];
const [waitingSuccessor] = !successor && !preparingSuccessor && task?.executionRunId && task.executionRunId !== owner.runId
? await tx.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, task.executionRunId), eq(heartbeatRuns.companyId, owner.companyId),
eq(heartbeatRuns.issueId, owner.issueId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"]),
notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : [];
if (!current.resultId || newWork || !task
|| (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor && !waitingSuccessor)) {
throw new Error("native_runner_checkpoint_owner_lost");
}
}
}
export async function assertNativeRunnerCheckpointOwner(tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, newWork: boolean) {
const locked = await lockNativeRunnerCheckpointOwner(tx, owner);
await validateNativeRunnerCheckpointOwner(tx, owner, locked, newWork);
}
export function nativeRunnerCheckpointWriter(db: Db, owner: NativeRunnerCheckpointOwner) {
const persistenceDb = checkpointDb(db);
let previousWork = new Set<string>();
return async (checkpoint: DurablePrpCheckpoint): Promise<void> => {
if (checkpoint.runId !== owner.runId) throw new Error("native_runner_checkpoint_binding_mismatch");
const changed = encryptParts(owner, checkpoint.changedRecords);
await persistenceDb.transaction(async tx => {
if (owner.runId !== owner.authorityRunId) {
await lockSourceRecoveryCoordinators(tx, owner.companyId, [owner.runId, owner.authorityRunId]);
await assertNativeSourceCheckpointConsumer(tx, owner, checkpoint);
}
await assertNativeRunnerCheckpointOwner(tx, owner, checkpoint.workCommandIds.some(id => !previousWork.has(id)));
if (checkpoint.reset || checkpoint.changedRecords.has("header") || checkpoint.removedRecords.includes("header")) {
const [header] = await tx.select({ material: nativeRunnerCheckpoints.material }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, owner.runId),
eq(nativeRunnerCheckpoints.part, "header"))).for("share").limit(1);
if (header && readNativeCheckpointRetirement(header.material, owner)) throw new Error("native_runner_checkpoint_retired");
}
if (checkpoint.predecessor) {
const [prior] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, checkpoint.predecessor.runId), eq(heartbeatRuns.companyId, owner.companyId),
eq(heartbeatRuns.nativeIssueId, owner.issueId),
eq(heartbeatRuns.nativeSessionId, checkpoint.predecessor.normalizedSessionId)));
if (!prior) throw new Error("native_runner_checkpoint_binding_mismatch");
}
await notifyLiveRunRecovery(tx, { companyId: owner.companyId, runId: owner.runId });
if (checkpoint.reset) await tx.delete(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, owner.runId)));
else if (checkpoint.removedRecords.length) await tx.delete(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, owner.runId),
inArray(nativeRunnerCheckpoints.part, [...checkpoint.removedRecords])));
if (changed.length) await tx.insert(nativeRunnerCheckpoints).values(changed)
.onConflictDoUpdate({ target: [nativeRunnerCheckpoints.runId, nativeRunnerCheckpoints.part],
set: { material: sql`excluded.material`, updatedAt: sql`now()` } });
});
previousWork = new Set(checkpoint.workCommandIds);
};
}
const authorities = new WeakMap<DurablePrpControlPlane, Map<string, { fence: string; write: ReturnType<typeof nativeRunnerCheckpointWriter> }>>();
export function nativeRunnerCheckpointPersistence(db: Db, bootId: string, authorityRunId: string | (() => string),
detach: (runIds: readonly string[], leaseOwner?: string) => Promise<unknown>) {
return { db, bootId, authorityRunId, onFailure: async (runId: string, leaseOwner: string) => {
try { await detach([runId], leaseOwner); }
finally {
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: sql`clock_timestamp()` })
.where(and(eq(nativeRunFinalizations.runId, runId), eq(nativeRunFinalizations.controllerBootId, bootId), eq(nativeRunFinalizations.leaseOwner, leaseOwner),
isNull(nativeRunFinalizations.resultId)));
}
} };
}
/** Attachment identity owns the stored records; current business authority owns
* the write lease. They differ when a completed warm session is recovered. */
export function nativeRunnerCheckpointRegistration(input: Omit<Parameters<typeof registerNativeRunnerCheckpoint>[0], "authority">) {
return (authority: DurablePrpControlPlane, identity?: { runId: string }) => registerNativeRunnerCheckpoint({
...input, authority, binding: { ...input.binding, runId: identity?.runId ?? input.binding.runId },
});
}
export async function registerNativeRunnerCheckpoint(input: {
db: Db; binding: Binding; bootId: string; authority: DurablePrpControlPlane;
authorityRunId?: string | (() => string);
onFailure?: (runId: string, leaseOwner: string) => Promise<void>;
}): Promise<void> {
const authorityRunId = typeof input.authorityRunId === "function"
? input.authorityRunId() : input.authorityRunId ?? input.binding.runId;
const [owner] = await input.db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.binding.companyId),
eq(nativeRunFinalizations.issueId, input.binding.issueId),
eq(nativeRunFinalizations.runId, authorityRunId),
eq(nativeRunFinalizations.controllerBootId, input.bootId),
gt(nativeRunFinalizations.leaseExpiresAt, sql`now()`)));
if (!owner?.leaseOwner) throw new Error("native_runner_checkpoint_owner_lost");
let writers = authorities.get(input.authority);
if (!writers) {
writers = new Map();
const boundWriters = writers;
input.authority.setCheckpointPersistence(async snapshot => {
const writer = boundWriters.get(snapshot.runId);
if (!writer) throw new Error("native_runner_checkpoint_owner_lost");
await writer.write(snapshot);
});
authorities.set(input.authority, writers);
}
const fence = JSON.stringify([authorityRunId, input.bootId, owner.controllerGeneration, owner.leaseOwner]);
if (writers.get(input.binding.runId)?.fence !== fence) {
const write = nativeRunnerCheckpointWriter(input.db, {
...input.binding, authorityRunId, bootId: input.bootId, generation: owner.controllerGeneration, leaseOwner: owner.leaseOwner,
});
writers.set(input.binding.runId, { fence, write: async checkpoint => {
try { await write(checkpoint); }
catch (error) {
void input.onFailure?.(authorityRunId, owner.leaseOwner!).catch(() => undefined);
throw error;
}
} });
}
await input.authority.flushCheckpoint();
}
/** Exact-run bounded metadata read; never loads command/event payload records. */
export async function loadNativeRunnerCheckpointHeader(db: Pick<Db, "select">, binding: Binding): Promise<string | null> {
const [row] = await db.select({ material: nativeRunnerCheckpoints.material }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId), eq(nativeRunnerCheckpoints.part, "header"))).limit(1);
if (!row) return null;
if (readNativeCheckpointRetirement(row.material, binding)) throw new Error("native_runner_checkpoint_retired");
try { return decryptPart(createLocalEncryptedCipher("read"), binding, "header", row.material); }
catch { return null; }
}
export async function loadNativeRunnerCheckpoint(db: Pick<Db, "select">, binding: Binding): Promise<DurablePrpCheckpoint | null> {
const rows = await db.select().from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId)));
if (!rows.length) return null;
const header = rows.find(row => row.part === "header");
if (header && readNativeCheckpointRetirement(header.material, binding)) throw new Error("native_runner_checkpoint_retired");
const cipher = createLocalEncryptedCipher("read");
const parts = new Map(rows.map(row =>
[row.part, decryptPart(cipher, binding, row.part, row.material)] as const));
const checkpoint = decodeDurablePrpCheckpoint(parts);
if (checkpoint.runId !== binding.runId) throw new Error("native_runner_checkpoint_binding_mismatch");
return checkpoint;
}
export async function loadNativeUnclaimedBootstrapCheckpoint(
db: Pick<Db, "select">, binding: Binding, identity: DurableRecoveryIdentity,
) {
try {
if (binding.runId !== identity.runId) return null;
const rows = await db.select().from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(5);
if (rows.length !== 4 || rows.reduce((bytes, row) => bytes + Buffer.byteLength(JSON.stringify(row.material)), 0)
> 2 * NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES) return null;
const cipher = createLocalEncryptedCipher("read");
const records = new Map(rows.map(row => [row.part, decryptPart(cipher, binding, row.part, row.material)] as const));
return inspectUnclaimedBootstrapCheckpoint(records, identity);
} catch { return null; }
}
export async function nativeRunnerNeverAuthenticated(
db: Pick<Db, "select">, binding: Binding,
identity: Pick<DurableRecoveryIdentity, "runId" | "normalizedSessionId" | "runnerInstanceId">,
): Promise<boolean> {
try {
if (binding.runId !== identity.runId) return false;
const rows = await db.select().from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + 3);
if (rows.length < 4 || rows.length > MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + 2
|| rows.reduce((bytes, row) => bytes + Buffer.byteLength(JSON.stringify(row.material)), 0)
> 2 * NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES) return false;
const cipher = createLocalEncryptedCipher("read");
const records = new Map(rows.map(row => [row.part, decryptPart(cipher, binding, row.part, row.material)] as const));
return inspectUnauthenticatedBootstrapCheckpoint(records, identity);
} catch { return false; }
}
const BOOTSTRAP_CHECKPOINT_PARTS = new Set(["header", "cursor"]);
const MAX_BOOTSTRAP_CHECKPOINT_COMMANDS = 8;
/** Session-open events are few; a longer journal is never a bootstrap. */
const MAX_BOOTSTRAP_CHECKPOINT_EVENTS = 16;
export type NativeRemoteBootstrapProof = "bootstrap_incomplete" | "session_opened_without_turn";
/**
* A remote runner's controller state lives only in this checkpoint, and every
* command is written here before the controller sends it. Once the previous
* controller's lease has expired, the checkpoint is therefore a complete record
* of what reached the sandbox runner. Returns which proof holds:
* `bootstrap_incomplete` when it holds only run.prepare and an unfinished
* session.open with no committed or acknowledged event, or
* `session_opened_without_turn` when session.open completed and every committed
* event is a session-open event. Either way no provider turn can exist. The
* sandbox runner journal is not on this host; the proof uses the controller
* record alone, as for a local root whose runner never journaled.
*/
export async function nativeRunnerCheckpointRemoteBootstrapProof(
db: Pick<Db, "select">, binding: Binding, expected: { runnerInstanceId: string; normalizedSessionId: string },
): Promise<NativeRemoteBootstrapProof | null> {
const evidence = await nativeRunnerCheckpointRemoteBootstrapEvidence(db, binding, expected.normalizedSessionId);
return evidence?.runnerInstanceId === expected.runnerInstanceId ? evidence.proof : null;
}
/**
* The same proof, with the runner instance the checkpoint names. A caller that
* must hold the proof across a new controller's registration reads it first.
*/
export async function nativeRunnerCheckpointRemoteBootstrapEvidence(
db: Pick<Db, "select">, binding: Binding, normalizedSessionId: string,
): Promise<{ proof: NativeRemoteBootstrapProof; runnerInstanceId: string } | null> {
try {
const limit = BOOTSTRAP_CHECKPOINT_PARTS.size + MAX_BOOTSTRAP_CHECKPOINT_COMMANDS + MAX_BOOTSTRAP_CHECKPOINT_EVENTS;
const parts = await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(limit + 1);
const eventParts = parts.map(row => row.part).filter(part => /^event:\d+$/.test(part));
const commandParts = parts.map(row => row.part).filter(part => /^command:\d+$/.test(part));
if (!parts.length || parts.length > limit || eventParts.length > MAX_BOOTSTRAP_CHECKPOINT_EVENTS
|| commandParts.length > MAX_BOOTSTRAP_CHECKPOINT_COMMANDS
|| parts.some(row => !BOOTSTRAP_CHECKPOINT_PARTS.has(row.part) && !/^(?:event|command):\d+$/.test(row.part))) return null;
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
if (!checkpoint || checkpoint.predecessor !== null
|| checkpoint.recoveryEvidence.normalizedSessionId !== normalizedSessionId) return null;
const runnerInstanceId = checkpoint.recoveryEvidence.runnerInstanceId;
// The decoder above validated these canonical v2 records. Rebuild the
// controller state in memory without writing files under recovery locks.
const read = (part: string) => {
const value = checkpoint.records.get(part);
return value === undefined ? undefined : JSON.parse(value) as Record<string, unknown>;
};
const header = read("header");
const cursor = read("cursor");
if (header?.schema !== "paperclip.runner.controller-checkpoint.v2" || !cursor
|| cursor.commandCount !== commandParts.length) return null;
const state: Record<string, unknown> = {
...(header.state as Record<string, unknown>),
ackedSourceSeq: cursor.ackedSourceSeq,
commands: commandParts.map(part => Number(part.slice("command:".length))).sort((a, b) => a - b)
.map(seq => read(`command:${seq}`)),
committedEvents: eventParts.map(part => Number(part.slice("event:".length))).sort((a, b) => a - b)
.map(seq => read(`event:${seq}`)),
};
if ((state.identity as { runId?: unknown } | undefined)?.runId !== binding.runId) return null;
if (runnerdStateIsIncompleteBootstrap(state)) return { proof: "bootstrap_incomplete", runnerInstanceId };
return runnerdStateHasNoProviderTurn(state) ? { proof: "session_opened_without_turn", runnerInstanceId } : null;
} catch {
return null;
}
}
export async function restoreNativeRunnerCheckpoint(db: Db, binding: Binding, root: string, normalizedSessionId?: string) {
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
if (!checkpoint) return null;
if (normalizedSessionId !== undefined && checkpoint.recoveryEvidence.normalizedSessionId !== normalizedSessionId) {
throw new Error("native_runner_checkpoint_binding_mismatch");
}
checkpoint.restoreSessionDirectory(root);
return checkpoint;
}
export async function hasNativeRunnerCheckpoint(db: Pick<Db, "select">, binding: Binding): Promise<boolean> {
const [row] = await db.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(1);
return row !== undefined;
}
/** Recovery history dispositions of a restart claim that took a run over. */
const NATIVE_RESTART_CLAIM_DISPOSITIONS = ["reattach_existing_runner", "reattach_remote_runner",
"resume_dead_runner", "bootstrap_incomplete"] as const;
export async function listExpiredNativeRunnerAuthorities(db: Db, now: Date, runIds: string[]): Promise<string[]> {
if (!runIds.length) return [];
// The caller supplies the finite expired active-run inventory. Return all of
// it: a blocked owner's unchanged row must not starve later recoverable runs.
const rows = await db.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
.where(and(inArray(nativeRunFinalizations.runId, runIds),
eq(heartbeatRuns.status, "running"), eq(nativeRunFinalizations.phase, "observed"),
isNull(nativeRunFinalizations.resultId),
or(isNull(nativeRunFinalizations.leaseExpiresAt), lte(nativeRunFinalizations.leaseExpiresAt, now)),
or(exists(db.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.runId, nativeRunFinalizations.runId),
eq(nativeRunnerCheckpoints.issueId, nativeRunFinalizations.issueId),
eq(nativeRunnerCheckpoints.companyId, nativeRunFinalizations.companyId)))),
and(sql`${heartbeatRuns.runnerProfileJson}->'nativePreparationFencingVersion' = ${JSON.stringify(NATIVE_PREPARATION_FENCING_VERSION)}::jsonb`,
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt)),
// A graceful shutdown handed this run to the next controller. That
// handoff exists before any checkpoint (for example, a run stopped
// just after its claim), and a follow-up may carry its predecessor's
// Runner identity. Without it nothing but a later server boot would
// look at the run again. The restart claim still proves the takeover.
// It stays eligible through ownership waits (the departing server may
// still be alive at the first look) until a claim follows the handoff.
and(inArray(nativeRunFinalizations.recoveryState, ["awaiting_runner_reattach", "awaiting_evidence"]),
sql`exists (select 1 from jsonb_array_elements(coalesce(${nativeRunFinalizations.recoveryHistory}, '[]'::jsonb))
with ordinality as handoff(item, ordinal)
where handoff.item ->> 'disposition' = 'restart_suspended'
and not exists (select 1 from jsonb_array_elements(${nativeRunFinalizations.recoveryHistory})
with ordinality as later(item, ordinal)
where later.ordinal > handoff.ordinal and later.item ->> 'disposition' in (${sql.join(
NATIVE_RESTART_CLAIM_DISPOSITIONS.map(disposition => sql`${disposition}`), sql`, `)})))`))));
return rows.map(row => row.runId);
}
/** Only a header that decrypted has a repeatable answer: a key or material failure may not recur. */
function inspectUpgradeHeaderMaterial(binding: Binding, material: Record<string, unknown>) {
let contents: string;
try { contents = decryptPart(createLocalEncryptedCipher("read"), binding, "header", material); }
catch { return { decrypted: false as const, identity: null }; }
return { decrypted: true as const, identity: inspectDurablePrpCheckpointHeader(contents) };
}
/** Inspect only the bounded encrypted header. Protocol interpretation stays in Runner. */
export function inspectNativeRunnerUpgradeHeader(binding: Binding, material: Record<string, unknown>) {
return inspectUpgradeHeaderMaterial(binding, material).identity;
}
/** The only header fields that rollout evidence compares, and that a memo keeps. */
export type NativeRunnerUpgradeHeaderIdentity = Pick<DurableRecoveryIdentity,
"runId" | "runnerInstanceId" | "normalizedSessionId" | "environmentLeaseId">;
const upgradeHeaderIdentity = (identity: DurableRecoveryIdentity | null): NativeRunnerUpgradeHeaderIdentity | null => identity && {
runId: identity.runId, runnerInstanceId: identity.runnerInstanceId,
normalizedSessionId: identity.normalizedSessionId, environmentLeaseId: identity.environmentLeaseId };
/** One read of header evidence. `known` is a JSON object from binding to the row
* revision this read answers without the material. Call `finish` after a completed read. */
export type NativeRunnerUpgradeHeaderRead = {
readonly known: string;
inspect(binding: Binding, revision: string, material: Record<string, unknown> | null): NativeRunnerUpgradeHeaderIdentity | null;
finish(): void;
};
/** PostgreSQL writes a new row version, with a new xmin or ctid, for every change to
* a row, so an unchanged revision means unchanged material. The write time and stored
* size are included so that no single field carries that guarantee. */
export function nativeRunnerCheckpointRevisionSql(checkpoint: SQL) {
return sql`${checkpoint}.xmin::text || ':' || ${checkpoint}.ctid::text
|| ':' || (extract(epoch from ${checkpoint}.updated_at) * 1000000)::bigint::text
|| ':' || pg_column_size(${checkpoint}.material)::text`;
}
/** `material` must already be null for a row the reader will not inspect. The statement
* that reads a run sends its header's material only for a revision the read does not know. */
export function nativeRunnerUpgradeHeaderSql(read: Pick<NativeRunnerUpgradeHeaderRead, "known">,
row: { companyId: SQL; issueId: SQL; runId: SQL; revision: SQL; material: SQL }) {
return {
join: sql`left join jsonb_each_text(${read.known}::jsonb) as known_header(binding, revision)
on known_header.binding = ${row.companyId}::text || '/' || ${row.issueId}::text || '/' || ${row.runId}::text`,
revision: sql`case when ${row.material} is not null then ${row.revision} end`,
material: sql`case when ${row.revision} is distinct from known_header.revision then ${row.material} end`,
};
}
/** A read that remembers nothing: every inspected header is sent and decrypted. */
export function directNativeRunnerUpgradeHeaderRead(): NativeRunnerUpgradeHeaderRead {
return { known: "{}", finish() {},
inspect: (binding, _revision, material) => material ? upgradeHeaderIdentity(inspectNativeRunnerUpgradeHeader(binding, material)) : null };
}
/** A digest of a value derived from the secrets key for this purpose alone. It opens nothing. */
function upgradeHeaderKeyIdentity(): string | null {
try {
return createHash("sha256").update(deriveLocalEncryptedInstanceSecret("native-runner-upgrade-header-memo",
process.env, { allowCreate: false })).digest("hex");
} catch { return null; }
}
/** Remembers the answer for a header that decrypted, so an unchanged header is not
* sent or decrypted again. An answer is reused only for the same binding, row revision
* and secrets key; a header that did not decrypt is read and tried on every read.
* An entry holds four identifiers, entries are bounded in number and bytes, and a
* read forgets every header it did not inspect, such as that of a run that ended. */
export function createNativeRunnerUpgradeHeaderMemo(limits: { maxEntries?: number; maxBytes?: number } = {}) {
const maxEntries = limits.maxEntries ?? 1_200, maxBytes = limits.maxBytes ?? 1024 * 1024;
type Answer = { revision: string; identity: NativeRunnerUpgradeHeaderIdentity | null };
const entries = new Map<string, Answer & { keyIdentity: string; bytes: number }>();
let bytes = 0;
const forget = (key: string) => {
const entry = entries.get(key);
if (entry && entries.delete(key)) bytes -= entry.bytes;
};
return {
get size() { return entries.size; },
get bytes() { return bytes; },
read(): NativeRunnerUpgradeHeaderRead {
let keyIdentity = entries.size ? upgradeHeaderKeyIdentity() : undefined;
const usable = new Map([...entries].filter(([, entry]) => entry.keyIdentity === keyIdentity));
const reused = new Set<string>(), decrypted = new Map<string, Answer>();
let keyChanged = false;
// A direct read loads the key for each header, after the query. An answer from memory
// is as conservative only if the key is loaded and unchanged when the answer is used.
const sameKey = () => {
const now = upgradeHeaderKeyIdentity();
if (keyIdentity === undefined) keyIdentity = now;
const same = now !== null && now === keyIdentity;
if (!same) keyChanged = true;
return same;
};
return {
known: JSON.stringify(Object.fromEntries([...usable].map(([key, entry]) => [key, entry.revision]))),
inspect(binding, revision, material) {
const key = `${binding.companyId}/${binding.issueId}/${binding.runId}`;
if (!material) {
// The query withholds material only for a revision this read offered.
const entry = usable.get(key);
if (entry?.revision !== revision || !sameKey()) return null;
reused.add(key);
return entry.identity;
}
const before = sameKey();
const result = inspectUpgradeHeaderMaterial(binding, material);
const identity = upgradeHeaderIdentity(result.identity);
if (result.decrypted && before && sameKey()) decrypted.set(key, { revision, identity });
else decrypted.delete(key);
return identity;
},
finish() {
// A read that saw the key missing or changed keeps and remembers nothing.
const current = keyChanged || !keyIdentity ? null : keyIdentity;
for (const [key, entry] of entries) {
if (!reused.has(key) || entry.keyIdentity !== current) forget(key);
}
for (const [key, answer] of current ? decrypted : []) {
const size = 2 * (key.length + answer.revision.length + current!.length
+ Object.values(answer.identity ?? {}).reduce((sum, value) => sum + value.length, 0));
forget(key);
entries.set(key, { ...answer, keyIdentity: current!, bytes: size });
bytes += size;
}
for (const key of entries.keys()) {
if (entries.size <= maxEntries && bytes <= maxBytes) break;
forget(key);
}
},
};
},
};
}