mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server stops and the incoming server takes over the live runs on remote Runners > - The incoming server reached those runs late: it waited up to 60 s for the outgoing server's controller lease to lapse, re-verified each run's whole log from object storage, and deferred a reattach when the agent directory was busy. Queued starts competed with the reattaches for the same resources. > - On hosted staging, the later a live run was reached, the more likely it was to fail. Runs reached in under 150 s never failed; runs reached after 420 s almost always failed. > - This pull request hands live runs over at the start of shutdown, makes a reattach skip work that the same run already did, and holds queued starts while handed-over runs are being reached, with a 180 s cap. > - The benefit is that live runs are reached within seconds of a swap instead of minutes, so they continue instead of failing. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each run that crossed a swap we measured the gap from the last Runner output to the moment the new server recorded the Runner's process identity: - gap under 150 s: 0 of 9 runs failed - gap over 420 s: 14 of 15 runs failed - median gap: 180 s for runs that resumed, 470 s for runs that failed In one traced run the incoming server waited on `remote_controller_lease_live` three times (09:14:21, 09:15:18, 09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the run's whole log from object storage. A control run spent 23 s in `task.prepare`. Queued starts used the same preparation path, the same 8 object-I/O slots and the same database pool, and nothing gave a reattach priority. The first-attempt failures were `PRP command session.snapshot timed out` and `codex_history_read_failed`. The retry then failed with `runner_harness_state_mismatch: resumed_not_suspended`, `runner_remote_process_identity_unavailable` or `provider_turn_lost_on_restore`. **Expected behavior** Every live run is reached within about 30 s of the swap. Queued work starts after that, or after a bounded wait. **Steps to reproduce** 1. Run a two-server release swap with live remote runs and queued runs. 2. Observe `native.recovery.transition` "waiting" events with `remote_controller_lease_live`, then the claim, then a long `heartbeat.prepare.run_output` span. 3. The new tests in `reattach-phase.test.ts` and `heartbeat-native-early-hand-off.test.ts` model this. **Paperclip version or commit** `master` at426e8a35dc. ## What Changed - Early hand-off at shutdown (`heartbeat/recovery.ts`, `server/src/index.ts`): in deploy-overlap mode, the outgoing server releases the remote controller lease of its live runs at SIGTERM (`recoveryState` `awaiting_runner_reattach`). A run another server already took over is left alone. The incoming server no longer waits for the lease to lapse. - Reattach fast path: - `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`: a reattach starts without waiting for the existing log to be re-verified. - `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that meets a busy agent directory waits in process (at most 30 s) instead of being deferred to a full re-preparation. An ordinary start is still deferred at once, with no added wait. - Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is a worker on the shared delivery coordinator, queue `reattach-phase`. - Trigger: one startup listing of a previous server's live runs; then the hand-over notification (`notifyReattachHandOver`, sent in the hand-over transaction); the earliest lease deadline it watches (a killed owner); and a 2 s re-check only while handed-over runs are still pending, which ends with the phase (cap 180 s). The bounded re-check exists because a run can also end on another path that sends no notification to this queue. - Source of truth: `native_run_finalizations` and `heartbeat_runs` rows. - Quiet condition: nothing pending and no watched lease, so no deadline and no SQL. It stops with the coordinator at shutdown. - Reached hook for every driver (`native-session-executor.ts`): a reattached run counts as reached when the native session runtime hands the executor its recovered or opened session (`onSession`). Every driver reaches that point only after the Runner answered this server's attach and recovery commands, including an adopted `RunnerdExternalDriver` Runner, which records no new process identity. The identity hook stays as an earlier signal. - Claim wake: when the reattach-phase worker finds runs that just became claimable (hand-over notification or a killed owner's lease deadline), it starts this server's restart-recovery claim for them at once. A validation cell showed the claim otherwise waited about 21.5 s after the hand-over. - `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase` return now sits before every action path of `act()`, except starting an overdue saved-work recovery (it resumes an existing run). - The in-transaction admission re-check of a queued claim (`confirmRunAdmissionStillOpen`) also honours the reattach phase. A phase that begins during a claim returns the run to the queue through the existing rollback. - A reattach whose log re-verification runs in the background awaits it before the log is finalized (`settleDeferredLogRecovery`, both finalization sites in `run-completion.ts`). - Reattach phase (`heartbeat/reattach-phase.ts`, `heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while runs that a previous controller handed over (lease released or observed expired, or claimed by this server's restart recovery) are claimable but not yet reached. A run is reached when its process identity is recorded. Authentication is logged as a mark. The cap (180 s, `PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs became claimable. Later batches join or re-arm the phase. Runs reached after 150 s are counted and logged. - Gates on the phase: - `claimQueuedRun`, start-next, resume-queued and dispatch in `heartbeat/queue.ts`, `run-lifecycle.ts` and `native-runner-execution-driver.ts` leave queued runs queued. - `execution-recovery-resolution.ts`: one line, the first line of `deliverReconciledExecutions`. - `recovery/assigned-work-sweep.ts`: one line, outcome `deferred_reattach_phase`. - `heartbeat/recovery.ts`: a one-line claim hook (`noteReattachRunsClaimed`) after restart recovery claims runs. - The owner of those three files agreed to these one-line gates. - `tests/restart-chaos/daytona-image.ts`: `PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a controller-only branch against an existing Daytona image. It accepts a content-ID mismatch only for the exact content ID of the named image, and it prints a TEST OVERRIDE warning. This is harness-only. ## Verification Red on master, green on this branch: - `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six two-server scenarios were red on the earlier phase timing: - `does not hold queued starts while the previous owner is alive and not handing over` - `closes the gate when the previous owner hands its runs over, and counts reach latency from the hand-over` - `runs the cap from the hand-over, so a wedged run ends the phase without the overlap eating it` - `begins when a killed owner's lease is observed expired, without any hand-over` - `re-arms for a later batch, including runs the previous owner started after discovery` - `closes the gate at once when this server claims a run between two lease reads, dated from the hand-over` - plus `claimQueuedRun leaves a queued run queued and unstarted, in the admission gate's shape`, `is the same gate the assigned-work sweep and reconciled continuation delivery read`, `ends at the cap, counted from when the runs became claimable, so one wedged Runner cannot hold run starts` - `heartbeat-native-early-hand-off.test.ts`: `releases the remote controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56 ms), `leaves a run another server already took over alone`, `holds the incoming server's queued starts only from the hand-over, not while the outgoing server still drives its runs` - `heartbeat/run-output.test.ts`: `lets a reattach start before its existing log is re-verified` - `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at once, with no wait added to the run-start path`, `lets a reattach wait its turn in process instead of a deferred full re-preparation`, `still defers a reattach once its bounded wait runs out` - `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only for the exact image content ID named by the test override` - Review fixes (red before the fix, green after): `reattach-phase.test.ts` `starts the claim of handed-over runs within a second of the hand-over notification`; `assigned-work-invariant.test.ts` `takes no action while handed-over live runs are being reached, and acts once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never lets the outgoing server claim back a run it handed off, while the incoming server may take it over`; `reattach-phase.test.ts` `retries a failed first listing instead of going quiet, and still finds the previous owner's runs`, `keeps a hand-over notification whose listing failed until a listing succeeds`, `moves the cap earlier when a claimed run turns out to have been handed over before its claim`; `queue.test.ts` `returns the claim to the queue when a reattach phase begins while the claim is being checked`; `run-completion.test.ts` `finalizes the run log in completeRun|failRun only after a reattach's deferred log recovery settles`. `native-session-executor.test.ts` `ends the reattach phase once a reattached %s session is driven by this server again, without the cap or manual settling` over codex_app_server, opencode_server, acpx_runtime, muse_external, grok_bot_mcp and openai_dot_mcp (6 of 6 red without the hook; the test drives the executor with a mocked session runtime, not each real driver). Lifecycle tests for the new worker: `does no database work and keeps no deadline on an instance with nothing to reattach`, `waits for the hand-over notification or the lease deadline instead of polling while the previous owner drives its runs`, `begins at a killed owner's lease deadline, without any hand-over or notification`, `stops with the delivery coordinator at shutdown and ignores a later hand-over`. Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing on this branch: `reattach-phase.test.ts`, `heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53 tests together), `agent-directory-busy.test.ts`, `heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30 to 96 on this shared machine, the full `queue.test.ts` and `recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres setup; CI supplies them. Daytona results that exist (local servers with Daytona sandboxes, release-swap cells). They ran on a combined branch with this PR, #21462, #21442 and #21362 (merged), so they cannot be credited to this PR alone: - G5 baseline (masterd1d71a7755): SIGTERM to PRP authentication 21.5 to 38 s at swap 1. - G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7 s at swap 1. All runs that crossed swap 2 were adopted on the same Runner process. The staging chain (`provider recovery timed out after 900000ms`, then `resumed_not_suspended`) did not recur. - Caveat: G5c ran before the phase timing fix in this PR. In G5c the phase opened at process start, its 180 s cap ran out during the overlap, and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease claims. The timing fix (the phase begins when the runs become claimable) has not run on Daytona. - G6c (12 live runs and 20 queued, two graceful swaps, no injection, combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11 survivors at both. - Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after SIGTERM. The 30 s target was met. - Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at 19 s, so the outgoing server drained gracefully. The incoming server did not claim on the hand-over; it claimed only when the outgoing server exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity 47 to 57 s after SIGTERM. The target was missed. Why the claims waited for the exit is not explained. - The host load was 49 to 70 from other work. 7 of 32 tasks timed out in the queue or budget after the swaps; we suspect the overload. ## Risks - Product trade-off: while handed-over live runs are being reached, queued starts wait, up to the 180 s cap. On an instance with no handed-over runs, nothing waits. - Not proven: two-server behavior with the phase timing fix and the notification-driven watch (no cell has run them); warm standby becoming active; runs that the outgoing server starts after discovery. - Early hand-off at SIGTERM orphans a server-side exec tool that is in flight. With #21442 the next controller settles it as interrupted with an unknown outcome; it is never run again. - Swap 2 in G6c: the incoming server claimed only at the outgoing server's exit. That is unexplained. - Cross-session ownership: the gates in `execution-recovery-resolution.ts` and `recovery/assigned-work-sweep.ts`, and the claim hook in `heartbeat/recovery.ts`, are in files that the restart-recovery work owns. They are one-line gates agreed with that owner. That owner said the gate did not need to be inside the claim transaction; that was before the phase could begin during a claim. After review, the in-transaction re-check in `heartbeat/queue.ts` also reads the phase, through the existing rollback. #21427 (open) also edits `execution-recovery-resolution.ts`, `heartbeat/queue.ts` and `recovery/assigned-work-sweep.ts`; whichever merges second needs a rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned by the restart-recovery work): a 3-line guard in `claimNativeRestartRecoveries` returns a run this process handed off at shutdown as `blocked` / `handed_off_to_next_controller` before takeover is evaluated, so the outgoing server never claims back a run it handed off. #21478 (merged) added a sweep path that starts an overdue saved-work recovery; it is deliberately not gated by the reattach phase, because it resumes an existing run (reattach-like work), which the phase makes room for rather than competes with. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
413 lines
16 KiB
TypeScript
413 lines
16 KiB
TypeScript
import { and, asc, desc, eq, gt, isNotNull, lte, not, or, sql } from "drizzle-orm";
|
|
import { StringDecoder } from "node:string_decoder";
|
|
import { heartbeatRunEvents, type Db } from "@paperclipai/db";
|
|
import { validatePrpEvent, type PrpEvent } from "../../vendor/paperclip-runner/index.js";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
import type { RunLogHandle, RunLogStore } from "../run-log-store.js";
|
|
import { compactRunLogChunk } from "../heartbeat/run-log.js";
|
|
import { MAX_RUN_LOG_RECORD_BYTES } from "../run-log-record-limits.js";
|
|
|
|
export function formatNativeRunLogEvent(event: PrpEvent) {
|
|
return `${JSON.stringify({ type: "paperclip.prp.event", event })}\n`;
|
|
}
|
|
export function nativeRunLogSourceWhere(companyId: string, runId: string) {
|
|
return and(
|
|
eq(heartbeatRunEvents.companyId, companyId),
|
|
eq(heartbeatRunEvents.runId, runId),
|
|
isNotNull(heartbeatRunEvents.sourceEventId),
|
|
not(
|
|
sql`coalesce((${or(
|
|
and(
|
|
eq(heartbeatRunEvents.eventType, "auth.credentials.updated"),
|
|
sql`${heartbeatRunEvents.sourceEventId} like 'credentials:%'`,
|
|
sql`${heartbeatRunEvents.sourceInstanceId} like '%:credentials'`,
|
|
),
|
|
and(
|
|
eq(heartbeatRunEvents.eventType, "native.cleanup.event"),
|
|
sql`${heartbeatRunEvents.sourceEventId} like 'cleanup:%'`,
|
|
sql`${heartbeatRunEvents.sourceInstanceId} like '%:cleanup:%'`,
|
|
),
|
|
and(
|
|
eq(heartbeatRunEvents.eventType, "workspace.diff.recorded"),
|
|
sql`${heartbeatRunEvents.sourceEventId} like 'provider-trace-reproject:%'`,
|
|
sql`${heartbeatRunEvents.sourceInstanceId} like 'provider-trace-reproject:%'`,
|
|
),
|
|
)}), false)`,
|
|
),
|
|
);
|
|
}
|
|
async function sourcePage(
|
|
db: Db,
|
|
companyId: string,
|
|
runId: string,
|
|
after: number,
|
|
through?: number,
|
|
signal?: AbortSignal,
|
|
) {
|
|
signal?.throwIfAborted();
|
|
const rows = await db
|
|
.select({
|
|
seq: heartbeatRunEvents.seq,
|
|
payload: heartbeatRunEvents.payload,
|
|
sourceInstanceId: heartbeatRunEvents.sourceInstanceId,
|
|
sourceEventId: heartbeatRunEvents.sourceEventId,
|
|
sourceSeq: heartbeatRunEvents.sourceSeq,
|
|
sha256: heartbeatRunEvents.sourcePayloadSha256,
|
|
})
|
|
.from(heartbeatRunEvents)
|
|
.where(
|
|
and(
|
|
nativeRunLogSourceWhere(companyId, runId),
|
|
gt(heartbeatRunEvents.seq, after),
|
|
through === undefined ? undefined : lte(heartbeatRunEvents.seq, through),
|
|
),
|
|
)
|
|
.orderBy(asc(heartbeatRunEvents.seq))
|
|
.limit(16);
|
|
signal?.throwIfAborted();
|
|
return rows.map((row) => {
|
|
const validated = validatePrpEvent(row.payload?.prpEvent);
|
|
if (
|
|
!validated.ok ||
|
|
validated.event.runId !== runId ||
|
|
validated.event.sourceInstanceId !== row.sourceInstanceId ||
|
|
validated.event.sourceEventId !== row.sourceEventId ||
|
|
validated.event.sourceSeq !== row.sourceSeq ||
|
|
nativeSha256(validated.event) !== row.sha256
|
|
)
|
|
throw new Error("native_run_log_source_integrity_error");
|
|
return { cursor: row.seq, event: validated.event };
|
|
});
|
|
}
|
|
|
|
function matchPublishedImageOmissions(source: unknown, published: unknown): unknown {
|
|
if (Array.isArray(source))
|
|
return source.map((value, index) => matchPublishedImageOmissions(value,
|
|
Array.isArray(published) ? published[index] : undefined));
|
|
if (!source || typeof source !== "object") return source;
|
|
const original = source as Record<string, unknown>;
|
|
const existing = published && typeof published === "object"
|
|
? published as Record<string, unknown> : {};
|
|
const result = Object.fromEntries(Object.entries(original).map(([key, value]) =>
|
|
[key, matchPublishedImageOmissions(value, existing[key])]));
|
|
const imageSource = original.source as Record<string, unknown> | undefined;
|
|
const publishedSource = existing.source as Record<string, unknown> | undefined;
|
|
if (original.type === "image" && imageSource?.type === "base64" &&
|
|
typeof imageSource.data === "string" && typeof publishedSource?.data === "string") {
|
|
const marker = /^\[omitted base64 image data: ([0-9]+) chars\]$/.exec(publishedSource.data);
|
|
if (marker && marker[1] === String(imageSource.data.length) && imageSource.data.length >= 1024 &&
|
|
/^[A-Za-z0-9+/=]+$/.test(imageSource.data)) {
|
|
// The existing scrubber depends on serialized key order, which JSONB
|
|
// does not retain. Honor only an omission already present at this path;
|
|
// published raw data still has to match the source exactly.
|
|
result.source = { ...(result.source as Record<string, unknown>), data: publishedSource.data };
|
|
}
|
|
}
|
|
return result;
|
|
}
|
|
|
|
function matchesPublishedEvent(source: unknown, published: unknown) {
|
|
return nativeSha256(published) === nativeSha256(matchPublishedImageOmissions(source, published));
|
|
}
|
|
|
|
/** Recover a compactor omission contained within one source-backed JSON string.
|
|
* Retained properties may have any order: JSONB does not retain writer ordering.
|
|
* Reconstruct the complete envelope, prove the exact compactor output, and then
|
|
* compare every field semantically. A marker or matching substring alone is not proof.
|
|
*/
|
|
function restoreCompactedEvent(chunk: string, source: unknown): unknown {
|
|
const marker = /\n\[paperclip truncated run log chunk: omitted ([1-9][0-9]*) chars\]\n/g;
|
|
const found = marker.exec(chunk);
|
|
if (!found || marker.exec(chunk)) return;
|
|
const omitted = Number(found[1]);
|
|
if (!Number.isSafeInteger(omitted)) return;
|
|
const head = chunk.slice(0, found.index), tail = chunk.slice(found.index + found[0].length);
|
|
let quoted = false, escaped = false, start = -1;
|
|
for (let index = 0; index < head.length; index++) {
|
|
const character = head[index];
|
|
if (!quoted) {
|
|
if (character === '"') { quoted = true; start = index; }
|
|
} else if (escaped) escaped = false;
|
|
else if (character === "\\") escaped = true;
|
|
else if (character === '"') quoted = false;
|
|
}
|
|
if (!quoted) return;
|
|
const leading = head.slice(start);
|
|
function* strings(value: unknown): Generator<string> {
|
|
if (typeof value === "string") yield value;
|
|
else if (Array.isArray(value)) for (const child of value) yield* strings(child);
|
|
else if (value && typeof value === "object")
|
|
for (const child of Object.values(value)) yield* strings(child);
|
|
}
|
|
for (const value of strings(source)) {
|
|
const encoded = JSON.stringify(value), end = leading.length + omitted;
|
|
if (end >= encoded.length || !encoded.startsWith(leading) || !tail.startsWith(encoded.slice(end))) continue;
|
|
const restored = head + encoded.slice(leading.length, end) + tail;
|
|
// Records written before invite tokens were redacted still hold the token;
|
|
// prove them with the compactor of that release.
|
|
if (![true, false].some(redactInvitations =>
|
|
compactRunLogChunk(restored, undefined, { redactInvitations }) === chunk)) continue;
|
|
try {
|
|
const envelope = JSON.parse(restored);
|
|
if (envelope.type === "paperclip.prp.event" && matchesPublishedEvent(source, envelope.event))
|
|
return envelope;
|
|
} catch { /* Another candidate cannot make malformed evidence authoritative. */ }
|
|
}
|
|
}
|
|
|
|
export function createNativeRunLogCoverageVerifier(
|
|
db: Db,
|
|
companyId: string,
|
|
runId: string,
|
|
options?: {
|
|
allowSourceSuffix?: boolean;
|
|
redactChunks?: (chunk: string) => Iterable<string>;
|
|
signal?: AbortSignal;
|
|
},
|
|
) {
|
|
const decoder = new StringDecoder("utf8");
|
|
let text = "",
|
|
cursor = 0;
|
|
let page: Awaited<ReturnType<typeof sourcePage>> = [];
|
|
const unproven = () => new Error("native_log_source_coverage_unproven");
|
|
async function next() {
|
|
if (!page.length) page = await sourcePage(db, companyId, runId, cursor, undefined, options?.signal);
|
|
return page[0];
|
|
}
|
|
async function line(value: string, newlineBytes = 1) {
|
|
if (!value.trim()) return;
|
|
if (Buffer.byteLength(value) + newlineBytes > MAX_RUN_LOG_RECORD_BYTES) throw unproven();
|
|
let chunk: unknown;
|
|
try {
|
|
chunk = JSON.parse(value).chunk;
|
|
} catch {
|
|
throw unproven();
|
|
}
|
|
if (typeof chunk !== "string") throw unproven();
|
|
let expected: Awaited<ReturnType<typeof next>>;
|
|
async function* sourceForms() {
|
|
expected ??= await next();
|
|
if (!expected) throw unproven();
|
|
if (!options?.redactChunks) { yield expected.event; return; }
|
|
for (const form of options.redactChunks(formatNativeRunLogEvent(expected.event))) {
|
|
let candidate;
|
|
try { candidate = JSON.parse(form); }
|
|
catch { continue; } // A malformed historical privacy projection proves nothing.
|
|
if (candidate?.type === "paperclip.prp.event" && candidate.event &&
|
|
typeof candidate.event === "object" && !Array.isArray(candidate.event))
|
|
yield candidate.event;
|
|
}
|
|
}
|
|
let envelope: { type?: unknown; event?: unknown };
|
|
try {
|
|
envelope = JSON.parse(chunk);
|
|
} catch {
|
|
if (!chunk.includes('"paperclip.prp.event"')) return;
|
|
let restored;
|
|
for await (const form of sourceForms()) {
|
|
restored = restoreCompactedEvent(chunk, form);
|
|
if (restored) break;
|
|
}
|
|
if (!restored) throw unproven();
|
|
envelope = restored;
|
|
}
|
|
if (envelope?.type !== "paperclip.prp.event") return;
|
|
let event: PrpEvent;
|
|
try {
|
|
const parsed = validatePrpEvent(envelope.event);
|
|
if (!parsed.ok) throw unproven();
|
|
event = parsed.event;
|
|
} catch {
|
|
throw unproven();
|
|
}
|
|
expected ??= await next();
|
|
if (
|
|
!expected ||
|
|
event.runId !== runId ||
|
|
event.sourceInstanceId !== expected.event.sourceInstanceId ||
|
|
event.sourceEventId !== expected.event.sourceEventId ||
|
|
event.sourceSeq !== expected.event.sourceSeq ||
|
|
event.normalizedSessionId !== expected.event.normalizedSessionId ||
|
|
event.schemaVersion !== expected.event.schemaVersion ||
|
|
event.eventType !== expected.event.eventType
|
|
)
|
|
throw unproven();
|
|
if (options?.redactChunks) {
|
|
// JSONB reorders object keys. Compare the complete redacted event
|
|
// semantically while retaining the original published bytes.
|
|
try {
|
|
let matched = false;
|
|
for await (const form of sourceForms()) {
|
|
if (matchesPublishedEvent(form, event)) { matched = true; break; }
|
|
}
|
|
if (!matched) throw unproven();
|
|
} catch {
|
|
throw unproven();
|
|
}
|
|
}
|
|
cursor = expected.cursor;
|
|
page.shift();
|
|
}
|
|
return {
|
|
async write(bytes: Buffer) {
|
|
text += decoder.write(bytes);
|
|
let boundary: number;
|
|
while ((boundary = text.indexOf("\n")) !== -1) {
|
|
await line(text.slice(0, boundary));
|
|
text = text.slice(boundary + 1);
|
|
}
|
|
if (Buffer.byteLength(text) > MAX_RUN_LOG_RECORD_BYTES) throw unproven();
|
|
},
|
|
async finish() {
|
|
text += decoder.end();
|
|
if (text) await line(text, 0);
|
|
if (!options?.allowSourceSuffix && await next()) throw unproven();
|
|
return cursor;
|
|
},
|
|
};
|
|
}
|
|
|
|
/** Validate committed provenance, or prove an older unwatermarked source prefix. */
|
|
export async function nativeRunLogPrefixCursor(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
runId: string;
|
|
store: Partial<Pick<RunLogStore, "read" | "recordVerifiedSourcePrefix">>;
|
|
handle: RunLogHandle;
|
|
redactChunks: (chunk: string) => Iterable<string>;
|
|
signal?: AbortSignal;
|
|
}) {
|
|
const bytes = input.handle.initialBytes ?? 0;
|
|
const watermark = input.handle.initialSourceCursor ?? 0;
|
|
const unproven = () => new Error("native_log_source_coverage_unproven");
|
|
if (!Number.isSafeInteger(bytes) || bytes < 0 || !Number.isSafeInteger(watermark) || watermark < 0)
|
|
throw unproven();
|
|
if (!bytes && !watermark) {
|
|
await input.store.recordVerifiedSourcePrefix?.(input.handle, 0);
|
|
return 0;
|
|
}
|
|
if (input.handle.initialSourcePrefixVerified) {
|
|
const sequence = input.handle.initialSequence ?? 0;
|
|
const [company, , run, extra] = input.handle.logRef.split("/");
|
|
if (input.handle.store !== "s3_segments" || company !== input.companyId || run !== input.runId || extra ||
|
|
!bytes || !Number.isSafeInteger(sequence) || sequence < 1) throw unproven();
|
|
// A separate immutable proof binds this exact journal index/end/sequence/cursor.
|
|
// Legacy maximum-cursor metadata alone never establishes ordered coverage.
|
|
// Only explicitly coverage-preserving appends can carry the proof forward;
|
|
// an older writer's changed prefix gets no matching proof and must scan once.
|
|
if (watermark) {
|
|
const [source] = await sourcePage(input.db, input.companyId, input.runId,
|
|
watermark - 1, watermark, input.signal);
|
|
if (!source || source.cursor !== watermark) throw unproven();
|
|
}
|
|
return watermark;
|
|
}
|
|
if (!input.store.read) throw unproven();
|
|
const verifier = createNativeRunLogCoverageVerifier(input.db, input.companyId, input.runId, {
|
|
allowSourceSuffix: true,
|
|
redactChunks: input.redactChunks,
|
|
signal: input.signal,
|
|
});
|
|
let offset = 0;
|
|
while (offset < bytes) {
|
|
input.signal?.throwIfAborted();
|
|
const page = await input.store.read(input.handle, {
|
|
offset, limitBytes: Math.min(256_000, bytes - offset), signal: input.signal, priority: "background",
|
|
});
|
|
const content = Buffer.from(page.content);
|
|
if (!content.length || offset + content.length > bytes) throw unproven();
|
|
await verifier.write(content);
|
|
offset += content.length;
|
|
}
|
|
const cursor = await verifier.finish();
|
|
if (watermark > cursor) throw unproven();
|
|
await input.store.recordVerifiedSourcePrefix?.(input.handle, cursor);
|
|
return cursor;
|
|
}
|
|
|
|
export function createNativeRunLogExporter(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
runId: string;
|
|
afterCursor: number;
|
|
enqueue: (cursor: number, chunk: string) => Promise<void>;
|
|
flush: () => Promise<void>;
|
|
signal?: AbortSignal;
|
|
}) {
|
|
let cursor = input.afterCursor;
|
|
let queue = Promise.resolve();
|
|
let queuedBytes = 0,
|
|
queuedRecords = 0;
|
|
const sourceWhere = nativeRunLogSourceWhere(input.companyId, input.runId);
|
|
async function replay(through?: number) {
|
|
input.signal?.throwIfAborted();
|
|
if (through === undefined) {
|
|
const [last] = await input.db
|
|
.select({ seq: heartbeatRunEvents.seq })
|
|
.from(heartbeatRunEvents)
|
|
.where(sourceWhere)
|
|
.orderBy(desc(heartbeatRunEvents.seq))
|
|
.limit(1);
|
|
through = last?.seq ?? cursor;
|
|
}
|
|
while (cursor < through) {
|
|
const rows = await sourcePage(
|
|
input.db,
|
|
input.companyId,
|
|
input.runId,
|
|
cursor,
|
|
through,
|
|
input.signal,
|
|
);
|
|
if (!rows.length) break;
|
|
for (const row of rows) {
|
|
input.signal?.throwIfAborted();
|
|
await input.enqueue(row.cursor, formatNativeRunLogEvent(row.event));
|
|
cursor = row.cursor;
|
|
}
|
|
await input.flush();
|
|
}
|
|
}
|
|
let trailingReplayThrough = 0;
|
|
let trailingReplay: Promise<void> | null = null;
|
|
const serialized = (work: () => Promise<void>) => {
|
|
const next = queue.then(work);
|
|
queue = next.catch(() => undefined);
|
|
return next;
|
|
};
|
|
const replayLater = (through: number) => {
|
|
trailingReplayThrough = Math.max(trailingReplayThrough, through);
|
|
return trailingReplay ??= serialized(async () => {
|
|
trailingReplay = null;
|
|
await replay(trailingReplayThrough);
|
|
});
|
|
};
|
|
return {
|
|
recover: () => serialized(() => replay()),
|
|
async record(sourceCursor: number, chunk: string) {
|
|
if (!Number.isSafeInteger(sourceCursor) || sourceCursor < 1)
|
|
throw new Error("invalid_native_run_log_cursor");
|
|
const bytes = Buffer.byteLength(chunk);
|
|
if (queuedRecords >= 32 || queuedBytes + bytes > MAX_RUN_LOG_RECORD_BYTES) {
|
|
await replayLater(sourceCursor);
|
|
return;
|
|
}
|
|
queuedRecords++;
|
|
queuedBytes += bytes;
|
|
const work = queue
|
|
.then(async () => {
|
|
if (sourceCursor <= cursor) return;
|
|
if (sourceCursor > cursor + 1) await replay(sourceCursor - 1);
|
|
await input.enqueue(sourceCursor, chunk);
|
|
cursor = sourceCursor;
|
|
})
|
|
.finally(() => {
|
|
queuedRecords--;
|
|
queuedBytes -= bytes;
|
|
});
|
|
queue = work.catch(() => undefined);
|
|
await work;
|
|
},
|
|
};
|
|
}
|