mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks and breaks the control plane on purpose, so failures show up before staging users see them > - A census of 48 hours of staging failures found runs whose result save-back failed because the sandbox disk was full (class F). The harness never filled a sandbox disk, so it missed that class > - #20758 made a full-disk save end recovery at once with cause `sandbox_disk_full` and a Retry. Nothing in chaos exercised that path end to end > - A real full disk cannot be made on demand. The owner chose a test-only hook that makes the save report a full disk, not a small disk > - This pull request adds the hook at the save's sandbox boundary and a DF1 fault that arms it, clears it, and grades the result > - The benefit is that every wave can now check the disk-full rest, the Retry offer, and that Retry works after space is freed ## Linked Issues or Issue Description Refs #20758 (product behaviour under test), #20678 (finalization hands a full-disk copy-back to recovery) **What is the problem?** The chaos harness had no fault for a full sandbox disk during result save-back. Staging saw this failure (census class F in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos could not reproduce it. **What is the expected behaviour?** A wave can schedule DF1. While DF1 is armed, the task's final workspace saves fail as a full disk. When DF1 is cleared, saves succeed again. The task must rest with `sandbox_disk_full` and a Retry, and that Retry must finish the task. **Why does it matter?** Without a fault, a regression in the disk-full path (no Retry, a wrong cause, or a Retry that does not work) reaches staging users before chaos sees it. ## What Changed - `server/src/services/sandbox-disk-full-fault.ts` (new, test-only): `sandboxDiskFullFaultTarget` wraps the target of a final save only when the harness armed that save. The capture commands then exit 1 with the exact message the in-sandbox script writes on ENOSPC, and the provider download fails with `No space left on device`. The gate follows `@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL` is read once at import and ignored under `NODE_ENV=production`. Disarmed, the hook is one comparison and returns the save's own target. - `server/src/services/remote-workspaces.ts`: the capture call in `saveRemoteWorkspace` passes its target through the hook. No other product change. `workspace-recovery.ts` is not touched. - `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the control-file seam (atomic writes), the rest evidence (`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`), the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry that must finish the task). - Harness wiring: `--faults DF1` (refused without `--env daytona`, with a storm, or with PV faults), no scheduled restart, arm at launch, clear at the first rest just before the Retry click, receipts in `<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: { faultedSaves, restedDiskFull, retryOffered, retryWorked }`. - Docs: a "Sandbox disk full (DF1)" section in `tests/restart-chaos/README.md`, and the census plan marks DF1 as built. ## Verification - `cd server && pnpm exec vitest run src/services/sandbox-disk-full-fault.test.ts`: the gate is off by default and under `NODE_ENV=production`, the save's own target comes back when disarmed, only a listed final save is faulted, the product capture turns it into the named disk-full failure, and clearing the file disarms the next save. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/disk-full-faults.test.ts tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts tests/restart-chaos/provider-faults.test.ts`: planning, the control file, and grading. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the server `tsc --noEmit`: clean. - Live wave: a Daytona DF1 wave is running. Its result is added here before merge. ## Risks - Low risk for production. The hook reads its variable once at startup and is ignored under `NODE_ENV=production`. Without the variable it does one comparison and returns the same target object. - The hook fails every sandbox command of an armed capture, including the read-only fallback's inventory. A real full disk can let that inventory succeed and fail later at the download. The outcome is the same (`sandbox_disk_full`), but the fallback's inventory step is not exercised under DF1. - DF1 runs only with `--env daytona`. A local run has no sandbox workspace save, so the harness refuses DF1 there. ## Model Used - Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
670 lines
35 KiB
TypeScript
670 lines
35 KiB
TypeScript
import { notifyNativeResultCommit } from "../native-result-commit-notifications.js";
|
|
import { notifyNativeWorkspaceFinalization } from "../native-workspace-finalization-notifications.js";
|
|
import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
|
|
import { and, asc, eq, gt, or, sql } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
issues,
|
|
nativeRunFinalizations,
|
|
nativeRunResults,
|
|
nativeRunnerCheckpoints,
|
|
toolInvocations,
|
|
} from "@paperclipai/db";
|
|
import type {
|
|
CompleteControlPlaneRunInput,
|
|
ControlPlanePort,
|
|
NativeRunEvent,
|
|
NativeRunResult,
|
|
OpenControlPlaneRunInput,
|
|
PersistedNativeSession,
|
|
PrpEvent,
|
|
PrpTerminalState,
|
|
ReplayControlPlaneEventsInput,
|
|
} from "../../vendor/paperclip-runner/index.js";
|
|
import {
|
|
normalizePrpResultSignals,
|
|
validatePrpEvent,
|
|
validatePrpStructuredRunResult,
|
|
} from "../../vendor/paperclip-runner/index.js";
|
|
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
|
import { publishLiveEvent } from "../live-events.js";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
import { redactRunnerApiEvent, redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
|
|
import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, type NativeControllerCompatibility } from "./native-controller-recovery.js";
|
|
import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY,
|
|
type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js";
|
|
import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js";
|
|
import { crashPoint } from "@paperclipai/shared/crash-points";
|
|
import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js";
|
|
|
|
export interface PaperclipControlPlaneBinding {
|
|
companyId: string;
|
|
issueId: string;
|
|
runId: string;
|
|
agentId: string;
|
|
sessionId: string;
|
|
completionContractId: string;
|
|
completionContractSha256: string;
|
|
sourceInstanceId: string;
|
|
controlPlaneSourceInstanceId: string;
|
|
}
|
|
|
|
const IDENTITY_DELTA_RETRY_WINDOW = 1_024;
|
|
const IDENTITY_DELTA_RETRY_BYTES = 4 * 1024 * 1024;
|
|
|
|
function isPrpEvent(value: NativeRunEvent | PrpEvent): value is PrpEvent {
|
|
return "schema" in value && [
|
|
"paperclip.prp.event.v1",
|
|
"paperclip.prp.event.v2",
|
|
"paperclip.prp.event.v3",
|
|
].includes(value.schema);
|
|
}
|
|
|
|
function isCompleteInput(value: NativeRunResult | CompleteControlPlaneRunInput): value is CompleteControlPlaneRunInput {
|
|
return "result" in value && "terminal" in value;
|
|
}
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
? value as Record<string, unknown>
|
|
: {};
|
|
}
|
|
|
|
function assertTerminal(value: unknown): asserts value is PrpTerminalState {
|
|
const terminal = value as Partial<PrpTerminalState> | null;
|
|
if (
|
|
terminal?.schema !== "paperclip.prp.terminal.v1"
|
|
|| !["completed", "failed", "interrupted", "cancelled"].includes(String(terminal.turnTerminalState))
|
|
|| !["succeeded", "failed", "cancelled"].includes(String(terminal.runTerminalState))
|
|
|| !["done", "blocked", "needs_review", "yielded"].includes(String(terminal.reportedWorkDisposition))
|
|
) {
|
|
throw new Error("native_terminal_schema_invalid");
|
|
}
|
|
}
|
|
|
|
/** Production implementation of the runner package's deliberately narrow persistence port. */
|
|
export class PaperclipControlPlanePort implements ControlPlanePort {
|
|
readonly #db: Db;
|
|
readonly #identityRedactor: ReturnType<typeof createAgentIdentityRedactor>;
|
|
readonly #redactedDeltas = new Map<string, { event: PrpEvent; originalSha: string; bytes: number }>();
|
|
#redactedDeltaBytes = 0;
|
|
readonly #binding: PaperclipControlPlaneBinding;
|
|
#sessionId: string | null = null;
|
|
readonly #assertControllerActive?: () => void;
|
|
readonly #assertRunAdmissionActive?: () => void;
|
|
readonly #onResultCommitted?: () => Promise<void>;
|
|
readonly #onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise<void>;
|
|
readonly #onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
|
|
readonly #onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
|
|
readonly #nativeRecovery?: { compatibility: NativeControllerCompatibility; owner: NativeRunnerCheckpointOwner };
|
|
readonly #taskSessionPublication?: NativeTaskSessionPublicationContext;
|
|
readonly #taskSessionPublicationOwner?: NativeRunnerCheckpointOwner;
|
|
readonly #deferredEventObservers = new Map<string, Promise<void>>();
|
|
#deferredObserverFailure: { error: unknown } | null = null;
|
|
|
|
constructor(
|
|
db: Db,
|
|
binding: PaperclipControlPlaneBinding,
|
|
options: {
|
|
/** Runtime-only secret; never part of the persisted binding. */
|
|
privateKeyPem?: string;
|
|
/** Synchronous revocation fence for a controller handing off on restart. */
|
|
assertControllerActive?: () => void;
|
|
/** Read-only admission may finish an existing startup solely for restart detachment. */
|
|
assertRunAdmissionActive?: () => void;
|
|
onResultCommitted?: () => Promise<void>;
|
|
onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise<void>;
|
|
onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
|
|
onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
|
|
nativeRecovery?: { compatibility: NativeControllerCompatibility; owner: NativeRunnerCheckpointOwner };
|
|
taskSessionPublication?: NativeTaskSessionPublicationContext;
|
|
taskSessionPublicationOwner?: NativeRunnerCheckpointOwner;
|
|
} = {},
|
|
) {
|
|
this.#db = db;
|
|
this.#assertControllerActive = options.assertControllerActive;
|
|
this.#assertRunAdmissionActive = options.assertRunAdmissionActive ?? options.assertControllerActive;
|
|
this.#identityRedactor = createAgentIdentityRedactor(options.privateKeyPem);
|
|
this.#binding = structuredClone(binding);
|
|
this.#onCommittedEvent = options.onCommittedEvent;
|
|
this.#onEventPersistence = options.onEventPersistence;
|
|
this.#onResultCommitted = options.onResultCommitted;
|
|
this.#onDuplicateEvent = options.onDuplicateEvent;
|
|
this.#nativeRecovery = options.nativeRecovery;
|
|
this.#taskSessionPublication = options.taskSessionPublication ? structuredClone(options.taskSessionPublication) : undefined;
|
|
this.#taskSessionPublicationOwner = options.taskSessionPublicationOwner;
|
|
}
|
|
|
|
#assertActive(options?: { signal: AbortSignal }): void {
|
|
this.#assertControllerActive?.();
|
|
options?.signal.throwIfAborted();
|
|
}
|
|
|
|
async #settleDeferredObservers(options?: { signal: AbortSignal }): Promise<void> {
|
|
const assertActive = () => {
|
|
this.#assertActive(options);
|
|
if (this.#deferredObserverFailure) throw this.#deferredObserverFailure.error;
|
|
};
|
|
assertActive();
|
|
if (this.#deferredEventObservers.size === 0) return;
|
|
const signal = options?.signal;
|
|
let removeAbort = () => {};
|
|
const aborted = new Promise<never>((_resolve, reject) => {
|
|
if (!signal) return;
|
|
const abort = () => reject(signal.reason ?? new Error("native event observation aborted"));
|
|
signal.addEventListener("abort", abort, { once: true });
|
|
removeAbort = () => signal.removeEventListener("abort", abort);
|
|
if (signal.aborted) abort();
|
|
});
|
|
try {
|
|
// Aborting a checkpoint revokes its writes, not already-owned accounting
|
|
// or output observation. Each retained promise remains observed below.
|
|
while (this.#deferredEventObservers.size > 0) {
|
|
await Promise.race([Promise.all([...this.#deferredEventObservers.values()]), aborted]);
|
|
assertActive();
|
|
}
|
|
} finally { removeAbort(); }
|
|
assertActive();
|
|
}
|
|
|
|
#matchesPersistedBinding(run: Pick<typeof heartbeatRuns.$inferSelect,
|
|
"companyId" | "agentId" | "runtimeMode" | "nativeIssueId" | "nativeSessionId" |
|
|
"runnerInstanceId" | "completionContractId" | "completionContractSha256">): boolean {
|
|
return run.companyId === this.#binding.companyId
|
|
&& run.agentId === this.#binding.agentId
|
|
&& run.runtimeMode === "native"
|
|
&& run.nativeIssueId === this.#binding.issueId
|
|
&& run.nativeSessionId === this.#binding.sessionId
|
|
&& run.runnerInstanceId === this.#binding.sourceInstanceId
|
|
&& run.completionContractId === this.#binding.completionContractId
|
|
&& run.completionContractSha256 === this.#binding.completionContractSha256;
|
|
}
|
|
|
|
async openRun(input: OpenControlPlaneRunInput): Promise<void> {
|
|
this.#assertRunAdmissionActive?.();
|
|
const identity = input.identity;
|
|
if (
|
|
identity.companyId !== this.#binding.companyId
|
|
|| identity.issueId !== this.#binding.issueId
|
|
|| identity.runId !== this.#binding.runId
|
|
|| identity.agentId !== this.#binding.agentId
|
|
|| identity.sessionId !== this.#binding.sessionId
|
|
|| input.sourceInstanceId !== this.#binding.sourceInstanceId
|
|
) {
|
|
throw new Error("native_open_run_binding_mismatch");
|
|
}
|
|
const run = await this.#db.select().from(heartbeatRuns)
|
|
.where(and(
|
|
eq(heartbeatRuns.id, this.#binding.runId),
|
|
eq(heartbeatRuns.companyId, this.#binding.companyId),
|
|
eq(heartbeatRuns.agentId, this.#binding.agentId),
|
|
))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!run || !this.#matchesPersistedBinding(run)) {
|
|
throw new Error("native_open_run_not_authorized");
|
|
}
|
|
this.#assertRunAdmissionActive?.();
|
|
this.#sessionId = identity.sessionId;
|
|
}
|
|
|
|
async loadSessionCheckpoint(): Promise<PersistedNativeSession | null> {
|
|
const run = await this.#db.select().from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, this.#binding.runId))
|
|
.limit(1).then((rows) => rows[0] ?? null);
|
|
if (
|
|
!run
|
|
|| !this.#matchesPersistedBinding(run)
|
|
) throw new Error("native_session_checkpoint_binding_mismatch");
|
|
const candidate = record(run.runnerProfileJson).sessionCheckpoint;
|
|
if (candidate === undefined || candidate === null) return null;
|
|
const snapshot = record(candidate) as Partial<PersistedNativeSession>;
|
|
const identity = record(snapshot.identity);
|
|
if (
|
|
typeof snapshot.sessionId !== "string"
|
|
|| identity.companyId !== this.#binding.companyId
|
|
|| identity.issueId !== this.#binding.issueId
|
|
|| identity.runId !== this.#binding.runId
|
|
|| identity.agentId !== this.#binding.agentId
|
|
|| identity.sessionId !== run.nativeSessionId
|
|
) throw new Error("native_session_checkpoint_invalid");
|
|
return structuredClone(snapshot as PersistedNativeSession);
|
|
}
|
|
|
|
normalizeReplayPayload(payload: PrpEvent["payload"]): PrpEvent["payload"] {
|
|
return redactRunnerApiPayloads(this.#identityRedactor.redact(payload));
|
|
}
|
|
|
|
async checkpointSession(snapshot: PersistedNativeSession, options?: { signal: AbortSignal }): Promise<void> {
|
|
this.#assertActive(options);
|
|
if (this.#deferredEventObservers.size > 0 || this.#deferredObserverFailure) await this.#settleDeferredObservers(options);
|
|
snapshot = redactRunnerApiPayloads(this.#identityRedactor.redact(snapshot));
|
|
const identity = snapshot.identity;
|
|
if (
|
|
identity.companyId !== this.#binding.companyId
|
|
|| identity.issueId !== this.#binding.issueId
|
|
|| identity.runId !== this.#binding.runId
|
|
|| identity.agentId !== this.#binding.agentId
|
|
|| this.#sessionId !== identity.sessionId
|
|
) throw new Error("native_session_checkpoint_binding_mismatch");
|
|
const publication = this.#taskSessionPublication && eligibleNativeTaskSessionPublication(snapshot)
|
|
? this.#taskSessionPublication : null;
|
|
if (publication && !this.#taskSessionPublicationOwner) throw new Error("native_task_session_publication_owner_required");
|
|
await this.#db.transaction(async (tx) => {
|
|
const recovery = this.#nativeRecovery;
|
|
const owner = recovery?.owner ?? (publication ? this.#taskSessionPublicationOwner : undefined);
|
|
if (publication && recovery && nativeSha256(recovery.owner) !== nativeSha256(this.#taskSessionPublicationOwner)) {
|
|
throw new Error("native_task_session_publication_owner_mismatch");
|
|
}
|
|
const ownerLock = owner ? nativeRunnerCheckpointOwnerLock(tx, owner) : null;
|
|
// NO KEY UPDATE serializes reset/admission fields while remaining compatible
|
|
// with completeRun's parent KEY SHARE. Keep coordinator → issue → run order.
|
|
const publicationIssue = publication ? tx.$with("publication_issue").as(tx.select({
|
|
id: issues.id, companyId: issues.companyId, executionRunId: issues.executionRunId,
|
|
conversationAgentId: issues.conversationAgentId, conversationSessionGeneration: issues.conversationSessionGeneration,
|
|
}).from(issues).where(and(eq(issues.id, this.#binding.issueId), eq(issues.companyId, this.#binding.companyId),
|
|
ownerLock ? sql`exists (select 1 from ${ownerLock})` : undefined)).for("no key update")) : null;
|
|
const lockedRun = tx.$with("snapshot_run").as(tx.select({
|
|
id: heartbeatRuns.id, status: heartbeatRuns.status,
|
|
contextSnapshot: (publication ? sql<Record<string, unknown>>`jsonb_build_object('conversationSessionGeneration', ${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration')` : sql<null>`null`).as("publication_context"),
|
|
resultJson: (publication ? sql<Record<string, unknown>>`jsonb_build_object('nativeCancellation', ${heartbeatRuns.resultJson}->'nativeCancellation',
|
|
'cancellation', ${heartbeatRuns.resultJson}->'cancellation', 'startupCancellation', ${heartbeatRuns.resultJson}->'startupCancellation')` : sql<null>`null`).as("publication_cancellation"),
|
|
startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt,
|
|
companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
|
runtimeMode: heartbeatRuns.runtimeMode, nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId, runnerInstanceId: heartbeatRuns.runnerInstanceId,
|
|
completionContractId: heartbeatRuns.completionContractId,
|
|
completionContractSha256: heartbeatRuns.completionContractSha256,
|
|
runnerProfileJson: heartbeatRuns.runnerProfileJson,
|
|
}).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, this.#binding.runId), ownerLock
|
|
? sql`exists (select 1 from ${ownerLock})` : undefined,
|
|
publicationIssue ? sql`exists (select 1 from ${publicationIssue})` : undefined)).for("update"));
|
|
// Project the locking CTE's tuples, including any update committed while
|
|
// the lock waited. A second scan of heartbeat_runs could see an old MVCC tuple.
|
|
// The run CTE depends on the coordinator CTE: both locks precede the
|
|
// outer clock sample, including when either lock waits.
|
|
const selected = await tx.with(...(ownerLock ? [ownerLock] : []), ...(publicationIssue ? [publicationIssue] : []), lockedRun).select({ run: {
|
|
id: lockedRun.id, status: lockedRun.status, contextSnapshot: lockedRun.contextSnapshot, resultJson: lockedRun.resultJson,
|
|
startedAt: lockedRun.startedAt, createdAt: lockedRun.createdAt,
|
|
companyId: lockedRun.companyId, agentId: lockedRun.agentId,
|
|
runtimeMode: lockedRun.runtimeMode, nativeIssueId: lockedRun.nativeIssueId,
|
|
nativeSessionId: lockedRun.nativeSessionId, runnerInstanceId: lockedRun.runnerInstanceId,
|
|
completionContractId: lockedRun.completionContractId,
|
|
completionContractSha256: lockedRun.completionContractSha256,
|
|
runnerProfileJson: lockedRun.runnerProfileJson,
|
|
}, now: sql<Date>`clock_timestamp()`, ...(ownerLock ? {
|
|
leaseOwner: sql<string | null>`(select ${ownerLock.leaseOwner} from ${ownerLock})`,
|
|
leaseExpiresAt: sql<Date | null>`(select ${ownerLock.leaseExpiresAt} from ${ownerLock})`.mapWith(nativeRunFinalizations.leaseExpiresAt),
|
|
resultId: sql<string | null>`(select ${ownerLock.resultId} from ${ownerLock})`,
|
|
} : {}), ...(recovery ? {
|
|
checkpointHeader: sql<Record<string, unknown> | null>`(select material from ${nativeRunnerCheckpoints}
|
|
where ${nativeRunnerCheckpoints.runId} = ${this.#binding.runId}
|
|
and ${nativeRunnerCheckpoints.companyId} = ${this.#binding.companyId}
|
|
and ${nativeRunnerCheckpoints.issueId} = ${this.#binding.issueId}
|
|
and ${nativeRunnerCheckpoints.part} = 'header')`,
|
|
} : {}), ...(publicationIssue ? { publicationIssue: sql<NativeTaskSessionPublicationIssue>`(select jsonb_build_object(
|
|
'id', ${publicationIssue.id}, 'companyId', ${publicationIssue.companyId}, 'executionRunId', ${publicationIssue.executionRunId},
|
|
'conversationAgentId', ${publicationIssue.conversationAgentId}, 'conversationSessionGeneration', ${publicationIssue.conversationSessionGeneration}) from ${publicationIssue})` } : {}),
|
|
}).from(lockedRun).limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const run = selected?.run;
|
|
this.#assertActive(options);
|
|
if (owner && !selected) throw new Error("native_runner_checkpoint_owner_lost");
|
|
if (!run || !this.#matchesPersistedBinding(run)) {
|
|
throw new Error("native_session_checkpoint_binding_mismatch");
|
|
}
|
|
// The run row lock may have waited beyond our lease. Generation cannot
|
|
// change while we hold the coordinator, but wall-clock expiry still can.
|
|
if (owner && selected) await validateNativeRunnerCheckpointOwner(tx, owner,
|
|
{ leaseOwner: selected.leaseOwner ?? null, leaseExpiresAt: selected.leaseExpiresAt ?? null,
|
|
resultId: selected.resultId ?? null, now: selected.now }, !snapshot.semanticResult);
|
|
const header = recovery && selected?.checkpointHeader
|
|
? inspectNativeRunnerUpgradeHeader(this.#binding, selected.checkpointHeader) : null;
|
|
if (header && (header.normalizedSessionId !== identity.sessionId || header.runnerInstanceId !== this.#binding.sourceInstanceId)) {
|
|
throw new Error("native_session_checkpoint_prp_binding_mismatch");
|
|
}
|
|
const profile: Record<string, unknown> = {
|
|
...record(run.runnerProfileJson),
|
|
sessionCheckpoint: structuredClone(snapshot) as unknown as Record<string, unknown>,
|
|
...(recovery ? { nativeControllerRecovery: createNativeControllerRecovery(snapshot, recovery.compatibility,
|
|
header ? { runId: this.#binding.runId, runnerInstanceId: header.runnerInstanceId,
|
|
normalizedSessionId: header.normalizedSessionId } : null) } : {}),
|
|
};
|
|
// Driver/live snapshots may already describe a terminal result before
|
|
// completeRun accepts it. Persist that observation without a handled
|
|
// receipt; the existing post-acceptance checkpoint publishes the pointer.
|
|
if (publication && selected?.resultId) {
|
|
const receipt = await publishNativeTaskSessionInTransaction(tx, { binding: this.#binding, context: publication,
|
|
snapshot, run, issue: selected.publicationIssue ?? null, resultId: selected.resultId ?? null,
|
|
recoveryReady: !recovery || header !== null });
|
|
profile[NATIVE_TASK_SESSION_PUBLICATION_KEY] = receipt;
|
|
if (Buffer.byteLength(JSON.stringify({ nativeControllerRecovery: profile.nativeControllerRecovery,
|
|
[NATIVE_TASK_SESSION_PUBLICATION_KEY]: receipt })) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) {
|
|
throw new Error("native_task_session_publication_metadata_too_large");
|
|
}
|
|
}
|
|
await tx.update(heartbeatRuns).set({
|
|
runnerProfileJson: profile,
|
|
sessionIdAfter: snapshot.providerSessionId ?? snapshot.sessionId,
|
|
nativePhase: snapshot.semanticResult ? "workspace_finalizing" : "observed",
|
|
nativePhaseUpdatedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
}).where(eq(heartbeatRuns.id, this.#binding.runId));
|
|
this.#assertActive(options);
|
|
});
|
|
}
|
|
|
|
async appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true }) {
|
|
const persistenceStartedAt = Date.now(), persistenceClock = performance.now();
|
|
this.#assertActive(options);
|
|
if (!isPrpEvent(value)) throw new Error("native_legacy_event_not_supported");
|
|
const validated = validatePrpEvent(value);
|
|
if (!validated.ok) throw new Error(`native_event_schema_invalid:${validated.issues[0]?.message ?? "unknown"}`);
|
|
let event = this.#identityRedactor.redact(validated.event);
|
|
if (event.runId !== this.#binding.runId || (this.#sessionId && event.normalizedSessionId !== this.#sessionId)) {
|
|
throw new Error("native_event_binding_mismatch");
|
|
}
|
|
const expectedSourceInstanceId = event.sourceKind === "control_plane"
|
|
? this.#binding.controlPlaneSourceInstanceId
|
|
: this.#binding.sourceInstanceId;
|
|
if (event.sourceInstanceId !== expectedSourceInstanceId) {
|
|
throw new Error("native_event_source_binding_mismatch");
|
|
}
|
|
if (options?.deferPostCommitObservers && (
|
|
!["turn.completed", "turn.interrupted", "turn.cancelled"].includes(event.eventType)
|
|
|| !event.turnId || event.payload.providerTerminalObserved === false
|
|
)) throw new Error("native_deferred_observer_terminal_required");
|
|
const terminalItem = /^item\.(completed|failed|cancelled)$/.test(event.eventType);
|
|
const terminalTurn = /^(turn|session)\.(completed|failed|cancelled|interrupted|closed)$/.test(event.eventType);
|
|
if ((event.eventType === "item.delta" || terminalItem || terminalTurn) && this.#identityRedactor.values.length > 0) {
|
|
const key = `${event.sourceInstanceId}:${event.sourceEventId}`;
|
|
const cached = this.#redactedDeltas.get(key);
|
|
const originalSha = nativeSha256(validated.event);
|
|
if (cached) {
|
|
if (cached.originalSha !== originalSha) throw new Error("native_event_source_payload_conflict");
|
|
event = cached.event;
|
|
} else {
|
|
const stream = `${event.sourceInstanceId}:${event.turnId}:${event.itemId}`;
|
|
event = { ...event, payload: event.eventType === "item.delta"
|
|
? this.#identityRedactor.delta(stream, validated.event.payload, event.itemId)
|
|
: this.#identityRedactor.settleDeltas(
|
|
terminalTurn ? (event.eventType.startsWith("session.") ? `${event.sourceInstanceId}:` : `${event.sourceInstanceId}:${event.turnId}:`) : stream,
|
|
event.payload, terminalTurn,
|
|
),
|
|
};
|
|
// Retries must use the same redacted payload without consuming a delta twice.
|
|
const bytes = Buffer.byteLength(JSON.stringify(event));
|
|
this.#redactedDeltas.set(key, { event, originalSha, bytes });
|
|
this.#redactedDeltaBytes += bytes;
|
|
// Match the bounded runner transcript retry window; large tool output
|
|
// also has a byte cap so long runs cannot retain another full transcript.
|
|
while (this.#redactedDeltas.size > IDENTITY_DELTA_RETRY_WINDOW || this.#redactedDeltaBytes > IDENTITY_DELTA_RETRY_BYTES) {
|
|
const oldest = this.#redactedDeltas.keys().next().value!;
|
|
this.#redactedDeltaBytes -= this.#redactedDeltas.get(oldest)!.bytes;
|
|
this.#redactedDeltas.delete(oldest);
|
|
}
|
|
}
|
|
}
|
|
event = redactRunnerApiEvent(event);
|
|
const observerKey = JSON.stringify([event.sourceInstanceId, event.sourceEventId]);
|
|
const pendingObservation = this.#deferredEventObservers.get(observerKey);
|
|
const persisted = await this.#db.transaction(async tx => {
|
|
this.#assertActive(options);
|
|
const receipt = await appendHeartbeatRunEvent(tx as unknown as Db, {
|
|
companyId: this.#binding.companyId,
|
|
runId: this.#binding.runId,
|
|
agentId: this.#binding.agentId,
|
|
eventType: event.eventType,
|
|
stream: "system",
|
|
level: event.eventType.includes("failed") ? "error" : "info",
|
|
payload: { prpEvent: event as unknown as Record<string, unknown> },
|
|
nativeSource: {
|
|
sourceInstanceId: event.sourceInstanceId,
|
|
sourceEventId: event.sourceEventId,
|
|
sourceSeq: event.sourceSeq,
|
|
protocolSchemaVersion: event.schemaVersion,
|
|
canonicalPayload: event as unknown as Record<string, unknown>,
|
|
},
|
|
});
|
|
// The nested append has no publication side effects. Revocation rolls
|
|
// back its event and sequence allocation before the outer commit.
|
|
this.#assertActive(options);
|
|
return receipt;
|
|
});
|
|
this.#assertActive(options);
|
|
const persistenceEndedAt = persistenceStartedAt + performance.now() - persistenceClock;
|
|
let observation = pendingObservation ?? this.#deferredEventObservers.get(observerKey);
|
|
if (!observation) observation = (async () => {
|
|
try { await this.#onEventPersistence?.(event, persistenceStartedAt, persistenceEndedAt); } catch {}
|
|
if (persisted.disposition === "committed") {
|
|
try {
|
|
publishLiveEvent({
|
|
companyId: this.#binding.companyId,
|
|
type: "heartbeat.run.event",
|
|
payload: {
|
|
runId: this.#binding.runId,
|
|
agentId: this.#binding.agentId,
|
|
issueId: this.#binding.issueId,
|
|
seq: persisted.row.seq,
|
|
eventType: persisted.row.eventType,
|
|
stream: persisted.row.stream,
|
|
level: persisted.row.level,
|
|
payload: persisted.row.payload,
|
|
},
|
|
});
|
|
} catch {
|
|
}
|
|
await this.#onCommittedEvent?.(event, persisted.row.seq);
|
|
} else {
|
|
// A recovered runner may replay the event whose durable side effects
|
|
// parked the prior attempt. Do not repeat those effects, but let the
|
|
// embedding runtime refresh observational state before appendEvent
|
|
// returns to its synchronous governed-wait boundary.
|
|
await this.#onDuplicateEvent?.(event, persisted.row.seq);
|
|
}
|
|
})();
|
|
if (options?.deferPostCommitObservers) {
|
|
this.#deferredEventObservers.set(observerKey, observation);
|
|
void observation.then(() => {
|
|
if (this.#deferredEventObservers.get(observerKey) === observation) this.#deferredEventObservers.delete(observerKey);
|
|
}, error => {
|
|
this.#deferredObserverFailure ??= { error };
|
|
if (this.#deferredEventObservers.get(observerKey) === observation) this.#deferredEventObservers.delete(observerKey);
|
|
});
|
|
} else {
|
|
await observation;
|
|
}
|
|
return {
|
|
cursor: persisted.row.seq,
|
|
highestContiguousSourceSeq: persisted.highestContiguousSourceSeq,
|
|
disposition: persisted.disposition,
|
|
};
|
|
}
|
|
|
|
async replayEvents(input: ReplayControlPlaneEventsInput) {
|
|
if (
|
|
input.runId !== this.#binding.runId
|
|
|| ![this.#binding.sourceInstanceId, this.#binding.controlPlaneSourceInstanceId].includes(input.sourceInstanceId)
|
|
) {
|
|
throw new Error("native_replay_binding_mismatch");
|
|
}
|
|
const rows = await this.#db
|
|
.select({ payload: heartbeatRunEvents.payload, sourceSeq: heartbeatRunEvents.sourceSeq })
|
|
.from(heartbeatRunEvents)
|
|
.where(and(
|
|
eq(heartbeatRunEvents.runId, this.#binding.runId),
|
|
eq(heartbeatRunEvents.sourceInstanceId, input.sourceInstanceId),
|
|
gt(heartbeatRunEvents.sourceSeq, input.afterSourceSeq),
|
|
))
|
|
.orderBy(asc(heartbeatRunEvents.sourceSeq))
|
|
.limit(Math.max(1, Math.min(input.limit, 1_000)));
|
|
const events = rows.flatMap((row) => {
|
|
const candidate = row.payload?.prpEvent;
|
|
const parsed = validatePrpEvent(candidate);
|
|
return parsed.ok ? [parsed.event] : [];
|
|
});
|
|
let cursor = input.afterSourceSeq;
|
|
for (const event of events) {
|
|
if (event.sourceSeq === cursor + 1) cursor += 1;
|
|
else if (event.sourceSeq > cursor + 1) break;
|
|
}
|
|
return { events, highestContiguousSourceSeq: cursor };
|
|
}
|
|
|
|
/**
|
|
* Proves from durable rows, across every runner instance, that this run has
|
|
* no provider event, source-sequenced runner event, tool invocation, or
|
|
* accepted result. Any such row, or a read failure, denies the proof.
|
|
*/
|
|
async provesNoRunProviderWork(options: { signal: AbortSignal }): Promise<boolean> {
|
|
options.signal.throwIfAborted();
|
|
const { companyId, runId } = this.#binding;
|
|
const providerEvents = sql.join([...NATIVE_PROVIDER_DURABLE_EVENT_TYPES].map(value => sql`${value}`), sql`, `);
|
|
const result = await this.#db.execute(sql`select
|
|
exists (select 1 from ${heartbeatRunEvents} event where event.company_id = ${companyId} and event.run_id = ${runId}
|
|
and (event.source_event_id is not null or event.source_instance_id is not null
|
|
or event.event_type in (${providerEvents})
|
|
or event.event_type ~ '^(turn|item|tool|semantic_tool|run[.]result)[.]')) as "providerEvent",
|
|
exists (select 1 from ${toolInvocations} invocation where invocation.company_id = ${companyId}
|
|
and invocation.run_id = ${runId}) as "toolInvocation",
|
|
exists (select 1 from ${nativeRunResults} result where result.company_id = ${companyId}
|
|
and result.run_id = ${runId}) as "acceptedResult",
|
|
exists (select 1 from ${nativeRunFinalizations} coordinator where coordinator.run_id = ${runId}
|
|
and coordinator.result_id is not null) as "finalizedResult"`);
|
|
options.signal.throwIfAborted();
|
|
const rows = (Array.isArray(result) ? result : (result as { rows?: unknown[] }).rows ?? []) as Array<Record<string, unknown>>;
|
|
const row = rows[0];
|
|
if (!row) return false;
|
|
return row.providerEvent === false && row.toolInvocation === false
|
|
&& row.acceptedResult === false && row.finalizedResult === false;
|
|
}
|
|
|
|
async completeRun(value: NativeRunResult | CompleteControlPlaneRunInput, options?: { signal: AbortSignal }): Promise<void> {
|
|
this.#assertActive(options);
|
|
if (this.#deferredEventObservers.size > 0 || this.#deferredObserverFailure) await this.#settleDeferredObservers(options);
|
|
value = redactRunnerApiPayloads(this.#identityRedactor.redact(value));
|
|
if (!isCompleteInput(value)) throw new Error("native_structured_result_required");
|
|
// Capture compatibility diagnostics before canonical validation removes
|
|
// legacy/non-actionable attention payloads. They are operator evidence,
|
|
// not part of the authoritative semantic result.
|
|
const normalizationDiagnostics = normalizePrpResultSignals(value.result);
|
|
const validated = validatePrpStructuredRunResult(value.result);
|
|
if (!validated.ok) throw new Error(`native_result_schema_invalid:${validated.issues[0]?.message ?? "unknown"}`);
|
|
assertTerminal(value.terminal);
|
|
if (validated.result.reportedWorkDisposition !== value.terminal.reportedWorkDisposition) {
|
|
throw new Error("native_result_terminal_disposition_mismatch");
|
|
}
|
|
const canonical = {
|
|
binding: this.#binding,
|
|
result: validated.result,
|
|
terminal: value.terminal,
|
|
turnId: value.turnId ?? null,
|
|
};
|
|
const canonicalSha256 = nativeSha256(canonical);
|
|
const serverFingerprint = nativeSha256({
|
|
runId: this.#binding.runId,
|
|
completionContractSha256: this.#binding.completionContractSha256,
|
|
canonicalSha256,
|
|
});
|
|
|
|
const insertedResult = await this.#db.transaction(async (tx) => {
|
|
// Status commit and snapshot publication lock coordinator → issue → run.
|
|
// A replay must follow that order too: holding the issue while waiting
|
|
// for the coordinator deadlocks a status commit's issue UPDATE lock.
|
|
const recovery = this.#nativeRecovery;
|
|
const owner = recovery ? await lockNativeRunnerCheckpointOwner(tx, recovery.owner) : null;
|
|
// Result insertion checks the task FK. Keep its parent lock before run.
|
|
await tx.select({ id: issues.id }).from(issues)
|
|
.where(and(eq(issues.id, this.#binding.issueId), eq(issues.companyId, this.#binding.companyId)))
|
|
.for("key share");
|
|
const run = await tx.select().from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, this.#binding.runId)).for("update").limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
this.#assertActive(options);
|
|
if (!run || !this.#matchesPersistedBinding(run)) {
|
|
throw new Error("native_result_binding_mismatch");
|
|
}
|
|
if (recovery && owner) {
|
|
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
|
|
await validateNativeRunnerCheckpointOwner(tx, recovery.owner, { ...owner, now: clock.now }, false);
|
|
}
|
|
|
|
const callerConditions = [eq(nativeRunResults.serverFingerprint, serverFingerprint)];
|
|
if (value.callerResultId) callerConditions.push(eq(nativeRunResults.callerResultId, value.callerResultId));
|
|
if (value.callerDedupeKey) callerConditions.push(eq(nativeRunResults.callerDedupeKey, value.callerDedupeKey));
|
|
const existing = await tx.select().from(nativeRunResults)
|
|
.where(and(eq(nativeRunResults.runId, this.#binding.runId), or(...callerConditions)))
|
|
.limit(1).then((rows) => rows[0] ?? null);
|
|
if (existing) {
|
|
this.#assertActive(options);
|
|
if (existing.canonicalSha256 !== canonicalSha256) throw new Error("structured_result_replay_conflict");
|
|
return false;
|
|
}
|
|
if (owner?.resultId) throw new Error("structured_result_replay_conflict");
|
|
const resultJson = {
|
|
result: validated.result,
|
|
terminal: value.terminal,
|
|
normalizationDiagnostics: {
|
|
ignoredAttentionRequests: normalizationDiagnostics.ignoredAttentionRequests,
|
|
},
|
|
} as unknown as Record<string, unknown>;
|
|
await notifyNativeResultCommit(tx, this.#binding);
|
|
await notifyNativeWorkspaceFinalization(tx, this.#binding);
|
|
const inserted = await tx.insert(nativeRunResults).values({
|
|
companyId: this.#binding.companyId,
|
|
issueId: this.#binding.issueId,
|
|
runId: this.#binding.runId,
|
|
turnId: value.turnId ?? null,
|
|
completionContractId: this.#binding.completionContractId,
|
|
callerResultId: value.callerResultId ?? null,
|
|
callerDedupeKey: value.callerDedupeKey ?? null,
|
|
serverFingerprint,
|
|
schemaStatus: "accepted",
|
|
resultJson,
|
|
canonicalSha256,
|
|
}).returning().then((rows) => rows[0]);
|
|
if (!inserted) throw new Error("native_result_not_persisted");
|
|
await tx.insert(nativeRunFinalizations).values({
|
|
runId: this.#binding.runId,
|
|
companyId: this.#binding.companyId,
|
|
issueId: this.#binding.issueId,
|
|
phase: "workspace_finalizing",
|
|
resultId: inserted.id,
|
|
}).onConflictDoUpdate({
|
|
target: nativeRunFinalizations.runId,
|
|
set: {
|
|
phase: "workspace_finalizing",
|
|
resultId: inserted.id,
|
|
recoveryState: null,
|
|
recoveryRequestId: null,
|
|
failureCode: null,
|
|
failureDetail: null,
|
|
nextAttemptAt: null,
|
|
updatedAt: new Date(),
|
|
},
|
|
});
|
|
await tx.update(heartbeatRuns).set({
|
|
nativePhase: "workspace_finalizing",
|
|
nativePhaseUpdatedAt: new Date(),
|
|
resultJson: {
|
|
...record(run.resultJson),
|
|
// Publish the accepted contract with its durable receipt. Recovery
|
|
// may settle this run without the original adapter ever returning.
|
|
nativeTerminal: value.terminal,
|
|
prpTurnTerminalState: value.terminal.turnTerminalState,
|
|
prpRunTerminalState: value.terminal.runTerminalState,
|
|
prpReportedWorkDisposition: value.terminal.reportedWorkDisposition,
|
|
},
|
|
updatedAt: new Date(),
|
|
}).where(eq(heartbeatRuns.id, this.#binding.runId));
|
|
this.#assertActive(options);
|
|
return true;
|
|
});
|
|
if (insertedResult) crashPoint("native.result.after_accept_before_finalize");
|
|
if (insertedResult) { try { await this.#onResultCommitted?.(); } catch {} }
|
|
}
|
|
}
|