mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 06:42:06 +01:00
fix(native): drain source events before durable controller handoff
Preserve source identity and accounting while reconstructing cold controllers. Keep a compact staged handoff through command delivery and authenticated adoption, and retire it with existing persistence after target checkpoint publication. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
9eb9ac6f9e
commit
768249cab0
23 files changed
+1819
-140
No files matched your search
@@ -2632,6 +2632,15 @@ An upgrade readiness report contains preliminary native evidence, not permission
|
||||
to remove a deployment volume. Keep the checkpoint pool bounded and measure all
|
||||
pools when qualifying database cost; the current size of two is a default.
|
||||
|
||||
For retained Codex event tails, keep source drain/accounting and target admission
|
||||
separate. `native-staged-attachment-recovery.ts` validates private source/target
|
||||
header evidence; `native-recovery-usage-delivery.ts` binds the source's accounting
|
||||
receiver. The transport persists its staged target before source bootstrap, then
|
||||
uses the existing authenticated attachment protocol after durable drain. Do not
|
||||
rotate the identity first, acknowledge before receipt persistence, or weaken the
|
||||
Runner's settled-provider check. A staged target does not yet certify a resumable
|
||||
session. Its eventual snapshot still uses the existing checkpoint transaction.
|
||||
|
||||
Custom source-mode HTTP launchers should create their listener with
|
||||
`createPaperclipHttpServer` from `server/src/http/server.ts`, passing the selected
|
||||
API origin as `apiUrl`. This registers the native Runner WebSocket transport
|
||||
|
||||
@@ -2724,6 +2724,47 @@ attachment IDs. A healthy warm handoff can create new attachment IDs within the
|
||||
retained Runner. Replacing that run's controller must preserve those IDs; deriving
|
||||
them again from the business run ID would request an invalid authority rotation.
|
||||
|
||||
Cold Codex continuation can restore a completed provider with a retained event
|
||||
suffix. Native remote controllers first persist a private staged attachment under
|
||||
the target run's existing checkpoint owner, then restore and drain the exact
|
||||
source authority before attaching the target. The source's run, turn and item
|
||||
identity stays on every old event. Receipt persistence precedes acknowledgement;
|
||||
unknown semantic work holds the handoff and never reaches the target's tool
|
||||
authority. The provider's settled-session attachment guard remains mandatory.
|
||||
|
||||
The staged controller format is
|
||||
`paperclip.runner.durable.control-plane-state.staged-attachment.v1`. It contains
|
||||
the source and target attachment tuples and one bound target attachment template
|
||||
inside the existing encrypted PRP authority, subject to its existing size limits.
|
||||
The header stores a bounded descriptor and digest. Before command delivery, the
|
||||
template is a separate changed-record part; after queueing, recovery uses the
|
||||
existing command or receipt. Control checkpoints do not rewrite another copy of
|
||||
instructions or tool context in the header.
|
||||
It is separate from the 4 KiB run-profile recovery envelope. Startup discovers
|
||||
this target obligation through bounded encrypted-header reads before a target
|
||||
session snapshot exists. It verifies both frozen runs, the accepted and released
|
||||
source, its original Runner artifact and the exact source header. Remote process
|
||||
verification then chooses authenticated source adoption or verified suspended
|
||||
source restoration. An in-memory rebound snapshot supplies normal admission;
|
||||
an existing target snapshot always takes precedence over that source baseline.
|
||||
The intent remains through command delivery and authenticated handoff, including
|
||||
an uncertain commit response. Source and target receipt phases require their
|
||||
exact physical authority; receipt replay cannot start a second Runner or attach
|
||||
again under a different endpoint. After the ordinary target snapshot commits,
|
||||
an in-memory notification permits intent retirement in the next existing command
|
||||
write. There is no separate metadata/snapshot transaction for that retirement.
|
||||
|
||||
Source accounting uses the source's immutable producer receipts and billing
|
||||
classification. Opening this receiver does not reset accounting ownership or
|
||||
reopen a finalized run. Equal finalized usage retains the existing audit policy;
|
||||
changed, incomplete or conflicting usage remains unacknowledged for
|
||||
reconciliation. The target's usage receiver never receives source events. Staging
|
||||
adds recovery control work only: text streaming and settled idle operation add no
|
||||
queries, snapshot transactions, pools or recurring scans. The first rollback
|
||||
release supporting this continuation must also read the staged controller format;
|
||||
older readers fail closed. Prepared-launch and other provider paths retain their
|
||||
existing admission and recovery contracts.
|
||||
|
||||
Accepted-result retirement still stops the old provider turn before draining and
|
||||
suspending its Runner. The checkpoint codec classifies `turn.stop` as lifecycle
|
||||
maintenance. Existing exact ownership, generation and accepted-result fences
|
||||
|
||||
@@ -41,6 +41,22 @@ retains the 32-candidate and newer-progress barrier checks. Snapshot transaction
|
||||
counts and stale-generation rejection are exercised by the control-plane port
|
||||
tests; incomplete-publication retention is exercised by the expiration tests.
|
||||
|
||||
The cold Codex transport regressions additionally retain a real provider event
|
||||
after completion, hold its persistence callback and require the original source
|
||||
tuple and Runner outbox to survive until acceptance. A separate case deletes the
|
||||
controller cache after its staged target is durably recorded and resumes through
|
||||
the same source before target attachment. Unknown semantic tails must stay held
|
||||
without target tool dispatch or a fabricated response. The staged restart helper
|
||||
and classifier tests cover target discovery before its first session snapshot,
|
||||
exact encrypted source/header/artifact bindings and the existing compatibility
|
||||
rebinder. Source accounting tests use real PostgreSQL to distinguish immutable
|
||||
receipt replay, equal finalized usage and late usage requiring reconciliation.
|
||||
Handoff crash tests must restore the exact queued command or authenticated
|
||||
receipt, reuse the verified physical Runner and preserve endpoint binding. Test
|
||||
both direct callbacks and the same guest listener when that ingress is enabled;
|
||||
changing an endpoint or losing physical proof must hold recovery.
|
||||
These component cases do not replace the normal three-controller Daytona gate.
|
||||
|
||||
Per-turn plugin continuation uses a stopped checkpoint on a replacement lease.
|
||||
The executor tests cover Claude and Codex, including reopening blocked work.
|
||||
The lease-authority integration test uses PostgreSQL to check that checkpoint
|
||||
|
||||
@@ -32,6 +32,8 @@ import {
|
||||
DurablePrpControlPlane,
|
||||
decodeDurablePrpCheckpoint,
|
||||
inspectDurablePrpCheckpointHeader,
|
||||
inspectStagedRunAttachment,
|
||||
inspectStagedRunAttachmentHeader,
|
||||
type DurablePrpCheckpoint,
|
||||
durableRecoveryInternals,
|
||||
SemanticToolNotDispatchedError,
|
||||
@@ -53,6 +55,66 @@ const identity: DurableRecoveryIdentity = {
|
||||
const expectedRunnerVersion = "0.3.0";
|
||||
const expectedRunnerDigest = `sha256:${"a".repeat(64)}`;
|
||||
|
||||
it("persists an undispatched source-to-target attachment before bootstrap and preserves it after cache loss", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-staged-attachment-"));
|
||||
const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" };
|
||||
const template = { provider: { provider: "codex", command: "fixture", instructions: "assigned instructions ".repeat(1_500) }, authorizedTools: {} };
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: resolve(root, "first"), identity, expectedRunnerVersion, expectedRunnerDigest });
|
||||
let checkpoint!: DurablePrpCheckpoint, templateWrites = 0;
|
||||
core.setCheckpointPersistence(async value => { checkpoint = value; if (value.changedRecords.has("staged-attachment-template")) templateWrites++; });
|
||||
let restored: DurablePrpControlPlane | undefined;
|
||||
let client: AuthenticatedClient | null = null;
|
||||
try {
|
||||
core.stageRunAttachment(target, template);
|
||||
await core.flushCheckpoint();
|
||||
expect(checkpoint.runId).toBe(target.runId);
|
||||
expect(checkpoint.predecessor).toEqual({ runId: identity.runId, normalizedSessionId: identity.normalizedSessionId });
|
||||
expect(checkpoint.workCommandIds).toEqual([]);
|
||||
const proof = inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!);
|
||||
expect(proof).toMatchObject({ attachment: { sourceIdentity: identity, targetIdentity: target, commandId: null } });
|
||||
expect(proof?.attachment).not.toHaveProperty("runAttachTemplate");
|
||||
expect(checkpoint.records.get("header")!.length).toBeLessThan(JSON.stringify(template).length / 2);
|
||||
expect(JSON.parse(checkpoint.records.get("staged-attachment-template")!)).toEqual(template);
|
||||
expect(proof?.runnerVersion).toBe(expectedRunnerVersion);
|
||||
expect(proof?.runnerDigest).toBe(expectedRunnerDigest);
|
||||
checkpoint.restoreSessionDirectory(resolve(root, "replacement"));
|
||||
restored = new DurablePrpControlPlane({ stateDirectory: resolve(root, "replacement", "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest });
|
||||
expect(inspectStagedRunAttachment(restored.store.state, target)).toMatchObject({ ...proof?.attachment, runAttachTemplate: template });
|
||||
expect(restored.store.state.commands).toEqual([]);
|
||||
expect(() => restored!.stageRunAttachment({ ...target, itemId: "foreign" }, template)).toThrow("conflicts");
|
||||
expect(() => restored!.queueCommand("turn.start", {})).toThrow("maintenance only");
|
||||
expect(() => restored!.queueCommand("run.attach", { ...template, paperclipNextAuthority: { identity: { ...target, itemId: "foreign" } } })).toThrow("conflicts");
|
||||
const corruptedRecords = new Map(checkpoint.records);
|
||||
corruptedRecords.set("staged-attachment-template", JSON.stringify({ provider: { command: "changed" } }));
|
||||
expect(() => decodeDurablePrpCheckpoint(corruptedRecords)).toThrow("native_runner_checkpoint_invalid");
|
||||
expect(inspectStagedRunAttachment(restored.store.state, { ...target, turnId: "foreign" })).toBeNull();
|
||||
for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"] as const) {
|
||||
const changed = JSON.parse(checkpoint.records.get("header")!);
|
||||
changed.state.stagedRunAttachment.targetIdentity[field] = "foreign";
|
||||
expect(inspectStagedRunAttachmentHeader(JSON.stringify(changed))).toBeNull();
|
||||
}
|
||||
const downgraded = JSON.parse(checkpoint.records.get("header")!);
|
||||
downgraded.state.schema = "paperclip.runner.durable.control-plane-state.v1";
|
||||
expect(inspectStagedRunAttachmentHeader(JSON.stringify(downgraded))).toBeNull();
|
||||
const beforeRetry = checkpoint;
|
||||
core.stageRunAttachment(target, template); await core.flushCheckpoint(); expect(checkpoint).toBe(beforeRetry);
|
||||
core.queueCommand("session.snapshot", {}); await core.flushCheckpoint();
|
||||
expect(checkpoint.changedRecords.has("staged-attachment-template")).toBe(false);
|
||||
await core.start(); client = await authenticate(core, core.issueBootstrapTicket());
|
||||
expect(client).not.toBeNull();
|
||||
core.queueCommand("run.attach", { ...template, paperclipNextAuthority: { identity: target,
|
||||
connection: { mode: "connect", connectUrl: core.connectUrl } } }, "staged-next-attach");
|
||||
await core.flushCheckpoint();
|
||||
expect(templateWrites).toBe(1);
|
||||
expect(checkpoint.records.has("staged-attachment-template")).toBe(false);
|
||||
expect(checkpoint.removedRecords).toContain("staged-attachment-template");
|
||||
expect(() => decodeDurablePrpCheckpoint(checkpoint.records)).not.toThrow();
|
||||
} finally {
|
||||
client?.socket.destroy();
|
||||
await restored?.stop(); await core.stop(); rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("continues externally persisted commands after losing the local controller directory", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-external-store-"));
|
||||
const options = { stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest };
|
||||
|
||||
@@ -54,6 +54,7 @@ const protocolVersion = RUNNER_PRP_PROTOCOL_MAX;
|
||||
const secureFrameSchema = "paperclip.runner.secure-frame.v1";
|
||||
const websocketGuid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
|
||||
const coreStateSchema = "paperclip.runner.durable.control-plane-state.v1";
|
||||
const stagedCoreStateSchema = "paperclip.runner.durable.control-plane-state.staged-attachment.v1";
|
||||
const externalCoreStateSchema = "paperclip.runner.external-control-plane-state.v1";
|
||||
const transitionCoreStateSchema =
|
||||
"paperclip.runner.durable.control-plane-state.warm-transition.v1";
|
||||
@@ -140,8 +141,17 @@ interface ConnectionLeaseRecord {
|
||||
revokedAt: string | null;
|
||||
}
|
||||
|
||||
export interface StagedRunAttachment {
|
||||
readonly sourceIdentity: DurableRecoveryIdentity;
|
||||
readonly targetIdentity: DurableRecoveryIdentity;
|
||||
readonly runAttachTemplate: Record<string, unknown>;
|
||||
readonly templateDigest: string;
|
||||
readonly commandId: string | null;
|
||||
}
|
||||
|
||||
interface StoredCoreState {
|
||||
schema: typeof coreStateSchema | typeof transitionCoreStateSchema;
|
||||
schema: typeof coreStateSchema | typeof transitionCoreStateSchema | typeof stagedCoreStateSchema;
|
||||
stagedRunAttachment?: Omit<StagedRunAttachment, "runAttachTemplate"> & { runAttachTemplate?: Record<string, unknown> };
|
||||
identity: DurableRecoveryIdentity;
|
||||
/** External checkpoints contain control state, not a duplicate output log. */
|
||||
checkpointEventPolicy?: "control-v1";
|
||||
@@ -311,10 +321,12 @@ type CheckpointHeader = {
|
||||
schema: "paperclip.runner.controller-checkpoint.v2";
|
||||
runnerVersion: string;
|
||||
runnerDigest: string;
|
||||
state: Omit<StoredCoreState, "commands" | "committedEvents" | "ackedSourceSeq" | "replayDeliveries">;
|
||||
state: Omit<StoredCoreState, "commands" | "committedEvents" | "ackedSourceSeq" | "replayDeliveries" | "stagedRunAttachment">
|
||||
& { stagedRunAttachment?: Omit<StagedRunAttachment, "runAttachTemplate"> };
|
||||
};
|
||||
|
||||
function checkpointRunId(state: StoredCoreState): string {
|
||||
if (state.stagedRunAttachment) return state.stagedRunAttachment.targetIdentity.runId;
|
||||
if (state.warmTransition) return state.warmTransition.receipt.newIdentity.runId;
|
||||
const attachment = state.commands.findLast(command => command.type === "run.attach");
|
||||
const next = attachment?.payload.paperclipNextAuthority as { identity?: { runId?: string } } | undefined;
|
||||
@@ -347,13 +359,15 @@ function checkpointChanges(records: ReadonlyMap<string, string>, runId: string,
|
||||
function captureCheckpoint(state: StoredCoreState, runnerVersion: string, runnerDigest: string): DurablePrpCheckpoint {
|
||||
const runId = checkpointRunId(state);
|
||||
const records = new Map<string, string>();
|
||||
const { commands, committedEvents, ackedSourceSeq: _cursor, replayDeliveries: _replays, ...metadata } = state;
|
||||
const { commands, committedEvents, ackedSourceSeq: _cursor, replayDeliveries: _replays, stagedRunAttachment, ...metadata } = state;
|
||||
const stagedDescriptor = stagedRunAttachment ? (({ runAttachTemplate: _template, ...descriptor }) => descriptor)(stagedRunAttachment) : undefined;
|
||||
if (stagedRunAttachment?.commandId === null) records.set("staged-attachment-template", JSON.stringify(stagedRunAttachment.runAttachTemplate));
|
||||
const latestUsage = committedEvents.findLast(event => event.eventType === "usage.reported");
|
||||
const recoveryEvents = committedEvents.filter(event => !isStreamingEvent(event) || event === latestUsage);
|
||||
for (const command of commands) records.set(`command:${command.controllerSeq}`, JSON.stringify(command));
|
||||
for (const event of recoveryEvents) records.set(`event:${event.sourceSeq}`, JSON.stringify(event));
|
||||
const header: CheckpointHeader = { schema: "paperclip.runner.controller-checkpoint.v2",
|
||||
runnerVersion, runnerDigest, state: { ...metadata, checkpointEventPolicy: "control-v1" } };
|
||||
runnerVersion, runnerDigest, state: { ...metadata, ...(stagedDescriptor ? { stagedRunAttachment: stagedDescriptor } : {}), checkpointEventPolicy: "control-v1" } };
|
||||
records.set("header", JSON.stringify(header));
|
||||
records.set("cursor", JSON.stringify({ ackedSourceSeq: state.ackedSourceSeq, replayDeliveries: state.replayDeliveries,
|
||||
commandCount: commands.length, eventCount: recoveryEvents.length }));
|
||||
@@ -397,11 +411,43 @@ export function inspectDurablePrpCheckpointHeader(contents: string): DurableReco
|
||||
const header = parseCheckpointHeader(contents);
|
||||
if (header.schema !== "paperclip.runner.controller-checkpoint.v2"
|
||||
|| header.state.checkpointEventPolicy !== "control-v1"
|
||||
|| ![coreStateSchema, transitionCoreStateSchema].includes(String(header.state.schema))) return null;
|
||||
|| ![coreStateSchema, transitionCoreStateSchema, stagedCoreStateSchema].includes(String(header.state.schema))
|
||||
|| (header.state.schema === stagedCoreStateSchema && !validStagedRunAttachment(header.state as unknown as StoredCoreState, true))) return null;
|
||||
return { ...header.state.identity };
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
/** Bounded checkpoint metadata inspection; never itself grants recovery authority. */
|
||||
export function inspectStagedRunAttachmentHeader(contents: string): {
|
||||
attachment: Omit<StagedRunAttachment, "runAttachTemplate">; runnerVersion: string; runnerDigest: string;
|
||||
protocolIdentity: DurableRecoveryIdentity;
|
||||
phase: "source_bootstrap" | "attachment_queued" | "awaiting_result" | "prepared" | "activated" | "completed";
|
||||
receipt: DurableWarmRunTransition | null;
|
||||
} | null {
|
||||
try {
|
||||
if (Buffer.byteLength(contents) > maxStateBytes) return null;
|
||||
const header = parseCheckpointHeader(contents), state = header.state as unknown as StoredCoreState;
|
||||
if (header.schema !== "paperclip.runner.controller-checkpoint.v2"
|
||||
|| state.checkpointEventPolicy !== "control-v1" || !validStagedRunAttachment(state, true)
|
||||
|| !state.stagedRunAttachment) return null;
|
||||
const transition = state.warmTransition;
|
||||
const completed = stagedTargetActivated(state) ? state.completedWarmTransition : undefined;
|
||||
const bound = transition ?? completed;
|
||||
if (bound) {
|
||||
const receipt = bound.receipt;
|
||||
const result = transition?.phase === "awaiting_result" ? transition.expectedResult : bound.command.result;
|
||||
const expected = warmTransitionReceipt(receipt.oldIdentity, bound.command, result!, receipt.oldAckedSourceSeq,
|
||||
{ leaseId: receipt.leaseId, expiresAtUnixMs: receipt.leaseExpiresAtUnixMs, revocationEpoch: receipt.leaseRevocationEpoch },
|
||||
header.runnerVersion, header.runnerDigest);
|
||||
if (canonicalJson(expected) !== canonicalJson(receipt)) return null;
|
||||
}
|
||||
return { attachment: structuredClone(state.stagedRunAttachment), protocolIdentity: structuredClone(state.identity),
|
||||
phase: transition?.phase ?? (completed ? "completed" : state.stagedRunAttachment.commandId ? "attachment_queued" : "source_bootstrap"),
|
||||
receipt: bound ? structuredClone(bound.receipt) : null,
|
||||
runnerVersion: header.runnerVersion, runnerDigest: header.runnerDigest };
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function checkpointState(records: ReadonlyMap<string, string>) {
|
||||
const header = parseCheckpointHeader(records.get("header") ?? "null");
|
||||
const read = (key: string): unknown => {
|
||||
@@ -423,9 +469,14 @@ function checkpointState(records: ReadonlyMap<string, string>) {
|
||||
}
|
||||
const cursor = v2 ? read("cursor") : header.state;
|
||||
if (!isRecord(cursor) || (v2 && (cursor.commandCount !== commandKeys.length || cursor.eventCount !== eventKeys.length
|
||||
|| records.size !== commandKeys.length + eventKeys.length + 2))) throw new Error("native_runner_checkpoint_incomplete");
|
||||
|| records.size !== commandKeys.length + eventKeys.length + 2 + (isRecord(header.state.stagedRunAttachment) && header.state.stagedRunAttachment.commandId === null ? 1 : 0)))) throw new Error("native_runner_checkpoint_incomplete");
|
||||
const state = { ...header.state, ackedSourceSeq: cursor.ackedSourceSeq, replayDeliveries: cursor.replayDeliveries,
|
||||
commands: commandKeys.map(read), committedEvents: eventKeys.map(read) };
|
||||
commands: commandKeys.map(read), committedEvents: eventKeys.map(read) } as unknown as StoredCoreState;
|
||||
if (state.stagedRunAttachment?.commandId === null) {
|
||||
const template = read("staged-attachment-template");
|
||||
if (!isRecord(template)) throw new Error("native_runner_checkpoint_invalid");
|
||||
state.stagedRunAttachment = { ...state.stagedRunAttachment, runAttachTemplate: template };
|
||||
}
|
||||
if (!isStoredCoreState(state, header.state.identity)) throw new Error("native_runner_checkpoint_invalid");
|
||||
if (state.commands.some((command, index) => commandKeys[index] !== `command:${command.controllerSeq}`)
|
||||
|| state.committedEvents.some((event, index) => eventKeys[index] !== `event:${event.sourceSeq}`)) {
|
||||
@@ -683,9 +734,9 @@ function warmTransitionReceipt(
|
||||
|
||||
function validStoredWarmTransition(state: StoredCoreState): boolean {
|
||||
const transition = state.warmTransition;
|
||||
if (!transition) return state.schema === coreStateSchema;
|
||||
if (!transition) return state.schema === coreStateSchema || (state.schema === stagedCoreStateSchema && validStagedRunAttachment(state));
|
||||
if (
|
||||
state.schema !== transitionCoreStateSchema ||
|
||||
(state.schema !== transitionCoreStateSchema && state.schema !== stagedCoreStateSchema) ||
|
||||
!["awaiting_result", "prepared", "activated"].includes(transition.phase) ||
|
||||
!isRecord(transition.receipt) ||
|
||||
!isRecord(transition.command) ||
|
||||
@@ -802,6 +853,76 @@ function unsettledSemanticInput(
|
||||
}
|
||||
}
|
||||
|
||||
function stagedCommandMatches(staged: Pick<StagedRunAttachment, "targetIdentity" | "templateDigest" | "commandId">, command: DurableRecoveryCoreCommand): boolean {
|
||||
if (command.type !== "run.attach" || command.commandId !== staged.commandId) return false;
|
||||
const { paperclipNextAuthority, ...template } = command.payload;
|
||||
return isRecord(paperclipNextAuthority)
|
||||
&& canonicalJson(paperclipNextAuthority.identity) === canonicalJson(staged.targetIdentity)
|
||||
&& `sha256:${canonicalDigest(template)}` === staged.templateDigest;
|
||||
}
|
||||
|
||||
function stagedAttachmentTemplate(state: StoredCoreState): Record<string, unknown> | null {
|
||||
const staged = state.stagedRunAttachment;
|
||||
if (!staged) return null;
|
||||
if (staged.commandId === null) return staged.runAttachTemplate ?? null;
|
||||
const command = state.commands?.find(command => command.commandId === staged.commandId)
|
||||
?? state.warmTransition?.command ?? state.completedWarmTransition?.command;
|
||||
if (!command || !stagedCommandMatches(staged, command)) return null;
|
||||
const { paperclipNextAuthority: _authority, ...template } = command.payload;
|
||||
return template;
|
||||
}
|
||||
|
||||
function stagedTargetActivated(state: StoredCoreState): boolean {
|
||||
const staged = state.stagedRunAttachment, completed = state.completedWarmTransition;
|
||||
return !!staged && !state.warmTransition && !!completed
|
||||
&& canonicalJson(state.identity) === canonicalJson(staged.targetIdentity)
|
||||
&& completed.receipt.commandId === staged.commandId
|
||||
&& canonicalJson(completed.receipt.oldIdentity) === canonicalJson(staged.sourceIdentity)
|
||||
&& canonicalJson(completed.receipt.newIdentity) === canonicalJson(staged.targetIdentity)
|
||||
&& stagedCommandMatches(staged, completed.command);
|
||||
}
|
||||
|
||||
function validStagedRunAttachment(state: StoredCoreState, headerOnly = false): boolean {
|
||||
const staged = state.stagedRunAttachment;
|
||||
if (staged === undefined) return state.schema !== stagedCoreStateSchema;
|
||||
if (!(state.schema === stagedCoreStateSchema && isRecord(staged)
|
||||
&& Object.keys(staged).sort().join(",") === (headerOnly || staged.commandId !== null ? "commandId,sourceIdentity,targetIdentity,templateDigest" : "commandId,runAttachTemplate,sourceIdentity,targetIdentity,templateDigest")
|
||||
&& exactIdentity(staged.sourceIdentity) && exactIdentity(staged.targetIdentity)
|
||||
&& staged.sourceIdentity.runId !== staged.targetIdentity.runId
|
||||
&& ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"].every(key =>
|
||||
staged.sourceIdentity[key as keyof DurableRecoveryIdentity] === staged.targetIdentity[key as keyof DurableRecoveryIdentity])
|
||||
&& /^sha256:[0-9a-f]{64}$/.test(staged.templateDigest)
|
||||
&& (headerOnly || staged.commandId !== null || (isRecord(staged.runAttachTemplate) && isRecord(staged.runAttachTemplate.provider)
|
||||
&& staged.runAttachTemplate.paperclipNextAuthority === undefined
|
||||
&& `sha256:${canonicalDigest(staged.runAttachTemplate)}` === staged.templateDigest
|
||||
&& Buffer.byteLength(JSON.stringify(staged.runAttachTemplate)) <= maxCommandBytes))
|
||||
&& (staged.commandId === null || (typeof staged.commandId === "string" && stableIdPattern.test(staged.commandId))))) return false;
|
||||
const sourceCurrent = canonicalJson(state.identity) === canonicalJson(staged.sourceIdentity);
|
||||
if (!sourceCurrent && canonicalJson(state.identity) !== canonicalJson(staged.targetIdentity)) return false;
|
||||
if (state.warmTransition) {
|
||||
const transition = state.warmTransition;
|
||||
return staged.commandId !== null && stagedCommandMatches(staged, transition.command)
|
||||
&& canonicalJson(transition.receipt.oldIdentity) === canonicalJson(staged.sourceIdentity)
|
||||
&& canonicalJson(transition.receipt.newIdentity) === canonicalJson(staged.targetIdentity)
|
||||
&& (sourceCurrent ? transition.phase !== "activated" : transition.phase === "activated");
|
||||
}
|
||||
if (!sourceCurrent) return stagedTargetActivated(state);
|
||||
if (staged.commandId === null) return true;
|
||||
// Header-only inspection cannot certify the separate command record. Full
|
||||
// restoration below requires its exact tuple, payload and command ID.
|
||||
return !Array.isArray(state.commands) || state.commands.some(command => stagedCommandMatches(staged, command));
|
||||
}
|
||||
|
||||
/** Inspection is evidence only; the caller still owns recovery authorization. */
|
||||
export function inspectStagedRunAttachment(state: unknown, expectedTarget?: DurableRecoveryIdentity): StagedRunAttachment | null {
|
||||
try {
|
||||
if (!isRecord(state) || !exactIdentity(state.identity) || !isStoredCoreState(state, state.identity)
|
||||
|| !state.stagedRunAttachment || (expectedTarget && canonicalJson(state.stagedRunAttachment.targetIdentity) !== canonicalJson(expectedTarget))) return null;
|
||||
const template = stagedAttachmentTemplate(state);
|
||||
return template ? { ...structuredClone(state.stagedRunAttachment), runAttachTemplate: structuredClone(template) } : null;
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function isStoredCoreState(
|
||||
value: unknown,
|
||||
identity: DurableRecoveryIdentity,
|
||||
@@ -811,7 +932,7 @@ function isStoredCoreState(
|
||||
const events = value.committedEvents;
|
||||
if (
|
||||
(value.schema !== coreStateSchema &&
|
||||
value.schema !== transitionCoreStateSchema) ||
|
||||
value.schema !== transitionCoreStateSchema && value.schema !== stagedCoreStateSchema) ||
|
||||
canonicalJson(value.identity) !== canonicalJson(identity) ||
|
||||
(value.checkpointEventPolicy !== undefined && value.checkpointEventPolicy !== "control-v1") ||
|
||||
!isRecord(value.tickets) ||
|
||||
@@ -828,6 +949,7 @@ function isStoredCoreState(
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
if (!validStagedRunAttachment(value as unknown as StoredCoreState)) return false;
|
||||
if (!validStoredWarmTransition(value as unknown as StoredCoreState))
|
||||
return false;
|
||||
if (value.completedWarmTransition !== undefined) {
|
||||
@@ -1758,6 +1880,7 @@ class AuthorityConnection {
|
||||
/** Authenticated, replay-safe PRP transport authority. Business operations are caller supplied. */
|
||||
export class DurablePrpControlPlane {
|
||||
#identity: DurableRecoveryIdentity;
|
||||
#stagedTargetCheckpointPublished: string | null = null;
|
||||
readonly #store: DurableCoreStore;
|
||||
#expectedRunnerVersion: string;
|
||||
#expectedRunnerDigest: string;
|
||||
@@ -1854,7 +1977,12 @@ export class DurablePrpControlPlane {
|
||||
cached.set(part, prior?.record === record ? prior : { record, command: JSON.parse(record) });
|
||||
}
|
||||
this.#publicationCommands = cached;
|
||||
return { ...header.state, ...cursor, commands: [...cached.values()].map(value => value.command),
|
||||
const descriptor = header.state.stagedRunAttachment;
|
||||
const workingStage = this.#store.state.stagedRunAttachment;
|
||||
const staged = descriptor?.commandId === null ? { ...descriptor,
|
||||
runAttachTemplate: workingStage?.templateDigest === descriptor.templateDigest && workingStage.runAttachTemplate
|
||||
? workingStage.runAttachTemplate : JSON.parse(snapshot.records.get("staged-attachment-template")!) as Record<string, unknown> } : descriptor;
|
||||
return { ...header.state, ...(staged ? { stagedRunAttachment: staged } : {}), ...cursor, commands: [...cached.values()].map(value => value.command),
|
||||
// Event records are immutable after insertion. Retain their array, not
|
||||
// another deep copy of the streaming payload window.
|
||||
committedEvents: this.#store.state.committedEvents };
|
||||
@@ -2323,6 +2451,23 @@ export class DurablePrpControlPlane {
|
||||
return ticket;
|
||||
}
|
||||
|
||||
/** Import exact durable handoff evidence for a caller-verified surviving participant; issues no credential. */
|
||||
prepareWarmTransitionAdoption(input: { transitionId: string; runnerState: Record<string, unknown> }): void {
|
||||
this.#store.assertWritable();
|
||||
const pending = input.runnerState.warmTransition;
|
||||
const proof = warmTransitionRecoveryProof({ controlPlaneState: this.#store.state, runnerState: input.runnerState,
|
||||
expectedNewIdentity: (isRecord(pending) && isRecord(pending.receipt) ? pending.receipt.newIdentity : null) as DurableRecoveryIdentity,
|
||||
expectedRunnerVersion: this.#expectedRunnerVersion, expectedRunnerDigest: this.#expectedRunnerDigest });
|
||||
if (!proof || proof.transition.receipt.transitionId !== input.transitionId || !this.#store.state.stagedRunAttachment) {
|
||||
throw new Error("Warm transition adoption snapshot proof is not authorized.");
|
||||
}
|
||||
if (this.#store.state.warmTransition && canonicalJson(this.#store.state.warmTransition) === canonicalJson(proof.transition)) return;
|
||||
const candidate = structuredClone(this.#store.state);
|
||||
candidate.schema = stagedCoreStateSchema;
|
||||
candidate.warmTransition = structuredClone(proof.transition);
|
||||
this.#store.commit(candidate);
|
||||
}
|
||||
|
||||
/** Caller-owned recovery admission is required; a receipt is not a credential. */
|
||||
issueWarmTransitionBootstrapTicket(
|
||||
input: {
|
||||
@@ -2361,7 +2506,7 @@ export class DurablePrpControlPlane {
|
||||
original.expiresAtUnixMs,
|
||||
);
|
||||
const candidate = structuredClone(this.#store.state);
|
||||
candidate.schema = transitionCoreStateSchema;
|
||||
candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema;
|
||||
candidate.warmTransition = structuredClone(transition);
|
||||
candidate.tickets[material.credentialId] = {
|
||||
recordId: `bootstrap_ticket_${randomUUID()}`,
|
||||
@@ -2380,6 +2525,36 @@ export class DurablePrpControlPlane {
|
||||
return ticket;
|
||||
}
|
||||
|
||||
/** Caller confirms its existing session-checkpoint transaction committed. No journal write. */
|
||||
markStagedTargetCheckpointPublished(identity: DurableRecoveryIdentity): boolean {
|
||||
if (!stagedTargetActivated(this.#store.state) || canonicalJson(identity) !== canonicalJson(this.#store.state.identity)) return false;
|
||||
this.#stagedTargetCheckpointPublished = canonicalJson(identity);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Durable intent precedes source restoration; it is never a dispatched command. */
|
||||
stageRunAttachment(targetIdentity: DurableRecoveryIdentity, runAttachTemplate: Record<string, unknown>): void {
|
||||
this.#store.assertWritable();
|
||||
const existing = this.#store.state.stagedRunAttachment;
|
||||
if (existing && !(stagedTargetActivated(this.#store.state)
|
||||
&& this.#stagedTargetCheckpointPublished === canonicalJson(this.#store.state.identity)
|
||||
&& targetIdentity.runId !== existing.targetIdentity.runId)) {
|
||||
if (!validStagedRunAttachment(this.#store.state)
|
||||
|| canonicalJson(existing.targetIdentity) !== canonicalJson(targetIdentity)
|
||||
|| canonicalJson(stagedAttachmentTemplate(this.#store.state)) !== canonicalJson(runAttachTemplate)) {
|
||||
throw new Error("Staged run attachment conflicts with its durable intent.");
|
||||
}
|
||||
return;
|
||||
}
|
||||
const candidate = structuredClone(this.#store.state);
|
||||
candidate.schema = stagedCoreStateSchema;
|
||||
candidate.stagedRunAttachment = { sourceIdentity: structuredClone(candidate.identity),
|
||||
targetIdentity: structuredClone(targetIdentity), runAttachTemplate: structuredClone(runAttachTemplate),
|
||||
templateDigest: `sha256:${canonicalDigest(runAttachTemplate)}`, commandId: null };
|
||||
if (!validStagedRunAttachment(candidate)) throw new Error("Staged run attachment binding is invalid.");
|
||||
this.#store.commit(candidate);
|
||||
}
|
||||
|
||||
queueCommand(
|
||||
type: string,
|
||||
payload: Record<string, unknown> = {},
|
||||
@@ -2387,6 +2562,21 @@ export class DurablePrpControlPlane {
|
||||
deliverImmediately = false,
|
||||
): DurableRecoveryCoreCommand {
|
||||
this.#store.assertWritable();
|
||||
const staged = this.#store.state.stagedRunAttachment;
|
||||
const retireStaged = stagedTargetActivated(this.#store.state) && (type === "turn.start"
|
||||
|| (this.#stagedTargetCheckpointPublished === canonicalJson(this.#store.state.identity)
|
||||
&& !["runner.drain", "session.snapshot", "turn.stop", "runner.suspend", "session.close"].includes(type)));
|
||||
if (staged && !retireStaged) {
|
||||
if (type === "run.attach") {
|
||||
const { paperclipNextAuthority, ...template } = payload;
|
||||
if (!isRecord(paperclipNextAuthority)
|
||||
|| canonicalJson(paperclipNextAuthority.identity) !== canonicalJson(staged.targetIdentity)
|
||||
|| `sha256:${canonicalDigest(template)}` !== staged.templateDigest
|
||||
|| (staged.commandId !== null && staged.commandId !== commandId)) throw new Error("Staged run attachment command conflicts.");
|
||||
} else if (!stagedTargetActivated(this.#store.state) && !["runner.drain", "session.open", "session.snapshot", "turn.stop", "runner.suspend", "session.close", "semantic_tool.result"].includes(type)) {
|
||||
throw new Error("Staged run attachment permits recovery maintenance only.");
|
||||
}
|
||||
}
|
||||
const transition = this.#store.state.warmTransition;
|
||||
if (transition && transition.phase !== "activated") {
|
||||
if (
|
||||
@@ -2461,6 +2651,16 @@ export class DurablePrpControlPlane {
|
||||
if (Buffer.byteLength(JSON.stringify(command)) > maxCommandBytes)
|
||||
throw new CommandJournalLimitError("command_payload_too_large");
|
||||
this.#store.state.commands.push(command);
|
||||
if (staged && !retireStaged && type === "run.attach") {
|
||||
const { runAttachTemplate: _template, ...descriptor } = staged;
|
||||
this.#store.state.stagedRunAttachment = { ...descriptor, commandId: command.commandId };
|
||||
} else if (retireStaged) {
|
||||
// Native runtime persists the recovered session before its first turn.
|
||||
// Read/goal controls during opening must not retire startup evidence.
|
||||
delete this.#store.state.stagedRunAttachment;
|
||||
this.#store.state.schema = coreStateSchema;
|
||||
this.#stagedTargetCheckpointPublished = null;
|
||||
}
|
||||
this.#store.save();
|
||||
void this.flushCheckpoint().catch(() => undefined);
|
||||
if (deliverImmediately) {
|
||||
@@ -2677,7 +2877,7 @@ export class DurablePrpControlPlane {
|
||||
}
|
||||
if (transition) {
|
||||
const candidate = structuredClone(this.#store.state);
|
||||
candidate.schema = coreStateSchema;
|
||||
candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : coreStateSchema;
|
||||
candidate.leases[transition.credentialId]!.identity = structuredClone(
|
||||
receipt.newIdentity,
|
||||
);
|
||||
@@ -3205,8 +3405,9 @@ export class DurablePrpControlPlane {
|
||||
}
|
||||
if (transition.phase === "prepared") {
|
||||
const candidate = initialCoreState(transition.receipt.newIdentity);
|
||||
candidate.stagedRunAttachment = structuredClone(this.#store.state.stagedRunAttachment);
|
||||
candidate.preparedLaunchConfigurations = structuredClone(this.#store.state.preparedLaunchConfigurations);
|
||||
candidate.schema = transitionCoreStateSchema;
|
||||
candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema;
|
||||
candidate.warmTransition = {
|
||||
...structuredClone(transition),
|
||||
phase: "activated",
|
||||
@@ -3507,7 +3708,7 @@ export class DurablePrpControlPlane {
|
||||
)!;
|
||||
completed.status = "completed";
|
||||
completed.result = structuredClone(result);
|
||||
candidate.schema = transitionCoreStateSchema;
|
||||
candidate.schema = candidate.stagedRunAttachment ? stagedCoreStateSchema : transitionCoreStateSchema;
|
||||
candidate.warmTransition = {
|
||||
receipt,
|
||||
phase: "prepared",
|
||||
|
||||
@@ -36,6 +36,10 @@ export interface RunnerExecutionDriver {
|
||||
checkpoint?: (settlement: "settled" | "unsettled") => Promise<void> | void;
|
||||
release: (reason?: "detach") => Promise<void> | void;
|
||||
}>;
|
||||
/** Register source and target checkpoint ownership before staging changes the stored run. */
|
||||
onStagedRunAttachment?: (authority: DurablePrpControlPlane, proof: {
|
||||
sourceIdentity: DurableRecoveryIdentity; targetIdentity: DurableRecoveryIdentity; templateDigest: string;
|
||||
}) => Promise<void>;
|
||||
/** Override process ownership; omission uses the SDK's local process owner. */
|
||||
runnerProcessLauncher?: (
|
||||
spec: RunnerProcessLaunchSpec,
|
||||
|
||||
@@ -37,6 +37,9 @@ export {
|
||||
RUNNER_PRP_PROTOCOL_MIN,
|
||||
RUNNER_PRP_PROTOCOL_MAX,
|
||||
inspectWarmRunTransition,
|
||||
inspectStagedRunAttachment,
|
||||
inspectStagedRunAttachmentHeader,
|
||||
type StagedRunAttachment,
|
||||
type DurablePrpControlPlaneOptions,
|
||||
type DurablePrpCheckpoint,
|
||||
decodeDurablePrpCheckpoint,
|
||||
|
||||
@@ -8239,7 +8239,7 @@ it("still fails closed when a real close grace period cannot fit a durable suspe
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("cold-restores a suspended provider session under its durable run binding", async () => {
|
||||
it.each([false, true, "restart-before-bootstrap", "unknown-semantic", "accounting-rejected"] as const)("cold-restores a suspended provider session under its durable run binding (pending tail: %s)", async (pendingTail) => {
|
||||
const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-cold-attach-"));
|
||||
const tracePath = join(stateDirectory, "provider-trace.ndjson");
|
||||
const skillRoot = join(stateDirectory, "runtime-skill");
|
||||
@@ -8332,14 +8332,57 @@ it("cold-restores a suspended provider session under its durable run binding", a
|
||||
throw new Error("cold attach fixture did not return a provider thread");
|
||||
}
|
||||
|
||||
// Model a retained provider-owned suffix: rotation must deliver and durably
|
||||
// accept it before attempting run.attach, rather than dropping or ACKing it.
|
||||
const retainedDelta = "cold-recovery-retained-tail";
|
||||
if (pendingTail) {
|
||||
const providerFile = join(stateDirectory, "runner", "codex-provider-state.json");
|
||||
const provider = JSON.parse(await readFile(providerFile, "utf8"));
|
||||
expect(provider.pendingEvents).toEqual([]);
|
||||
provider.pendingEvents.push({
|
||||
executorEventId: `codex_provider_${String(provider.nextProviderEventSeq++).padStart(16, "0")}`,
|
||||
eventType: pendingTail === "unknown-semantic" ? "semantic_tool.input" : "usage.reported",
|
||||
priority: "p0",
|
||||
payload: pendingTail === "unknown-semantic" ? { semantic_tool: {
|
||||
schema: "paperclip.prp.semantic_tool.v1", schemaVersion: 1, phase: "input",
|
||||
callId: "unknown-source-call", operationId: "get_task_context", input: {}, idempotencyKey: null,
|
||||
correlation: { runId: baseIdentity.runId, normalizedSessionId: baseIdentity.normalizedSessionId,
|
||||
turnId: baseIdentity.turnId, itemId: baseIdentity.itemId },
|
||||
content: { digest: `sha256:${createHash("sha256").update("{}").digest("hex")}`, redactionDisposition: "digest_only", references: [] },
|
||||
} } : { provider: "codex", kind: "usage", delta: retainedDelta,
|
||||
providerTurnId: provider.completed_provider_turn_id, runDelta: { inputTokens: 20, outputTokens: 2 },
|
||||
runDeltaAvailable: true, runDeltaBasis: "run" },
|
||||
});
|
||||
await writeFile(providerFile, JSON.stringify(provider));
|
||||
}
|
||||
const retainedEvents: PrpEvent[] = [];
|
||||
let releaseRetained!: () => void, retainedEntered!: () => void;
|
||||
const retainedAccepted = new Promise<void>(done => { releaseRetained = done; });
|
||||
const retainedAdmission = new Promise<void>(done => { retainedEntered = done; });
|
||||
const secondIdentity = {
|
||||
...baseIdentity,
|
||||
runId: "run-cold-second",
|
||||
turnId: "turn-cold-second",
|
||||
itemId: "item-cold-second",
|
||||
};
|
||||
const rotated = createCapabilityRunnerdCodexTransport({
|
||||
let stagedCheckpoint: durableControlPlane.DurablePrpCheckpoint | undefined;
|
||||
let rejectStagedCommit = pendingTail === "restart-before-bootstrap";
|
||||
const rotatedOptions = {
|
||||
...options,
|
||||
onStagedRunAttachment: pendingTail === "restart-before-bootstrap" ? async (authority: DurablePrpControlPlane) => {
|
||||
authority.setCheckpointPersistence(async checkpoint => {
|
||||
stagedCheckpoint = checkpoint;
|
||||
if (rejectStagedCommit) throw new Error("fixture lost controller after durable intent");
|
||||
});
|
||||
} : undefined,
|
||||
onRecoveryAccountingEvent: pendingTail ? async (event: PrpEvent) => {
|
||||
if (event.payload.delta === retainedDelta) {
|
||||
retainedEvents.push(structuredClone(event));
|
||||
retainedEntered();
|
||||
if (pendingTail === "accounting-rejected") throw new Error("source accounting receipt unavailable");
|
||||
await retainedAccepted;
|
||||
}
|
||||
} : undefined,
|
||||
environment: { ...options.environment, PAPERCLIP_GITHUB_BROKER_TOKEN: "test-run-B-capability" },
|
||||
codexArgs: options.codexArgs.map((arg) => arg.replace('/run/A', '/run/B')),
|
||||
resumeDynamicTools: dynamicTools,
|
||||
@@ -8348,14 +8391,81 @@ it("cold-restores a suspended provider session under its durable run binding", a
|
||||
criterionIds: ["criterion-second"],
|
||||
},
|
||||
prpIdentity: secondIdentity,
|
||||
});
|
||||
rotated.transport.setServerRequestHandler(async () => ({
|
||||
success: true,
|
||||
contentItems: [],
|
||||
}));
|
||||
};
|
||||
let rotated = createCapabilityRunnerdCodexTransport(rotatedOptions);
|
||||
if (pendingTail === "restart-before-bootstrap") {
|
||||
await expect(rotated.transport.request("thread/read", {})).rejects.toThrow("fixture lost controller after durable intent");
|
||||
expect(rotated.evidence().runnerPid).toBeNull();
|
||||
expect(stagedCheckpoint?.runId).toBe(secondIdentity.runId);
|
||||
expect([...stagedCheckpoint!.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value).type)).not.toContain("run.attach");
|
||||
await rotated.detachControllerForRestart().catch(() => undefined);
|
||||
rejectStagedCommit = false;
|
||||
await rm(join(stateDirectory, "control-plane"), { recursive: true, force: true });
|
||||
stagedCheckpoint!.restoreSessionDirectory(stateDirectory);
|
||||
rotated = createCapabilityRunnerdCodexTransport(rotatedOptions);
|
||||
}
|
||||
const currentHandler = vi.fn(async () => ({ success: true, contentItems: [] }));
|
||||
rotated.transport.setServerRequestHandler(currentHandler);
|
||||
if (pendingTail === "accounting-rejected") {
|
||||
const reading = rotated.transport.request("thread/read", {});
|
||||
void reading.catch(() => undefined);
|
||||
try {
|
||||
await Promise.race([retainedAdmission, reading]);
|
||||
const source = JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8"));
|
||||
expect(source.identity).toEqual(baseIdentity);
|
||||
expect(source.stagedRunAttachment.targetIdentity).toEqual(secondIdentity);
|
||||
expect(source.commands.some((command: { type: string }) => command.type === "run.attach")).toBe(false);
|
||||
const retainedRunner = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8"));
|
||||
expect(retainedRunner.outbox.some((entry: { envelope: { payload: PrpEvent } }) =>
|
||||
entry.envelope.payload.sourceEventId === retainedEvents[0]!.sourceEventId)).toBe(true);
|
||||
expect(currentHandler).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await expect(rotated.transport.close()).rejects.toMatchObject({ code: "native_session_close_unrecoverable" });
|
||||
await expect(reading).rejects.toThrow();
|
||||
await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills"));
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (pendingTail === "unknown-semantic") {
|
||||
try {
|
||||
await expect(rotated.transport.request("thread/read", {})).rejects.toThrow("native_runner_source_semantic_reconciliation_required");
|
||||
expect(currentHandler).not.toHaveBeenCalled();
|
||||
const retained = JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8"));
|
||||
expect(retained.identity).toEqual(baseIdentity);
|
||||
expect(retained.stagedRunAttachment.targetIdentity).toEqual(secondIdentity);
|
||||
expect(retained.commands.some((command: { type: string }) => ["run.attach", "semantic_tool.result"].includes(command.type))).toBe(false);
|
||||
expect(retained.committedEvents.some((event: PrpEvent) => event.eventType === "semantic_tool.input")).toBe(true);
|
||||
} finally {
|
||||
await expect(rotated.transport.close()).rejects.toMatchObject({ code: "native_session_close_unrecoverable" });
|
||||
await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills"));
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const read = await rotated.transport.request("thread/read", {});
|
||||
const reading = rotated.transport.request("thread/read", {});
|
||||
void reading.catch(() => undefined);
|
||||
if (pendingTail) {
|
||||
await Promise.race([retainedAdmission, reading]);
|
||||
const retainedRunner = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8"));
|
||||
expect(retainedRunner.outbox.some((entry: { envelope: { payload: PrpEvent } }) =>
|
||||
entry.envelope.payload.sourceEventId === retainedEvents[0]!.sourceEventId)).toBe(true);
|
||||
const retainedCore = stagedCheckpoint ? {
|
||||
identity: JSON.parse(stagedCheckpoint.records.get("header")!).state.identity,
|
||||
commands: [...stagedCheckpoint.records].filter(([key]) => key.startsWith("command:")).map(([, value]) => JSON.parse(value)),
|
||||
} : JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8"));
|
||||
expect(retainedCore.identity).toEqual(baseIdentity);
|
||||
expect(retainedCore.commands.some((command: { type: string }) => command.type === "run.attach")).toBe(false);
|
||||
}
|
||||
releaseRetained();
|
||||
const read = await reading;
|
||||
expect(retainedEvents).toHaveLength(pendingTail ? 1 : 0);
|
||||
if (pendingTail) expect(retainedEvents[0]).toMatchObject({
|
||||
runId: baseIdentity.runId, turnId: baseIdentity.turnId, itemId: baseIdentity.itemId,
|
||||
});
|
||||
const persistedProvider = JSON.parse(await readFile(join(stateDirectory, "runner", "codex-provider-state.json"), "utf8"));
|
||||
expect(persistedProvider.pendingEvents).toEqual([]);
|
||||
expect(persistedProvider.config.args.join("\n")).toContain('/run/B');
|
||||
expect(JSON.stringify(persistedProvider)).not.toContain("test-run-B-capability");
|
||||
expect(read.thread).toMatchObject({
|
||||
@@ -8372,8 +8482,14 @@ it("cold-restores a suspended provider session under its durable run binding", a
|
||||
expect(rotated.evidence().diagnostics).toContain(
|
||||
"runnerd attached the durable provider session to a fresh PRP run authority",
|
||||
);
|
||||
expect(await stat(join(stateDirectory, "authority-epochs"))).toBeDefined();
|
||||
if (pendingTail) {
|
||||
const target = stagedCheckpoint ? JSON.parse(stagedCheckpoint.records.get("header")!).state
|
||||
: JSON.parse(await readFile(join(stateDirectory, "control-plane", "control-plane-state.json"), "utf8"));
|
||||
expect(target.stagedRunAttachment).toBeUndefined();
|
||||
expect(target.completedWarmTransition.receipt).toMatchObject({ oldIdentity: baseIdentity, newIdentity: secondIdentity });
|
||||
} else expect(await stat(join(stateDirectory, "authority-epochs"))).toBeDefined();
|
||||
} finally {
|
||||
releaseRetained();
|
||||
await rotated.transport.close();
|
||||
}
|
||||
const requestFrames = (await readFile(tracePath, "utf8"))
|
||||
@@ -8410,6 +8526,12 @@ it("cold-restores a suspended provider session under its durable run binding", a
|
||||
);
|
||||
}
|
||||
|
||||
if (pendingTail) {
|
||||
await releaseMaterializedNativeRuntimeSkills(join(stateDirectory, "codex-home", "skills"));
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
return;
|
||||
}
|
||||
|
||||
// A remote process owner keeps runner-state outside the controller's local
|
||||
// session root. Resume must defer to its explicit state reader instead of
|
||||
// rejecting recovery before the remote checkpoint can be made available.
|
||||
|
||||
@@ -69,6 +69,7 @@ import type {
|
||||
} from "../contracts/harness-driver.js";
|
||||
import {
|
||||
DurablePrpControlPlane,
|
||||
inspectStagedRunAttachment,
|
||||
SemanticToolNotDispatchedError,
|
||||
durableRecoveryInternals,
|
||||
inspectWarmRunTransition,
|
||||
@@ -1241,6 +1242,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD
|
||||
observeCredentials?: boolean;
|
||||
onCredentialsUpdated?: (event: PrpEvent) => Promise<void>;
|
||||
onAccountingEvent?: (event: PrpEvent) => Promise<void>;
|
||||
onRecoveryAccountingEvent?: (event: PrpEvent, sourceIdentity: DurableRecoveryIdentity) => Promise<void>;
|
||||
recoverSemanticToolInput?: DurablePrpControlPlaneOptions["recoverSemanticToolInput"];
|
||||
opencodePermissionMode?: NativeOpenCodePermissionMode;
|
||||
acpxAgent?: QualifiedAcpxAgent;
|
||||
@@ -1379,7 +1381,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD
|
||||
itemId: string;
|
||||
};
|
||||
/** Existing server-owned routes only; never provision two ingress owners. */
|
||||
warmTransitionRegistrationMode?: "routed_connect";
|
||||
warmTransitionRegistrationMode?: "routed_connect" | "routed_same_endpoint";
|
||||
/** Rechecks the server-owned recovery claim after each asynchronous boundary. */
|
||||
authorizeWarmTransitionRecovery?: (
|
||||
stage: "before_bootstrap" | "before_spawn" | "before_authentication",
|
||||
@@ -3562,6 +3564,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
#pump: NodeJS.Timeout | null = null;
|
||||
#eventSourceSeq = 0;
|
||||
#eventIdentity: DurableRecoveryIdentity | null = null;
|
||||
#sourceRecoveryIdentity: DurableRecoveryIdentity | null = null;
|
||||
#sourceAttachmentDeadline: number | null = null;
|
||||
#pendingWarmRecoveryCompletion: {
|
||||
transitionId: string;
|
||||
identity: DurableRecoveryIdentity;
|
||||
@@ -3790,6 +3794,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
): Promise<Record<string, unknown>> {
|
||||
if (this.#closed) throw new Error("PRP Codex transport is closed");
|
||||
this.#throwIfFailed();
|
||||
if (this.#sourceRecoveryIdentity && !["initialize", "collaborationMode/list"].includes(method)) {
|
||||
throw new Error("native_runner_source_attachment_pending");
|
||||
}
|
||||
if (
|
||||
this.#pendingWarmRecoveryCompletion !== null &&
|
||||
!["thread/read", "initialize", "collaborationMode/list"].includes(method)
|
||||
@@ -4099,7 +4106,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
}
|
||||
|
||||
#recoverPendingSemanticInputsWhenReady(): void {
|
||||
if (this.#startupComplete && this.#handlerInstalled) {
|
||||
if (this.#startupComplete && this.#handlerInstalled && this.#sourceRecoveryIdentity === null && this.#pendingWarmRecoveryCompletion === null) {
|
||||
void Promise.resolve().then(() => this.#core?.recoverPendingSemanticInputs())
|
||||
.catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error))));
|
||||
}
|
||||
@@ -4109,7 +4116,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
// Remote runner ingress already has a bounded reconnect budget. Reuse the
|
||||
// same budget here so a transient tunnel reconnect cannot trip the shorter
|
||||
// generic command timeout and replace an otherwise healthy warm runner.
|
||||
const reconnectGraceMs = this.options.runnerReconnectGraceMs ?? 5_000;
|
||||
const reconnectGraceMs = Math.min(this.options.runnerReconnectGraceMs ?? 5_000,
|
||||
this.#sourceAttachmentDeadline === null ? Infinity : Math.max(0, this.#sourceAttachmentDeadline - Date.now()));
|
||||
await waitForWarmAttachmentReadiness({
|
||||
graceMs: reconnectGraceMs,
|
||||
waitForConnection: (deadline) => this.#awaitWarmRunnerConnection(deadline),
|
||||
@@ -4239,7 +4247,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
.update(`${prior.runId}:${desired.runId}:${desired.turnId}`)
|
||||
.digest("hex")
|
||||
.slice(0, 32)}`;
|
||||
const runAttachTemplate = this.#runAttachTemplate
|
||||
const stagedCandidate = inspectStagedRunAttachment(core.store.state);
|
||||
const staged = stagedCandidate && recoveryIdentityMatches(stagedCandidate.targetIdentity, desired) ? stagedCandidate : undefined;
|
||||
const runAttachTemplate = staged ? structuredClone(staged.runAttachTemplate) : this.#runAttachTemplate
|
||||
? retargetRunAttachPayload(
|
||||
this.#runAttachTemplate,
|
||||
desired,
|
||||
@@ -4256,7 +4266,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
currentRunGrant?.runtimeContext,
|
||||
currentRunGrant ? { text: currentRunGrant.instructions, context: currentRunGrant.runtimeContext } : undefined,
|
||||
);
|
||||
if (this.options.prepareLaunch) {
|
||||
if (this.options.prepareLaunch && !staged) {
|
||||
const priorProvider = record(runAttachTemplate.provider);
|
||||
const priorRequirements = new RunnerLaunchPreparationStore(this.#root).load(prior)?.requirements;
|
||||
await this.#prepareOwnedProvider(runAttachTemplate, desired,
|
||||
@@ -4266,6 +4276,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
}
|
||||
this.#assertStartupOpen();
|
||||
this.#runAttachTemplate = structuredClone(runAttachTemplate);
|
||||
if (!staged && (this.options.provider ?? "codex") === "codex" && !this.options.prepareLaunch && this.options.onStagedRunAttachment) {
|
||||
// Registration already installed the target writer. Stage and queue
|
||||
// share the existing attachment checkpoint; no duplicate owner reads.
|
||||
core.stageRunAttachment(desired, runAttachTemplate);
|
||||
}
|
||||
const payload = {
|
||||
...runAttachTemplate,
|
||||
paperclipNextAuthority: { identity: desired, connection },
|
||||
@@ -4278,7 +4293,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
await this.#waitCommand(
|
||||
"run.attach",
|
||||
commandId,
|
||||
this.#coldAdmissionDeadline(),
|
||||
this.#sourceAttachmentDeadline === null ? this.#coldAdmissionDeadline()
|
||||
: Math.min(Date.now() + 30_000, this.#sourceAttachmentDeadline),
|
||||
true,
|
||||
);
|
||||
const attached = core.getCommand(commandId);
|
||||
@@ -4288,6 +4304,21 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
|
||||
activationStarted = true;
|
||||
core.rotateRunIdentity(desired, runAttachTemplate);
|
||||
if (core.store.state.stagedRunAttachment && registration?.connection?.mode === "listen" && this.options.readRunnerState) {
|
||||
const deadline = Math.min(Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity);
|
||||
const artifact = this.options.runnerArtifact ?? approvedRunnerArtifact(this.options.runnerBinary ?? defaultCapabilityRunnerdBinary());
|
||||
for (;;) {
|
||||
this.#assertStartupOpen();
|
||||
const physical = await this.options.readRunnerState();
|
||||
const proof = inspectWarmRunTransition({ controlPlaneState: core.store.state, runnerState: physical,
|
||||
expectedNewIdentity: desired, expectedRunnerVersion: artifact.version, expectedRunnerDigest: artifact.digest });
|
||||
if (proof && recoveryIdentityMatches(proof.runnerIdentity, desired)) {
|
||||
await previousRelease?.(); previousReleased = true; break;
|
||||
}
|
||||
if (Date.now() >= deadline || await this.#runnerHasExited()) throw new Error("native_runner_warm_transition_activation_pending");
|
||||
await new Promise<void>(done => setTimeout(done, 100));
|
||||
}
|
||||
}
|
||||
await registration?.activate?.();
|
||||
if (registration?.failure) {
|
||||
void registration.failure.catch((error: unknown) => {
|
||||
@@ -4297,13 +4328,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
});
|
||||
}
|
||||
await this.#awaitRegistrationReady(registration?.ready);
|
||||
const activationDeadline =
|
||||
Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000);
|
||||
const activationDeadline = Math.min(
|
||||
Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity);
|
||||
while (
|
||||
!recoveryIdentityMatches(core.store.state.identity, desired) ||
|
||||
core.store.state.warmTransition !== undefined ||
|
||||
core.activeRunnerConnectionCount() === 0
|
||||
) {
|
||||
this.#assertStartupOpen();
|
||||
if (
|
||||
Date.now() >= activationDeadline ||
|
||||
(await this.#runnerHasExited())
|
||||
@@ -4312,11 +4344,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
}
|
||||
await new Promise<void>((resolveWait) => setTimeout(resolveWait, 10));
|
||||
}
|
||||
this.#assertStartupOpen();
|
||||
this.#eventIdentity = structuredClone(desired);
|
||||
this.#eventSourceSeq = 0;
|
||||
this.#deferredTurnStartEvents = [];
|
||||
this.#durableTurnId = desired.turnId;
|
||||
await previousRelease?.();
|
||||
if (!previousReleased) await previousRelease?.();
|
||||
previousReleased = true;
|
||||
this.#controlPlaneRelease = registration?.release ?? null;
|
||||
} catch (error) {
|
||||
@@ -5244,7 +5277,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
onSemanticToolInput: (call) => this.#handleSemanticToolInput(call),
|
||||
recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call),
|
||||
onCommittedEvent: async (event) => {
|
||||
await this.options.onAccountingEvent?.(event);
|
||||
if (this.#sourceRecoveryIdentity && event.runId === this.#sourceRecoveryIdentity.runId) {
|
||||
if (!this.options.onRecoveryAccountingEvent && this.options.onAccountingEvent) throw new Error("native_runner_source_accounting_unavailable");
|
||||
await this.options.onRecoveryAccountingEvent?.(event, this.#sourceRecoveryIdentity);
|
||||
} else await this.options.onAccountingEvent?.(event);
|
||||
if (event.eventType === "auth.credentials.updated") {
|
||||
await this.options.onCredentialsUpdated?.(event);
|
||||
}
|
||||
@@ -5386,7 +5422,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
|
||||
async #resumeWork(): Promise<void> {
|
||||
if (this.options.acpxCandidateAdmission) throw new Error("ACPX candidate durable runner recovery is not admitted");
|
||||
const desiredIdentity = this.options.prpIdentity;
|
||||
let desiredIdentity = this.options.prpIdentity;
|
||||
if (desiredIdentity === undefined) {
|
||||
throw new Error("PRP provider resume state is unavailable");
|
||||
}
|
||||
@@ -5450,6 +5486,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
) {
|
||||
candidateRunnerState = await this.options.readRunnerState();
|
||||
}
|
||||
const staged = inspectStagedRunAttachment(controlPlaneState, desiredIdentity);
|
||||
if (controlPlaneState?.stagedRunAttachment !== undefined && staged === null) {
|
||||
throw new Error("native_runner_staged_attachment_binding_mismatch");
|
||||
}
|
||||
let warmRecovery: {
|
||||
runnerState: Record<string, unknown>;
|
||||
runnerIdentity: DurableRecoveryIdentity;
|
||||
@@ -5475,8 +5515,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
!controlPlaneState ||
|
||||
!candidateRunnerState ||
|
||||
(this.options.controlPlaneRegistration !== undefined &&
|
||||
this.options.warmTransitionRegistrationMode !== "routed_connect") ||
|
||||
this.options.adoptExistingRunner !== undefined
|
||||
this.options.warmTransitionRegistrationMode !== "routed_connect"
|
||||
&& !(staged && this.options.adoptExistingRunner && this.options.warmTransitionRegistrationMode === "routed_same_endpoint")) ||
|
||||
(this.options.adoptExistingRunner !== undefined && !staged)
|
||||
) {
|
||||
throw new Error(
|
||||
"native_runner_warm_transition_requires_exact_owned_endpoint",
|
||||
@@ -5501,14 +5542,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
? new URL(connection.connectUrl)
|
||||
: null;
|
||||
if (
|
||||
connection.mode !== "connect" ||
|
||||
endpoint === null ||
|
||||
endpoint.search ||
|
||||
endpoint.hash ||
|
||||
endpoint.username ||
|
||||
endpoint.password ||
|
||||
(connection.mode !== "connect" && !(connection.mode === "listen" && staged
|
||||
&& this.options.adoptExistingRunner && this.options.controlPlaneRegistration
|
||||
&& this.options.warmTransitionRegistrationMode === "routed_same_endpoint")) ||
|
||||
(connection.mode === "connect" && (endpoint === null || endpoint.search || endpoint.hash || endpoint.username || endpoint.password)) ||
|
||||
(!this.options.controlPlaneRegistration &&
|
||||
(endpoint.protocol !== "ws:" ||
|
||||
(endpoint === null || endpoint.protocol !== "ws:" ||
|
||||
endpoint.hostname !== "127.0.0.1" ||
|
||||
endpoint.pathname !== "/durableRecovery/connect" ||
|
||||
!endpoint.port))
|
||||
@@ -5522,7 +5561,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
transitionId: receipt.transitionId,
|
||||
commandId: receipt.commandId,
|
||||
...(!this.options.controlPlaneRegistration
|
||||
? { port: Number(endpoint.port) }
|
||||
? { port: Number(endpoint!.port) }
|
||||
: {}),
|
||||
};
|
||||
// Fence concurrent public work before exposing the core or awaiting
|
||||
@@ -5532,6 +5571,40 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
identity: structuredClone(desiredIdentity),
|
||||
};
|
||||
}
|
||||
const completedStaged = staged && recoveryIdentityMatches(identity, staged.targetIdentity)
|
||||
&& record(record(controlPlaneState?.completedWarmTransition).receipt).commandId === staged.commandId;
|
||||
const queuedSourceAttachment = staged && staged.commandId !== null && warmRecovery === null
|
||||
&& recoveryIdentityMatches(identity, staged.sourceIdentity) ? staged : null;
|
||||
if (queuedSourceAttachment && !this.options.adoptExistingRunner) {
|
||||
throw new Error("native_runner_queued_attachment_requires_verified_live_source");
|
||||
}
|
||||
let sourceAttachmentTarget: DurableRecoveryIdentity | null = null;
|
||||
if (controlPlaneState && warmRecovery === null && (this.options.provider ?? "codex") === "codex"
|
||||
&& !this.options.prepareLaunch && (staged || this.options.onStagedRunAttachment || this.options.onRecoveryAccountingEvent)
|
||||
&& identity.runId !== desiredIdentity.runId) {
|
||||
// Restore and acknowledge the source under its own authority. The target
|
||||
// remains a durable, undispatched intent until every old obligation settles.
|
||||
if (!this.options.adoptExistingRunner) {
|
||||
if (localStateOwner) {
|
||||
if (!candidateRunnerState) throw new Error("native_runner_authority_rotation_state_unavailable");
|
||||
assertSuspendedRunnerState(candidateRunnerState, identity);
|
||||
} else {
|
||||
if (!this.options.readRunnerState || !this.options.prepareExternalRunnerState) throw new Error("native_runner_prp_run_rotation_unavailable");
|
||||
await this.options.prepareExternalRunnerState();
|
||||
assertSuspendedRunnerState(await this.options.readRunnerState(), identity);
|
||||
}
|
||||
}
|
||||
sourceAttachmentTarget = queuedSourceAttachment ? null : desiredIdentity;
|
||||
desiredIdentity = identity;
|
||||
this.#sourceRecoveryIdentity = structuredClone(identity);
|
||||
this.#sourceAttachmentDeadline = Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000);
|
||||
} else if (staged && !completedStaged && !warmRecovery) {
|
||||
throw new Error("native_runner_staged_attachment_profile_unsupported");
|
||||
}
|
||||
if (warmRecovery && staged) {
|
||||
this.#sourceRecoveryIdentity = structuredClone(staged.sourceIdentity);
|
||||
this.#sourceAttachmentDeadline = Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000);
|
||||
}
|
||||
const exactAuthority =
|
||||
controlPlaneState !== null &&
|
||||
identity.runnerInstanceId === desiredIdentity.runnerInstanceId &&
|
||||
@@ -5697,7 +5770,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
onSemanticToolInput: (call) => this.#handleSemanticToolInput(call),
|
||||
recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call),
|
||||
onCommittedEvent: async (event) => {
|
||||
await this.options.onAccountingEvent?.(event);
|
||||
if (this.#sourceRecoveryIdentity && event.runId === this.#sourceRecoveryIdentity.runId) {
|
||||
if (!this.options.onRecoveryAccountingEvent && this.options.onAccountingEvent) throw new Error("native_runner_source_accounting_unavailable");
|
||||
await this.options.onRecoveryAccountingEvent?.(event, this.#sourceRecoveryIdentity);
|
||||
} else await this.options.onAccountingEvent?.(event);
|
||||
if (event.eventType === "auth.credentials.updated") {
|
||||
await this.options.onCredentialsUpdated?.(event);
|
||||
}
|
||||
@@ -5768,14 +5844,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
preparedContext, preparedInstructions, saved?.prepared, saved?.requirements);
|
||||
this.#runAttachTemplate = structuredClone(template);
|
||||
}
|
||||
if (rotatedAuthority) {
|
||||
if (rotatedAuthority || sourceAttachmentTarget) {
|
||||
if (provider === "acpx" && this.options.acpxAgent === "copilot"
|
||||
&& typeof this.options.baseInstructions !== "string") {
|
||||
throw new Error("native_runner_current_instruction_grant_unavailable");
|
||||
}
|
||||
const attachmentIdentity = sourceAttachmentTarget ?? desiredIdentity;
|
||||
const runAttachTemplate = rotatedRunAttachPayload(
|
||||
controlPlaneState,
|
||||
desiredIdentity,
|
||||
attachmentIdentity,
|
||||
this.#authorizedTools,
|
||||
this.options.resumeCompletionContract,
|
||||
runtimeContext,
|
||||
@@ -5816,8 +5893,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
runAttachTemplate.provider = environmentPrepared.provider;
|
||||
}
|
||||
this.#runAttachTemplate = structuredClone(runAttachTemplate);
|
||||
core.persistRunAttachTemplate(runAttachTemplate);
|
||||
core.queueCommand("run.attach", runAttachTemplate);
|
||||
if (sourceAttachmentTarget) {
|
||||
await this.options.onStagedRunAttachment?.(core, { sourceIdentity: desiredIdentity,
|
||||
targetIdentity: sourceAttachmentTarget, templateDigest: commandDigest(runAttachTemplate) });
|
||||
core.stageRunAttachment(sourceAttachmentTarget, runAttachTemplate);
|
||||
} else {
|
||||
core.persistRunAttachTemplate(runAttachTemplate);
|
||||
core.queueCommand("run.attach", runAttachTemplate);
|
||||
}
|
||||
}
|
||||
const committedEvents = core.store.state.committedEvents;
|
||||
const runAttachment = recoveredRunAttachment(core.store.state);
|
||||
@@ -5829,7 +5912,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
? `command_resume_probe_${randomUUID().replaceAll("-", "")}`
|
||||
: null;
|
||||
if (recoveryProbeCommandId !== null) {
|
||||
core.queueCommand("runner.drain", {}, recoveryProbeCommandId);
|
||||
core.queueCommand(sourceAttachmentTarget ? "session.open" : "runner.drain", {}, recoveryProbeCommandId);
|
||||
}
|
||||
// A controller retry can open the exact authority after run.attach has
|
||||
// already reached a durable outcome. Re-observe that command instead of
|
||||
@@ -5878,6 +5961,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
: "restored provider identity from the exact durable checkpoint; awaiting live confirmation",
|
||||
);
|
||||
}
|
||||
if (staged && !sourceAttachmentTarget) {
|
||||
await this.options.onStagedRunAttachment?.(core, { sourceIdentity: staged.sourceIdentity,
|
||||
targetIdentity: staged.targetIdentity, templateDigest: staged.templateDigest });
|
||||
if (warmRecovery && this.options.adoptExistingRunner) core.prepareWarmTransitionAdoption({
|
||||
transitionId: warmRecovery.transitionId, runnerState: warmRecovery.runnerState });
|
||||
}
|
||||
type RecoveryRegistration = Awaited<
|
||||
ReturnType<
|
||||
NonNullable<
|
||||
@@ -5888,67 +5977,48 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
let oldTransitionRegistration: RecoveryRegistration | null = null;
|
||||
let newTransitionRegistration: RecoveryRegistration | null = null;
|
||||
this.#assertStartupOpen();
|
||||
if (warmRecovery && this.options.controlPlaneRegistration) {
|
||||
const recoveringAttachment = warmRecovery || queuedSourceAttachment;
|
||||
const queuedCommand = queuedSourceAttachment ? core.getCommand(queuedSourceAttachment.commandId!) : undefined;
|
||||
const expectedRecoveryConnection = (warmRecovery?.proof.receipt.connection
|
||||
?? record(queuedCommand?.payload.paperclipNextAuthority).connection) as RunnerProcessConnection | undefined;
|
||||
const recoverySource = warmRecovery?.proof.receipt.oldIdentity ?? queuedSourceAttachment?.sourceIdentity;
|
||||
const recoveryTarget = warmRecovery?.proof.receipt.newIdentity ?? queuedSourceAttachment?.targetIdentity;
|
||||
if (recoveringAttachment && this.options.controlPlaneRegistration) {
|
||||
try {
|
||||
oldTransitionRegistration = await this.options.controlPlaneRegistration(
|
||||
core,
|
||||
warmRecovery.proof.receipt.oldIdentity,
|
||||
);
|
||||
const oldConnection = oldTransitionRegistration.connection ?? {
|
||||
mode: "connect",
|
||||
connectUrl: oldTransitionRegistration.connectUrl,
|
||||
};
|
||||
if (
|
||||
oldConnection.mode !== "connect" ||
|
||||
typeof oldConnection.connectUrl !== "string"
|
||||
) {
|
||||
throw new Error(
|
||||
"native_runner_warm_transition_registered_endpoint_mismatch",
|
||||
);
|
||||
if (!recoverySource || !recoveryTarget || !expectedRecoveryConnection
|
||||
|| (expectedRecoveryConnection.mode !== "connect" && !(staged && this.options.adoptExistingRunner
|
||||
&& this.options.warmTransitionRegistrationMode === "routed_same_endpoint"))) {
|
||||
throw new Error("native_runner_warm_transition_requires_exact_owned_endpoint");
|
||||
}
|
||||
oldTransitionRegistration = await this.options.controlPlaneRegistration(core, recoverySource);
|
||||
this.#assertStartupOpen();
|
||||
newTransitionRegistration = await this.options.controlPlaneRegistration(
|
||||
core,
|
||||
warmRecovery.proof.receipt.newIdentity,
|
||||
);
|
||||
const newConnection = newTransitionRegistration.connection ?? {
|
||||
mode: "connect",
|
||||
connectUrl: newTransitionRegistration.connectUrl,
|
||||
};
|
||||
if (
|
||||
newConnection.mode !== "connect" ||
|
||||
durableRecoveryInternals.canonicalJson(newConnection) !==
|
||||
durableRecoveryInternals.canonicalJson(
|
||||
warmRecovery.proof.receipt.connection,
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
"native_runner_warm_transition_registered_endpoint_mismatch",
|
||||
);
|
||||
newTransitionRegistration = await this.options.controlPlaneRegistration(core, recoveryTarget);
|
||||
const connection = newTransitionRegistration.connection ?? { mode: "connect", connectUrl: newTransitionRegistration.connectUrl };
|
||||
if (durableRecoveryInternals.canonicalJson(connection) !== durableRecoveryInternals.canonicalJson(expectedRecoveryConnection)) {
|
||||
throw new Error("native_runner_warm_transition_registered_endpoint_mismatch");
|
||||
}
|
||||
} catch (error) {
|
||||
await Promise.allSettled(
|
||||
[oldTransitionRegistration, newTransitionRegistration].map((entry) =>
|
||||
Promise.resolve().then(() => entry?.release()),
|
||||
),
|
||||
);
|
||||
await Promise.allSettled([oldTransitionRegistration, newTransitionRegistration].map(entry => Promise.resolve().then(() => entry?.release())));
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
const registration =
|
||||
warmRecovery && oldTransitionRegistration && newTransitionRegistration
|
||||
? recoveryIdentityMatches(
|
||||
warmRecovery.runnerIdentity,
|
||||
warmRecovery.proof.receipt.oldIdentity,
|
||||
)
|
||||
? oldTransitionRegistration
|
||||
: newTransitionRegistration
|
||||
: this.options.controlPlaneRegistration
|
||||
? await this.options.controlPlaneRegistration(core)
|
||||
: null;
|
||||
const registration = oldTransitionRegistration && newTransitionRegistration
|
||||
? recoveryIdentityMatches(warmRecovery?.runnerIdentity ?? queuedSourceAttachment!.sourceIdentity, recoverySource!)
|
||||
? oldTransitionRegistration : newTransitionRegistration
|
||||
: this.options.controlPlaneRegistration
|
||||
? await this.options.controlPlaneRegistration(core, this.#sourceRecoveryIdentity ?? undefined) : null;
|
||||
this.#startupFailureCode =
|
||||
registration?.startupFailureCode ?? "runner_local_connect_failed";
|
||||
if (registration === null) await core.start(warmRecovery?.port);
|
||||
if (registration === null) {
|
||||
let queuedPort: number | undefined;
|
||||
if (queuedSourceAttachment) {
|
||||
const endpoint = expectedRecoveryConnection?.mode === "connect" ? new URL(expectedRecoveryConnection.connectUrl) : null;
|
||||
if (!endpoint || endpoint.protocol !== "ws:" || endpoint.hostname !== "127.0.0.1" || endpoint.pathname !== "/durableRecovery/connect"
|
||||
|| !endpoint.port || endpoint.search || endpoint.hash || endpoint.username || endpoint.password) throw new Error("native_runner_warm_transition_snapshot_mismatch");
|
||||
queuedPort = Number(endpoint.port);
|
||||
}
|
||||
await core.start(warmRecovery?.port ?? queuedPort);
|
||||
}
|
||||
else {
|
||||
this.#controlPlaneCheckpoint = registration.checkpoint ?? null;
|
||||
this.#controlPlaneRelease =
|
||||
@@ -5969,7 +6039,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
this.#assertStartupOpen();
|
||||
// Route and immutable endpoint admission precedes every launch-material
|
||||
// write. A refused transition leaves its retained home/context untouched.
|
||||
if (runtimeContext !== null && !this.options.prepareLaunch) {
|
||||
if (runtimeContext !== null && !this.options.prepareLaunch && this.#sourceRecoveryIdentity === null) {
|
||||
writeFileSync(
|
||||
localRuntimeContextPath,
|
||||
`${JSON.stringify(runtimeContext)}\n`,
|
||||
@@ -5978,7 +6048,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
}
|
||||
if (provider === "aws_agentcore" && !this.options.prepareLaunch)
|
||||
mkdirSync(codexHome, { recursive: true, mode: 0o700 });
|
||||
if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured") {
|
||||
if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured" && this.#sourceRecoveryIdentity === null) {
|
||||
await releaseMaterializedNativeRuntimeSkills(
|
||||
resolve(localCodexHome, "skills"),
|
||||
);
|
||||
@@ -6018,7 +6088,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
}
|
||||
// A replacement executor restores its cold provider before acknowledging recovery.
|
||||
this.#assertStartupOpen();
|
||||
const admissionDeadline = this.#coldAdmissionDeadline(adoptedRunner === undefined);
|
||||
const admissionDeadline = this.#sourceAttachmentDeadline === null
|
||||
? this.#coldAdmissionDeadline(adoptedRunner === undefined)
|
||||
: Math.min(Date.now() + 30_000, this.#sourceAttachmentDeadline);
|
||||
const handle = adoptedRunner
|
||||
? null
|
||||
: spawnRunner({
|
||||
@@ -6064,18 +6136,30 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
await this.#publishSpawnedProcess(handle);
|
||||
}
|
||||
this.#assertStartupOpen();
|
||||
let targetRecoveryActivated = registration === newTransitionRegistration;
|
||||
if (oldTransitionRegistration && newTransitionRegistration) {
|
||||
await oldTransitionRegistration.activate?.();
|
||||
// A guest listener still serves only its actual physical run. Open that
|
||||
// route first; probing the future/retired path cannot prove readiness.
|
||||
await registration?.activate?.();
|
||||
this.#assertStartupOpen();
|
||||
await newTransitionRegistration.activate?.();
|
||||
if (registration === oldTransitionRegistration) {
|
||||
// Source authentication can consume its receipt and disappear before
|
||||
// a sampled count sees it. The durable/physical proof loop below owns
|
||||
// that progress; only target authentication completes recovery.
|
||||
if (expectedRecoveryConnection?.mode !== "listen") {
|
||||
await newTransitionRegistration.activate?.(); targetRecoveryActivated = true;
|
||||
}
|
||||
}
|
||||
} else await registration?.activate?.();
|
||||
this.#assertStartupOpen();
|
||||
if (registration?.failure) {
|
||||
void registration.failure.catch((error: unknown) => {
|
||||
this.#failTransport(
|
||||
error instanceof Error ? error : new Error(String(error)),
|
||||
);
|
||||
});
|
||||
if (registration?.failure && !recoveringAttachment) {
|
||||
void registration.failure.catch((error: unknown) => this.#failTransport(error instanceof Error ? error : new Error(String(error))));
|
||||
} else if (recoveringAttachment) {
|
||||
// The old listener can retire normally during recovery. Authentication
|
||||
// and the shared deadline, rather than its stale readiness promise, own
|
||||
// the transition. Only the target remains live after activation.
|
||||
void oldTransitionRegistration?.failure?.catch(() => undefined);
|
||||
void newTransitionRegistration?.failure?.catch(() => undefined);
|
||||
}
|
||||
if (!adoptedRunner) await this.#awaitRegistrationReady(registration?.ready);
|
||||
this.#assertStartupOpen();
|
||||
@@ -6088,22 +6172,40 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
this.#evidence.runnerProcessGroupId = handle?.processGroupId ?? null;
|
||||
}
|
||||
this.#publish();
|
||||
if (warmRecovery) {
|
||||
const deadline =
|
||||
Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000);
|
||||
if (recoveringAttachment) {
|
||||
const deadline = Math.min(Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000), this.#sourceAttachmentDeadline ?? Infinity);
|
||||
let sourceRouteRetired = false, nextPhysicalInspection = 0, previousConnections = core.activeRunnerConnectionCount();
|
||||
while (
|
||||
!recoveryIdentityMatches(core.store.state.identity, desiredIdentity) ||
|
||||
core.store.state.warmTransition !== undefined
|
||||
!recoveryIdentityMatches(core.store.state.identity, recoveryTarget!) ||
|
||||
core.store.state.warmTransition !== undefined || core.activeRunnerConnectionCount() !== 1
|
||||
) {
|
||||
this.#assertStartupOpen();
|
||||
const connections = core.activeRunnerConnectionCount();
|
||||
if (connections === 0 && previousConnections > 0) nextPhysicalInspection = 0;
|
||||
previousConnections = connections;
|
||||
if (!sourceRouteRetired && expectedRecoveryConnection?.mode === "listen" && oldTransitionRegistration
|
||||
&& newTransitionRegistration && this.options.readRunnerState && Date.now() >= nextPhysicalInspection) {
|
||||
nextPhysicalInspection = Date.now() + 250;
|
||||
const physical = await this.options.readRunnerState();
|
||||
const proof = inspectWarmRunTransition({ controlPlaneState: core.store.state, runnerState: physical,
|
||||
expectedNewIdentity: recoveryTarget!, expectedRunnerVersion: runnerArtifact.version, expectedRunnerDigest: runnerArtifact.digest });
|
||||
if (proof && recoveryIdentityMatches(proof.runnerIdentity, recoveryTarget!)) {
|
||||
// The same participant has consumed the old ACK and now serves
|
||||
// the target path. Stop obsolete ingress retries before they can
|
||||
// compete with the target's authenticated connection.
|
||||
await oldTransitionRegistration.release(); sourceRouteRetired = true;
|
||||
if (!targetRecoveryActivated) { await newTransitionRegistration.activate?.(); targetRecoveryActivated = true; }
|
||||
}
|
||||
}
|
||||
if (Date.now() >= deadline || (await this.#runnerHasExited())) {
|
||||
throw new Error("native_runner_warm_transition_recovery_pending");
|
||||
}
|
||||
await new Promise<void>((resolveWait) => setTimeout(resolveWait, 10));
|
||||
}
|
||||
if (core.getCommand(warmRecovery.commandId)?.status !== "completed") {
|
||||
if (core.getCommand(warmRecovery?.commandId ?? queuedSourceAttachment!.commandId!)?.status !== "completed") {
|
||||
throw new Error("native_runner_warm_transition_result_unproven");
|
||||
}
|
||||
desiredIdentity = recoveryTarget!;
|
||||
this.#durableTurnId = desiredIdentity.turnId;
|
||||
this.#eventIdentity = structuredClone(desiredIdentity);
|
||||
this.#eventSourceSeq = 0;
|
||||
@@ -6112,8 +6214,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
await oldTransitionRegistration.release();
|
||||
this.#assertStartupOpen();
|
||||
this.#controlPlaneRelease = newTransitionRegistration.release;
|
||||
this.#controlPlaneCheckpoint =
|
||||
newTransitionRegistration.checkpoint ?? null;
|
||||
this.#controlPlaneCheckpoint = newTransitionRegistration.checkpoint ?? null;
|
||||
void newTransitionRegistration.failure?.catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error))));
|
||||
}
|
||||
}
|
||||
this.#assertStartupOpen();
|
||||
@@ -6121,7 +6223,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
if (adoptedRunner) {
|
||||
await this.#awaitAdoptedRunnerConnection(
|
||||
adoptedRunner,
|
||||
registration?.ready,
|
||||
newTransitionRegistration?.ready ?? registration?.ready,
|
||||
);
|
||||
}
|
||||
// Only a replacement executor can be restoring a cold Pi process. Share
|
||||
@@ -6130,7 +6232,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
await this.#waitCommand("run.attach", runAttachment.commandId, admissionDeadline, true);
|
||||
}
|
||||
if (recoveryProbeCommandId !== null) {
|
||||
await this.#waitCommand("runner.drain", recoveryProbeCommandId, admissionDeadline, true);
|
||||
await this.#waitCommand(sourceAttachmentTarget ? "session.open" : "runner.drain", recoveryProbeCommandId, admissionDeadline, true);
|
||||
if (this.#checkpointProviderIdentityExpectation !== null) {
|
||||
// A replacement runner can restore the exact provider while its fresh
|
||||
// session.resumed event is compacted or delayed behind the completed
|
||||
@@ -6159,8 +6261,20 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
admissionDeadline,
|
||||
);
|
||||
this.#startupComplete = true;
|
||||
if (recoveringAttachment && staged) {
|
||||
this.#sourceRecoveryIdentity = null;
|
||||
this.#sourceAttachmentDeadline = null;
|
||||
}
|
||||
if (sourceAttachmentTarget) {
|
||||
await this.#awaitWarmRunAttachmentReady();
|
||||
if (!core.semanticToolResultsSettled()) throw new Error("native_runner_source_semantic_reconciliation_required");
|
||||
if (runtimeContext !== null) writeFileSync(localRuntimeContextPath, `${JSON.stringify(runtimeContext)}\n`, { mode: 0o600 });
|
||||
await this.#attachRunWork(sourceAttachmentTarget);
|
||||
this.#sourceRecoveryIdentity = null;
|
||||
this.#sourceAttachmentDeadline = null;
|
||||
}
|
||||
this.#diagnostic(
|
||||
rotatedAuthority
|
||||
rotatedAuthority || sourceAttachmentTarget
|
||||
? "runnerd attached the durable provider session to a fresh PRP run authority"
|
||||
: "runnerd restored its durable PRP session and provider thread",
|
||||
);
|
||||
@@ -6169,6 +6283,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
}
|
||||
|
||||
async #recoverSemanticToolInput(call: Parameters<NonNullable<DurablePrpControlPlaneOptions["onSemanticToolInput"]>>[0]) {
|
||||
if (this.#sourceRecoveryIdentity) throw new Error("native_runner_source_semantic_reconciliation_required");
|
||||
if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") {
|
||||
return this.#handleSemanticToolInput(call);
|
||||
}
|
||||
@@ -6181,6 +6296,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
NonNullable<DurablePrpControlPlaneOptions["onSemanticToolInput"]>
|
||||
>[0],
|
||||
) {
|
||||
if (this.#sourceRecoveryIdentity) throw new Error("native_runner_source_semantic_reconciliation_required");
|
||||
try {
|
||||
this.#throwIfFailed();
|
||||
} catch {
|
||||
@@ -6522,6 +6638,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
await completing;
|
||||
if (this.#pendingWarmRecoveryCompletion === completion) {
|
||||
this.#pendingWarmRecoveryCompletion = null;
|
||||
this.#recoverPendingSemanticInputsWhenReady();
|
||||
}
|
||||
} finally {
|
||||
if (this.#warmRecoveryCompletionInFlight === completing) {
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { createServer, request } from "node:http";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import {
|
||||
DurablePrpControlPlane, inspectStagedRunAttachmentHeader,
|
||||
type DurablePrpCheckpoint, type RunnerProcessConnection, type PrpWireConnection,
|
||||
} from "../control-plane/durable-prp-control-plane.js";
|
||||
import {
|
||||
createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary,
|
||||
readRunnerdArtifactBinding,
|
||||
} from "./runnerd-codex-transport.js";
|
||||
|
||||
// The guest rejects WebSocket extensions. Use the same uncompressed wire
|
||||
// contract as production, without importing the server connector into Runner.
|
||||
async function openOwnedWire(url: string): Promise<{ wire: PrpWireConnection; closed: Promise<void>; close(): void }> {
|
||||
return new Promise((resolvePeer, reject) => {
|
||||
const key = randomBytes(16).toString("base64");
|
||||
const req = request(url.replace("ws:", "http:"), { headers: {
|
||||
Connection: "Upgrade", Upgrade: "websocket", "Sec-WebSocket-Version": "13", "Sec-WebSocket-Key": key,
|
||||
} });
|
||||
req.on("error", reject); req.on("response", response => { response.resume(); reject(new Error(`Guest upgrade refused: ${response.statusCode}`)); });
|
||||
req.setTimeout(1_000, () => req.destroy(new Error("Guest upgrade timeout")));
|
||||
req.on("upgrade", (response, socket, head) => {
|
||||
socket.setTimeout(0);
|
||||
const expected = createHash("sha1").update(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").digest("base64");
|
||||
if (response.headers["sec-websocket-accept"] !== expected) { socket.destroy(); reject(new Error("Invalid guest upgrade")); return; }
|
||||
const listeners: Array<(value: unknown) => void> = [], closes: Array<() => void> = [];
|
||||
let buffer: Buffer = head, ended = false;
|
||||
let closeResolve!: () => void;
|
||||
const closed = new Promise<void>(done => { closeResolve = done; });
|
||||
const finish = () => { if (ended) return; ended = true; for (const close of closes) close(); closeResolve(); };
|
||||
const send = (payload: Buffer, opcode: number) => {
|
||||
const mask = randomBytes(4), prefix = payload.length < 126 ? Buffer.from([0x80 | opcode, 0x80 | payload.length]) : Buffer.alloc(payload.length <= 65535 ? 4 : 10);
|
||||
if (payload.length >= 126) { prefix[0] = 0x80 | opcode; prefix[1] = 0x80 | (payload.length <= 65535 ? 126 : 127);
|
||||
if (payload.length <= 65535) prefix.writeUInt16BE(payload.length, 2); else prefix.writeBigUInt64BE(BigInt(payload.length), 2); }
|
||||
const masked = Buffer.from(payload); for (let index = 0; index < masked.length; index++) masked[index] ^= mask[index % 4]!;
|
||||
socket.write(Buffer.concat([prefix, mask, masked]));
|
||||
};
|
||||
const parse = () => {
|
||||
while (buffer.length >= 2) {
|
||||
const opcode = buffer[0]! & 15, initial = buffer[1]! & 127;
|
||||
if (buffer.length < (initial === 126 ? 4 : initial === 127 ? 10 : 2)) return;
|
||||
const size = initial === 126 ? buffer.readUInt16BE(2) : initial === 127 ? Number(buffer.readBigUInt64BE(2)) : initial;
|
||||
const offset = initial === 126 ? 4 : initial === 127 ? 10 : 2;
|
||||
if (buffer.length < offset + size) return;
|
||||
const payload = buffer.subarray(offset, offset + size); buffer = buffer.subarray(offset + size);
|
||||
if (opcode === 8) { socket.destroy(); finish(); return; }
|
||||
if (opcode === 9) send(payload, 10);
|
||||
if (opcode === 1 || opcode === 2) for (const listener of listeners) listener(JSON.parse(payload.toString("utf8")));
|
||||
}
|
||||
};
|
||||
socket.on("data", bytes => { buffer = Buffer.concat([buffer, bytes]); parse(); }); socket.on("close", finish); socket.on("error", finish);
|
||||
const close = () => { socket.destroy(); finish(); };
|
||||
resolvePeer({ wire: { sendJson: value => send(Buffer.from(JSON.stringify(value)), 1), close,
|
||||
onJson: listener => { listeners.push(listener); }, onClose: listener => { closes.push(() => listener({})); } }, closed, close });
|
||||
queueMicrotask(parse);
|
||||
}); req.end();
|
||||
});
|
||||
}
|
||||
|
||||
// Real encrypted PRP and real Runner/provider processes; no model or server imports.
|
||||
it.each(["connect", "listen"] as const)("recovers a staged %s handoff at every durable boundary without another process", async mode => {
|
||||
const root = await mkdtemp(join(tmpdir(), "runner-staged-handoff-"));
|
||||
const shared = join(root, "shared"), calls = join(root, "calls.log");
|
||||
const source = { runnerInstanceId: "runner-staged", environmentLeaseId: "lease-staged", runId: "source-run",
|
||||
normalizedSessionId: "session-staged", turnId: "source-turn", itemId: "source-item" };
|
||||
const target = { ...source, runId: "target-run", turnId: "target-turn", itemId: "target-item" };
|
||||
const http = createServer();
|
||||
const routes = new Map<string, { core: DurablePrpControlPlane; token: symbol }>();
|
||||
http.on("upgrade", (request, socket, head) => {
|
||||
const route = routes.get(request.url ?? "");
|
||||
if (route) route.core.handleUpgrade(request, socket, request.url!, head); else socket.destroy();
|
||||
});
|
||||
await new Promise<void>(done => http.listen(0, "127.0.0.1", done));
|
||||
const address = http.address();
|
||||
if (!address || typeof address === "string") throw new Error("Missing owned endpoint");
|
||||
const guest = createServer();
|
||||
await new Promise<void>(done => guest.listen(0, "127.0.0.1", done));
|
||||
const guestAddress = guest.address();
|
||||
if (!guestAddress || typeof guestAddress === "string") throw new Error("Missing owned guest endpoint");
|
||||
const guestPort = guestAddress.port;
|
||||
await new Promise<void>(done => guest.close(() => done()));
|
||||
const within = async <T>(promise: Promise<T>, label: string, ms = 8_000): Promise<T> => {
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
try { return await Promise.race([promise, new Promise<never>((_, reject) => { timer = setTimeout(() => reject(new Error(`staged fixture timeout: ${label}`)), ms); })]); }
|
||||
finally { clearTimeout(timer); }
|
||||
};
|
||||
const dead = (pid: number) => { try { process.kill(pid, 0); return false; } catch (error) { return (error as NodeJS.ErrnoException).code === "ESRCH"; } };
|
||||
// Each boundary uses a fresh company-free fixture so no previous command or
|
||||
// activation can satisfy a later assertion.
|
||||
try {
|
||||
for (const boundary of ["queued", "prepared", "activated", "completed"] as const) {
|
||||
const directory = join(shared, boundary), callLog = `${calls}-${boundary}`;
|
||||
let checkpoint!: DurablePrpCheckpoint, currentCore!: DurablePrpControlPlane;
|
||||
let lost = false, release!: () => void, reached!: () => void;
|
||||
const held = new Promise<void>(done => { release = done; });
|
||||
const atBoundary = new Promise<void>(done => { reached = done; });
|
||||
const cores = new Set<DurablePrpControlPlane>();
|
||||
const registered = new WeakSet<DurablePrpControlPlane>();
|
||||
let generation = 0;
|
||||
const registration = async (core: DurablePrpControlPlane, identity = core.store.state.identity) => {
|
||||
currentCore = core;
|
||||
if (!registered.has(core)) {
|
||||
registered.add(core); cores.add(core);
|
||||
const epoch = generation;
|
||||
core.setCheckpointPersistence(async saved => {
|
||||
if (epoch !== generation) throw new Error("retired fixture controller");
|
||||
checkpoint = saved;
|
||||
const proof = inspectStagedRunAttachmentHeader(saved.records.get("header")!);
|
||||
if (!lost && proof?.phase === (boundary === "queued" ? "attachment_queued" : boundary)) {
|
||||
lost = true; reached(); await held;
|
||||
if (epoch !== generation) throw new Error("retired fixture controller");
|
||||
}
|
||||
});
|
||||
}
|
||||
const path = `/api/runner/v1/connect/${identity.runId}`, token = Symbol(path);
|
||||
const connection: RunnerProcessConnection = mode === "connect"
|
||||
? { mode: "connect", connectUrl: `ws://127.0.0.1:${address.port}${path}` }
|
||||
: { mode: "listen", listenAddress: "0.0.0.0", listenPort: guestPort, listenPath: path };
|
||||
if (mode === "connect") {
|
||||
routes.set(path, { core, token });
|
||||
return { connection, release: () => { if (routes.get(path)?.token === token) routes.delete(path); } };
|
||||
}
|
||||
let stopped = false, socket: { close(): void } | undefined;
|
||||
let readyResolve!: () => void, readyReject!: (error: Error) => void;
|
||||
const ready = new Promise<void>((done, reject) => { readyResolve = done; readyReject = reject; });
|
||||
void ready.catch(() => undefined);
|
||||
const connect = async () => {
|
||||
const deadline = Date.now() + 8_000;
|
||||
while (!stopped && Date.now() < deadline) {
|
||||
try {
|
||||
const peer = await openOwnedWire(`ws://127.0.0.1:${guestPort}${path}`); socket = peer;
|
||||
if (stopped) { peer.close(); return; }
|
||||
core.attachWireConnection(peer.wire); readyResolve(); await peer.closed;
|
||||
} catch { /* The target listener appears only after the old receipt ACK. */ }
|
||||
// Match the production ingress connector's first retry delay (250ms).
|
||||
if (!stopped) await new Promise<void>(done => setTimeout(done, 250));
|
||||
}
|
||||
if (!stopped) readyReject(new Error("Owned listener did not become available"));
|
||||
};
|
||||
return { connection, activate: () => { void connect(); }, ready: () => ready,
|
||||
release: () => { stopped = true; socket?.close(); } };
|
||||
};
|
||||
const options = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
|
||||
codexArgs: ["--call-log", callLog, "--record-process-start", "--state-file", join(directory, "fake.json")],
|
||||
stateDirectory: directory, readRunnerState: async () => JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8")), lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 },
|
||||
runnerReconnectGraceMs: 5_000, prpIdentity: source, controlPlaneRegistration: registration,
|
||||
onStagedRunAttachment: async () => undefined, onRecoveryAccountingEvent: async () => undefined };
|
||||
const first = createCapabilityRunnerdCodexTransport(options);
|
||||
let replacement: ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
|
||||
let runnerPid: number | null = null, providerPid: number | null = null;
|
||||
try {
|
||||
const opened = await within(first.transport.request("thread/start", { cwd: root, dynamicTools: [] }), "source open");
|
||||
runnerPid = first.evidence().runnerPid; providerPid = first.evidence().providerPid;
|
||||
const startedAt = first.transport.processInfo().startedAt;
|
||||
const providerStart = execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim();
|
||||
expect(runnerPid).toBeGreaterThan(0); expect(providerPid).toBeGreaterThan(0);
|
||||
const attach = first.transport.attachRun!({ runId: target.runId, turnId: target.turnId, itemId: target.itemId });
|
||||
void attach.catch(() => undefined);
|
||||
await within(atBoundary, `reach ${boundary}`);
|
||||
expect(inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!)?.attachment.targetIdentity).toEqual(target);
|
||||
generation += 1;
|
||||
// Disconnect the controller only. The bounded failed commit is released
|
||||
// after its route is retired; a stale callback cannot publish generation B.
|
||||
const detached = first.detachControllerForRestart(); void detached.catch(() => undefined);
|
||||
release(); await within(detached, "detach").catch(() => undefined);
|
||||
const next = join(root, `replacement-${boundary}`);
|
||||
checkpoint.restoreSessionDirectory(next);
|
||||
const noSpawn = vi.fn(() => { throw new Error("Duplicate Runner launch"); });
|
||||
const recoveryOptions: Parameters<typeof createCapabilityRunnerdCodexTransport>[0] = { ...options, stateDirectory: next,
|
||||
prpIdentity: target, runnerArtifact: readRunnerdArtifactBinding(options.runnerBinary),
|
||||
runnerStateDirectory: join(directory, "runner"), readRunnerState: async () => JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8")),
|
||||
prepareExternalRunnerState: async () => { throw new Error("Verified live handoff must not restore a backup"); },
|
||||
runnerProcessLauncher: noSpawn, warmTransitionRegistrationMode: mode === "listen" ? "routed_same_endpoint" : "routed_connect",
|
||||
resumeProviderSession: { driverSessionId: String((opened.thread as { id: string }).id), providerSessionId: String((opened.thread as { sessionId: string }).sessionId) },
|
||||
adoptExistingRunner: { pid: runnerPid, processGroupId: runnerPid, startedAt,
|
||||
isAlive: () => !dead(runnerPid!), signal: signal => { if (dead(runnerPid!)) return true; process.kill(-runnerPid!, signal); return true; } } };
|
||||
if (boundary === "queued") {
|
||||
const noProof = createCapabilityRunnerdCodexTransport({ ...recoveryOptions, adoptExistingRunner: undefined });
|
||||
await expect(noProof.transport.request("thread/read", {})).rejects.toThrow("queued_attachment_requires_verified_live_source");
|
||||
await noProof.detachControllerForRestart().catch(() => undefined);
|
||||
checkpoint.restoreSessionDirectory(next);
|
||||
const wrongEndpoint = createCapabilityRunnerdCodexTransport({ ...recoveryOptions,
|
||||
controlPlaneRegistration: async (core, identity) => {
|
||||
const registered = await registration(core, identity);
|
||||
if (identity?.runId !== target.runId) return registered;
|
||||
return { ...registered, connection: mode === "listen"
|
||||
? { mode: "listen", listenAddress: "0.0.0.0", listenPort: guestPort + 1, listenPath: `/api/runner/v1/connect/${target.runId}` }
|
||||
: { mode: "connect", connectUrl: `ws://127.0.0.1:${address.port}/changed` } };
|
||||
} });
|
||||
await expect(wrongEndpoint.transport.request("thread/read", {})).rejects.toThrow("registered_endpoint_mismatch");
|
||||
await wrongEndpoint.detachControllerForRestart().catch(() => undefined);
|
||||
expect(noSpawn).not.toHaveBeenCalled();
|
||||
checkpoint.restoreSessionDirectory(next);
|
||||
}
|
||||
replacement = createCapabilityRunnerdCodexTransport(recoveryOptions);
|
||||
const read = await within(replacement.transport.request("thread/read", {}), `recover ${boundary}`);
|
||||
expect(read.thread).toMatchObject({ id: (opened.thread as { id: string }).id,
|
||||
sessionId: (opened.thread as { sessionId: string }).sessionId });
|
||||
expect(dead(providerPid!)).toBe(false);
|
||||
expect(execFileSync("ps", ["-p", String(providerPid), "-o", "lstart="], { encoding: "utf8" }).trim()).toBe(providerStart);
|
||||
expect(replacement.evidence().runnerPid).toBe(runnerPid);
|
||||
expect(currentCore.activeRunnerConnectionCount()).toBe(1);
|
||||
expect(currentCore.store.state.identity).toEqual(target);
|
||||
expect(currentCore.store.state.stagedRunAttachment).toBeDefined();
|
||||
expect(currentCore.markStagedTargetCheckpointPublished({ ...target, itemId: "wrong" })).toBe(false);
|
||||
await currentCore.flushCheckpoint();
|
||||
const beforeMark = checkpoint;
|
||||
expect(currentCore.markStagedTargetCheckpointPublished(target)).toBe(true);
|
||||
await currentCore.flushCheckpoint(); expect(checkpoint).toBe(beforeMark);
|
||||
await within(replacement.transport.request("thread/goal/get", {}), "post-checkpoint read");
|
||||
expect(currentCore.store.state.stagedRunAttachment).toBeUndefined();
|
||||
expect(noSpawn).not.toHaveBeenCalled();
|
||||
const effects = (await readFile(callLog, "utf8")).trim().split(/\r?\n/);
|
||||
expect(effects.filter(line => line === "process-start")).toHaveLength(1);
|
||||
expect(effects.filter(line => line === "thread/start")).toHaveLength(1);
|
||||
expect(effects.filter(line => line === "turn/start")).toHaveLength(0);
|
||||
} catch (error) {
|
||||
const physical = JSON.parse(await readFile(join(directory, "runner/runner-state.json"), "utf8").catch(() => "{}"));
|
||||
throw new Error(`${mode}/${boundary}: ${String(error)}; ${JSON.stringify({
|
||||
active: currentCore?.activeRunnerConnectionCount(), coreRun: currentCore?.store.state.identity.runId,
|
||||
coreWarm: currentCore?.store.state.warmTransition?.phase, header: checkpoint && inspectStagedRunAttachmentHeader(checkpoint.records.get("header")!)?.phase,
|
||||
runnerRun: physical.runId, runnerWarm: physical.warmTransition?.phase, lifecycle: physical.lifecycle,
|
||||
failureCode: physical.recoverableFailure?.code, failureKeys: Object.keys(physical.recoverableFailure ?? {}),
|
||||
})}`, { cause: error });
|
||||
} finally {
|
||||
release();
|
||||
if (!runnerPid && !replacement) { await first.transport.close().catch(() => undefined); await first.retireProbeResources?.(1_000).catch(() => undefined); }
|
||||
else await first.detachControllerForRestart().catch(() => undefined);
|
||||
await replacement?.transport.close().catch(() => undefined);
|
||||
for (const pid of [providerPid, runnerPid]) if (pid && !dead(-pid)) { try { process.kill(-pid, "SIGKILL"); } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; } }
|
||||
for (const core of cores) await core.stop().catch(() => undefined);
|
||||
for (const pid of [providerPid, runnerPid]) if (pid) await vi.waitFor(() => expect(dead(pid)).toBe(true));
|
||||
}
|
||||
}
|
||||
} finally { http.closeAllConnections(); await new Promise<void>(done => http.close(() => done())); await rm(root, { recursive: true, force: true }); }
|
||||
}, 90_000);
|
||||
@@ -0,0 +1,92 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { PgDialect } from "drizzle-orm/pg-core";
|
||||
import type { SQL } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { DurableRecoveryIdentity, PrpEvent } from "../../vendor/paperclip-runner/index.js";
|
||||
import { buildNativeExecutionInput } from "./native-execution-input.js";
|
||||
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
|
||||
import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js";
|
||||
|
||||
const accounting = vi.hoisted(() => ({ open: vi.fn(), capture: vi.fn() }));
|
||||
vi.mock("./native-usage-delivery.js", () => ({ createNativeUsageContinuationDelivery: accounting.open }));
|
||||
beforeEach(() => { vi.clearAllMocks(); accounting.capture.mockResolvedValue(undefined); accounting.open.mockResolvedValue({ capture: accounting.capture }); });
|
||||
|
||||
function fixture() {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), sessionId = randomUUID();
|
||||
const source: DurableRecoveryIdentity = { runId: randomUUID(), runnerInstanceId: randomUUID(), environmentLeaseId: randomUUID(),
|
||||
normalizedSessionId: sessionId, turnId: randomUUID(), itemId: randomUUID() };
|
||||
const build = (runId: string) => buildNativeExecutionInput({ companyId, agentId, runId, normalizedSessionId: sessionId,
|
||||
provider: "codex", issue: { id: issueId, identifier: "REC-1", title: "Continue", description: null, workMode: "standard" },
|
||||
taskPrompt: "Continue once", workspace: { id: randomUUID(), cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
||||
completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
|
||||
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Preserve the result" }] } },
|
||||
runtimeContext: nativeRuntimeContextFixture() });
|
||||
const execution = build(randomUUID()), previous = build(source.runId);
|
||||
const row = { run: { status: "succeeded", finishedAt: new Date() as Date | null, runtimeMode: "native", runnerInstanceId: source.runnerInstanceId,
|
||||
nativeSessionId: sessionId, runnerProfileJson: { nativeExecutionInput: previous } },
|
||||
owner: { phase: "committed", resultId: randomUUID() as string | null, leaseOwner: null as string | null, companyId, issueId } };
|
||||
const where = vi.fn(async (_predicate: SQL) => [row]);
|
||||
const select = vi.fn(() => ({ from: () => ({ innerJoin: () => ({ where }) }) }));
|
||||
const db = { select } as unknown as Db;
|
||||
const matchesExecution = vi.fn(() => true);
|
||||
const deliver = () => createNativeRecoveryUsageDelivery({ db, execution, sourceIdentity: { ...source }, matchesExecution });
|
||||
const event: PrpEvent = { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", sourceInstanceId: source.runnerInstanceId,
|
||||
sourceEventId: "source-usage", sourceSeq: 1, runId: source.runId, normalizedSessionId: sessionId, turnId: source.turnId,
|
||||
eventType: "usage.reported", priority: 0, emittedAt: new Date().toISOString(), payload: {} };
|
||||
return { source, execution, previous, row, select, where, matchesExecution, deliver, event };
|
||||
}
|
||||
|
||||
describe("source accounting recovery authority", () => {
|
||||
it("pins the relational source tuple and hydrates once without binding target accounting", async () => {
|
||||
const f = fixture(), deliver = f.deliver();
|
||||
await deliver(f.event, f.source);
|
||||
await deliver({ ...f.event, sourceEventId: "source-terminal", sourceSeq: 2, eventType: "turn.completed" }, f.source);
|
||||
expect(f.select).toHaveBeenCalledOnce();
|
||||
expect(new PgDialect().sqlToQuery(f.where.mock.calls[0][0]).params).toEqual([
|
||||
f.source.runId, f.execution.binding.companyId, f.execution.binding.agentId, f.execution.binding.issueId,
|
||||
]);
|
||||
expect(f.matchesExecution).toHaveBeenCalledWith(f.previous);
|
||||
expect(accounting.open).toHaveBeenCalledExactlyOnceWith(expect.anything(), {
|
||||
companyId: f.execution.binding.companyId, runId: f.source.runId, runnerInstanceId: f.source.runnerInstanceId, provider: f.previous.provider,
|
||||
});
|
||||
expect(accounting.capture).toHaveBeenCalledTimes(2);
|
||||
expect(accounting.capture).toHaveBeenCalledWith(f.event);
|
||||
});
|
||||
|
||||
it.each(["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const)(
|
||||
"rejects a changed pinned %s before source hydration", async key => {
|
||||
const f = fixture();
|
||||
await expect(f.deliver()(f.event, { ...f.source, [key]: randomUUID() })).rejects.toThrow("native_recovery_usage_source_unproven");
|
||||
expect(f.select).not.toHaveBeenCalled();
|
||||
expect(accounting.open).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["runId", "sourceInstanceId", "normalizedSessionId"] as const)("rejects mismatched event %s before source hydration", async key => {
|
||||
const f = fixture();
|
||||
await expect(f.deliver()({ ...f.event, [key]: randomUUID() }, f.source)).rejects.toThrow("native_recovery_usage_source_unproven");
|
||||
expect(f.select).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["nonterminal", "unfinished", "uncommitted", "no-result", "held", "company", "issue", "scope"])(
|
||||
"keeps an unproven source held: %s", async change => {
|
||||
const f = fixture();
|
||||
if (change === "nonterminal") f.row.run.status = "running";
|
||||
if (change === "unfinished") f.row.run.finishedAt = null;
|
||||
if (change === "uncommitted") f.row.owner.phase = "observed";
|
||||
if (change === "no-result") f.row.owner.resultId = null;
|
||||
if (change === "held") f.row.owner.leaseOwner = "live-owner";
|
||||
if (change === "company") f.row.owner.companyId = randomUUID();
|
||||
if (change === "issue") f.row.owner.issueId = randomUUID();
|
||||
if (change === "scope") f.matchesExecution.mockReturnValue(false);
|
||||
await expect(f.deliver()(f.event, f.source)).rejects.toThrow("native_recovery_usage_source_unproven");
|
||||
expect(accounting.open).not.toHaveBeenCalled();
|
||||
expect(accounting.capture).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps a durable source receipt failure unacknowledged", async () => {
|
||||
const f = fixture();
|
||||
accounting.capture.mockRejectedValue(new Error("native_usage_after_finalization_requires_reconciliation"));
|
||||
await expect(f.deliver()(f.event, f.source)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,47 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { parseNativeExecutionInput, type DurableRecoveryIdentity, type NativeExecutionInput, type PrpEvent } from "../../vendor/paperclip-runner/index.js";
|
||||
import { createNativeUsageContinuationDelivery } from "./native-usage-delivery.js";
|
||||
|
||||
const identityKeys = ["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const;
|
||||
|
||||
/** A cold handoff may discover accounting that belongs to its completed source.
|
||||
* Hydrate that producer once; never route its receipts through the new run. */
|
||||
export function createNativeRecoveryUsageDelivery(input: {
|
||||
db: Db;
|
||||
execution: NativeExecutionInput;
|
||||
sourceIdentity: Record<string, unknown>;
|
||||
matchesExecution(previous: NativeExecutionInput): boolean;
|
||||
}) {
|
||||
let delivery: ReturnType<typeof createNativeUsageContinuationDelivery> | undefined;
|
||||
const deny = (): never => { throw new Error("native_recovery_usage_source_unproven"); };
|
||||
const open = async (source: DurableRecoveryIdentity) => {
|
||||
const { binding } = input.execution;
|
||||
const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations })
|
||||
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
||||
.where(and(eq(heartbeatRuns.id, source.runId), eq(heartbeatRuns.companyId, binding.companyId),
|
||||
eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId)));
|
||||
if (!prior || prior.run.status !== "succeeded" || prior.run.finishedAt === null ||
|
||||
prior.run.runtimeMode !== "native" || prior.run.runnerInstanceId !== source.runnerInstanceId ||
|
||||
prior.run.nativeSessionId !== source.normalizedSessionId || prior.owner.phase !== "committed" ||
|
||||
!prior.owner.resultId || prior.owner.leaseOwner || prior.owner.companyId !== binding.companyId ||
|
||||
prior.owner.issueId !== binding.issueId) return deny();
|
||||
const previous = parseNativeExecutionInput(prior.run.runnerProfileJson?.nativeExecutionInput);
|
||||
if (previous.binding.companyId !== binding.companyId || previous.binding.agentId !== binding.agentId ||
|
||||
previous.binding.issueId !== binding.issueId || previous.binding.runId !== source.runId ||
|
||||
!input.matchesExecution(previous)) return deny();
|
||||
return createNativeUsageContinuationDelivery(input.db, {
|
||||
companyId: binding.companyId, runId: source.runId,
|
||||
runnerInstanceId: source.runnerInstanceId, provider: previous.provider,
|
||||
});
|
||||
};
|
||||
return async (event: PrpEvent, source: DurableRecoveryIdentity) => {
|
||||
if (source.runId === input.execution.binding.runId ||
|
||||
identityKeys.some(key => typeof input.sourceIdentity[key] !== "string" ||
|
||||
!input.sourceIdentity[key] || source[key] !== input.sourceIdentity[key]) ||
|
||||
event.runId !== source.runId || event.sourceInstanceId !== source.runnerInstanceId ||
|
||||
event.normalizedSessionId !== source.normalizedSessionId) return deny();
|
||||
delivery ??= open(source);
|
||||
await (await delivery).capture(event);
|
||||
};
|
||||
}
|
||||
@@ -8,6 +8,18 @@ import {
|
||||
} from "./native-restart-recovery.js";
|
||||
|
||||
describe("native restart recovery classification", () => {
|
||||
it.each([false, true])("recognizes verified staged remote evidence with target snapshot=%s", hasCheckpoint => {
|
||||
const evidence = { remoteSandbox: true, runnerPidAlive: false, runnerGroupAlive: false,
|
||||
processStartMatches: false, hasCheckpoint, hasProviderEvidence: false };
|
||||
expect(classifyNativeRunnerRecoveryEvidence(evidence).claimKind).toBeNull();
|
||||
expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true })).toEqual({
|
||||
claimKind: "reattach_remote_runner", reason: "staged_remote_attachment_requires_sandbox_verification",
|
||||
});
|
||||
expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, stagedAttachmentVerified: true,
|
||||
checkpointFailed: true }).claimKind).toBeNull();
|
||||
expect(classifyNativeRunnerRecoveryEvidence({ ...evidence, remoteSandbox: false,
|
||||
stagedAttachmentVerified: true }).claimKind).toBe(hasCheckpoint ? null : "bootstrap_incomplete");
|
||||
});
|
||||
it.each([true, false])("uses the remote fenced lease instead of the old container PID (live=%s)", async live => {
|
||||
const now = new Date();
|
||||
const probe = vi.fn(() => { throw new Error("must not probe another container's PID"); });
|
||||
|
||||
@@ -7,7 +7,8 @@ import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-pre
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js";
|
||||
import { hasNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js";
|
||||
import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js";
|
||||
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
|
||||
import { activeNativeExecutionRunIds, NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js";
|
||||
import {
|
||||
@@ -69,6 +70,8 @@ export type NativeRestartRecoveryClaim =
|
||||
recoveryRequestId: string | null;
|
||||
remote: { kind?: "sandbox"; providerLeaseId: string; remoteCwd: string }
|
||||
| { kind: "plugin_task"; leaseId: string };
|
||||
/** Discovery evidence only; executor revalidates the encrypted header. */
|
||||
stagedAttachment?: Pick<NativeStagedAttachmentRecovery, "sourceIdentity" | "targetIdentity" | "templateDigest">;
|
||||
}
|
||||
| {
|
||||
kind: "resume_dead_runner";
|
||||
@@ -322,6 +325,7 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
|
||||
checkpointIdentityMatches?: boolean;
|
||||
hasProviderEvidence: boolean;
|
||||
checkpointFailed?: boolean;
|
||||
stagedAttachmentVerified?: boolean;
|
||||
providerAttempt?: number;
|
||||
}): {
|
||||
claimKind: NativeRestartRecoveryClaim["kind"] | null;
|
||||
@@ -336,6 +340,9 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
|
||||
// inspection; the executor must prove the original lease and authenticate
|
||||
// the surviving runner before it can resume delivery.
|
||||
if (input.remoteSandbox) {
|
||||
if (input.stagedAttachmentVerified === true) {
|
||||
return { claimKind: "reattach_remote_runner", reason: "staged_remote_attachment_requires_sandbox_verification" };
|
||||
}
|
||||
if (input.unclaimedBootstrap === true && !input.hasCheckpoint && !input.hasProviderEvidence) {
|
||||
return { claimKind: "bootstrap_incomplete", reason: "remote_execution_never_claimed" };
|
||||
}
|
||||
@@ -677,6 +684,17 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
} as const;
|
||||
}
|
||||
|
||||
// Only the classic remote path stages attachments. After the owner is
|
||||
// eligible for takeover, inspect its bounded header even if activation
|
||||
// already published a runtime snapshot. Never scan command/event payloads.
|
||||
const stagedHeader = remoteWorkspace && !pluginTask && durableAuthority
|
||||
? await loadNativeRunnerCheckpointHeader(tx, {
|
||||
companyId: row.run.companyId, issueId: row.coordinator.issueId, runId: row.run.id,
|
||||
}) : null;
|
||||
const stagedAttachment = stagedHeader ? await resolveNativeStagedAttachmentRecovery({
|
||||
db: tx, currentRun: row.run, header: stagedHeader,
|
||||
}) : null;
|
||||
|
||||
const runnerPidAlive =
|
||||
!remoteProcess && processIsAlive(row.run.processPid);
|
||||
const runnerGroupAlive =
|
||||
@@ -865,6 +883,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
knownProviderProcessIdentityAmbiguous:
|
||||
providerProcesses.ambiguousLivePids.length > 0,
|
||||
hasCheckpoint,
|
||||
stagedAttachmentVerified: stagedAttachment !== null,
|
||||
checkpointIdentityMatches:
|
||||
checkpointIdentityMatches &&
|
||||
(!remoteSandbox || remoteWorkspace !== null),
|
||||
@@ -1115,6 +1134,10 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
return {
|
||||
kind: claimKind,
|
||||
...common,
|
||||
...(stagedAttachment ? { stagedAttachment: {
|
||||
sourceIdentity: stagedAttachment.sourceIdentity, targetIdentity: stagedAttachment.targetIdentity,
|
||||
templateDigest: stagedAttachment.templateDigest,
|
||||
} } : {}),
|
||||
remote: pluginTask ? { kind: "plugin_task", leaseId: pluginTask.id } : {
|
||||
providerLeaseId: remoteWorkspace!.providerLeaseId,
|
||||
remoteCwd: remoteWorkspace!.remoteCwd,
|
||||
|
||||
@@ -180,6 +180,16 @@ export async function registerNativeRunnerCheckpoint(input: {
|
||||
await input.authority.flushCheckpoint();
|
||||
}
|
||||
|
||||
/** Exact-run bounded metadata read; never loads command/event payload records. */
|
||||
export async function loadNativeRunnerCheckpointHeader(db: Pick<Db, "select">, binding: Binding): Promise<string | null> {
|
||||
const [row] = await db.select({ material: nativeRunnerCheckpoints.material }).from(nativeRunnerCheckpoints).where(and(
|
||||
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
|
||||
eq(nativeRunnerCheckpoints.runId, binding.runId), eq(nativeRunnerCheckpoints.part, "header"))).limit(1);
|
||||
if (!row) return null;
|
||||
try { return decryptPart(createLocalEncryptedCipher("read"), binding, "header", row.material); }
|
||||
catch { return null; }
|
||||
}
|
||||
|
||||
export async function loadNativeRunnerCheckpoint(db: Pick<Db, "select">, binding: Binding): Promise<DurablePrpCheckpoint | null> {
|
||||
const rows = await db.select().from(nativeRunnerCheckpoints).where(and(
|
||||
eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
|
||||
|
||||
@@ -6,9 +6,12 @@ import { prepareNativeRemoteRunner, createNativeRunnerExecutionDriver } from "./
|
||||
const transport = vi.hoisted(() => ({
|
||||
register: vi.fn(),
|
||||
connect: vi.fn(),
|
||||
checkpoint: vi.fn(),
|
||||
checkpointRegistration: vi.fn(),
|
||||
}));
|
||||
vi.mock("../../realtime/runner-prp-ws.js", () => ({ registerRunnerPrpAuthority: transport.register }));
|
||||
vi.mock("../../realtime/runner-prp-outbound.js", () => ({ connectRunnerPrpIngress: transport.connect }));
|
||||
vi.mock("./native-runner-checkpoint.js", () => ({ nativeRunnerCheckpointRegistration: transport.checkpointRegistration }));
|
||||
|
||||
const binding = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run-a" };
|
||||
const authority = {} as Parameters<NonNullable<ReturnType<typeof createNativeRunnerExecutionDriver>["controlPlaneRegistration"]>>[0];
|
||||
@@ -30,9 +33,41 @@ beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
transport.register.mockResolvedValue({ connectUrl: "ws://127.0.0.1/core", release: vi.fn() });
|
||||
transport.connect.mockReturnValue({ ready: Promise.resolve(), close: vi.fn(async () => {}) });
|
||||
transport.checkpointRegistration.mockReturnValue(transport.checkpoint);
|
||||
transport.checkpoint.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
describe("shared Runner execution driver", () => {
|
||||
it("installs both staged checkpoint writers under the business lease without acquiring endpoints", async () => {
|
||||
const prepare = vi.fn();
|
||||
const persistence = { db: {} as never, bootId: "boot-c", authorityRunId: "run-c" };
|
||||
const driver = createNativeRunnerExecutionDriver({ target: sandbox(), binding: { ...binding, runId: "run-c" },
|
||||
controllerPersistence: persistence, prepareRemote: prepare });
|
||||
const sourceIdentity = { runnerInstanceId: "runner", environmentLeaseId: "lease", normalizedSessionId: "session",
|
||||
runId: "run-a", turnId: "turn-a", itemId: "item-a" };
|
||||
const targetIdentity = { ...sourceIdentity, runId: "run-c", turnId: "turn-c", itemId: "item-c" };
|
||||
await driver.onStagedRunAttachment!(authority, { sourceIdentity, targetIdentity, templateDigest: `sha256:${"a".repeat(64)}` });
|
||||
expect(transport.checkpointRegistration).toHaveBeenCalledExactlyOnceWith({ ...persistence,
|
||||
binding: { companyId: binding.companyId, issueId: binding.issueId, runId: "run-c" } });
|
||||
expect(transport.checkpoint.mock.calls).toEqual([[authority, sourceIdentity], [authority, targetIdentity]]);
|
||||
expect(prepare).not.toHaveBeenCalled();
|
||||
expect(transport.register).not.toHaveBeenCalled();
|
||||
expect(transport.connect).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("installs the target writer first when restoring an already persisted stage", async () => {
|
||||
const sourceIdentity = { runnerInstanceId: "runner", environmentLeaseId: "lease", normalizedSessionId: "session",
|
||||
runId: "run-a", turnId: "turn-a", itemId: "item-a" };
|
||||
const targetIdentity = { ...sourceIdentity, runId: "run-c", turnId: "turn-c", itemId: "item-c" };
|
||||
const restored = { store: { state: { stagedRunAttachment: { sourceIdentity, targetIdentity } } } } as unknown as typeof authority;
|
||||
const driver = createNativeRunnerExecutionDriver({ target: sandbox(), binding: { ...binding, runId: "run-c" },
|
||||
controllerPersistence: { db: {} as never, bootId: "boot-c", authorityRunId: "run-c" } });
|
||||
await driver.onStagedRunAttachment!(restored, { sourceIdentity, targetIdentity, templateDigest: `sha256:${"a".repeat(64)}` });
|
||||
expect(transport.checkpoint.mock.calls).toEqual([[restored, targetIdentity], [restored, sourceIdentity]]);
|
||||
expect(transport.register).not.toHaveBeenCalled();
|
||||
expect(transport.connect).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps local execution on the registered loopback route", async () => {
|
||||
const driver = createNativeRunnerExecutionDriver({ target: { kind: "local" }, binding });
|
||||
const connection = await driver.controlPlaneRegistration!(authority);
|
||||
|
||||
@@ -92,6 +92,20 @@ export async function prepareNativeRemoteRunner(input: RemotePreparation) {
|
||||
export function createNativeRunnerExecutionDriver(input: NativeRunnerExecutionDriverInput): RunnerExecutionDriver {
|
||||
return {
|
||||
...input.process,
|
||||
onStagedRunAttachment: input.controllerPersistence && input.binding.issueId
|
||||
? async (authority, proof) => {
|
||||
// Both journals use the current business lease. Install persistence
|
||||
// before staging; endpoint acquisition still belongs to attachment.
|
||||
const register = nativeRunnerCheckpointRegistration({ ...input.controllerPersistence!,
|
||||
binding: { companyId: input.binding.companyId, issueId: input.binding.issueId!, runId: input.binding.runId },
|
||||
});
|
||||
// A restored stage already publishes under the target. Its writer
|
||||
// must exist before registration flushes that restored revision.
|
||||
const identities = authority.store?.state.stagedRunAttachment
|
||||
? [proof.targetIdentity, proof.sourceIdentity] : [proof.sourceIdentity, proof.targetIdentity];
|
||||
for (const identity of identities) await register(authority, identity);
|
||||
}
|
||||
: undefined,
|
||||
controlPlaneRegistration: async (authority, attachmentIdentity) => {
|
||||
await input.beforeConnect?.(attachmentIdentity, authority);
|
||||
if (input.controllerPersistence && input.binding.issueId) {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { rehydratePinnedSkillAssets } from "./runtime-context.js";
|
||||
import { createNativeUsageDelivery } from "./native-usage-delivery.js";
|
||||
import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js";
|
||||
import { createNativeTurnAccounting, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent } from "./native-usage-accounting.js";
|
||||
export { nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage } from "./native-usage-accounting.js";
|
||||
import { measureRunnerOperation, recordRunnerTimingReceipt } from "./runner-operation-timing.js";
|
||||
@@ -105,7 +106,7 @@ import {
|
||||
} from "node:fs";
|
||||
import { tmpdir, userInfo } from "node:os";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { loadNativeRunnerCheckpointHeader, nativeRunnerCheckpointPersistence, nativeRunnerCheckpointRegistration, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { REMOTE_CONTROLLER_LEASE_MS, REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js";
|
||||
import type { NativeTaskSessionPublicationContext } from "./native-task-session-publication.js";
|
||||
import { basename, dirname, join, posix, resolve } from "node:path";
|
||||
@@ -116,6 +117,7 @@ import type {
|
||||
import type { NativeFinalizationResult } from "@paperclipai/shared";
|
||||
import type {
|
||||
DurablePrpControlPlane,
|
||||
DurableRecoveryIdentity,
|
||||
HarnessRuntimeRequestResolution,
|
||||
NativeExecutionInput,
|
||||
NativeRuntimeContextSnapshot,
|
||||
@@ -140,6 +142,7 @@ import {
|
||||
createRunnerdCodexTransport,
|
||||
executeNativeSession,
|
||||
applyNativeSessionGoalControl,
|
||||
inspectStagedRunAttachment,
|
||||
inspectWarmRunTransition,
|
||||
parseNativeExecutionInput,
|
||||
parsePaperclipQuestionSet,
|
||||
@@ -213,6 +216,7 @@ import {
|
||||
} from "./remote-runner-recovery-diagnostics.js";
|
||||
import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js";
|
||||
import { inspectRemoteWarmSessionRecovery, type RemoteWarmSessionRecovery } from "./remote-warm-session-recovery.js";
|
||||
import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js";
|
||||
import { emitStartupReuse, environmentStartupReuse } from "@paperclipai/adapter-utils/startup-reuse";
|
||||
import { persistActivity, publishActivity } from "../activity-log.js";
|
||||
import { commitNativeStatusDecision } from "./status-decision-committer.js";
|
||||
@@ -564,6 +568,7 @@ export type NativeInstructionWorkingCopy = {
|
||||
|
||||
type WarmNativeSession = {
|
||||
onReleased: () => void;
|
||||
sessionCheckpointPersisted?: (snapshot: PersistedNativeSession) => void;
|
||||
pluginTaskId?: string;
|
||||
observabilityCompatibilityKey?: string;
|
||||
runId: string;
|
||||
@@ -1805,6 +1810,14 @@ class SessionAuthorityEpoch {
|
||||
this.#assertCurrent();
|
||||
}
|
||||
|
||||
async acceptRecoveryAccountingEvent(event: PrpEvent, capture: () => Promise<void>) {
|
||||
if (!isNativeUsageAccountingEvent(event)) return;
|
||||
this.#assertCurrent();
|
||||
if (event.runId === this.runId) throw new Error("native_recovery_usage_source_unproven");
|
||||
await capture();
|
||||
this.#assertCurrent();
|
||||
}
|
||||
|
||||
definitions() {
|
||||
this.#assertCurrent();
|
||||
return this.#authority.definitions();
|
||||
@@ -5114,6 +5127,12 @@ async function migrateRunnerdStateRootForExecution(input: {
|
||||
if (!isSafeNativeStateDirectory(scoped)) {
|
||||
throw new Error("runner_state_directory_unsafe");
|
||||
}
|
||||
const staged = await verifyStagedAttachmentRestart(input);
|
||||
if (staged) {
|
||||
// Physical inspection belongs to backend admission after lease renewal
|
||||
// starts. This phase only validates the immutable source/target proof.
|
||||
return;
|
||||
}
|
||||
if (hasRetainedWarmTransitionEvidence(scoped)) {
|
||||
// A copied receipt is not restart/process authority. Keep both valid
|
||||
// unsupported and invalid forward evidence out of destructive legacy
|
||||
@@ -5626,6 +5645,11 @@ function readRunnerdDurableIdentity(
|
||||
).toString("utf8"),
|
||||
),
|
||||
);
|
||||
if (state.schema === "paperclip.runner.durable.control-plane-state.staged-attachment.v1") {
|
||||
const staged = inspectStagedRunAttachment(state);
|
||||
if (!staged) throw new Error("native_runner_staged_attachment_binding_mismatch");
|
||||
return { ...staged.sourceIdentity };
|
||||
}
|
||||
if (state.schema !== RUNNERD_CONTROL_PLANE_STATE_SCHEMA &&
|
||||
!(allowExternal && state.schema === "paperclip.runner.external-control-plane-state.v1")) return null;
|
||||
return record(state.identity);
|
||||
@@ -5634,6 +5658,50 @@ function readRunnerdDurableIdentity(
|
||||
}
|
||||
}
|
||||
|
||||
function readStagedRunnerdAttachment(root: string) {
|
||||
const path = resolve(root, "control-plane", "control-plane-state.json");
|
||||
if (!existsSync(path)) return null;
|
||||
const state = record(JSON.parse(readBoundedNativeFile(path, NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
|
||||
"runner_durable_identity_too_large").toString("utf8")));
|
||||
if (state.stagedRunAttachment === undefined &&
|
||||
state.schema !== "paperclip.runner.durable.control-plane-state.staged-attachment.v1") return null;
|
||||
const staged = inspectStagedRunAttachment(state);
|
||||
if (!staged) throw new Error("native_runner_staged_attachment_binding_mismatch");
|
||||
return staged;
|
||||
}
|
||||
|
||||
/** A staged attachment has not published a target session. Re-establish its
|
||||
* exact source and target from private durable evidence before any remote RPC. */
|
||||
async function verifyStagedAttachmentRestart(input: {
|
||||
db: Db; execution: NativeExecutionInput; restartRecovery?: NativeRestartRecoveryClaim;
|
||||
runnerExecutionTarget?: AdapterExecutionTarget | null;
|
||||
}, expected?: NativeStagedAttachmentRecovery): Promise<NativeStagedAttachmentRecovery | null> {
|
||||
const root = scopedRunnerdStateRoot(input.execution);
|
||||
const localStage = readStagedRunnerdAttachment(root);
|
||||
const staged = expected ?? localStage;
|
||||
if (!staged) return null;
|
||||
const deny = (): never => { throw new Error("native_runner_staged_attachment_recovery_unproven"); };
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (input.restartRecovery?.kind !== "reattach_remote_runner" || input.restartRecovery.remote.kind === "plugin_task" ||
|
||||
target?.kind !== "remote" || target.transport !== "sandbox" || target.runnerTask ||
|
||||
target.sandboxLeaseAcquisition?.outcome !== "resumed") return deny();
|
||||
const [current] = await input.db.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, input.execution.binding.runId), eq(heartbeatRuns.companyId, input.execution.binding.companyId),
|
||||
eq(heartbeatRuns.agentId, input.execution.binding.agentId), eq(heartbeatRuns.nativeIssueId, input.execution.binding.issueId)));
|
||||
const header = await loadNativeRunnerCheckpointHeader(input.db, input.execution.binding);
|
||||
if (!current || current.status !== "running" || !header) return deny();
|
||||
const verified = await resolveNativeStagedAttachmentRecovery({ db: input.db, currentRun: current, header,
|
||||
expectedTarget: staged.targetIdentity,
|
||||
matchesExecution: (previous, execution) => nativeSessionScopeKey(previous) === nativeSessionScopeKey(execution) &&
|
||||
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(execution, "remote"),
|
||||
});
|
||||
if (!verified || nativeSha256(verified.currentExecution) !== nativeSha256(input.execution) ||
|
||||
verified.templateDigest !== staged.templateDigest ||
|
||||
nativeSha256(verified.sourceIdentity) !== nativeSha256(staged.sourceIdentity) ||
|
||||
(localStage && nativeSha256(readStagedRunnerdAttachment(root)) !== nativeSha256(localStage))) return deny();
|
||||
return verified;
|
||||
}
|
||||
|
||||
function durableIdentityMatchesExecution(
|
||||
identity: Record<string, unknown> | null,
|
||||
execution: NativeExecutionInput,
|
||||
@@ -8180,6 +8248,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
const restoredController = input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
? await restoreNativeRunnerCheckpoint(input.db, input.execution.binding, scopedRunnerdStateRoot(input.execution))
|
||||
: null;
|
||||
const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
? await verifyStagedAttachmentRestart(input) : null;
|
||||
if (input.useRunnerd) await restoreWarmRunnerdController(input);
|
||||
const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY);
|
||||
const redactIdentityText = (text: string) => redactSensitiveText(
|
||||
@@ -9086,7 +9156,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
let managedCredentialSession: NativeSession | undefined;
|
||||
let githubAccess: NativeGitHubAccess | undefined;
|
||||
let releaseGitHubRun: (() => void) | undefined;
|
||||
let persistedWarmSession: PersistedNativeSession | null | undefined;
|
||||
let persistedWarmSession: PersistedNativeSession | null | undefined = stagedRestart && !stagedRestart.hasTargetCheckpoint
|
||||
? stagedRestart.checkpoint : undefined;
|
||||
if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) {
|
||||
const entry = warmNativeSessions.get(warmSessionId);
|
||||
if (entry) {
|
||||
@@ -9468,6 +9539,12 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
requireSessionCloseBeforeReturn: runnerdBackend !== null || input.instructionWorkingCopy !== undefined,
|
||||
onSessionClosed: input.instructionWorkingCopy?.collectStopped,
|
||||
onCheckpoint: async (snapshot) => {
|
||||
// The runtime invokes this only after its existing database
|
||||
// snapshot transaction commits. Retirement coalesces into the
|
||||
// next PRP command; this notification performs no persistence.
|
||||
const checkpointOwner = warmSessionId === null ? undefined
|
||||
: warmNativeSessions.get(warmSessionId)?.sessionCheckpointPersisted;
|
||||
(checkpointOwner ?? runnerdBackend?.sessionCheckpointPersisted)?.(snapshot);
|
||||
if (snapshot.governedWait) {
|
||||
const startup = nativeSessionStartups.get(input.execution.binding.runId);
|
||||
if (startup) startup.governedWait = true;
|
||||
@@ -9564,11 +9641,15 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
if (existing.ownerToken !== warmSessionOwnerToken) {
|
||||
throw new Error("native_session_supervisor_busy");
|
||||
}
|
||||
if (existing.session !== session) {
|
||||
existing.sessionCheckpointPersisted = runnerdBackend?.sessionCheckpointPersisted;
|
||||
}
|
||||
existing.session = session;
|
||||
existing.runId = input.execution.binding.runId;
|
||||
existing.issueId = input.execution.binding.issueId;
|
||||
} else
|
||||
warmNativeSessions.set(warmSessionId, {
|
||||
sessionCheckpointPersisted: runnerdBackend?.sessionCheckpointPersisted,
|
||||
onReleased: () => releaseRetainedNativeSessionCleanup(input.db, {
|
||||
companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId,
|
||||
}),
|
||||
@@ -11442,6 +11523,62 @@ export async function verifyRemoteRunnerReattachment(input: {
|
||||
};
|
||||
}
|
||||
|
||||
/** A crash before source bootstrap leaves a suspended Runner; a crash during
|
||||
* its drain leaves the same live source. Neither permits a replacement until
|
||||
* the physical process and all six persisted identity fields are proven. */
|
||||
async function verifyStagedRemoteRunner(input: {
|
||||
recovery: NativeStagedAttachmentRecovery;
|
||||
execution: NativeExecutionInput;
|
||||
target: AdapterExecutionTarget | null | undefined;
|
||||
claim: Extract<NativeRestartRecoveryClaim, { kind: "reattach_remote_runner" }>;
|
||||
}) {
|
||||
const { target } = input;
|
||||
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) {
|
||||
throw new Error("native_runner_staged_attachment_profile_unsupported");
|
||||
}
|
||||
const sourceIdentity = input.recovery.sourceIdentity;
|
||||
const stateDirectory = posix.join(target.remoteCwd, ".paperclip-runtime", "paperclip-runner", "sessions",
|
||||
createHash("sha256").update(sourceIdentity.normalizedSessionId).digest("hex"), "runner");
|
||||
const read = () => readRemoteRunnerState({ runner: target.runner!, stateDirectory });
|
||||
const state = await read();
|
||||
const control = record(JSON.parse(readBoundedNativeFile(resolve(scopedRunnerdStateRoot(input.execution),
|
||||
"control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
|
||||
"runner_durable_identity_too_large").toString("utf8")));
|
||||
const warm = inspectWarmRunTransition({ controlPlaneState: control, runnerState: state,
|
||||
expectedNewIdentity: input.recovery.targetIdentity, expectedRunnerVersion: input.recovery.runnerVersion,
|
||||
expectedRunnerDigest: input.recovery.runnerDigest });
|
||||
if (!warm && (control.warmTransition !== undefined || state.warmTransition !== undefined)) {
|
||||
throw new Error("native_runner_staged_attachment_recovery_unproven");
|
||||
}
|
||||
const identity = warm?.runnerIdentity ?? input.recovery.protocolIdentity;
|
||||
const sameIdentity = (value: Record<string, unknown>) =>
|
||||
Object.entries(identity).every(([key, expected]) => value[key] === expected);
|
||||
if (!sameIdentity(state)) throw new Error("runner_remote_recovery_identity_mismatch");
|
||||
const claim = { ...input.claim, runId: identity.runId };
|
||||
const verify = () => verifyRemoteRunnerReattachment({ claim, target, identity: { ...identity },
|
||||
runId: identity.runId, normalizedSessionId: identity.normalizedSessionId });
|
||||
if (state.lifecycle !== "suspended") return verify();
|
||||
const settled = (value: Record<string, unknown>) => sameIdentity(value) && value.lifecycle === "suspended" &&
|
||||
Array.isArray(value.outbox) && value.outbox.length === 0 && value.pendingTerminalDelivery == null &&
|
||||
value.pendingProviderCleanup == null && value.warmTransition == null;
|
||||
if (!settled(state)) throw new Error("native_runner_staged_attachment_source_unsettled");
|
||||
try {
|
||||
const retiring = await verify();
|
||||
const deadline = Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS;
|
||||
while (await retiring.isAlive()) {
|
||||
if (Date.now() >= deadline) throw new Error("native_runner_staged_attachment_source_retiring");
|
||||
await new Promise<void>(resolveWait => setTimeout(resolveWait, 100));
|
||||
}
|
||||
} catch (error) {
|
||||
const diagnostic = readRemoteRunnerRecoveryError(error);
|
||||
if (diagnostic?.stage !== "process_verification" || diagnostic.reason !== "process_not_running") throw error;
|
||||
}
|
||||
// An absent/unsafe marker or a changed process generation never proves exit.
|
||||
// The package rechecks this same suspended source before launching it.
|
||||
if (!settled(await read())) throw new Error("native_runner_staged_attachment_source_unsettled");
|
||||
return undefined;
|
||||
}
|
||||
|
||||
async function waitForRemoteRunnerProcessIdentity(input: {
|
||||
runner: CommandManagedRuntimeRunner;
|
||||
identityPath: string;
|
||||
@@ -11870,7 +12007,7 @@ export async function createRunnerdBackend(input: {
|
||||
contextSnapshot: Record<string, unknown>;
|
||||
},
|
||||
) => Promise<unknown>;
|
||||
}): Promise<NativeSessionBackend & { bindManagedSession(session: NativeSession): NativeSession }> {
|
||||
}): Promise<ManagedRunnerdBackend> {
|
||||
const sessionScopeId = nativeSessionScopeKey(input.execution);
|
||||
const scopeOwner = executingRunnerdSessionScopes.get(sessionScopeId);
|
||||
if (scopeOwner && scopeOwner !== input.execution.binding.runId) {
|
||||
@@ -12020,11 +12157,16 @@ async function prepareRemoteExternalRunner({ input, target, runner, root, identi
|
||||
return { runnerBinary, readProviderState, ...executionDriver };
|
||||
}
|
||||
|
||||
type ManagedRunnerdBackend = NativeSessionBackend & {
|
||||
bindManagedSession(session: NativeSession): NativeSession;
|
||||
sessionCheckpointPersisted?(snapshot: PersistedNativeSession): void;
|
||||
};
|
||||
|
||||
async function createRunnerdBackendWithinSessionClaim(
|
||||
input: Parameters<typeof createRunnerdBackend>[0],
|
||||
sessionScopeId: string,
|
||||
retainedTransition?: VerifiedWarmTransitionBinding,
|
||||
): Promise<NativeSessionBackend & { bindManagedSession(session: NativeSession): NativeSession }> {
|
||||
): Promise<ManagedRunnerdBackend> {
|
||||
const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller");
|
||||
let recoveryPending = retainedTransition !== undefined;
|
||||
const target = input.runnerExecutionTarget ?? { kind: "local" as const };
|
||||
@@ -12157,6 +12299,17 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
: durableIdentityMatchesSession(durableIdentity, input.execution)
|
||||
? durableIdentity
|
||||
: null;
|
||||
const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
? await verifyStagedAttachmentRestart(input) : null;
|
||||
let stagedRecoveryPending = stagedRecovery !== null;
|
||||
const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity;
|
||||
const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
|
||||
? createNativeRecoveryUsageDelivery({
|
||||
db: input.db, execution: input.execution, sourceIdentity: recoveryUsageIdentity,
|
||||
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
|
||||
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
|
||||
})
|
||||
: undefined;
|
||||
const effectiveRunnerInstanceId =
|
||||
durableBinding?.runnerInstanceId ?? input.runnerInstanceId;
|
||||
const effectiveEnvironmentLeaseId =
|
||||
@@ -13663,7 +13816,9 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
? input.restartRecovery.process
|
||||
: null;
|
||||
const adoptedProcess =
|
||||
input.remoteWarmRecovery
|
||||
stagedRecovery && input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
? await verifyStagedRemoteRunner({ recovery: stagedRecovery, execution: input.execution, target, claim: input.restartRecovery })
|
||||
: input.remoteWarmRecovery
|
||||
? await (async () => {
|
||||
await input.remoteWarmRecovery!.revalidate();
|
||||
return verifyRemoteRunnerReattachment({
|
||||
@@ -13698,6 +13853,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
) => {
|
||||
const current = sessionToolAuthorityEpochs.get(sessionScopeId);
|
||||
if (!current) throw new Error("native_session_tool_authority_unavailable");
|
||||
const attachmentRunId = runnerControlPlane?.store?.state.identity.runId;
|
||||
if (attachmentRunId && attachmentRunId !== current.runId) {
|
||||
throw new Error("native_runner_source_semantic_reconciliation_required");
|
||||
}
|
||||
return current.execute(call);
|
||||
};
|
||||
if (requiresRemoteProviderPack) {
|
||||
@@ -13745,8 +13904,19 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
onAccountingEvent: input.onAccountingEvent ? async event => {
|
||||
const current = sessionToolAuthorityEpochs.get(sessionScopeId);
|
||||
if (!current) throw new Error("native_session_authority_unavailable");
|
||||
if (input.remoteWarmRecovery && event.runId === input.remoteWarmRecovery.identity.runId) {
|
||||
if (!recoveryUsage) throw new Error("native_recovery_usage_source_unproven");
|
||||
await current.acceptRecoveryAccountingEvent(event, () => recoveryUsage(event,
|
||||
input.remoteWarmRecovery!.identity as DurableRecoveryIdentity));
|
||||
return;
|
||||
}
|
||||
await current.acceptAccountingEvent(event);
|
||||
} : undefined,
|
||||
onRecoveryAccountingEvent: input.onAccountingEvent ? async (event, source) => {
|
||||
const current = sessionToolAuthorityEpochs.get(sessionScopeId);
|
||||
if (!current || !recoveryUsage) throw new Error("native_recovery_usage_source_unproven");
|
||||
await current.acceptRecoveryAccountingEvent(event, () => recoveryUsage(event, source));
|
||||
} : undefined,
|
||||
onCredentialsUpdated: (event) => appendNativeCredentialUpdate(input.db, {
|
||||
companyId: input.execution.binding.companyId,
|
||||
agentId: input.execution.binding.agentId,
|
||||
@@ -13920,7 +14090,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
input.execution.interactionResponses.length > 0
|
||||
? "allow_replacement_after_governed_wait"
|
||||
: undefined),
|
||||
prpIdentity: input.remoteWarmRecovery ? {
|
||||
prpIdentity: stagedRecovery ? stagedRecovery.targetIdentity : input.remoteWarmRecovery ? {
|
||||
runnerInstanceId: input.remoteWarmRecovery.identity.runnerInstanceId,
|
||||
environmentLeaseId: input.remoteWarmRecovery.identity.environmentLeaseId,
|
||||
runId: input.remoteWarmRecovery.identity.runId,
|
||||
@@ -13947,10 +14117,19 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
turnId: `turn-${input.execution.binding.runId}`,
|
||||
itemId: `item-${input.execution.binding.runId}`,
|
||||
},
|
||||
warmTransitionRegistrationMode: retainedTransition
|
||||
? "routed_connect"
|
||||
: undefined,
|
||||
authorizeWarmTransitionRecovery: retainedTransition
|
||||
warmTransitionRegistrationMode: stagedRecovery && remoteTarget?.transport === "sandbox" &&
|
||||
remoteTarget.effectiveCapabilities?.runnerWebSocketIngress === true
|
||||
? "routed_same_endpoint"
|
||||
: retainedTransition || stagedRecovery ? "routed_connect" : undefined,
|
||||
authorizeWarmTransitionRecovery: stagedRecovery
|
||||
? async () => {
|
||||
if (!stagedRecoveryPending) return;
|
||||
const current = await verifyStagedAttachmentRestart(input, stagedRecovery);
|
||||
if (!current || current.templateDigest !== stagedRecovery.templateDigest) {
|
||||
throw new Error("native_runner_staged_attachment_recovery_unproven");
|
||||
}
|
||||
}
|
||||
: retainedTransition
|
||||
? async (
|
||||
stage:
|
||||
"before_bootstrap" | "before_spawn" | "before_authentication",
|
||||
@@ -14025,6 +14204,13 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
beforeConnect: async (_identity, authority) => {
|
||||
runnerControlPlane = authority;
|
||||
await input.remoteWarmRecovery?.revalidate();
|
||||
if (stagedRecovery && authority.store.state.stagedRunAttachment) {
|
||||
const current = await verifyStagedAttachmentRestart(input, stagedRecovery);
|
||||
if (!current || current.templateDigest !== stagedRecovery.templateDigest ||
|
||||
nativeSha256(current.targetIdentity) !== nativeSha256(stagedRecovery.targetIdentity)) {
|
||||
throw new Error("native_runner_staged_attachment_recovery_unproven");
|
||||
}
|
||||
}
|
||||
if (retainedTransition && recoveryPending) {
|
||||
const current = await verifyWarmTransitionRestart(input);
|
||||
if (current.stateFingerprint !== retainedTransition.stateFingerprint) {
|
||||
@@ -14116,6 +14302,15 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
};
|
||||
return {
|
||||
bindManagedSession: wrapManagedSession,
|
||||
sessionCheckpointPersisted: snapshot => {
|
||||
const staged = runnerControlPlane?.store.state.stagedRunAttachment;
|
||||
if (staged && snapshot.identity.runId === staged.targetIdentity.runId &&
|
||||
snapshot.identity.sessionId === staged.targetIdentity.normalizedSessionId) {
|
||||
if (runnerControlPlane!.markStagedTargetCheckpointPublished(staged.targetIdentity)) {
|
||||
stagedRecoveryPending = false;
|
||||
}
|
||||
}
|
||||
},
|
||||
descriptor: () => backend.descriptor(),
|
||||
openSession: async (sessionInput) => {
|
||||
const session = await backend.openSession(sessionInput);
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { DurablePrpControlPlane, type DurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js";
|
||||
import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js";
|
||||
import { buildNativeExecutionInput } from "./native-execution-input.js";
|
||||
import { nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
|
||||
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
|
||||
import { resolveNativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js";
|
||||
|
||||
const companyId = "10000000-0000-4000-8000-000000000001";
|
||||
const issueId = "20000000-0000-4000-8000-000000000002";
|
||||
const agentId = "30000000-0000-4000-8000-000000000003";
|
||||
const sourceRunId = "40000000-0000-4000-8000-000000000004";
|
||||
const currentRunId = "50000000-0000-4000-8000-000000000005";
|
||||
const normalizedSessionId = "60000000-0000-4000-8000-000000000006";
|
||||
const runnerInstanceId = "70000000-0000-4000-8000-000000000007";
|
||||
const sourceIdentity = { runId: sourceRunId, normalizedSessionId, runnerInstanceId,
|
||||
environmentLeaseId: "environment-1", turnId: "opaque-source-turn", itemId: "opaque-source-item" };
|
||||
const targetIdentity = { ...sourceIdentity, runId: currentRunId, turnId: "opaque-target-turn", itemId: "opaque-target-item" };
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
||||
|
||||
function execution(runId: string) {
|
||||
return buildNativeExecutionInput({ companyId, runId, agentId, normalizedSessionId,
|
||||
issue: { id: issueId, identifier: "TEST-1", title: "Continue", description: null, workMode: "standard" },
|
||||
taskPrompt: "Continue normally", provider: "codex", workspace: { id: "workspace-1", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
||||
completionContract: { id: "80000000-0000-4000-8000-000000000008", sha256: `sha256:${"a".repeat(64)}`,
|
||||
schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Continue", criteria: [{ id: "done", requirement: "Answer" }] } },
|
||||
runtimeContext: nativeRuntimeContextFixture(),
|
||||
});
|
||||
}
|
||||
|
||||
async function fixture() {
|
||||
const root = await mkdtemp(join(tmpdir(), "native-staged-proof-")); roots.push(root);
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: root, identity: sourceIdentity,
|
||||
expectedRunnerVersion: "0.0.0", expectedRunnerDigest: `sha256:${"f".repeat(64)}` });
|
||||
let saved!: DurablePrpCheckpoint;
|
||||
core.setCheckpointPersistence(async checkpoint => { saved = checkpoint; });
|
||||
core.persistRunAttachTemplate({ provider: { provider: "codex", command: "codex" }, authorizedTools: {} });
|
||||
await core.flushCheckpoint();
|
||||
const sourceHeader = saved.records.get("header")!;
|
||||
core.stageRunAttachment(targetIdentity, { provider: { provider: "codex", command: "codex" }, authorizedTools: {} });
|
||||
await core.flushCheckpoint();
|
||||
const header = saved.records.get("header")!;
|
||||
const base = { companyId, agentId, issueId, nativeIssueId: issueId, runnerInstanceId,
|
||||
nativeSessionId: normalizedSessionId, runtimeMode: "native", status: "running", finishedAt: null as Date | null };
|
||||
const currentRun = { ...base, id: currentRunId, runnerProfileJson: { nativeExecutionInput: execution(currentRunId) } };
|
||||
const sourceRun = { ...base, id: sourceRunId, status: "succeeded", finishedAt: new Date(), runnerProfileJson: {
|
||||
nativeExecutionInput: execution(sourceRunId), nativeToolContractFingerprint: nativeToolContractFingerprintForTarget("remote", "codex"),
|
||||
sessionCheckpoint: { sessionId: "provider-thread-1", providerSessionId: "provider-thread-1", identity: {
|
||||
companyId, issueId, agentId, runId: sourceRunId, sessionId: normalizedSessionId },
|
||||
},
|
||||
} };
|
||||
const source = { run: sourceRun, coordinator: { runId: sourceRunId, companyId, issueId,
|
||||
phase: "committed", resultId: "result-1", leaseOwner: null, leaseExpiresAt: null }, resultId: "result-1", resultStatus: "accepted" };
|
||||
function database(sourceRows = [source], contents = sourceHeader, envelope = {}) {
|
||||
const material = createLocalEncryptedCipher("write").encrypt(JSON.stringify({ companyId, issueId, runId: sourceRunId, part: "header", contents, ...envelope }));
|
||||
const batches: unknown[][] = [sourceRows, [{ material }]];
|
||||
const select = vi.fn(() => {
|
||||
const chain = { from: () => chain, innerJoin: () => chain, where: () => chain,
|
||||
limit: vi.fn(async () => batches.shift() ?? []) };
|
||||
return chain;
|
||||
});
|
||||
return { db: { select } as unknown as Pick<Db, "select">, select };
|
||||
}
|
||||
return { header, sourceHeader, source, sourceRun, currentRun, database };
|
||||
}
|
||||
|
||||
describe("private staged attachment restart evidence", () => {
|
||||
it("proves both frozen run bindings before the target has a session snapshot", async () => {
|
||||
const f = await fixture(); const { db, select } = f.database();
|
||||
const matchesExecution = vi.fn(() => true);
|
||||
const result = await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun,
|
||||
expectedTarget: targetIdentity, matchesExecution });
|
||||
expect(result).toMatchObject({ sourceIdentity, targetIdentity, sourceRun: { id: sourceRunId }, hasTargetCheckpoint: false, targetCheckpoint: null,
|
||||
protocolIdentity: sourceIdentity, phase: "source_bootstrap", receipt: null });
|
||||
expect(matchesExecution).toHaveBeenCalledOnce();
|
||||
expect(select).toHaveBeenCalledTimes(2); // one source point lookup and one encrypted header point lookup
|
||||
expect(f.currentRun.runnerProfileJson).not.toHaveProperty("sessionCheckpoint");
|
||||
});
|
||||
|
||||
it("retains staged evidence when the target runtime has already published its snapshot", async () => {
|
||||
const f = await fixture();
|
||||
const snapshot = structuredClone(f.sourceRun.runnerProfileJson.sessionCheckpoint);
|
||||
snapshot.identity.runId = currentRunId;
|
||||
Object.assign(snapshot, { cursor: "target-cursor", activeTurnId: "target-current-turn", pendingRuntimeRequests: [{ requestId: "target-request" }] });
|
||||
Object.assign(f.currentRun.runnerProfileJson, { sessionCheckpoint: snapshot });
|
||||
const result = await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun });
|
||||
expect(result).toMatchObject({ sourceIdentity, targetIdentity, checkpoint: { identity: { runId: currentRunId } },
|
||||
hasTargetCheckpoint: true, targetCheckpoint: snapshot });
|
||||
expect(f.currentRun.runnerProfileJson).toHaveProperty("sessionCheckpoint", snapshot);
|
||||
});
|
||||
|
||||
it("returns the codec's queued phase without assuming the Runner has activated the target", async () => {
|
||||
const f = await fixture(); const header = JSON.parse(f.header);
|
||||
// Header inspection deliberately does not certify separate command records.
|
||||
// Full restoration/authentication owns that later gate; no RPC runs here.
|
||||
header.state.stagedRunAttachment.commandId = "staged-attach-command";
|
||||
const result = await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: JSON.stringify(header), currentRun: f.currentRun });
|
||||
expect(result).toMatchObject({ commandId: "staged-attach-command", protocolIdentity: sourceIdentity,
|
||||
phase: "attachment_queued", receipt: null, targetIdentity });
|
||||
});
|
||||
|
||||
it.each(["runId", "companyId", "issueId", "agentId", "sessionId", "providerSessionId"])(
|
||||
"rejects a present target checkpoint with changed %s instead of replaying the source", async key => {
|
||||
const f = await fixture(); const snapshot = structuredClone(f.sourceRun.runnerProfileJson.sessionCheckpoint);
|
||||
snapshot.identity.runId = currentRunId;
|
||||
if (key === "providerSessionId") snapshot.providerSessionId = "foreign";
|
||||
else Object.assign(snapshot.identity, { [key]: "foreign" });
|
||||
Object.assign(f.currentRun.runnerProfileJson, { sessionCheckpoint: snapshot });
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["runId", "runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "turnId", "itemId"] as const)(
|
||||
"rejects a different expected target %s before querying the source", async key => {
|
||||
const f = await fixture(); const { db, select } = f.database();
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun,
|
||||
expectedTarget: { ...targetIdentity, [key]: "foreign" } })).toBeNull();
|
||||
expect(select).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["agentId", "companyId", "nativeIssueId", "nativeSessionId", "runnerInstanceId"] as const)(
|
||||
"rejects mixed source DB %s", async key => {
|
||||
const f = await fixture(); Object.assign(f.source.run, { [key]: "foreign" });
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["binding", "schema", "runner", "session"])("rejects current frozen evidence mismatch: %s", async kind => {
|
||||
const f = await fixture(); const { db, select } = f.database();
|
||||
if (kind === "binding") f.currentRun.runnerProfileJson.nativeExecutionInput.binding.agentId = sourceRunId;
|
||||
if (kind === "schema") Object.assign(f.currentRun.runnerProfileJson.nativeExecutionInput, { schema: "unsupported" });
|
||||
if (kind === "runner") f.currentRun.runnerInstanceId = sourceRunId;
|
||||
if (kind === "session") f.currentRun.nativeSessionId = sourceRunId;
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ db, header: f.header, currentRun: f.currentRun })).toBeNull();
|
||||
expect(select).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["phase", "leaseOwner", "resultStatus", "status", "finishedAt", "frozenRun", "workspace", "provider", "sourceTuple", "artifactVersion", "artifactDigest"])(
|
||||
"rejects unsupported or stale source evidence: %s", async kind => {
|
||||
const f = await fixture(); let sourceHeader = f.sourceHeader;
|
||||
if (kind === "phase") f.source.coordinator.phase = "observed";
|
||||
if (kind === "leaseOwner") Object.assign(f.source.coordinator, { leaseOwner: "another-owner", leaseExpiresAt: new Date(0) });
|
||||
if (kind === "resultStatus") f.source.resultStatus = "rejected";
|
||||
if (kind === "status") f.source.run.status = "running";
|
||||
if (kind === "finishedAt") Object.assign(f.source.run, { finishedAt: null });
|
||||
if (kind === "frozenRun") f.source.run.runnerProfileJson.nativeExecutionInput.binding.runId = currentRunId;
|
||||
if (kind === "workspace") f.source.run.runnerProfileJson.nativeExecutionInput.binding.executionWorkspaceId = "foreign";
|
||||
if (kind === "provider") Object.assign(f.source.run.runnerProfileJson.nativeExecutionInput.provider, { model: "another-model" });
|
||||
if (kind === "sourceTuple") { const h = JSON.parse(sourceHeader); h.state.identity.turnId = "foreign"; sourceHeader = JSON.stringify(h); }
|
||||
if (kind === "artifactVersion") { const h = JSON.parse(sourceHeader); h.runnerVersion = "other-version"; sourceHeader = JSON.stringify(h); }
|
||||
if (kind === "artifactDigest") { const h = JSON.parse(sourceHeader); h.runnerDigest = `sha256:${"d".repeat(64)}`; sourceHeader = JSON.stringify(h); }
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ ...f.database([f.source], sourceHeader), header: f.header, currentRun: f.currentRun })).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps private executor scope/config verification mandatory when supplied", async () => {
|
||||
const f = await fixture();
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun,
|
||||
matchesExecution: () => false })).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["companyId", "issueId", "runId", "part"])("rejects encrypted source header with another %s", async key => {
|
||||
const f = await fixture();
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ ...f.database([f.source], f.sourceHeader, { [key]: "foreign" }),
|
||||
header: f.header, currentRun: f.currentRun })).toBeNull();
|
||||
});
|
||||
|
||||
it("does not accept a source lacking the current tool contract or provider identity", async () => {
|
||||
const f = await fixture();
|
||||
f.source.run.runnerProfileJson.nativeToolContractFingerprint = "old-contract";
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull();
|
||||
f.source.run.runnerProfileJson.nativeToolContractFingerprint = nativeToolContractFingerprintForTarget("remote", "codex");
|
||||
f.source.run.runnerProfileJson.sessionCheckpoint.providerSessionId = "";
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ ...f.database(), header: f.header, currentRun: f.currentRun })).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects corrupted or ordinary headers instead of promoting them to staged evidence", async () => {
|
||||
const f = await fixture();
|
||||
for (const header of [f.sourceHeader, "{}", "not-json"]) {
|
||||
const { db, select } = f.database();
|
||||
expect(await resolveNativeStagedAttachmentRecovery({ db, header, currentRun: f.currentRun })).toBeNull();
|
||||
expect(select).not.toHaveBeenCalled();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,124 @@
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
|
||||
import {
|
||||
inspectDurablePrpCheckpointHeader,
|
||||
inspectStagedRunAttachmentHeader,
|
||||
parseNativeExecutionInput,
|
||||
type DurableRecoveryIdentity,
|
||||
type NativeExecutionInput,
|
||||
type PersistedNativeSession,
|
||||
type StagedRunAttachment,
|
||||
} from "../../vendor/paperclip-runner/index.js";
|
||||
import { loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js";
|
||||
import { rebindNativeSessionCheckpoint } from "./native-session-resume.js";
|
||||
|
||||
type FrozenRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "companyId" | "agentId" | "nativeIssueId" | "nativeSessionId" | "runnerInstanceId" |
|
||||
"runtimeMode" | "status" | "runnerProfileJson" | "finishedAt">;
|
||||
|
||||
type StagedHeaderProof = NonNullable<ReturnType<typeof inspectStagedRunAttachmentHeader>>;
|
||||
export type NativeStagedAttachmentRecovery = Omit<StagedRunAttachment, "runAttachTemplate"> &
|
||||
Pick<StagedHeaderProof, "protocolIdentity" | "phase" | "receipt"> & {
|
||||
sourceRun: FrozenRun;
|
||||
sourceExecution: NativeExecutionInput;
|
||||
currentExecution: NativeExecutionInput;
|
||||
/** In-memory input only. It is not a published/resumable target snapshot. */
|
||||
checkpoint: PersistedNativeSession;
|
||||
/** A published target snapshot takes precedence over the source rebound. */
|
||||
hasTargetCheckpoint: boolean;
|
||||
targetCheckpoint: PersistedNativeSession | null;
|
||||
runnerVersion: string;
|
||||
runnerDigest: string;
|
||||
};
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value !== null && typeof value === "object" && !Array.isArray(value)
|
||||
? value as Record<string, unknown> : {};
|
||||
}
|
||||
|
||||
function boundTargetCheckpoint(run: FrozenRun, source: PersistedNativeSession): PersistedNativeSession | null {
|
||||
const snapshot = record(run.runnerProfileJson?.sessionCheckpoint), identity = record(snapshot.identity);
|
||||
if (typeof snapshot.sessionId !== "string" || !snapshot.sessionId || identity.runId !== run.id ||
|
||||
identity.companyId !== run.companyId || identity.issueId !== run.nativeIssueId || identity.agentId !== run.agentId ||
|
||||
identity.sessionId !== run.nativeSessionId) return null;
|
||||
// A target checkpoint may advance cursor/goal/tool progress, but cannot change
|
||||
// the provider session retained by this attachment. Do not rebound it again.
|
||||
if ((source.providerSessionId != null && source.providerSessionId !== snapshot.providerSessionId) ||
|
||||
(source.providerIdentity != null && !isDeepStrictEqual(source.providerIdentity, snapshot.providerIdentity))) return null;
|
||||
return structuredClone(snapshot) as unknown as PersistedNativeSession;
|
||||
}
|
||||
|
||||
function frozenExecution(run: FrozenRun): NativeExecutionInput | null {
|
||||
try {
|
||||
const execution = parseNativeExecutionInput(run.runnerProfileJson?.nativeExecutionInput);
|
||||
return run.runtimeMode === "native" && execution.binding.runId === run.id &&
|
||||
execution.binding.companyId === run.companyId && execution.binding.agentId === run.agentId &&
|
||||
execution.binding.issueId === run.nativeIssueId && execution.session.normalizedSessionId === run.nativeSessionId
|
||||
? execution : null;
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function matchesRun(identity: DurableRecoveryIdentity, run: FrozenRun): boolean {
|
||||
return identity.runId === run.id && identity.runnerInstanceId === run.runnerInstanceId &&
|
||||
identity.normalizedSessionId === run.nativeSessionId;
|
||||
}
|
||||
|
||||
/** Staging proves which prior attachment may be inspected, not that it is alive
|
||||
* or that work may be dispatched. Classification reads only two indexed point
|
||||
* lookups: the named source/result and its encrypted header. The executor must
|
||||
* re-read this proof with its private scope/config matcher before any RPC, then
|
||||
* independently prove the physical Runner and authenticate the target tuple. */
|
||||
export async function resolveNativeStagedAttachmentRecovery(input: {
|
||||
db: Pick<Db, "select">;
|
||||
currentRun: FrozenRun;
|
||||
header: string;
|
||||
expectedTarget?: DurableRecoveryIdentity;
|
||||
matchesExecution?: (previous: NativeExecutionInput, current: NativeExecutionInput) => boolean;
|
||||
}): Promise<NativeStagedAttachmentRecovery | null> {
|
||||
const proof = inspectStagedRunAttachmentHeader(input.header);
|
||||
const current = frozenExecution(input.currentRun);
|
||||
if (!proof || !current || current.provider.kind !== "codex" || !["running", "failed"].includes(input.currentRun.status) ||
|
||||
!matchesRun(proof.attachment.targetIdentity, input.currentRun) ||
|
||||
(input.expectedTarget && !isDeepStrictEqual(proof.attachment.targetIdentity, input.expectedTarget))) return null;
|
||||
const { attachment } = proof;
|
||||
const [source] = await input.db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations,
|
||||
resultId: nativeRunResults.id, resultStatus: nativeRunResults.schemaStatus }).from(heartbeatRuns)
|
||||
.innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.runId, heartbeatRuns.id),
|
||||
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId), eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId)))
|
||||
.innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId),
|
||||
eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.companyId, heartbeatRuns.companyId),
|
||||
eq(nativeRunResults.issueId, heartbeatRuns.nativeIssueId)))
|
||||
.where(and(eq(heartbeatRuns.id, attachment.sourceIdentity.runId), eq(heartbeatRuns.companyId, input.currentRun.companyId),
|
||||
eq(heartbeatRuns.agentId, input.currentRun.agentId), eq(heartbeatRuns.nativeIssueId, current.binding.issueId)))
|
||||
.limit(1);
|
||||
if (!source || source.run.companyId !== current.binding.companyId || source.run.agentId !== current.binding.agentId ||
|
||||
source.run.nativeIssueId !== current.binding.issueId || !matchesRun(attachment.sourceIdentity, source.run) ||
|
||||
source.run.status !== "succeeded" || source.run.finishedAt === null || source.coordinator.phase !== "committed" ||
|
||||
source.coordinator.companyId !== current.binding.companyId || source.coordinator.issueId !== current.binding.issueId ||
|
||||
source.coordinator.runId !== source.run.id || !source.coordinator.resultId ||
|
||||
source.coordinator.resultId !== source.resultId || source.resultStatus !== "accepted" ||
|
||||
source.coordinator.leaseOwner !== null || source.coordinator.leaseExpiresAt !== null) return null;
|
||||
const previous = frozenExecution(source.run);
|
||||
if (!previous || (input.matchesExecution && !input.matchesExecution(previous, current))) return null;
|
||||
// Reuse the existing provider, tool-contract, workspace, runtime-context and
|
||||
// normalized-session compatibility owner; no second set of digest formulas.
|
||||
const checkpoint = rebindNativeSessionCheckpoint({ previousRun: source.run, currentExecution: current, executionTargetKind: "remote" });
|
||||
if (!checkpoint || !(typeof checkpoint.providerSessionId === "string" && checkpoint.providerSessionId.length > 0 ||
|
||||
checkpoint.providerIdentity && Object.keys(checkpoint.providerIdentity).length > 0)) return null;
|
||||
const hasTargetCheckpoint = input.currentRun.runnerProfileJson?.sessionCheckpoint != null;
|
||||
const targetCheckpoint = hasTargetCheckpoint ? boundTargetCheckpoint(input.currentRun, checkpoint) : null;
|
||||
if (hasTargetCheckpoint && !targetCheckpoint) return null;
|
||||
const sourceHeader = await loadNativeRunnerCheckpointHeader(input.db, {
|
||||
companyId: source.run.companyId, issueId: current.binding.issueId, runId: source.run.id,
|
||||
});
|
||||
if (!sourceHeader || !isDeepStrictEqual(inspectDurablePrpCheckpointHeader(sourceHeader), attachment.sourceIdentity)) return null;
|
||||
// The codec above has validated this bounded header's artifact fields. A
|
||||
// staged target cannot silently select different bytes from its source.
|
||||
const sourceArtifact = JSON.parse(sourceHeader) as { runnerVersion: string; runnerDigest: string };
|
||||
if (sourceArtifact.runnerVersion !== proof.runnerVersion || sourceArtifact.runnerDigest !== proof.runnerDigest) return null;
|
||||
return { ...attachment, sourceRun: source.run, sourceExecution: previous, currentExecution: current,
|
||||
checkpoint, hasTargetCheckpoint, targetCheckpoint,
|
||||
protocolIdentity: proof.protocolIdentity, phase: proof.phase, receipt: proof.receipt,
|
||||
runnerVersion: proof.runnerVersion, runnerDigest: proof.runnerDigest };
|
||||
}
|
||||
@@ -12,8 +12,8 @@ import { agents, budgetPolicies, companies, costEvents, createDb, heartbeatRuns,
|
||||
import { type PrpEvent } from "../../vendor/paperclip-runner/index.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { accountRunCost } from "../run-cost-accounting.js";
|
||||
import { usageReceiptSpoolPath } from "../usage-receipts.js";
|
||||
import { createNativeUsageDelivery } from "./native-usage-delivery.js";
|
||||
import { usageReceiptSpoolPath, type UsageReceiptEnvelope } from "../usage-receipts.js";
|
||||
import { createNativeUsageContinuationDelivery, createNativeUsageDelivery } from "./native-usage-delivery.js";
|
||||
|
||||
const runnerRoot = fileURLToPath(new URL("../../../../packages/paperclip-runner/runner/", import.meta.url));
|
||||
const fixtureProfile = process.env.CI ? "release" : "debug";
|
||||
@@ -44,6 +44,89 @@ describe("producer-owned native usage", () => {
|
||||
payload: type === "usage.reported" ? { runDeltaAvailable: true, runDeltaBasis: "run", runDelta: { inputTokens: 120, outputTokens: 10, providerCostUsd: amount } } : {},
|
||||
} as PrpEvent);
|
||||
|
||||
const finalizedSource = async () => {
|
||||
const input = await seed();
|
||||
await db.update(heartbeatRuns).set({ runnerInstanceId: input.runnerInstanceId }).where(eq(heartbeatRuns.id, input.runId));
|
||||
const delivery = await createNativeUsageDelivery(db, input);
|
||||
const report = event(input, 1, "usage.reported"), done = event(input, 2, "turn.completed");
|
||||
await delivery.capture(report);
|
||||
await delivery.capture(done);
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.runId));
|
||||
expect(await accountRunCost(db, input.runId)).toBe(true);
|
||||
const { billing: _billing, ...source } = input;
|
||||
return { input, source, report, done };
|
||||
};
|
||||
|
||||
it("continues exact finalized source receipts without acquiring ownership or resetting accounting", async () => {
|
||||
const { input, source, report, done } = await finalizedSource();
|
||||
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
|
||||
await expect(createNativeUsageDelivery(db, input)).rejects.toThrow("native_usage_run_not_available");
|
||||
const continuation = await createNativeUsageContinuationDelivery(db, source);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before);
|
||||
await continuation.capture(report);
|
||||
await continuation.capture(done);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before);
|
||||
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(2);
|
||||
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId))).toHaveLength(1);
|
||||
await expect(continuation.capture({ ...report, sourceInstanceId: randomUUID() })).rejects.toThrow("native_usage_source_mismatch");
|
||||
await expect(continuation.capture({ ...report, payload: { ...report.payload as object, runDeltaAvailable: false } }))
|
||||
.rejects.toThrow("native_usage_source_conflict");
|
||||
});
|
||||
|
||||
it("journals an equal finalized source tail using the source billing without repricing", async () => {
|
||||
const { input, source } = await finalizedSource();
|
||||
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
|
||||
// Even a structurally supplied extra current-run classification cannot own
|
||||
// the old receipt: the continuation derives billing from saved source data.
|
||||
const withCurrentBilling = { ...source, billing: { provider: "anthropic", biller: "current-grant", billingType: "subscription_included" } };
|
||||
const continuation = await createNativeUsageContinuationDelivery(db, withCurrentBilling);
|
||||
await continuation.capture(event(input, 3, "usage.reported"));
|
||||
const receipts = await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId));
|
||||
expect(receipts).toHaveLength(3);
|
||||
expect(receipts.find(row => row.sequence === 3)?.receiptJson.receipt).toMatchObject(billing);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before);
|
||||
expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId)))[0].costCents).toBe(25);
|
||||
});
|
||||
|
||||
it.each(["tokens", "cost", "incomplete", "unpriceable", "basis"])(
|
||||
"holds a differing finalized source tail before acknowledgement: %s", async change => {
|
||||
const { input, source } = await finalizedSource();
|
||||
const continuation = await createNativeUsageContinuationDelivery(db, source);
|
||||
const late = event(input, 3, "usage.reported");
|
||||
if (change === "incomplete") (late.payload as Record<string, unknown>).runDeltaAvailable = false;
|
||||
if (change === "basis") delete (late.payload as Record<string, unknown>).runDeltaBasis;
|
||||
if (["tokens", "cost", "unpriceable"].includes(change)) (late.payload as Record<string, unknown>).runDelta = {
|
||||
inputTokens: change === "tokens" ? 150 : change === "unpriceable" ? "unknown" : 120,
|
||||
outputTokens: 10, providerCostUsd: change === "cost" ? 0.5 : 0.25,
|
||||
};
|
||||
await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation");
|
||||
await expect(continuation.capture(late)).rejects.toThrow("native_usage_after_finalization_requires_reconciliation");
|
||||
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(2);
|
||||
expect((await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, input.runId)))[0].costCents).toBe(25);
|
||||
});
|
||||
|
||||
it.each(["company", "run", "runner", "source"])("rejects a changed continuation binding without mutation: %s", async change => {
|
||||
const { input, source } = await finalizedSource();
|
||||
if (change === "source") await db.update(heartbeatRuns).set({ usageJson: sql`${heartbeatRuns.usageJson} || ${JSON.stringify({ accountingReceiptSourceId: randomUUID() })}::jsonb` })
|
||||
.where(eq(heartbeatRuns.id, input.runId));
|
||||
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
|
||||
await expect(createNativeUsageContinuationDelivery(db, { ...source,
|
||||
...(change === "company" ? { companyId: randomUUID() } : {}),
|
||||
...(change === "run" ? { runId: randomUUID() } : {}),
|
||||
...(change === "runner" ? { runnerInstanceId: randomUUID() } : {}),
|
||||
})).rejects.toThrow("native_usage_continuation_source_mismatch");
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))).toEqual(before);
|
||||
});
|
||||
|
||||
it("rejects changed immutable source receipt facts during hydration", async () => {
|
||||
const { input, source } = await finalizedSource();
|
||||
const [receipt] = await db.select().from(runUsageReceipts).where(and(eq(runUsageReceipts.runId, input.runId), eq(runUsageReceipts.sequence, 1)));
|
||||
const changed = structuredClone(receipt.receiptJson) as UsageReceiptEnvelope;
|
||||
changed.producerEvent!.usage!.runDelta = { inputTokens: 999, outputTokens: 10 };
|
||||
await db.update(runUsageReceipts).set({ receiptJson: changed }).where(eq(runUsageReceipts.id, receipt.id));
|
||||
await expect(createNativeUsageContinuationDelivery(db, source)).rejects.toThrow("native_usage_source_conflict");
|
||||
});
|
||||
|
||||
it("retains the same Runner report through a failed Postgres receipt and an empty replacement receiver", async () => {
|
||||
const input = await seed();
|
||||
expect(existsSync(usageReceiptSpoolPath())).toBe(false);
|
||||
|
||||
@@ -16,11 +16,13 @@ const uuid = (identity: string) => {
|
||||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-a${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
|
||||
};
|
||||
|
||||
export async function createNativeUsageDelivery(db: Db, input: {
|
||||
type NativeUsageDeliveryInput = {
|
||||
companyId: string; runId: string; runnerInstanceId: string;
|
||||
provider: NativeExecutionInput["provider"];
|
||||
billing: Pick<AdapterUsageCheckpoint, "provider" | "biller" | "billingType">;
|
||||
}) {
|
||||
};
|
||||
|
||||
export async function createNativeUsageDelivery(db: Db, input: NativeUsageDeliveryInput) {
|
||||
const sourceId = uuid(`${input.runId}:${input.runnerInstanceId}`);
|
||||
const [bound] = await db.update(heartbeatRuns).set({ usageJson: sql`
|
||||
coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || '{"accountingReceiptOwner":"native_producer"}'::jsonb ||
|
||||
@@ -29,9 +31,39 @@ export async function createNativeUsageDelivery(db: Db, input: {
|
||||
}).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId),
|
||||
isNull(heartbeatRuns.costAccountedAt))).returning({ id: heartbeatRuns.id });
|
||||
if (!bound) throw new Error("native_usage_run_not_available");
|
||||
const saved = await db.select({ receipt: runUsageReceipts.receiptJson }).from(runUsageReceipts).where(and(
|
||||
return createNativeUsageReceiver(db, input, sourceId, await loadNativeUsageReceipts(db, input, sourceId));
|
||||
}
|
||||
|
||||
/** Drain the authenticated previous producer's accounting without acquiring or
|
||||
* resetting its ownership. Finalized sources retain normal receipt equality
|
||||
* and reconciliation rules; this grants no business/tool execution authority. */
|
||||
export async function createNativeUsageContinuationDelivery(db: Db, input: Omit<NativeUsageDeliveryInput, "billing">) {
|
||||
const sourceId = uuid(`${input.runId}:${input.runnerInstanceId}`);
|
||||
const [source] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId, usage: heartbeatRuns.usageJson })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
|
||||
if (source?.runnerInstanceId !== input.runnerInstanceId || source.usage?.accountingReceiptOwner !== "native_producer"
|
||||
|| source.usage.accountingReceiptSourceId !== sourceId) throw new Error("native_usage_continuation_source_mismatch");
|
||||
const saved = await loadNativeUsageReceipts(db, input, sourceId);
|
||||
if (!saved.length) throw new Error("native_usage_continuation_receipts_missing");
|
||||
const classification = (receipt: AdapterUsageCheckpoint) => ({
|
||||
provider: receipt.provider, biller: receipt.biller, billingType: receipt.billingType,
|
||||
});
|
||||
const receipts = saved.map(row => (row.receipt as UsageReceiptEnvelope).receipt as AdapterUsageCheckpoint);
|
||||
const billing = classification(receipts[0]);
|
||||
if (receipts.some(receipt => receiptFingerprint(classification(receipt)) !== receiptFingerprint(billing)
|
||||
|| receipt.model !== (input.provider.model ?? "unknown"))) throw new Error("native_usage_source_conflict");
|
||||
return createNativeUsageReceiver(db, { ...input, billing }, sourceId, saved, true);
|
||||
}
|
||||
|
||||
function loadNativeUsageReceipts(db: Db, input: Pick<NativeUsageDeliveryInput, "companyId" | "runId">, sourceId: string) {
|
||||
return db.select({ receipt: runUsageReceipts.receiptJson, hash: runUsageReceipts.receiptHash,
|
||||
sequence: runUsageReceipts.sequence }).from(runUsageReceipts).where(and(
|
||||
eq(runUsageReceipts.companyId, input.companyId), eq(runUsageReceipts.runId, input.runId),
|
||||
eq(runUsageReceipts.sourceId, sourceId))).orderBy(asc(runUsageReceipts.sequence));
|
||||
}
|
||||
|
||||
function createNativeUsageReceiver(db: Db, input: NativeUsageDeliveryInput, sourceId: string,
|
||||
saved: Awaited<ReturnType<typeof loadNativeUsageReceipts>>, verifySaved = false) {
|
||||
const accepted = new Map<string, UsageReceiptEnvelope>();
|
||||
const turnAccounting = ["opencode", "acpx"].includes(input.provider.kind) ? createNativeTurnAccounting(input.provider) : null;
|
||||
let observedUsage: Record<string, unknown> | null = null;
|
||||
@@ -60,6 +92,12 @@ export async function createNativeUsageDelivery(db: Db, input: {
|
||||
for (const row of saved) {
|
||||
const envelope = row.receipt as UsageReceiptEnvelope;
|
||||
if (!envelope.producerEvent) throw new Error("native_usage_receipt_missing_producer");
|
||||
if (verifySaved && (envelope.companyId !== input.companyId || envelope.runId !== input.runId
|
||||
|| envelope.sourceId !== sourceId || envelope.sequence !== row.sequence || envelope.adapterType !== "paperclip_runner"
|
||||
|| envelope.id !== uuid(`${input.runId}:${input.runnerInstanceId}:${envelope.producerEvent.sourceEventId}`)
|
||||
|| receiptFingerprint(envelope) !== row.hash || accepted.has(envelope.producerEvent.sourceEventId))) {
|
||||
throw new Error("native_usage_source_conflict");
|
||||
}
|
||||
accepted.set(envelope.producerEvent.sourceEventId, envelope);
|
||||
observe(envelope);
|
||||
}
|
||||
|
||||
+3
@@ -94,6 +94,9 @@ export const RUNNER_PRP_PROTOCOL_MIN = runner.RUNNER_PRP_PROTOCOL_MIN;
|
||||
export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX;
|
||||
export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit;
|
||||
export const inspectWarmRunTransition = runner.inspectWarmRunTransition;
|
||||
export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment;
|
||||
export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader;
|
||||
export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment;
|
||||
export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding;
|
||||
export const readLinuxProcessStartedAt = runner.readLinuxProcessStartedAt;
|
||||
export const NativeSessionCloseUnrecoverableError =
|
||||
|
||||
Reference in new issue
Block a user