mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
feat(runner): add hidden server PRP coordinator (#12176)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip Runner needs a narrow server trust boundary before an adapter can start it. > - The package has durable runner transport, but the server does not host or authorize that transport. > - Native persistence exists, but no writer connects PRP events to those records. > - A direct adapter must not enter this path by accident. > - This pull request adds a hidden, run-bound PRP server coordinator. > - The benefit is a recoverable server boundary that remains unavailable to normal execution. ## Linked Issues or Issue Description Refs #11962 Refs #12129 Refs #12169 **Subsystem affected** Cross-cutting. The change affects the runner package and server orchestration. **Problem or motivation** The server cannot authenticate runnerd, commit PRP events before ACK, authorize semantic tools, or enter native finalization from a durable runner result. The application must have this hidden boundary before a guarded adapter can use the runner. **Proposed solution** Add an authenticated PRP WebSocket authority and register it only for one exact persisted native Codex run. Bind each connection and event to the company, issue, agent, run, runner, session, turn, item, and verified runner identity. Commit each event before its cumulative ACK. Project only authorized same-task read tools. Rebuild the accepted result and finalization record from durable result and terminal events. **Alternatives considered** The server could expose a broad runner API key or route semantic calls through existing adapter endpoints. Those options grant too much authority and weaken replay recovery. The server could also add the user-facing adapter in this pull request. That option would mix rollout selection with the transport trust boundary and make legacy compatibility harder to review. **Roadmap alignment** This work supports the shipped enforced-outcomes, governed-tool, and self-healing-run milestones. It does not add a new roadmap surface. ## What Changed - Add the durable PRP server authority with one-use bootstrap tickets, reconnect leases, encrypted frames, bounded state, cumulative ACKs, and idempotent commands. - Add `/api/runner/v1/connect/:runId`. Derive its `ws://` or `wss://` URL from the configured Paperclip API URL. - Register one authority only after the coordinator verifies the complete native Codex run binding. - Commit validated PRP events to `heartbeat_run_events` before ACK. Reject source gaps and conflicting replays. - Rebuild accepted results and finalization records from durable result and terminal events. Enforce finalization owner leases and retry times. - Project five same-task read operations. Recheck run, agent, task, and company authority for each call. - Keep the route hidden. No adapter selects this coordinator, and no code starts runnerd. - Vendor the compiled runner TypeScript runtime into the server package while keeping the workspace package development-only for the server. - Document the package, database writer, run-log payload, and credential exclusions. ## Verification - Run `pnpm --filter @paperclipai/paperclip-runner check:all`. All TypeScript protocol checks and 69 Vitest tests pass, including commit-before-ACK crash recovery. All 43 Rust unit tests and 13 Rust integration tests pass. Conformance and replay parity pass. - Run the focused server WebSocket, coordinator, package-build, and startup-wiring suites. All 26 tests pass, including a clean-checkout reproduction with the runner `dist` directory absent. - Run `pnpm -r typecheck`. - Run `pnpm test:run`. - Run `pnpm build`. - Confirm that the diff contains 19 files. Confirm that it contains no workflow or `pnpm-lock.yaml` change. ## Risks - The server installs the WebSocket route at startup. An unregistered or malformed run path fails closed and creates no native record. - Bootstrap tickets are one use. The private state directory uses mode `0700`, and the state file uses mode `0600`. The file stores derived authentication verifiers and never stores raw tickets or lease tokens. - The journal has explicit frame, command, event-window, and file-size bounds. A bound violation closes the runner connection or rejects the command. - A runner event reaches the database before its ACK. A crash between event commit and ACK causes a byte-equivalent replay, not a second logical effect. - The coordinator accepts only an existing queued or running native Codex row with exact company, task, agent, runner, session, and completion-contract ownership. - Existing direct adapters do not call this service. They keep their current execution, transcript, result, and finalization paths. - The server has no production dependency on the private runner package. Its build copies the compiled runtime into `server/dist`; the workspace link is development-only. This adds no external package and does not change the lockfile. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with GPT-5. The exact deployment ID and context-window size are not exposed. The model used agentic reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta authored and GitHub committed
1 parent
86fe9339e1
commit
9964b034bb
19 files changed
+4129
-15
No files matched your search
@@ -2,7 +2,7 @@
|
||||
|
||||
This private workspace package contains the staged Paperclip Runner work.
|
||||
|
||||
The package currently exposes only the language-neutral PRP v1 TypeScript
|
||||
The package currently exposes the language-neutral PRP v1 TypeScript
|
||||
contract, provider-neutral structured questions and responses, deterministic
|
||||
fixture validation/replay, structured-result normalization, and the session
|
||||
reducer oracle. It also contains a package-local Rust runner, scripted fake
|
||||
@@ -11,27 +11,35 @@ PRP transport. The transport authenticates and encrypts loopback WebSocket
|
||||
sessions, persists an ACK-driven outbox and command journal, and reconnects with
|
||||
a short-lived lease. The Rust runner now includes a Codex-only app-server
|
||||
provider bridge with durable thread resume, cancellation, structured questions,
|
||||
and provider-neutral event normalization. No server code starts or invokes it.
|
||||
and provider-neutral event normalization. The root surface now also exposes an
|
||||
authenticated durable PRP authority for server-side use. It stores only
|
||||
bootstrap and reconnect credential digests, validates immutable run identity on
|
||||
every connection and event, and persists commands and cumulative event ACK
|
||||
state across server restarts.
|
||||
The package also publishes the canonical semantic action declarations and
|
||||
their input and output schemas. Its package-local dispatcher projects only
|
||||
bound, run-authorized actions and emits redacted semantic receipts. It does not
|
||||
add application bindings, a server adapter, or production Paperclip behavior.
|
||||
|
||||
The first and only installed provider is Codex. Dynamic semantic tools remain
|
||||
undiscoverable because no production application binding or server authority
|
||||
has landed. Catalog membership alone does not grant authority. See
|
||||
undiscoverable unless the hidden server coordinator projects one of the five
|
||||
same-task read bindings for an already persisted native Codex run. Catalog
|
||||
membership alone does not grant authority, and no production adapter can create
|
||||
or start such a run yet. See
|
||||
[`SEMANTIC_ACTIONS.md`](SEMANTIC_ACTIONS.md) for the catalog boundary.
|
||||
|
||||
The package has two initial public surfaces:
|
||||
|
||||
- `@paperclipai/paperclip-runner` contains runtime contracts, validation,
|
||||
replay/reducer logic, the semantic catalog, and the authorization dispatcher.
|
||||
replay/reducer logic, the semantic catalog, the authorization dispatcher, and
|
||||
the Node-only durable server authority.
|
||||
- `@paperclipai/paperclip-runner/testing` adds Node-only fixture loading and a
|
||||
provider-neutral semantic conformance kit for deterministic test adapters.
|
||||
|
||||
No SDK, browser, React, eval, live-console, lab, or provider-experiment entry
|
||||
point is exported. The package remains private in this wave, and no production
|
||||
adapter starts it yet.
|
||||
point is exported. The package remains private in this wave. The server route
|
||||
at `/api/runner/v1/connect/:runId` has no authority until the hidden coordinator
|
||||
registers an exact existing run binding, and no production adapter starts it.
|
||||
|
||||
Run the complete contract gate with:
|
||||
|
||||
|
||||
@@ -0,0 +1,618 @@
|
||||
import {
|
||||
createCipheriv,
|
||||
createDecipheriv,
|
||||
createHash,
|
||||
createHmac,
|
||||
} from "node:crypto";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { connect, type Socket } from "node:net";
|
||||
import { tmpdir } from "node:os";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { validatePrpEvent } from "../protocol/replay-contract.js";
|
||||
import { digestPaperclipSemanticContent } from "../semantic-tools/receipts.js";
|
||||
import { DurablePrpControlPlane } from "./durable-prp-control-plane.js";
|
||||
import type { DurableRecoveryIdentity } from "./prp-transport-types.js";
|
||||
|
||||
const identity: DurableRecoveryIdentity = {
|
||||
runnerInstanceId: "runner-test-1",
|
||||
environmentLeaseId: "environment-test-1",
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
normalizedSessionId: "session-test-1",
|
||||
turnId: "turn-test-1",
|
||||
itemId: "item-test-1",
|
||||
};
|
||||
const expectedRunnerVersion = "0.3.0";
|
||||
const expectedRunnerDigest = `sha256:${"a".repeat(64)}`;
|
||||
|
||||
function domainDigest(domain: string, parts: readonly Buffer[]): Buffer {
|
||||
const digest = createHash("sha256")
|
||||
.update(domain)
|
||||
.update(Buffer.from([0]));
|
||||
for (const part of parts) {
|
||||
const length = Buffer.alloc(8);
|
||||
length.writeBigUInt64BE(BigInt(part.length));
|
||||
digest.update(length).update(part);
|
||||
}
|
||||
return digest.digest();
|
||||
}
|
||||
|
||||
function domainHmac(
|
||||
key: Buffer,
|
||||
domain: string,
|
||||
parts: readonly Buffer[],
|
||||
): Buffer {
|
||||
const digest = createHmac("sha256", key)
|
||||
.update(domain)
|
||||
.update(Buffer.from([0]));
|
||||
for (const part of parts) {
|
||||
const length = Buffer.alloc(8);
|
||||
length.writeBigUInt64BE(BigInt(part.length));
|
||||
digest.update(length).update(part);
|
||||
}
|
||||
return digest.digest();
|
||||
}
|
||||
|
||||
function canonicalJson(value: unknown): string {
|
||||
if (Array.isArray(value)) return `[${value.map(canonicalJson).join(",")}]`;
|
||||
if (typeof value === "object" && value !== null) {
|
||||
const record = value as Record<string, unknown>;
|
||||
return `{${Object.keys(record)
|
||||
.sort()
|
||||
.map((key) => `${JSON.stringify(key)}:${canonicalJson(record[key])}`)
|
||||
.join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function credentialMaterial(token: string): {
|
||||
credentialId: string;
|
||||
authKey: Buffer;
|
||||
} {
|
||||
const bytes = Buffer.from(token);
|
||||
const authKey = domainDigest("paperclip-runner-auth-key-v1", [bytes]);
|
||||
return {
|
||||
credentialId: `sha256:${domainDigest("paperclip-runner-credential-id-v1", [bytes]).toString("hex")}`,
|
||||
authKey,
|
||||
};
|
||||
}
|
||||
|
||||
class ServerFrameReader {
|
||||
#buffer = Buffer.alloc(0);
|
||||
#frames: Array<Record<string, unknown> | null> = [];
|
||||
#waiters: Array<(value: Record<string, unknown> | null) => void> = [];
|
||||
|
||||
constructor(socket: Socket) {
|
||||
socket.on("data", (chunk: Buffer) => {
|
||||
this.#buffer = Buffer.concat([this.#buffer, chunk]);
|
||||
this.#drain();
|
||||
});
|
||||
socket.once("close", () => this.#publish(null));
|
||||
}
|
||||
|
||||
next(): Promise<Record<string, unknown> | null> {
|
||||
const queued = this.#frames.shift();
|
||||
if (queued !== undefined) return Promise.resolve(queued);
|
||||
return new Promise((resolveFrame, rejectFrame) => {
|
||||
const timer = setTimeout(
|
||||
() => rejectFrame(new Error("Server WebSocket frame timed out.")),
|
||||
2_000,
|
||||
);
|
||||
this.#waiters.push((value) => {
|
||||
clearTimeout(timer);
|
||||
resolveFrame(value);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
#publish(value: Record<string, unknown> | null): void {
|
||||
const waiter = this.#waiters.shift();
|
||||
if (waiter) waiter(value);
|
||||
else this.#frames.push(value);
|
||||
}
|
||||
|
||||
#drain(): void {
|
||||
while (this.#buffer.length >= 2) {
|
||||
let length = this.#buffer[1]! & 0x7f;
|
||||
let cursor = 2;
|
||||
if (length === 126) {
|
||||
if (this.#buffer.length < 4) return;
|
||||
length = this.#buffer.readUInt16BE(2);
|
||||
cursor = 4;
|
||||
} else if (length === 127) {
|
||||
if (this.#buffer.length < 10) return;
|
||||
length = Number(this.#buffer.readBigUInt64BE(2));
|
||||
cursor = 10;
|
||||
}
|
||||
if (this.#buffer.length < cursor + length) return;
|
||||
const payload = this.#buffer.subarray(cursor, cursor + length);
|
||||
this.#buffer = this.#buffer.subarray(cursor + length);
|
||||
this.#publish(
|
||||
JSON.parse(payload.toString("utf8")) as Record<string, unknown>,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function upgradeSocket(url: string): Promise<{
|
||||
socket: Socket;
|
||||
reader: ServerFrameReader;
|
||||
}> {
|
||||
const parsed = new URL(url);
|
||||
const socket = await new Promise<Socket>((resolveSocket, rejectSocket) => {
|
||||
const candidate = connect(Number(parsed.port), parsed.hostname);
|
||||
let response = Buffer.alloc(0);
|
||||
const onData = (chunk: Buffer): void => {
|
||||
response = Buffer.concat([response, chunk]);
|
||||
const boundary = response.indexOf("\r\n\r\n");
|
||||
if (boundary < 0) return;
|
||||
candidate.off("data", onData);
|
||||
const status = Number(
|
||||
response.toString("utf8").match(/^HTTP\/1\.1 (\d{3})/)?.[1],
|
||||
);
|
||||
if (status !== 101) {
|
||||
candidate.destroy();
|
||||
rejectSocket(new Error(`WebSocket upgrade returned ${String(status)}`));
|
||||
} else {
|
||||
resolveSocket(candidate);
|
||||
}
|
||||
};
|
||||
candidate.once("error", rejectSocket);
|
||||
candidate.once("connect", () => {
|
||||
candidate.write(
|
||||
[
|
||||
`GET ${parsed.pathname} HTTP/1.1`,
|
||||
`Host: ${parsed.host}`,
|
||||
"Upgrade: websocket",
|
||||
"Connection: Upgrade",
|
||||
"Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==",
|
||||
"Sec-WebSocket-Version: 13",
|
||||
"\r\n",
|
||||
].join("\r\n"),
|
||||
);
|
||||
});
|
||||
candidate.on("data", onData);
|
||||
});
|
||||
return { socket, reader: new ServerFrameReader(socket) };
|
||||
}
|
||||
|
||||
function sendMaskedJson(socket: Socket, value: unknown): void {
|
||||
const payload = Buffer.from(JSON.stringify(value));
|
||||
const mask = Buffer.from([0x11, 0x22, 0x33, 0x44]);
|
||||
const header: number[] = [0x81];
|
||||
if (payload.length <= 125) {
|
||||
header.push(0x80 | payload.length);
|
||||
} else if (payload.length <= 0xffff) {
|
||||
header.push(0x80 | 126, payload.length >>> 8, payload.length & 0xff);
|
||||
} else {
|
||||
throw new Error("Test client frame exceeds the supported size.");
|
||||
}
|
||||
const masked = Buffer.from(payload);
|
||||
for (let index = 0; index < masked.length; index += 1) {
|
||||
masked[index] = masked[index]! ^ mask[index % mask.length]!;
|
||||
}
|
||||
socket.write(Buffer.concat([Buffer.from(header), mask, masked]));
|
||||
}
|
||||
|
||||
interface AuthenticatedClient {
|
||||
socket: Socket;
|
||||
reader: ServerFrameReader;
|
||||
authKey: Buffer;
|
||||
sessionId: string;
|
||||
sendCounter: bigint;
|
||||
receiveCounter: bigint;
|
||||
leaseToken: string | null;
|
||||
welcome: Record<string, unknown>;
|
||||
}
|
||||
|
||||
function authHello(
|
||||
credentialId: string,
|
||||
selectedIdentity: DurableRecoveryIdentity = identity,
|
||||
): Record<string, unknown> {
|
||||
return {
|
||||
protocol: "paperclip.runner",
|
||||
version: 1,
|
||||
kind: "auth_hello",
|
||||
payload: {
|
||||
credentialId,
|
||||
clientNonce: "client-nonce-test",
|
||||
protocolMin: 1,
|
||||
protocolMax: 1,
|
||||
...selectedIdentity,
|
||||
runnerVersion: expectedRunnerVersion,
|
||||
runnerDigest: expectedRunnerDigest,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function authenticate(
|
||||
controlPlane: DurablePrpControlPlane,
|
||||
token: string,
|
||||
selectedIdentity: DurableRecoveryIdentity = identity,
|
||||
): Promise<AuthenticatedClient | null> {
|
||||
const { socket, reader } = await upgradeSocket(controlPlane.connectUrl);
|
||||
const material = credentialMaterial(token);
|
||||
sendMaskedJson(socket, authHello(material.credentialId, selectedIdentity));
|
||||
const challenge = await reader.next();
|
||||
if (challenge === null) return null;
|
||||
const challengePayload = challenge.payload as Record<string, unknown>;
|
||||
const serverProof = challengePayload.serverProof;
|
||||
if (typeof serverProof !== "string") throw new Error("Missing server proof.");
|
||||
const canonicalPayload = { ...challengePayload };
|
||||
delete canonicalPayload.serverProof;
|
||||
const canonicalChallenge = canonicalJson(canonicalPayload);
|
||||
const expectedServerProof = domainHmac(
|
||||
material.authKey,
|
||||
"paperclip-runner-server-proof-v1",
|
||||
[Buffer.from(canonicalChallenge)],
|
||||
).toString("hex");
|
||||
expect(serverProof).toBe(expectedServerProof);
|
||||
const clientProof = domainHmac(
|
||||
material.authKey,
|
||||
"paperclip-runner-client-proof-v1",
|
||||
[Buffer.from(canonicalChallenge), Buffer.from(serverProof)],
|
||||
).toString("hex");
|
||||
sendMaskedJson(socket, {
|
||||
protocol: "paperclip.runner",
|
||||
version: 1,
|
||||
kind: "auth_response",
|
||||
payload: {
|
||||
credentialId: material.credentialId,
|
||||
clientNonce: challengePayload.clientNonce,
|
||||
serverNonce: challengePayload.serverNonce,
|
||||
clientProof,
|
||||
},
|
||||
});
|
||||
const binding = domainDigest("paperclip-runner-session-binding-v1", [
|
||||
Buffer.from(canonicalChallenge),
|
||||
Buffer.from(serverProof),
|
||||
Buffer.from(clientProof),
|
||||
]);
|
||||
const client: AuthenticatedClient = {
|
||||
socket,
|
||||
reader,
|
||||
authKey: material.authKey,
|
||||
sessionId: `sha256:${binding.toString("hex")}`,
|
||||
sendCounter: 0n,
|
||||
receiveCounter: 0n,
|
||||
leaseToken: null,
|
||||
welcome: {},
|
||||
};
|
||||
const welcome = await receiveSecure(client);
|
||||
if (welcome === null) return null;
|
||||
expect(welcome.kind).toBe("welcome");
|
||||
client.welcome = welcome;
|
||||
const leaseToken = (welcome.payload as Record<string, unknown>)
|
||||
.connectionLeaseToken;
|
||||
client.leaseToken = typeof leaseToken === "string" ? leaseToken : null;
|
||||
return client;
|
||||
}
|
||||
|
||||
function secureNonce(prefix: "P3C1" | "P3S1", counter: bigint): Buffer {
|
||||
const nonce = Buffer.alloc(12);
|
||||
nonce.write(prefix, 0, "ascii");
|
||||
nonce.writeBigUInt64BE(counter, 4);
|
||||
return nonce;
|
||||
}
|
||||
|
||||
function secureAad(
|
||||
client: AuthenticatedClient,
|
||||
direction: "client_to_core" | "core_to_client",
|
||||
counter: bigint,
|
||||
): Buffer {
|
||||
return Buffer.from(
|
||||
`paperclip.runner.secure-frame.v1\0${client.sessionId}\0${direction}\0${counter}`,
|
||||
);
|
||||
}
|
||||
|
||||
function sendSecure(
|
||||
client: AuthenticatedClient,
|
||||
value: Record<string, unknown>,
|
||||
): void {
|
||||
const binding = Buffer.from(client.sessionId.slice("sha256:".length), "hex");
|
||||
const key = domainHmac(
|
||||
client.authKey,
|
||||
"paperclip-runner-client-to-core-key-v1",
|
||||
[binding],
|
||||
);
|
||||
const counter = client.sendCounter;
|
||||
const cipher = createCipheriv(
|
||||
"aes-256-gcm",
|
||||
key,
|
||||
secureNonce("P3C1", counter),
|
||||
);
|
||||
cipher.setAAD(secureAad(client, "client_to_core", counter));
|
||||
const encrypted = Buffer.concat([
|
||||
cipher.update(Buffer.from(JSON.stringify(value))),
|
||||
cipher.final(),
|
||||
cipher.getAuthTag(),
|
||||
]);
|
||||
sendMaskedJson(client.socket, {
|
||||
schema: "paperclip.runner.secure-frame.v1",
|
||||
counter: Number(counter),
|
||||
ciphertext: encrypted.toString("hex"),
|
||||
});
|
||||
client.sendCounter += 1n;
|
||||
}
|
||||
|
||||
async function receiveSecure(
|
||||
client: AuthenticatedClient,
|
||||
): Promise<Record<string, unknown> | null> {
|
||||
const frame = await client.reader.next();
|
||||
if (frame === null) return null;
|
||||
const counter = BigInt(frame.counter as number);
|
||||
expect(counter).toBe(client.receiveCounter);
|
||||
const binding = Buffer.from(client.sessionId.slice("sha256:".length), "hex");
|
||||
const key = domainHmac(
|
||||
client.authKey,
|
||||
"paperclip-runner-core-to-client-key-v1",
|
||||
[binding],
|
||||
);
|
||||
const sealed = Buffer.from(String(frame.ciphertext), "hex");
|
||||
const decipher = createDecipheriv(
|
||||
"aes-256-gcm",
|
||||
key,
|
||||
secureNonce("P3S1", counter),
|
||||
);
|
||||
decipher.setAAD(secureAad(client, "core_to_client", counter));
|
||||
decipher.setAuthTag(sealed.subarray(-16));
|
||||
const value = JSON.parse(
|
||||
Buffer.concat([
|
||||
decipher.update(sealed.subarray(0, -16)),
|
||||
decipher.final(),
|
||||
]).toString("utf8"),
|
||||
) as Record<string, unknown>;
|
||||
client.receiveCounter += 1n;
|
||||
return value;
|
||||
}
|
||||
|
||||
function semanticInputEvent(sourceSeq = 1): Record<string, unknown> {
|
||||
return {
|
||||
protocol: "paperclip.runner",
|
||||
version: 1,
|
||||
kind: "event",
|
||||
runnerInstanceId: identity.runnerInstanceId,
|
||||
environmentLeaseId: identity.environmentLeaseId,
|
||||
runId: identity.runId,
|
||||
normalizedSessionId: identity.normalizedSessionId,
|
||||
turnId: identity.turnId,
|
||||
itemId: identity.itemId,
|
||||
payload: {
|
||||
sourceSeq,
|
||||
sourceEventId: `semantic-event-${sourceSeq}`,
|
||||
sourceInstanceId: identity.runnerInstanceId,
|
||||
sourceKind: "runner",
|
||||
runId: identity.runId,
|
||||
normalizedSessionId: identity.normalizedSessionId,
|
||||
turnId: identity.turnId,
|
||||
itemId: identity.itemId,
|
||||
eventType: "semantic_tool.input",
|
||||
schema: "paperclip.prp.event.v1",
|
||||
schemaVersion: 1,
|
||||
priority: 0,
|
||||
emittedAt: "2026-08-25T18:00:00.000Z",
|
||||
payload: {
|
||||
semantic_tool: {
|
||||
schema: "paperclip.prp.semantic_tool.v1",
|
||||
schemaVersion: 1,
|
||||
phase: "input",
|
||||
callId: "call-1",
|
||||
operationId: "get_task_context",
|
||||
correlation: {
|
||||
runId: identity.runId,
|
||||
normalizedSessionId: identity.normalizedSessionId,
|
||||
turnId: identity.turnId,
|
||||
itemId: identity.itemId,
|
||||
},
|
||||
idempotencyKey: null,
|
||||
content: {
|
||||
digest: digestPaperclipSemanticContent({}),
|
||||
redactionDisposition: "digest_only",
|
||||
references: [],
|
||||
},
|
||||
input: {},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe.sequential("DurablePrpControlPlane", () => {
|
||||
it("exchanges a one-use bootstrap for a run-bound reconnect lease", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "paperclip-prp-auth-"));
|
||||
const controlPlane = new DurablePrpControlPlane({
|
||||
stateDirectory: root,
|
||||
identity,
|
||||
expectedRunnerVersion,
|
||||
expectedRunnerDigest,
|
||||
});
|
||||
try {
|
||||
await controlPlane.start();
|
||||
const cancel = controlPlane.queueCommand(
|
||||
"run.cancel",
|
||||
{ reason: "test" },
|
||||
"command-cancel-1",
|
||||
);
|
||||
expect(
|
||||
controlPlane.queueCommand(
|
||||
"run.cancel",
|
||||
{ reason: "test" },
|
||||
"command-cancel-1",
|
||||
),
|
||||
).toEqual(cancel);
|
||||
expect(() =>
|
||||
controlPlane.queueCommand(
|
||||
"run.cancel",
|
||||
{ reason: "different" },
|
||||
"command-cancel-1",
|
||||
),
|
||||
).toThrow("command replay conflicts");
|
||||
expect(() => controlPlane.queueCommand("unknown.command")).toThrow(
|
||||
"command is invalid",
|
||||
);
|
||||
const ticket = controlPlane.issueBootstrapTicket();
|
||||
const first = await authenticate(controlPlane, ticket);
|
||||
expect(first?.leaseToken).toEqual(expect.any(String));
|
||||
first?.socket.destroy();
|
||||
|
||||
const reused = await authenticate(controlPlane, ticket);
|
||||
expect(reused).toBeNull();
|
||||
|
||||
const lease = await authenticate(controlPlane, first!.leaseToken!);
|
||||
expect(lease).not.toBeNull();
|
||||
expect(lease?.leaseToken).toBeNull();
|
||||
lease?.socket.destroy();
|
||||
|
||||
const wrongRun = await authenticate(
|
||||
controlPlane,
|
||||
controlPlane.issueBootstrapTicket(),
|
||||
{
|
||||
...identity,
|
||||
runId: "00000000-0000-4000-8000-000000000999",
|
||||
},
|
||||
);
|
||||
expect(wrongRun).toBeNull();
|
||||
} finally {
|
||||
await controlPlane.stop();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("recovers one semantic call from the durable event after a coordinator restart", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "paperclip-prp-recovery-"));
|
||||
let firstCalls = 0;
|
||||
const first = new DurablePrpControlPlane({
|
||||
stateDirectory: root,
|
||||
identity,
|
||||
expectedRunnerVersion,
|
||||
expectedRunnerDigest,
|
||||
onSemanticToolInput: async () => {
|
||||
firstCalls += 1;
|
||||
return new Promise(() => undefined);
|
||||
},
|
||||
});
|
||||
let leaseToken: string;
|
||||
try {
|
||||
await first.start();
|
||||
const client = await authenticate(first, first.issueBootstrapTicket());
|
||||
leaseToken = client!.leaseToken!;
|
||||
const validation = validatePrpEvent(semanticInputEvent().payload);
|
||||
expect(validation, JSON.stringify(validation)).toMatchObject({
|
||||
ok: true,
|
||||
});
|
||||
sendSecure(client!, semanticInputEvent());
|
||||
const ack = await receiveSecure(client!);
|
||||
expect(ack).toMatchObject({ kind: "ack" });
|
||||
expect(firstCalls).toBe(1);
|
||||
client?.socket.destroy();
|
||||
} finally {
|
||||
await first.stop();
|
||||
}
|
||||
|
||||
let recoveredCalls = 0;
|
||||
const recovered = new DurablePrpControlPlane({
|
||||
stateDirectory: root,
|
||||
identity,
|
||||
expectedRunnerVersion,
|
||||
expectedRunnerDigest,
|
||||
onSemanticToolInput: async (call) => {
|
||||
recoveredCalls += 1;
|
||||
return {
|
||||
result: { ok: true, operationId: call.operationId },
|
||||
};
|
||||
},
|
||||
});
|
||||
try {
|
||||
await recovered.start();
|
||||
const client = await authenticate(recovered, leaseToken!);
|
||||
sendSecure(client!, semanticInputEvent());
|
||||
const outcomes = [
|
||||
await receiveSecure(client!),
|
||||
await receiveSecure(client!),
|
||||
];
|
||||
const command = outcomes.find((outcome) => outcome?.kind === "command");
|
||||
expect(outcomes.some((outcome) => outcome?.kind === "ack")).toBe(true);
|
||||
expect(command?.payload).toMatchObject({
|
||||
type: "semantic_tool.result",
|
||||
payload: {
|
||||
callId: "call-1",
|
||||
operationId: "get_task_context",
|
||||
result: { ok: true, operationId: "get_task_context" },
|
||||
isError: false,
|
||||
},
|
||||
});
|
||||
expect(recoveredCalls).toBe(1);
|
||||
|
||||
const tampered = semanticInputEvent(2);
|
||||
const tamperedEvent = tampered.payload as Record<string, unknown>;
|
||||
const tamperedPayload = tamperedEvent.payload as Record<string, unknown>;
|
||||
const tamperedSemantic = tamperedPayload.semantic_tool as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
tamperedSemantic.content = {
|
||||
...(tamperedSemantic.content as Record<string, unknown>),
|
||||
digest: `sha256:${"0".repeat(64)}`,
|
||||
};
|
||||
sendSecure(client!, tampered);
|
||||
await expect(receiveSecure(client!)).resolves.toBeNull();
|
||||
client?.socket.destroy();
|
||||
} finally {
|
||||
await recovered.stop();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("does not acknowledge an event before the caller commits it", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "paperclip-prp-commit-order-"));
|
||||
const first = new DurablePrpControlPlane({
|
||||
stateDirectory: root,
|
||||
identity,
|
||||
expectedRunnerVersion,
|
||||
expectedRunnerDigest,
|
||||
onSemanticToolInput: async () => ({ result: { ok: true } }),
|
||||
onCommittedEvent: async () => {
|
||||
throw new Error("database commit failed");
|
||||
},
|
||||
});
|
||||
let leaseToken: string;
|
||||
try {
|
||||
await first.start();
|
||||
const client = await authenticate(first, first.issueBootstrapTicket());
|
||||
leaseToken = client!.leaseToken!;
|
||||
sendSecure(client!, semanticInputEvent());
|
||||
await expect(receiveSecure(client!)).resolves.toBeNull();
|
||||
} finally {
|
||||
await first.stop();
|
||||
}
|
||||
|
||||
let committed = 0;
|
||||
const recovered = new DurablePrpControlPlane({
|
||||
stateDirectory: root,
|
||||
identity,
|
||||
expectedRunnerVersion,
|
||||
expectedRunnerDigest,
|
||||
onSemanticToolInput: async () => ({ result: { ok: true } }),
|
||||
onCommittedEvent: async () => {
|
||||
committed += 1;
|
||||
},
|
||||
});
|
||||
try {
|
||||
await recovered.start();
|
||||
const client = await authenticate(recovered, leaseToken!);
|
||||
expect(client?.welcome.payload).toMatchObject({ ackedSourceSeq: 0 });
|
||||
sendSecure(client!, semanticInputEvent());
|
||||
await expect(receiveSecure(client!)).resolves.toMatchObject({
|
||||
kind: "ack",
|
||||
payload: { ackedSourceSeq: 1 },
|
||||
});
|
||||
expect(committed).toBe(1);
|
||||
client?.socket.destroy();
|
||||
} finally {
|
||||
await recovered.stop();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,29 @@
|
||||
export interface DurableRecoveryIdentity {
|
||||
runnerInstanceId: string;
|
||||
environmentLeaseId: string;
|
||||
runId: string;
|
||||
normalizedSessionId: string;
|
||||
turnId: string;
|
||||
itemId: string;
|
||||
}
|
||||
|
||||
export interface DurableRecoveryCoreCommand {
|
||||
schema: "paperclip.prp.command.v1";
|
||||
commandId: string;
|
||||
controllerSeq: number;
|
||||
type: string;
|
||||
issuedAt: string;
|
||||
payload: Record<string, unknown>;
|
||||
status: "pending" | "completed" | "failed" | "rejected";
|
||||
result: Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
export interface DurableRecoveryCommittedEvent {
|
||||
sourceSeq: number;
|
||||
sourceEventId: string;
|
||||
eventType: string;
|
||||
priority: 0 | 1 | 2;
|
||||
envelope: Record<string, unknown>;
|
||||
deliveryCount: number;
|
||||
logicalEffectCount: number;
|
||||
}
|
||||
@@ -1,6 +1,11 @@
|
||||
export * from "./catalog/index.js";
|
||||
export * from "./contracts/completion-result.js";
|
||||
export * from "./contracts/question-set.js";
|
||||
export {
|
||||
DurablePrpControlPlane,
|
||||
type DurablePrpControlPlaneOptions,
|
||||
} from "./control-plane/durable-prp-control-plane.js";
|
||||
export type { DurableRecoveryIdentity } from "./control-plane/prp-transport-types.js";
|
||||
export * from "./protocol/replay-contract.js";
|
||||
export * from "./protocol/result-normalization.js";
|
||||
export * from "./reducer/session-reducer.js";
|
||||
|
||||
Reference in new issue
Block a user