From 125ddcabe51b2850cac2c64ee111a529d0578673 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:19:36 -0500 Subject: [PATCH] perf(runner): keep event journal writes off the HTTP thread (#17745) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - The HTTP server also owns durable Runner state. > - Each committed event rewrites and flushes that state on the HTTP thread. > - Large journals and slow writes delay unrelated navigation and saves. > - Event acknowledgements must still wait for durable storage. > - This PR stages output writes asynchronously and joins them before writer replacement. > - The board can serve requests while Runner output waits for storage. ## Linked Issues or Issue Description Refs: #17507. Related: #17514 reduces writes per event burst. This PR moves individual output writes and file flushes outside the HTTP thread. **What happened?** A live CPU profile spent about three seconds of a 13-second sample serializing and synchronously writing Runner state. Representative journals were about 9 MB. Unrelated task requests had multi-second delays. **Expected behavior** A slow output-journal write must leave unrelated HTTP requests responsive. An event must not be acknowledged before it is durable. **Steps to reproduce** Hold an authenticated output event's file flush in the durable protocol test. Send an unrelated HTTP request during that flush. Also save a command, reconnect, or retire the controller during the write. ## What Changed - Stage `item.delta` snapshots with asynchronous file writes and file flushes. Keep lifecycle, control, authentication, and tool events synchronous so transport readers observe durable boundaries. - Fence publication by saved revision. An older snapshot cannot replace newer commands or connection state. A failed write fences the writer. - Write compact JSON and share private-file checks. Existing journals remain readable. - Stop external ingress before joining admitted writes. Checkpoint only after retirement succeeds. Close reports a failed detach consistently and releases notification consumers. - Add real HTTP and Rust transport cases for held writes, reconnects, commands, failure, restart, close, and timeout. - Add immutable Copilot source identity v61 for the changed shared transport. Preserve v60, historical decoding, exact-current launch checks, and qualification flags. No new live Copilot qualification is claimed. ## Verification - Head `6f5c92e778`: workspace `pnpm -r typecheck` and `pnpm build` passed. All CI gates passed in [run 40356159484](https://github.com/paperclipai/paperclip/actions/runs/40356159484), including both full Runner shards. Fresh Greptile review is 5/5; all findings are resolved. - Current-base focused verification passed 300 cases across protocol, real Rust restart/continuation, permission, backend, and immutable-profile suites. The complete transport suite also passed 286 cases before the base refresh. - Earlier iterations exposed lifecycle visibility, late writer publication, and close-order failures. Controlled regressions reproduced each before repair. CI also found missing test-peer drain methods and a stale source fingerprint; both are corrected. An older broad local test run was stopped after its source was superseded. No full local-suite pass is claimed; full coverage passed in isolated CI. - A synthetic 9 MB journal comparison used 20 writes per sample and concurrent HTTP probes, with alternating baseline/treatment order. HTTP median changed from 31 ms to 2–3 ms; p95 changed from 51–85 ms to 12–21 ms. Parsed output was equal. The asynchronous loop served 125–153 probes versus 19–20, and took longer overall (992–1069 ms versus 675–678 ms). This proves local I/O responsiveness, not deployed application latency. - Deployment and live save/archive acceptance remain pending. ## Risks Persistence interleavings affect restart authority. The tests cover concurrent commands, replacement, failed storage, and reopening the journal. JSON serialization, atomic rename, directory flush, and control-event writes remain synchronous. This change still rewrites whole journals. Incomplete retirement retains recovery evidence and fails the close. A new exact-current Copilot source identity can reject an older launch identity; historical decoding remains available. No migration is required. ## Model Used OpenAI GPT-6 (Codex), with reasoning, tool use, SSH and browser diagnostics, code editing, and test execution. The exact serving model suffix and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../runner-copilot-capabilities.md | 10 +- packages/paperclip-runner/acpx-profiles.json | 4 +- .../paperclip-runner/docs/durable-recovery.md | 16 ++ .../src/generated_acpx_profiles.rs | 2 +- .../scripts/copilot-profile-identity.mjs | 4 +- .../backends/native-backend-factory.test.ts | 1 + .../durable-prp-control-plane.test.ts | 113 +++++++++++++ .../durable-prp-control-plane.ts | 62 +++++--- .../src/control-plane/private-state-file.ts | 15 ++ .../src/control-plane/staged-private-write.ts | 33 ++++ .../drivers/acpx/copilot-installation.test.ts | 19 ++- .../src/drivers/acpx/generated-profiles.ts | 4 +- .../src/drivers/acpx/profile-compatibility.ts | 4 +- ...runnerd-acpx-permission-forwarding.test.ts | 1 + .../runnerd-acpx-permission-ordering.test.ts | 1 + .../src/live/runnerd-codex-transport.ts | 50 +++++- .../live/runnerd-detach-durability.test.ts | 150 ++++++++++++++++++ .../copilot-profile-v61-identity.json | 85 ++++++++++ 18 files changed, 534 insertions(+), 40 deletions(-) create mode 100644 packages/paperclip-runner/src/control-plane/private-state-file.ts create mode 100644 packages/paperclip-runner/src/control-plane/staged-private-write.ts create mode 100644 packages/paperclip-runner/src/live/runnerd-detach-durability.test.ts create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json diff --git a/doc/architecture/runner-copilot-capabilities.md b/doc/architecture/runner-copilot-capabilities.md index fcbe1a6f847..36e2f8ea81c 100644 --- a/doc/architecture/runner-copilot-capabilities.md +++ b/doc/architecture/runner-copilot-capabilities.md @@ -1,6 +1,14 @@ # Copilot 1.0.88 rich ACP capability audit -Current source declaration (2026-10-26): **Copilot profile v60** binds the shared +Current source declaration (2026-10-26): **Copilot profile v61** binds controller +journal retirement and the historical profile decoder. Its digest is +`sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3`. +The v60 declaration remains unchanged and readable as history. This source +identity update changes no Copilot executable, arguments, permission policy, +tool receipts, or qualification flags. Offline compatibility checks supply no +new live qualification; prior live outcomes retain their original identities. + +Historical source declaration (2026-10-26): **Copilot profile v60** binds the shared sources that carry Codex's explicit hosted-search restriction. Its digest is `sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa`. The v59 declaration remains unchanged and readable as history. Exact-current diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index 6635ab026cb..1e6e821a8b2 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 60, + "agentProfileVersion": 61, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa", + "commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index ed505bc7712..a95cbd22b04 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -175,6 +175,22 @@ The ACK means that every source event through sequence 9 is committed or deduplicated. The runner rejects an ACK that moves backward or beyond its produced source cursor. It removes only events at or below a valid ACK. +The controller stages `item.delta` output snapshots with asynchronous file writes and file +flushes. A slow volume does not block unrelated HTTP requests during those +operations. The controller still publishes the snapshot atomically, flushes its +directory, and waits for persistence before acknowledging an event. +A newer command or authenticated connection can save the +same event while staging is pending; the older snapshot cannot replace that +newer durable state. A failed write fences further authority until reload. +Lifecycle, control, authentication, and semantic tool events retain synchronous +persistence before transport readers can observe their state. Semantic tool +admission and dispatch cannot yield to a replacement connection between those operations. +Controller replacement must still stop ingress and drain admitted processing +before a successor reuses the journal. External control routes stop their +reconnect loop before the controller joins admitted work. An incomplete drain +cannot certify a reusable checkpoint. Snapshots use compact JSON; existing +indented snapshots remain readable. + For the lost-ACK fault, the mock commits an event, drops the ACK and socket, and reports the prior cursor once after reconnect. The runner sends the same bytes again. The mock increments delivery count, keeps one logical event, and sends diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index e847662ad88..64dc17a0c88 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa", + "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index d6f02953739..354f06c837a 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v60-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v61-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 60 }; +const declaration = { ...historical.declaration, agentProfileVersion: 61 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts index 95ad646cfbf..d6366addb8b 100644 --- a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts +++ b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts @@ -658,6 +658,7 @@ describe("native backend factory", () => { ["cursor", "../../test/fixtures/cursor-acp/profile-v7-identity.json"], ["cursor", "../../test/fixtures/cursor-acp/profile-v10-identity.json"], ["copilot", "../../test/fixtures/copilot-profile-v14-identity.json"], + ["copilot", "../../test/fixtures/copilot-profile-v60-identity.json"], ["pi", "../../test-fixtures/pi-acp/profile-v13-identity.json"], ["pi", "../../test-fixtures/pi-acp/profile-v14-identity.json"], ["pi", "../../test-fixtures/pi-acp/profile-v18-identity.json"], diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index ff4812fcb94..bcbc3e6f243 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -13,6 +13,8 @@ import { } from "node:fs"; import { connect, type Socket } from "node:net"; import nodeFs from "node:fs"; +import fsPromises from "node:fs/promises"; +import { createServer } from "node:http"; import { syncBuiltinESMExports } from "node:module"; import { tmpdir } from "node:os"; import { resolve } from "node:path"; @@ -47,6 +49,117 @@ const identity: DurableRecoveryIdentity = { const expectedRunnerVersion = "0.3.0"; const expectedRunnerDigest = `sha256:${"a".repeat(64)}`; +it.each(["event", "newer_command", "replacement", "write_failure", "drain", "semantic_replacement"] as const)( + "keeps HTTP responsive during a durable event write and preserves %s", async (mode) => { + const root = mkdtempSync(resolve(tmpdir(), "runner-event-io-")); + const dispatch = vi.fn(async () => { + expect(JSON.parse(readFileSync(resolve(root, "control-plane-state.json"), "utf8")).ackedSourceSeq).toBe(1); + return { result: { ok: true } }; + }); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, + expectedRunnerVersion, expectedRunnerDigest, onCommittedEvent: async () => {}, onSemanticToolInput: dispatch }); + const http = createServer((_req, res) => res.end("responsive")); + let release!: () => void; + const barrier = new Promise(resolveWrite => { release = resolveWrite; }); + const originalOpen = fsPromises.open; + let staged = false; + let holdFirst = true; + let client: AuthenticatedClient | null = null; + let replacement: AuthenticatedClient | null = null; + const open = vi.spyOn(fsPromises, "open").mockImplementation(async (...args) => { + const file = await originalOpen(...args); + if (holdFirst && String(args[0]).includes(".control-plane-state.json.")) { + holdFirst = false; + const sync = file.sync.bind(file); + file.sync = async () => { + staged = true; + await barrier; + if (mode === "write_failure") throw new Error("controlled disk failure"); + await sync(); + }; + } + return file; + }); + try { + await core.start(); + await new Promise(resolveListen => http.listen(0, "127.0.0.1", resolveListen)); + client = (await authenticate(core, core.issueBootstrapTicket()))!; + const event = semanticInputEvent(); + const payload = event.payload as Record; + if (mode !== "semantic_replacement") { + payload.eventType = "item.delta"; + payload.priority = 2; + payload.payload = { delta: "output while the volume is slow" }; + } + expect(validatePrpEvent(payload)).toMatchObject({ ok: true }); + sendSecure(client, event); + await vi.waitFor(() => expect(staged || dispatch.mock.calls.length > 0).toBe(true)); + if (mode === "semantic_replacement") { + replacement = (await authenticate(core, client.leaseToken!))!; + release(); + await vi.waitFor(() => expect(dispatch).toHaveBeenCalledTimes(1)); + await vi.waitFor(() => expect(core.store.state.commands.some(command => command.type === "semantic_tool.result")).toBe(true)); + return; + } + // The actual socket request completes while the real file's fsync is held. + const address = http.address() as { port: number }; + expect(await (await fetch(`http://127.0.0.1:${address.port}`)).text()).toBe("responsive"); + expect(JSON.parse(readFileSync(core.store.path, "utf8")).ackedSourceSeq).toBe(0); + if (mode === "newer_command") core.queueCommand("session.snapshot", {}, "command_during_event"); + let drained = false; + const retirement = mode === "drain" ? (async () => { + await core.stop(); + await core.drainPendingConnectionProcessing(); + drained = true; + })() : null; + if (retirement) { + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(drained).toBe(false); + } + if (mode === "replacement") { + replacement = (await authenticate(core, client.leaseToken!))!; + const second = structuredClone(event); + const next = second.payload as Record; + next.sourceSeq = 2; + next.sourceEventId = "output-event-2"; + sendSecure(replacement, second); + expect(await receiveSecure(replacement)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 2 } }); + } + release(); + if (retirement) { + expect(await receiveSecure(client)).toBeNull(); + await retirement; + expect(drained).toBe(true); + } else if (mode === "write_failure") { + expect(await receiveSecure(client)).toBeNull(); + expect(() => core.queueCommand("session.snapshot")).toThrow("indeterminate"); + } else if (mode !== "replacement") { + expect(await receiveSecure(client)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 1 } }); + } + await core.stop(); + await core.drainPendingConnectionProcessing(); + const restored = new DurablePrpControlPlane({ stateDirectory: root, identity, + expectedRunnerVersion, expectedRunnerDigest }); + const expectedEvents = mode === "replacement" ? 2 : mode === "write_failure" ? 0 : 1; + expect(restored.store.state.ackedSourceSeq).toBe(expectedEvents); + expect(restored.store.state.committedEvents).toHaveLength(expectedEvents); + if (mode === "newer_command") { + expect(restored.store.state.commands.some(command => command.commandId === "command_during_event")).toBe(true); + } + expect(nodeFs.readdirSync(root).filter(name => name.endsWith(".tmp"))).toEqual([]); + } finally { + release(); + open.mockRestore(); + client?.socket.destroy(); + replacement?.socket.destroy(); + await core.stop(); + await core.drainPendingConnectionProcessing(); + await new Promise(resolveClose => http.close(() => resolveClose())); + rmSync(root, { recursive: true, force: true }); + } + }, +); + it("commits credential updates before ACK and preserves replay identity", async () => { const root = mkdtempSync(resolve(tmpdir(), "runner-credential-update-")); let release!: () => void; diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index 2556c47f48c..2ce1af85df8 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -23,12 +23,13 @@ import { renameSync, unlinkSync, writeFileSync, - type Stats, } from "node:fs"; import { createServer, type IncomingMessage, type Server } from "node:http"; import { dirname, resolve } from "node:path"; import type { Duplex } from "node:stream"; import { fileURLToPath } from "node:url"; +import { withStagedPrivateWrite } from "./staged-private-write.js"; +import { verifyPrivateRegularFile } from "./private-state-file.js"; import { NativeSessionProtocolIntegrityError } from "../contracts/native-session-backend.js"; import { ACPX_CREDENTIAL_BINDING_ENV, ACPX_CREDENTIAL_NAMES, CLAUDE_ROUTING_ENV_KEYS, createAcpxSidecarHostEnvironment } from "../drivers/acpx/environment.js"; @@ -1122,22 +1123,6 @@ function verifyPrivateDirectory(path: string): void { } } -function verifyPrivateRegularFile(file: Stats, path: string): void { - if (!file.isFile()) { - throw new Error(`Private state path is not a regular file: ${path}`); - } - if (process.platform !== "win32") { - if ((file.mode & 0o777) !== 0o600) { - throw new Error(`Private state file does not use mode 0600: ${path}`); - } - if (process.geteuid !== undefined && file.uid !== process.geteuid()) { - throw new Error( - `Private state file is not owned by the daemon user: ${path}`, - ); - } - } -} - export function readPrivateFile(path: string): string | null { let descriptor: number; try { @@ -1218,6 +1203,8 @@ class DurableCoreStore { readonly path: string; #state: StoredCoreState; #writeIndeterminate = false; + #revision = 0; + #savedRevision = 0; constructor(directory: string, identity: DurableRecoveryIdentity) { try { @@ -1255,7 +1242,33 @@ class DurableCoreStore { save(): void { this.assertWritable(); - atomicPrivateWrite(this.path, `${JSON.stringify(this.#state, null, 2)}\n`); + const revision = ++this.#revision; + try { + atomicPrivateWrite(this.path, `${JSON.stringify(this.#state)}\n`); + this.#savedRevision = revision; + } catch (error) { + this.#writeIndeterminate = true; + throw error; + } + } + + async saveEvent(): Promise { + this.assertWritable(); + const revision = ++this.#revision; + try { + await withStagedPrivateWrite(this.path, `${JSON.stringify(this.#state)}\n`, (temporary) => { + this.assertWritable(); + // A command or replacement connection can persist this event and newer + // authority while its file is staged. Never roll that state back. + if (this.#savedRevision >= revision) return; + renameSync(temporary, this.path); + syncParentDirectory(this.path); + this.#savedRevision = revision; + }); + } catch (error) { + this.#writeIndeterminate = true; + throw error; + } } assertWritable(): void { @@ -1268,9 +1281,11 @@ class DurableCoreStore { /** Persist a complete candidate before publishing any new authority in memory. */ commit(candidate: StoredCoreState): void { this.assertWritable(); + const revision = ++this.#revision; try { - atomicPrivateWrite(this.path, `${JSON.stringify(candidate, null, 2)}\n`); + atomicPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`); this.#state = candidate; + this.#savedRevision = revision; } catch (error) { // Rename may already have succeeded before directory fsync failed. // Never overwrite that possibly durable receipt using stale memory. @@ -3292,7 +3307,14 @@ export class DurablePrpControlPlane { }); this.#store.state.ackedSourceSeq = sourceSeq; } - this.#store.save(); + // Output deltas have no lifecycle or control effects. Other events must + // become durable before transport readers can observe them in memory. + if (eventType === "item.delta") { + await this.#store.saveEvent(); + if (this.#eventOwner !== connection || this.#identity !== identity || this.#protocolIntegrityError !== null) return; + } else { + this.#store.save(); + } if ( isSemanticInput && diff --git a/packages/paperclip-runner/src/control-plane/private-state-file.ts b/packages/paperclip-runner/src/control-plane/private-state-file.ts new file mode 100644 index 00000000000..788cce5cbfd --- /dev/null +++ b/packages/paperclip-runner/src/control-plane/private-state-file.ts @@ -0,0 +1,15 @@ +import type { Stats } from "node:fs"; + +export function verifyPrivateRegularFile(file: Stats, path: string): void { + if (!file.isFile()) { + throw new Error(`Private state path is not a regular file: ${path}`); + } + if (process.platform !== "win32") { + if ((file.mode & 0o777) !== 0o600) { + throw new Error(`Private state file does not use mode 0600: ${path}`); + } + if (process.geteuid !== undefined && file.uid !== process.geteuid()) { + throw new Error(`Private state file is not owned by the daemon user: ${path}`); + } + } +} diff --git a/packages/paperclip-runner/src/control-plane/staged-private-write.ts b/packages/paperclip-runner/src/control-plane/staged-private-write.ts new file mode 100644 index 00000000000..fed62c54ba4 --- /dev/null +++ b/packages/paperclip-runner/src/control-plane/staged-private-write.ts @@ -0,0 +1,33 @@ +import { randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import fs from "node:fs/promises"; +import { basename, dirname, resolve } from "node:path"; +import { verifyPrivateRegularFile } from "./private-state-file.js"; + +/** Stage durable bytes off the event loop; the owner publishes them without yielding. */ +export async function withStagedPrivateWrite( + path: string, + contents: string, + publish: (temporary: string) => void, +): Promise { + const temporary = resolve(dirname(path), `.${basename(path)}.${randomUUID()}.tmp`); + const file = await fs.open(temporary, constants.O_WRONLY | constants.O_CREAT | + constants.O_EXCL | (constants.O_NOFOLLOW ?? 0), 0o600); + try { + try { + if (process.platform !== "win32") await file.chmod(0o600); + const metadata = await file.stat(); + verifyPrivateRegularFile(metadata, temporary); + await file.writeFile(contents, "utf8"); + await file.sync(); + } finally { + await file.close(); + } + publish(temporary); + } finally { + // Publication can either rename this file or retain a newer durable write. + await fs.unlink(temporary).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + }); + } +} diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 8180acde238..c1c175f2994 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,24 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(60); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(61); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3"); + }); + + it("preserves v60 while binding controller journal retirement in v61", async () => { + const priorBytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v60-identity.json", import.meta.url)); + expect(createHash("sha256").update(priorBytes).digest("hex")).toBe("e3c45b0311517d05f06ed10f357db713ae157cfa87e4a8becf087c5b6dfa7003"); + const prior = JSON.parse(priorBytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v61-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + expect(isSupportedAcpxProfileVersion("copilot", 60)).toBe(true); + expect(current.commandDigest).not.toBe(prior.commandDigest); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 60, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); }); it("preserves v59 while binding the Codex-only hosted search restriction in v60", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index a5d86f192f2..f52273f437c 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 60, + "agentProfileVersion": 61, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa", + "commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 82856d06617..6a1888d273c 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts b/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts index 71b37c8ff25..14e6351c9c9 100644 --- a/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts @@ -27,6 +27,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) } async start() {} async stop() {} + async drainPendingConnectionProcessing() {} disconnectActiveRunner() {} persistRunAttachTemplate() {} issueBootstrapTicket() { return {}; } diff --git a/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts b/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts index 3e0c5314b78..1dcd3de3875 100644 --- a/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts @@ -25,6 +25,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) } async start() {} async stop() {} + async drainPendingConnectionProcessing() {} disconnectActiveRunner() {} persistRunAttachTemplate() {} issueBootstrapTicket() { return {}; } diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index d0d51df3077..d088b3c7916 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -726,6 +726,7 @@ async function releaseRunnerProcessOwnership(input: { ((settlement: "settled" | "unsettled") => Promise | void) | null; forceKill: () => Promise | void; release: (() => Promise | void) | null; + retireController?: () => Promise; }): Promise { let releaseFailure: unknown; try { @@ -735,6 +736,9 @@ async function releaseRunnerProcessOwnership(input: { } let checkpointFailure: unknown; try { + // External ingress must stop reconnecting before admitted journal writes + // can be joined. A failed retirement cannot authorize a new checkpoint. + await input.retireController?.(); if (input.checkpoint !== null) { // Release only the authenticated control route first. In provider-ingress // mode this stops its reconnect loop; it does not release the sandbox @@ -3569,6 +3573,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { readonly #closedSignal: Promise; #resolveClosedSignal!: () => void; #closePromise: Promise | null = null; + #detachPromise: Promise | null = null; #startupOperation: Promise | null = null; #attachmentOperation: Promise | null = null; #startupDrainTimedOut = false; @@ -4466,7 +4471,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { close(reason?: string): Promise { // Detachment relinquishes process ownership. A late execution finalizer // must not suspend or signal the runner now owned by the next controller. - if (this.#controllerDetachedForRestart) return Promise.resolve(); + if (this.#controllerDetachedForRestart) + return this.#detachPromise ?? Promise.resolve(); if (reason) { this.#diagnostic( `runner transport close requested: ${reason.replaceAll(/[\r\n]/g, " ").slice(0, 1_000)}`, @@ -4476,8 +4482,33 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { return this.#closePromise; } - async detachControllerForRestart(): Promise { + detachControllerForRestart(): Promise { + this.#detachPromise ??= this.#detachControllerOnce(); + return this.#detachPromise; + } + + async #stopAndDrainAuthority(deadline: number): Promise { + const core = this.#core; + if (!core) return; + await core.stop(); + let timer: ReturnType | undefined; + try { + await Promise.race([ + core.drainPendingConnectionProcessing(), + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error( + "Controller journal drain timed out; preserve the session before replacing its writer.", + )), Math.max(0, deadline - Date.now())); + }), + ]); + } finally { + clearTimeout(timer); + } + } + + async #detachControllerOnce(): Promise { if (this.#closed) return; + const deadline = Date.now() + runnerCloseGraceMs(this.options); this.#controllerDetachedForRestart = true; this.#closed = true; this.#resolveClosedSignal(); @@ -4495,13 +4526,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { `controller route release failed during restart detach: ${String(error)}`, ); }); - await this.#core?.stop().catch((error: unknown) => { - this.#diagnostic( - `controller authority stop failed during restart detach: ${String(error)}`, - ); - }); + // A successor must not share this journal with an admitted async write. + // A timeout rejects detachment, including repeated calls, instead of + // reporting that the old writer has retired. + try { + await this.#stopAndDrainAuthority(deadline); + } finally { + this.#queue.close(); + } this.#handle = null; - this.#queue.close(); this.#diagnostic( "controller authority detached for restart; durable runner left alive", ); @@ -4725,6 +4758,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#evidence.runnerSignal = result.signal as NodeJS.Signals | null; }, release: this.#controlPlaneRelease, + retireController: () => this.#stopAndDrainAuthority(closeDeadline), }); } finally { await this.#core?.stop(); diff --git a/packages/paperclip-runner/src/live/runnerd-detach-durability.test.ts b/packages/paperclip-runner/src/live/runnerd-detach-durability.test.ts new file mode 100644 index 00000000000..d49672cf964 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-detach-durability.test.ts @@ -0,0 +1,150 @@ +import { spawn, type ChildProcess } from "node:child_process"; +import fs from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary, readRunnerdArtifactBinding, runnerdRecoveryInternals } from "./runnerd-codex-transport.js"; + +it.each(["resolved", "rejected"] as const)("keeps the checkpoint behind controller retirement (%s)", async outcome => { + const steps: string[] = []; + let release!: () => void; + let reject!: (error: Error) => void; + const barrier = new Promise((resolveDrain, rejectDrain) => { release = resolveDrain; reject = rejectDrain; }); + const operation = runnerdRecoveryInternals.releaseRunnerProcessOwnership({ + runnerSettled: true, + release: () => { steps.push("route"); }, + retireController: async () => { steps.push("drain"); await barrier; }, + checkpoint: () => { steps.push("checkpoint"); }, + forceKill: () => { steps.push("containment"); }, + }); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(steps).toEqual(["route", "drain"]); + if (outcome === "rejected") { + const failed = expect(operation).rejects.toThrow("journal unavailable"); + reject(new Error("journal unavailable")); + await failed; + expect(steps).toEqual(["route", "drain", "containment"]); + } else { + release(); + await operation; + expect(steps).toEqual(["route", "drain", "checkpoint", "containment"]); + } +}); + +it.each(["drain", "timeout", "external_close"] as const)("joins an output flush before controller retirement permits a successor writer (%s)", async mode => { + const root = await fs.mkdtemp(join(tmpdir(), "runnerd-detach-write-")); + await fs.mkdir(join(root, "diagnostics"), { mode: 0o700 }); + const binary = defaultCapabilityRunnerdBinary(); + const artifact = readRunnerdArtifactBinding(binary); + let core!: DurablePrpControlPlane; + const start = DurablePrpControlPlane.prototype.start; + const startSpy = vi.spyOn(DurablePrpControlPlane.prototype, "start").mockImplementation(function (port) { + core = this; + return start.call(this, port); + }); + let release!: () => void; + const barrier = new Promise(resolveWrite => { release = resolveWrite; }); + let staged = false; + const originalOpen = fs.open; + const openSpy = vi.spyOn(fs, "open").mockImplementation(async (...args) => { + const file = await originalOpen(...args); + if (!staged && String(args[0]).includes(".control-plane-state.json.")) { + const sync = file.sync.bind(file); + file.sync = async () => { staged = true; await barrier; await sync(); }; + } + return file; + }); + let child: ChildProcess | undefined; + let exit: Promise | undefined; + const closeSteps: string[] = []; + let stopSpy: ReturnType | undefined; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: binary, + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider-state.json"), "--split-event-burst", "--split-event-prefix-count", "1"], + stateDirectory: root, + closeGraceMs: mode === "timeout" ? 200 : 1_500, + ...(mode === "external_close" ? { controlPlaneRegistration: async (authority: DurablePrpControlPlane) => { + await authority.start(); + const stop = authority.stop.bind(authority); + stopSpy = vi.spyOn(authority, "stop").mockImplementation(async () => { + closeSteps.push("stop"); + release(); + await stop(); + }); + return { connectUrl: authority.connectUrl, release: () => { closeSteps.push("release"); } }; + } } : {}), + runnerProcessLauncher: spec => { + child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: ["ignore", "pipe", "pipe"] }); + const owned = child; + let stderr = ""; + owned.stderr?.on("data", chunk => { stderr += String(chunk); }); + exit = new Promise(resolveExit => owned.once("exit", () => resolveExit())); + const completion = new Promise<{ code: number | null; signal: string | null; stdout: string; stderr: string }>((resolveExit, reject) => { + owned.once("error", reject); + owned.once("exit", (code, signal) => resolveExit({ code, signal, stdout: "", stderr })); + }); + return { child: owned, completion }; + }, + }); + bundle.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] })); + let detached: Promise | undefined; + try { + await bundle.transport.request("thread/start", { cwd: root, dynamicTools: [{ + name: "get_task_context", description: "Read fixture context.", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + }] }); + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Exercise a held output flush." }] }); + await vi.waitFor(() => expect(staged).toBe(true), { timeout: 5_000 }); + if (mode === "external_close") { + await bundle.transport.close().catch(error => { + expect(error.message).toMatch(/journal drain timed out|did not durably suspend/); + }); + expect(closeSteps.indexOf("release")).toBeGreaterThanOrEqual(0); + expect(closeSteps.indexOf("release")).toBeLessThan(closeSteps.indexOf("stop")); + return; + } + let settled = false; + detached = bundle.detachControllerForRestart().then(() => { settled = true; }); + await vi.waitFor(() => expect(core.activeRunnerConnectionCount()).toBe(0)); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(settled).toBe(false); + if (mode === "timeout") { + await expect(detached).rejects.toThrow("journal drain timed out"); + await expect(bundle.detachControllerForRestart()).rejects.toThrow("journal drain timed out"); + await expect(bundle.transport.close()).rejects.toThrow("journal drain timed out"); + release(); + await core.drainPendingConnectionProcessing(); + } else { + release(); + await detached; + } + const successor = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), + identity: core.store.state.identity, expectedRunnerVersion: artifact.version, expectedRunnerDigest: artifact.digest }); + successor.queueCommand("session.snapshot", {}, "successor_command"); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + const persisted = JSON.parse(await fs.readFile(successor.store.path, "utf8")); + expect(persisted.commands.some((command: { commandId: string }) => command.commandId === "successor_command")).toBe(true); + } catch (error) { + const diagnostic = await fs.readFile(join(root, "diagnostics", "runnerd.stderr.log"), "utf8").catch(() => ""); + throw new Error(`${String(error)}\n${diagnostic}`, { cause: error }); + } finally { + release(); + await detached?.catch(() => undefined); + openSpy.mockRestore(); + startSpy.mockRestore(); + await bundle.detachControllerForRestart().catch(() => undefined); + const providerPid = bundle.evidence().codexPid; + if (providerPid) { + try { process.kill(providerPid, "SIGKILL"); } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; + } + } + child?.kill("SIGKILL"); + await exit; + await core?.drainPendingConnectionProcessing(); + stopSpy?.mockRestore(); + await fs.rm(root, { recursive: true, force: true }); + } +}, 15_000); diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json new file mode 100644 index 00000000000..6a641986fce --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 61, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "e6c98128d98a4177c42d68a04c1f749554fd5ec844da1bbbdb3f682670189445", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "f9670b63d68e8b84cb79ab18425028f1ff781adb398cb9955ca80b8e0c9264c7", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +}