perf(runner): keep event journal writes off the HTTP thread (#17745)

## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - The HTTP server also owns durable Runner state.
> - Each committed event rewrites and flushes that state on the HTTP
thread.
> - Large journals and slow writes delay unrelated navigation and saves.
> - Event acknowledgements must still wait for durable storage.
> - This PR stages output writes asynchronously and joins them before
writer replacement.
> - The board can serve requests while Runner output waits for storage.

## Linked Issues or Issue Description

Refs: #17507. Related: #17514 reduces writes per event burst. This PR
moves individual output writes and file flushes outside the HTTP thread.

**What happened?**
A live CPU profile spent about three seconds of a 13-second sample
serializing and synchronously writing Runner state. Representative
journals were about 9 MB. Unrelated task requests had multi-second
delays.

**Expected behavior**
A slow output-journal write must leave unrelated HTTP requests
responsive. An event must not be acknowledged before it is durable.

**Steps to reproduce**
Hold an authenticated output event's file flush in the durable protocol
test. Send an unrelated HTTP request during that flush. Also save a
command, reconnect, or retire the controller during the write.

## What Changed

- Stage `item.delta` snapshots with asynchronous file writes and file
flushes. Keep lifecycle, control, authentication, and tool events
synchronous so transport readers observe durable boundaries.
- Fence publication by saved revision. An older snapshot cannot replace
newer commands or connection state. A failed write fences the writer.
- Write compact JSON and share private-file checks. Existing journals
remain readable.
- Stop external ingress before joining admitted writes. Checkpoint only
after retirement succeeds. Close reports a failed detach consistently
and releases notification consumers.
- Add real HTTP and Rust transport cases for held writes, reconnects,
commands, failure, restart, close, and timeout.
- Add immutable Copilot source identity v61 for the changed shared
transport. Preserve v60, historical decoding, exact-current launch
checks, and qualification flags. No new live Copilot qualification is
claimed.

## Verification

- Head `6f5c92e778`: workspace `pnpm -r typecheck` and `pnpm build`
passed. All CI gates passed in [run
40356159484](https://github.com/paperclipai/paperclip/actions/runs/40356159484),
including both full Runner shards. Fresh Greptile review is 5/5; all
findings are resolved.
- Current-base focused verification passed 300 cases across protocol,
real Rust restart/continuation, permission, backend, and
immutable-profile suites. The complete transport suite also passed 286
cases before the base refresh.
- Earlier iterations exposed lifecycle visibility, late writer
publication, and close-order failures. Controlled regressions reproduced
each before repair. CI also found missing test-peer drain methods and a
stale source fingerprint; both are corrected. An older broad local test
run was stopped after its source was superseded. No full local-suite
pass is claimed; full coverage passed in isolated CI.
- A synthetic 9 MB journal comparison used 20 writes per sample and
concurrent HTTP probes, with alternating baseline/treatment order. HTTP
median changed from 31 ms to 2–3 ms; p95 changed from 51–85 ms to 12–21
ms. Parsed output was equal. The asynchronous loop served 125–153 probes
versus 19–20, and took longer overall (992–1069 ms versus 675–678 ms).
This proves local I/O responsiveness, not deployed application latency.
- Deployment and live save/archive acceptance remain pending.

## Risks

Persistence interleavings affect restart authority. The tests cover
concurrent commands, replacement, failed storage, and reopening the
journal. JSON serialization, atomic rename, directory flush, and
control-event writes remain synchronous. This change still rewrites
whole journals. Incomplete retirement retains recovery evidence and
fails the close. A new exact-current Copilot source identity can reject
an older launch identity; historical decoding remains available. No
migration is required.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, tool use, SSH and browser
diagnostics, code editing, and test execution. The exact serving model
suffix and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-26 11:19:36 -05:00
1 parent 063a498b54
commit 125ddcabe5
18 files changed
+534 -40

No files matched your search

@@ -1,6 +1,14 @@
# Copilot 1.0.88 rich ACP capability audit
Current source declaration (2026-10-26): **Copilot profile v60** binds the shared
Current source declaration (2026-10-26): **Copilot profile v61** binds controller
journal retirement and the historical profile decoder. Its digest is
`sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3`.
The v60 declaration remains unchanged and readable as history. This source
identity update changes no Copilot executable, arguments, permission policy,
tool receipts, or qualification flags. Offline compatibility checks supply no
new live qualification; prior live outcomes retain their original identities.
Historical source declaration (2026-10-26): **Copilot profile v60** binds the shared
sources that carry Codex's explicit hosted-search restriction. Its digest is
`sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa`.
The v59 declaration remains unchanged and readable as history. Exact-current
+2 -2
View File
@@ -32,12 +32,12 @@
"requiresProviderPolicy": true
},
"copilot": {
"agentProfileVersion": 60,
"agentProfileVersion": 61,
"agentServerPackage": "@github/copilot",
"agentServerVersion": "1.0.88",
"agentRuntimePackage": null,
"agentRuntimeVersion": null,
"commandDigest": "sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa",
"commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3",
"requiresProviderPolicy": true
},
"claude": {
@@ -175,6 +175,22 @@ The ACK means that every source event through sequence 9 is committed or
deduplicated. The runner rejects an ACK that moves backward or beyond its
produced source cursor. It removes only events at or below a valid ACK.
The controller stages `item.delta` output snapshots with asynchronous file writes and file
flushes. A slow volume does not block unrelated HTTP requests during those
operations. The controller still publishes the snapshot atomically, flushes its
directory, and waits for persistence before acknowledging an event.
A newer command or authenticated connection can save the
same event while staging is pending; the older snapshot cannot replace that
newer durable state. A failed write fences further authority until reload.
Lifecycle, control, authentication, and semantic tool events retain synchronous
persistence before transport readers can observe their state. Semantic tool
admission and dispatch cannot yield to a replacement connection between those operations.
Controller replacement must still stop ingress and drain admitted processing
before a successor reuses the journal. External control routes stop their
reconnect loop before the controller joins admitted work. An incomplete drain
cannot certify a reusable checkpoint. Snapshots use compact JSON; existing
indented snapshots remain readable.
For the lost-ACK fault, the mock commits an event, drops the ACK and socket, and
reports the prior cursor once after reconnect. The runner sends the same bytes
again. The mock increments delivery count, keeps one logical event, and sends
@@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option<AcpxReleaseProfile> {
agent_runtime_package: None,
agent_runtime_version: None,
command_digest:
"sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa",
"sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3",
requires_provider_policy: true,
},
"claude" => AcpxReleaseProfile {
@@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url";
import { resolve, dirname } from "node:path";
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const fixture = resolve(root, "test/fixtures/copilot-profile-v60-identity.json");
const fixture = resolve(root, "test/fixtures/copilot-profile-v61-identity.json");
const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8"));
const declaration = { ...historical.declaration, agentProfileVersion: 60 };
const declaration = { ...historical.declaration, agentProfileVersion: 61 };
delete declaration.permissionIdentitySourceSha256;
const sources = {
displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts",
@@ -658,6 +658,7 @@ describe("native backend factory", () => {
["cursor", "../../test/fixtures/cursor-acp/profile-v7-identity.json"],
["cursor", "../../test/fixtures/cursor-acp/profile-v10-identity.json"],
["copilot", "../../test/fixtures/copilot-profile-v14-identity.json"],
["copilot", "../../test/fixtures/copilot-profile-v60-identity.json"],
["pi", "../../test-fixtures/pi-acp/profile-v13-identity.json"],
["pi", "../../test-fixtures/pi-acp/profile-v14-identity.json"],
["pi", "../../test-fixtures/pi-acp/profile-v18-identity.json"],
@@ -13,6 +13,8 @@ import {
} from "node:fs";
import { connect, type Socket } from "node:net";
import nodeFs from "node:fs";
import fsPromises from "node:fs/promises";
import { createServer } from "node:http";
import { syncBuiltinESMExports } from "node:module";
import { tmpdir } from "node:os";
import { resolve } from "node:path";
@@ -47,6 +49,117 @@ const identity: DurableRecoveryIdentity = {
const expectedRunnerVersion = "0.3.0";
const expectedRunnerDigest = `sha256:${"a".repeat(64)}`;
it.each(["event", "newer_command", "replacement", "write_failure", "drain", "semantic_replacement"] as const)(
"keeps HTTP responsive during a durable event write and preserves %s", async (mode) => {
const root = mkdtempSync(resolve(tmpdir(), "runner-event-io-"));
const dispatch = vi.fn(async () => {
expect(JSON.parse(readFileSync(resolve(root, "control-plane-state.json"), "utf8")).ackedSourceSeq).toBe(1);
return { result: { ok: true } };
});
const core = new DurablePrpControlPlane({ stateDirectory: root, identity,
expectedRunnerVersion, expectedRunnerDigest, onCommittedEvent: async () => {}, onSemanticToolInput: dispatch });
const http = createServer((_req, res) => res.end("responsive"));
let release!: () => void;
const barrier = new Promise<void>(resolveWrite => { release = resolveWrite; });
const originalOpen = fsPromises.open;
let staged = false;
let holdFirst = true;
let client: AuthenticatedClient | null = null;
let replacement: AuthenticatedClient | null = null;
const open = vi.spyOn(fsPromises, "open").mockImplementation(async (...args) => {
const file = await originalOpen(...args);
if (holdFirst && String(args[0]).includes(".control-plane-state.json.")) {
holdFirst = false;
const sync = file.sync.bind(file);
file.sync = async () => {
staged = true;
await barrier;
if (mode === "write_failure") throw new Error("controlled disk failure");
await sync();
};
}
return file;
});
try {
await core.start();
await new Promise<void>(resolveListen => http.listen(0, "127.0.0.1", resolveListen));
client = (await authenticate(core, core.issueBootstrapTicket()))!;
const event = semanticInputEvent();
const payload = event.payload as Record<string, unknown>;
if (mode !== "semantic_replacement") {
payload.eventType = "item.delta";
payload.priority = 2;
payload.payload = { delta: "output while the volume is slow" };
}
expect(validatePrpEvent(payload)).toMatchObject({ ok: true });
sendSecure(client, event);
await vi.waitFor(() => expect(staged || dispatch.mock.calls.length > 0).toBe(true));
if (mode === "semantic_replacement") {
replacement = (await authenticate(core, client.leaseToken!))!;
release();
await vi.waitFor(() => expect(dispatch).toHaveBeenCalledTimes(1));
await vi.waitFor(() => expect(core.store.state.commands.some(command => command.type === "semantic_tool.result")).toBe(true));
return;
}
// The actual socket request completes while the real file's fsync is held.
const address = http.address() as { port: number };
expect(await (await fetch(`http://127.0.0.1:${address.port}`)).text()).toBe("responsive");
expect(JSON.parse(readFileSync(core.store.path, "utf8")).ackedSourceSeq).toBe(0);
if (mode === "newer_command") core.queueCommand("session.snapshot", {}, "command_during_event");
let drained = false;
const retirement = mode === "drain" ? (async () => {
await core.stop();
await core.drainPendingConnectionProcessing();
drained = true;
})() : null;
if (retirement) {
await new Promise<void>(resolveTurn => setImmediate(resolveTurn));
expect(drained).toBe(false);
}
if (mode === "replacement") {
replacement = (await authenticate(core, client.leaseToken!))!;
const second = structuredClone(event);
const next = second.payload as Record<string, unknown>;
next.sourceSeq = 2;
next.sourceEventId = "output-event-2";
sendSecure(replacement, second);
expect(await receiveSecure(replacement)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 2 } });
}
release();
if (retirement) {
expect(await receiveSecure(client)).toBeNull();
await retirement;
expect(drained).toBe(true);
} else if (mode === "write_failure") {
expect(await receiveSecure(client)).toBeNull();
expect(() => core.queueCommand("session.snapshot")).toThrow("indeterminate");
} else if (mode !== "replacement") {
expect(await receiveSecure(client)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 1 } });
}
await core.stop();
await core.drainPendingConnectionProcessing();
const restored = new DurablePrpControlPlane({ stateDirectory: root, identity,
expectedRunnerVersion, expectedRunnerDigest });
const expectedEvents = mode === "replacement" ? 2 : mode === "write_failure" ? 0 : 1;
expect(restored.store.state.ackedSourceSeq).toBe(expectedEvents);
expect(restored.store.state.committedEvents).toHaveLength(expectedEvents);
if (mode === "newer_command") {
expect(restored.store.state.commands.some(command => command.commandId === "command_during_event")).toBe(true);
}
expect(nodeFs.readdirSync(root).filter(name => name.endsWith(".tmp"))).toEqual([]);
} finally {
release();
open.mockRestore();
client?.socket.destroy();
replacement?.socket.destroy();
await core.stop();
await core.drainPendingConnectionProcessing();
await new Promise<void>(resolveClose => http.close(() => resolveClose()));
rmSync(root, { recursive: true, force: true });
}
},
);
it("commits credential updates before ACK and preserves replay identity", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-credential-update-"));
let release!: () => void;
@@ -23,12 +23,13 @@ import {
renameSync,
unlinkSync,
writeFileSync,
type Stats,
} from "node:fs";
import { createServer, type IncomingMessage, type Server } from "node:http";
import { dirname, resolve } from "node:path";
import type { Duplex } from "node:stream";
import { fileURLToPath } from "node:url";
import { withStagedPrivateWrite } from "./staged-private-write.js";
import { verifyPrivateRegularFile } from "./private-state-file.js";
import { NativeSessionProtocolIntegrityError } from "../contracts/native-session-backend.js";
import { ACPX_CREDENTIAL_BINDING_ENV, ACPX_CREDENTIAL_NAMES, CLAUDE_ROUTING_ENV_KEYS, createAcpxSidecarHostEnvironment } from "../drivers/acpx/environment.js";
@@ -1122,22 +1123,6 @@ function verifyPrivateDirectory(path: string): void {
}
}
function verifyPrivateRegularFile(file: Stats, path: string): void {
if (!file.isFile()) {
throw new Error(`Private state path is not a regular file: ${path}`);
}
if (process.platform !== "win32") {
if ((file.mode & 0o777) !== 0o600) {
throw new Error(`Private state file does not use mode 0600: ${path}`);
}
if (process.geteuid !== undefined && file.uid !== process.geteuid()) {
throw new Error(
`Private state file is not owned by the daemon user: ${path}`,
);
}
}
}
export function readPrivateFile(path: string): string | null {
let descriptor: number;
try {
@@ -1218,6 +1203,8 @@ class DurableCoreStore {
readonly path: string;
#state: StoredCoreState;
#writeIndeterminate = false;
#revision = 0;
#savedRevision = 0;
constructor(directory: string, identity: DurableRecoveryIdentity) {
try {
@@ -1255,7 +1242,33 @@ class DurableCoreStore {
save(): void {
this.assertWritable();
atomicPrivateWrite(this.path, `${JSON.stringify(this.#state, null, 2)}\n`);
const revision = ++this.#revision;
try {
atomicPrivateWrite(this.path, `${JSON.stringify(this.#state)}\n`);
this.#savedRevision = revision;
} catch (error) {
this.#writeIndeterminate = true;
throw error;
}
}
async saveEvent(): Promise<void> {
this.assertWritable();
const revision = ++this.#revision;
try {
await withStagedPrivateWrite(this.path, `${JSON.stringify(this.#state)}\n`, (temporary) => {
this.assertWritable();
// A command or replacement connection can persist this event and newer
// authority while its file is staged. Never roll that state back.
if (this.#savedRevision >= revision) return;
renameSync(temporary, this.path);
syncParentDirectory(this.path);
this.#savedRevision = revision;
});
} catch (error) {
this.#writeIndeterminate = true;
throw error;
}
}
assertWritable(): void {
@@ -1268,9 +1281,11 @@ class DurableCoreStore {
/** Persist a complete candidate before publishing any new authority in memory. */
commit(candidate: StoredCoreState): void {
this.assertWritable();
const revision = ++this.#revision;
try {
atomicPrivateWrite(this.path, `${JSON.stringify(candidate, null, 2)}\n`);
atomicPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`);
this.#state = candidate;
this.#savedRevision = revision;
} catch (error) {
// Rename may already have succeeded before directory fsync failed.
// Never overwrite that possibly durable receipt using stale memory.
@@ -3292,7 +3307,14 @@ export class DurablePrpControlPlane {
});
this.#store.state.ackedSourceSeq = sourceSeq;
}
this.#store.save();
// Output deltas have no lifecycle or control effects. Other events must
// become durable before transport readers can observe them in memory.
if (eventType === "item.delta") {
await this.#store.saveEvent();
if (this.#eventOwner !== connection || this.#identity !== identity || this.#protocolIntegrityError !== null) return;
} else {
this.#store.save();
}
if (
isSemanticInput &&
@@ -0,0 +1,15 @@
import type { Stats } from "node:fs";
export function verifyPrivateRegularFile(file: Stats, path: string): void {
if (!file.isFile()) {
throw new Error(`Private state path is not a regular file: ${path}`);
}
if (process.platform !== "win32") {
if ((file.mode & 0o777) !== 0o600) {
throw new Error(`Private state file does not use mode 0600: ${path}`);
}
if (process.geteuid !== undefined && file.uid !== process.geteuid()) {
throw new Error(`Private state file is not owned by the daemon user: ${path}`);
}
}
}
@@ -0,0 +1,33 @@
import { randomUUID } from "node:crypto";
import { constants } from "node:fs";
import fs from "node:fs/promises";
import { basename, dirname, resolve } from "node:path";
import { verifyPrivateRegularFile } from "./private-state-file.js";
/** Stage durable bytes off the event loop; the owner publishes them without yielding. */
export async function withStagedPrivateWrite(
path: string,
contents: string,
publish: (temporary: string) => void,
): Promise<void> {
const temporary = resolve(dirname(path), `.${basename(path)}.${randomUUID()}.tmp`);
const file = await fs.open(temporary, constants.O_WRONLY | constants.O_CREAT |
constants.O_EXCL | (constants.O_NOFOLLOW ?? 0), 0o600);
try {
try {
if (process.platform !== "win32") await file.chmod(0o600);
const metadata = await file.stat();
verifyPrivateRegularFile(metadata, temporary);
await file.writeFile(contents, "utf8");
await file.sync();
} finally {
await file.close();
}
publish(temporary);
} finally {
// Publication can either rename this file or retain a newer durable write.
await fs.unlink(temporary).catch((error: NodeJS.ErrnoException) => {
if (error.code !== "ENOENT") throw error;
});
}
}
@@ -87,9 +87,24 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi
describe("Copilot build-owned installation", () => {
it("admits the current profile with environment launch preparation", () => {
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(60);
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(61);
expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined();
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa");
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3");
});
it("preserves v60 while binding controller journal retirement in v61", async () => {
const priorBytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v60-identity.json", import.meta.url));
expect(createHash("sha256").update(priorBytes).digest("hex")).toBe("e3c45b0311517d05f06ed10f357db713ae157cfa87e4a8becf087c5b6dfa7003");
const prior = JSON.parse(priorBytes.toString("utf8"));
const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v61-identity.json", import.meta.url), "utf8"));
expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([
"agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256",
]);
expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort());
expect(isSupportedAcpxProfileVersion("copilot", 60)).toBe(true);
expect(current.commandDigest).not.toBe(prior.commandDigest);
await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot,
agentProfileVersion: 60, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned");
});
it("preserves v59 while binding the Codex-only hosted search restriction in v60", async () => {
@@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = {
"protocolVersion": 1,
"acpxVersion": "0.13.1",
"agent": "copilot",
"agentProfileVersion": 60,
"agentProfileVersion": 61,
"agentServerPackage": "@github/copilot",
"agentServerVersion": "1.0.88",
"agentRuntimePackage": null,
"agentRuntimeVersion": null,
"commandDigest": "sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa",
"commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3",
"permissionPolicy": "interactive"
},
"claude": {
@@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js";
/** Decodable historical profile revisions, not permission to launch them.
* Exact current profile/package/digest admission is checked separately. */
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59;
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60;
type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"];
export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion;
const historicalVersions: Readonly<Record<string, readonly HistoricalAcpxProfileVersion[]>> = {
pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5],
grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60],
cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
};
@@ -27,6 +27,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal)
}
async start() {}
async stop() {}
async drainPendingConnectionProcessing() {}
disconnectActiveRunner() {}
persistRunAttachTemplate() {}
issueBootstrapTicket() { return {}; }
@@ -25,6 +25,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal)
}
async start() {}
async stop() {}
async drainPendingConnectionProcessing() {}
disconnectActiveRunner() {}
persistRunAttachTemplate() {}
issueBootstrapTicket() { return {}; }
@@ -726,6 +726,7 @@ async function releaseRunnerProcessOwnership(input: {
((settlement: "settled" | "unsettled") => Promise<void> | void) | null;
forceKill: () => Promise<void> | void;
release: (() => Promise<void> | void) | null;
retireController?: () => Promise<void>;
}): Promise<void> {
let releaseFailure: unknown;
try {
@@ -735,6 +736,9 @@ async function releaseRunnerProcessOwnership(input: {
}
let checkpointFailure: unknown;
try {
// External ingress must stop reconnecting before admitted journal writes
// can be joined. A failed retirement cannot authorize a new checkpoint.
await input.retireController?.();
if (input.checkpoint !== null) {
// Release only the authenticated control route first. In provider-ingress
// mode this stops its reconnect loop; it does not release the sandbox
@@ -3569,6 +3573,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
readonly #closedSignal: Promise<void>;
#resolveClosedSignal!: () => void;
#closePromise: Promise<void> | null = null;
#detachPromise: Promise<void> | null = null;
#startupOperation: Promise<unknown> | null = null;
#attachmentOperation: Promise<void> | null = null;
#startupDrainTimedOut = false;
@@ -4466,7 +4471,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
close(reason?: string): Promise<void> {
// Detachment relinquishes process ownership. A late execution finalizer
// must not suspend or signal the runner now owned by the next controller.
if (this.#controllerDetachedForRestart) return Promise.resolve();
if (this.#controllerDetachedForRestart)
return this.#detachPromise ?? Promise.resolve();
if (reason) {
this.#diagnostic(
`runner transport close requested: ${reason.replaceAll(/[\r\n]/g, " ").slice(0, 1_000)}`,
@@ -4476,8 +4482,33 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
return this.#closePromise;
}
async detachControllerForRestart(): Promise<void> {
detachControllerForRestart(): Promise<void> {
this.#detachPromise ??= this.#detachControllerOnce();
return this.#detachPromise;
}
async #stopAndDrainAuthority(deadline: number): Promise<void> {
const core = this.#core;
if (!core) return;
await core.stop();
let timer: ReturnType<typeof setTimeout> | undefined;
try {
await Promise.race([
core.drainPendingConnectionProcessing(),
new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(new Error(
"Controller journal drain timed out; preserve the session before replacing its writer.",
)), Math.max(0, deadline - Date.now()));
}),
]);
} finally {
clearTimeout(timer);
}
}
async #detachControllerOnce(): Promise<void> {
if (this.#closed) return;
const deadline = Date.now() + runnerCloseGraceMs(this.options);
this.#controllerDetachedForRestart = true;
this.#closed = true;
this.#resolveClosedSignal();
@@ -4495,13 +4526,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
`controller route release failed during restart detach: ${String(error)}`,
);
});
await this.#core?.stop().catch((error: unknown) => {
this.#diagnostic(
`controller authority stop failed during restart detach: ${String(error)}`,
);
});
// A successor must not share this journal with an admitted async write.
// A timeout rejects detachment, including repeated calls, instead of
// reporting that the old writer has retired.
try {
await this.#stopAndDrainAuthority(deadline);
} finally {
this.#queue.close();
}
this.#handle = null;
this.#queue.close();
this.#diagnostic(
"controller authority detached for restart; durable runner left alive",
);
@@ -4725,6 +4758,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
this.#evidence.runnerSignal = result.signal as NodeJS.Signals | null;
},
release: this.#controlPlaneRelease,
retireController: () => this.#stopAndDrainAuthority(closeDeadline),
});
} finally {
await this.#core?.stop();
@@ -0,0 +1,150 @@
import { spawn, type ChildProcess } from "node:child_process";
import fs from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { expect, it, vi } from "vitest";
import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary, readRunnerdArtifactBinding, runnerdRecoveryInternals } from "./runnerd-codex-transport.js";
it.each(["resolved", "rejected"] as const)("keeps the checkpoint behind controller retirement (%s)", async outcome => {
const steps: string[] = [];
let release!: () => void;
let reject!: (error: Error) => void;
const barrier = new Promise<void>((resolveDrain, rejectDrain) => { release = resolveDrain; reject = rejectDrain; });
const operation = runnerdRecoveryInternals.releaseRunnerProcessOwnership({
runnerSettled: true,
release: () => { steps.push("route"); },
retireController: async () => { steps.push("drain"); await barrier; },
checkpoint: () => { steps.push("checkpoint"); },
forceKill: () => { steps.push("containment"); },
});
await new Promise<void>(resolveTurn => setImmediate(resolveTurn));
expect(steps).toEqual(["route", "drain"]);
if (outcome === "rejected") {
const failed = expect(operation).rejects.toThrow("journal unavailable");
reject(new Error("journal unavailable"));
await failed;
expect(steps).toEqual(["route", "drain", "containment"]);
} else {
release();
await operation;
expect(steps).toEqual(["route", "drain", "checkpoint", "containment"]);
}
});
it.each(["drain", "timeout", "external_close"] as const)("joins an output flush before controller retirement permits a successor writer (%s)", async mode => {
const root = await fs.mkdtemp(join(tmpdir(), "runnerd-detach-write-"));
await fs.mkdir(join(root, "diagnostics"), { mode: 0o700 });
const binary = defaultCapabilityRunnerdBinary();
const artifact = readRunnerdArtifactBinding(binary);
let core!: DurablePrpControlPlane;
const start = DurablePrpControlPlane.prototype.start;
const startSpy = vi.spyOn(DurablePrpControlPlane.prototype, "start").mockImplementation(function (port) {
core = this;
return start.call(this, port);
});
let release!: () => void;
const barrier = new Promise<void>(resolveWrite => { release = resolveWrite; });
let staged = false;
const originalOpen = fs.open;
const openSpy = vi.spyOn(fs, "open").mockImplementation(async (...args) => {
const file = await originalOpen(...args);
if (!staged && String(args[0]).includes(".control-plane-state.json.")) {
const sync = file.sync.bind(file);
file.sync = async () => { staged = true; await barrier; await sync(); };
}
return file;
});
let child: ChildProcess | undefined;
let exit: Promise<void> | undefined;
const closeSteps: string[] = [];
let stopSpy: ReturnType<typeof vi.spyOn> | undefined;
const bundle = createCapabilityRunnerdCodexTransport({
runnerBinary: binary,
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
codexArgs: ["--state-file", join(root, "provider-state.json"), "--split-event-burst", "--split-event-prefix-count", "1"],
stateDirectory: root,
closeGraceMs: mode === "timeout" ? 200 : 1_500,
...(mode === "external_close" ? { controlPlaneRegistration: async (authority: DurablePrpControlPlane) => {
await authority.start();
const stop = authority.stop.bind(authority);
stopSpy = vi.spyOn(authority, "stop").mockImplementation(async () => {
closeSteps.push("stop");
release();
await stop();
});
return { connectUrl: authority.connectUrl, release: () => { closeSteps.push("release"); } };
} } : {}),
runnerProcessLauncher: spec => {
child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: ["ignore", "pipe", "pipe"] });
const owned = child;
let stderr = "";
owned.stderr?.on("data", chunk => { stderr += String(chunk); });
exit = new Promise(resolveExit => owned.once("exit", () => resolveExit()));
const completion = new Promise<{ code: number | null; signal: string | null; stdout: string; stderr: string }>((resolveExit, reject) => {
owned.once("error", reject);
owned.once("exit", (code, signal) => resolveExit({ code, signal, stdout: "", stderr }));
});
return { child: owned, completion };
},
});
bundle.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
let detached: Promise<void> | undefined;
try {
await bundle.transport.request("thread/start", { cwd: root, dynamicTools: [{
name: "get_task_context", description: "Read fixture context.",
inputSchema: { type: "object", properties: {}, additionalProperties: false },
}] });
await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Exercise a held output flush." }] });
await vi.waitFor(() => expect(staged).toBe(true), { timeout: 5_000 });
if (mode === "external_close") {
await bundle.transport.close().catch(error => {
expect(error.message).toMatch(/journal drain timed out|did not durably suspend/);
});
expect(closeSteps.indexOf("release")).toBeGreaterThanOrEqual(0);
expect(closeSteps.indexOf("release")).toBeLessThan(closeSteps.indexOf("stop"));
return;
}
let settled = false;
detached = bundle.detachControllerForRestart().then(() => { settled = true; });
await vi.waitFor(() => expect(core.activeRunnerConnectionCount()).toBe(0));
await new Promise<void>(resolveTurn => setImmediate(resolveTurn));
expect(settled).toBe(false);
if (mode === "timeout") {
await expect(detached).rejects.toThrow("journal drain timed out");
await expect(bundle.detachControllerForRestart()).rejects.toThrow("journal drain timed out");
await expect(bundle.transport.close()).rejects.toThrow("journal drain timed out");
release();
await core.drainPendingConnectionProcessing();
} else {
release();
await detached;
}
const successor = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"),
identity: core.store.state.identity, expectedRunnerVersion: artifact.version, expectedRunnerDigest: artifact.digest });
successor.queueCommand("session.snapshot", {}, "successor_command");
await new Promise<void>(resolveTurn => setImmediate(resolveTurn));
const persisted = JSON.parse(await fs.readFile(successor.store.path, "utf8"));
expect(persisted.commands.some((command: { commandId: string }) => command.commandId === "successor_command")).toBe(true);
} catch (error) {
const diagnostic = await fs.readFile(join(root, "diagnostics", "runnerd.stderr.log"), "utf8").catch(() => "");
throw new Error(`${String(error)}\n${diagnostic}`, { cause: error });
} finally {
release();
await detached?.catch(() => undefined);
openSpy.mockRestore();
startSpy.mockRestore();
await bundle.detachControllerForRestart().catch(() => undefined);
const providerPid = bundle.evidence().codexPid;
if (providerPid) {
try { process.kill(providerPid, "SIGKILL"); } catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error;
}
}
child?.kill("SIGKILL");
await exit;
await core?.drainPendingConnectionProcessing();
stopSpy?.mockRestore();
await fs.rm(root, { recursive: true, force: true });
}
}, 15_000);
@@ -0,0 +1,85 @@
{
"commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3",
"declaration": {
"schema": "paperclip.acpx_profile_declaration.v1",
"agent": "copilot",
"agentProfileVersion": 61,
"acpxVersion": "0.13.1",
"agentServerVersion": "1.0.88",
"protocolPolicyRevision": "copilot-agent-manual-v1",
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
"agentFilesBinding": "registered-runtime-context-v1",
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
"messageIdentityContract": "copilot-native-message-id-v1",
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
"permissionContextContract": "copilot-edit-permission-context-v1",
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
"semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629",
"semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103",
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
"semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31",
"semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9",
"semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603",
"semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236",
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
"nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72",
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
"harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412",
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
"nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37",
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
"nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a",
"providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c",
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
"runtimeRequestBridgeSourceSha256": "e6c98128d98a4177c42d68a04c1f749554fd5ec844da1bbbdb3f682670189445",
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
"nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918",
"profileCompatibilitySourceSha256": "f9670b63d68e8b84cb79ab18425028f1ff781adb398cb9955ca80b8e0c9264c7",
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
"credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161",
"sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80",
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
"semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0",
"runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec",
"runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8",
"runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea",
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
}
}