mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 09:37:19 +01:00
## Thinking Path > - Paperclip manages AI agents and their work. > - Native warm sessions retain a Runner owner between turns. > - Closing that owner must stop execution and preserve valid recovery state. > - A failed suspension can leave a rejected close promise after the task has stopped. > - A session reset then retries that same promise and blocks every fresh session. > - This PR separates task retirement proof from checkpoint reuse for a new session. > - A reset can proceed after exact stop proof, completed handoff, and file collection. ## Linked Issues or Issue Description Refs #18011. That change preserves checkpoint authority during warm preparation. This change handles a stopped owner whose checkpoint cannot be reused. Related: #14695 covers managed files retained with warm processes. A search found no duplicate stopped-owner reset fix. **What happened?** A warm Runner can stop and complete its task handoff while close reports `runner did not durably suspend before checkpoint`. The transport caches that rejected close promise. Core retains the warm owner. Each session reset retries the same promise before it can launch a provider. A controller restart clears the in-memory owner and permits recovery. **Expected behavior** A reset to a new normalized session should release the old owner when its exact task has proved that execution stopped and its handoff completed. Core must collect stopped instruction files before it admits the new session. An unconfirmed owner must continue to block replacement. **Steps to reproduce** 1. Complete a warm native run through a plugin task provider. 2. Change configuration so Core retires the warm owner. 3. Let the task stop and complete, but fail durable suspension proof. Close keeps its rejected promise. 4. Reset the session and start another run on the same task environment. 5. Observe the old close error before any new provider work starts. ## What Changed - Keep a lease-bound retirement receipt in the plugin task owner. It requires `executionStopped: true` and successful task completion after credential handoff. - Accept an unrecoverable suspension error only during retirement for a different normalized session, with that exact owner's receipt and settled Core tool results. - Collect stopped instruction files before releasing the warm owner. Preserve old checkpoint evidence. - Keep same-session, active-owner, incomplete-handoff, unrelated-owner, unresolved Core tool results, and other close-error fences. - Report Core tool settlement on the close error after ingress retirement and drain. Default missing proof to false. Expose the error through the server's Runner shim. Add regression tests and document the recovery boundary. ## Verification - The reset regression fails with the new handling disabled and passes with it enabled. - Both affected suites passed: 1,032 tests. The final 15 focused owner/reset cases pass after adding the tool-settlement guard. - The assembled regression uses the real task-owner and backend binding. Its controlled task provider proves stop and completion. Close returns the same rejected promise on every call. - Negative cases cover incomplete handoff, other close errors, same-session continuation, provider change with the same session ID, unrelated receipts, unresolved Core tool results, and failed file collection. Collection can later succeed without replaying stop or reading the old checkpoint. - Five real-process transport cases pass. They cover an insufficient suspension deadline, the cached rejected promise, late tool completion within and beyond the deadline, interruption, and result-persistence failure. Unsettled callbacks produce `semanticToolsSettled: false`. - `pnpm -r typecheck` passed with the repository's pinned Rust toolchain on PATH. After the final contract change, Runner TypeScript build and server `tsc --noEmit` passed. - The local full-suite run was stopped at the requester's direction. Full regression and build coverage are delegated to GitHub CI. - Full GitHub CI passed on `ef0040d80af679ce6d4a130468e15f75678c1f4c`. Standard Greptile reviewed the same commit: 5/5, zero comments. Apex then reviewed that same commit: 5/5, zero comments. Final checks confirm no unresolved review threads and no merge conflicts. - Initial full CI on `00c002fd5a` failed one scheduled-retry route test at 18/20 cancellations. Controlled cancellation latency reproduced the short poll-budget failure. Before the next cycle, master advanced with #18322, which coordinates that test at the first completed batch. The separate local timeout commit was dropped during rebase. The upstream test passes locally. The rebased run passed 46 targeted owner/reset and scheduled-retry route cases. This PR has no scheduled-retry source or test diff. - No deployed provider run or tenant rollout is claimed by these source tests. ## Risks The exception applies only to a new normalized session and an exact completed plugin task. The old checkpoint remains evidence and cannot authorize reuse through this path. The original suspension failure can remain visible on its run. This fix prevents repeated startup failures after a reset; it does not hide unconfirmed cleanup or make every suspension succeed. Missing task receipts and failed file collection continue to block replacement. No schema migration, plugin protocol change, Rust Runner binary change, or new polling is required. Tests use a controlled task boundary; they do not certify every external provider implementation. ## Model Used OpenAI Codex, GPT-6 family, with tool use and local code execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal issue or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] Greptile Apex is 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
7962 lines
320 KiB
TypeScript
7962 lines
320 KiB
TypeScript
import { confirmAntigravityProbeNativeRetirement } from "../drivers/acpx/antigravity-probe-retirement.js";
|
|
import { resolveGrokDisableWebSearch, admittedGrokWebSearchPolicy } from "../drivers/acpx/grok-web-search.js";
|
|
import { PRP_COMPLETION_TOOL_NAME, PRP_COMPLETION_RESULT_BRIDGE_INPUT_SCHEMA } from "../contracts/completion-result.js";
|
|
import { RunnerLaunchPreparationStore } from "./runner-launch-preparation-store.js";
|
|
import type { PreparedRunnerProviderLaunch, RunnerProviderLaunchRequirements } from "./runner-provider-launch.js";
|
|
import { preparedRunnerLaunchWorkingDirectory, validatePreparedRunnerLaunch, runnerProviderLaunchRequirements, createRunnerProviderLaunch, createRunnerdCodexAppServerArgs, trustedRuntimeReadOnlyRoots } from "./runner-provider-launch.js";
|
|
export { createRunnerdCodexAppServerArgs, resolveRunnerdAcpxPermissionMode, trustedRuntimeReadOnlyRoots, withCodexCollaborationRuntimeInstructions } from "./runner-provider-launch.js";
|
|
import { applyProviderObservabilityEnvironment, redactObservabilityDiagnostic } from "../drivers/agent-observability.js";
|
|
import type { RunnerExecutionDriver } from "../control-plane/runner-execution-driver.js";
|
|
import { resolvePinnedOpenCodeCommand } from "../drivers/opencode/opencode-server-driver.js";
|
|
import { bindAcpxCandidateAdmission, parseAcpxCandidateAdmission, type AcpxCandidateAdmission, type AcpxCandidateStartupAdmission } from "../drivers/acpx/candidate-admission.js";
|
|
import { ACPX_CAPABILITY_PROFILES } from "../drivers/acpx/capability-profiles.js";
|
|
import { resolveClaudeEffort, admittedClaudeEffort } from "../drivers/acpx/claude-effort.js";
|
|
import { resolveGrokEffort, admittedGrokEffort } from "../drivers/acpx/grok-effort.js";
|
|
import { admittedPiThinkingLevel, resolvePiThinkingLevel } from "../drivers/acpx/pi-thinking.js";
|
|
import { configuredEnvironment } from "../configured-environment.js";
|
|
import { githubCredentialEnvironment } from "../github-credential-environment.js";
|
|
import { resolveAcpxProviderMode } from "../drivers/acpx/provider-mode.js";
|
|
import { preconfiguredHomeEnvironment, preconfiguredCodexHome, usesPreconfiguredHome } from "../drivers/preconfigured-home.js";
|
|
import { isAcpxCanonicalInputMethod } from "../drivers/acpx/profile-extensions.js";
|
|
import { RunnerdTraceFrameIndex } from "./runnerd-trace-frame-index.js";
|
|
import { waitForWarmAttachmentReadiness } from "./warm-attachment-readiness.js";
|
|
import { resolveCodexCommand } from "../drivers/codex/codex-command.js";
|
|
import { isCanonicalProviderEventType } from "../provider-events.js";
|
|
import { execFileSync } from "node:child_process";
|
|
import { readLinuxProcessStartedAt } from "./linux-process-start.js";
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import {
|
|
appendFileSync,
|
|
existsSync,
|
|
lstatSync,
|
|
mkdirSync,
|
|
mkdtempSync,
|
|
readFileSync,
|
|
readdirSync,
|
|
renameSync,
|
|
rmSync,
|
|
statSync,
|
|
writeFileSync,
|
|
} from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { basename, dirname, posix, resolve } from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
import type {
|
|
CodexAppServerTransport,
|
|
CodexRpcNotification,
|
|
CodexRpcServerRequest,
|
|
CodexServerRequestHandler,
|
|
CodexTraceInterpretation,
|
|
CodexTransportProcessInfo,
|
|
} from "../drivers/codex/app-server-transport.js";
|
|
import { createSanitizedCodexEnvironment } from "../drivers/codex/app-server-transport.js";
|
|
import {
|
|
CodexAppServerDriver,
|
|
codexSemanticToolSpecs,
|
|
} from "../drivers/codex/codex-app-server-driver.js";
|
|
import type {
|
|
DurableRecoveryCommittedEvent,
|
|
DurableRecoveryIdentity,
|
|
} from "../contracts/durable-recovery.js";
|
|
import type { NativeRunIdentity, NativeTurnControlCapabilities } from "../contracts/types.js";
|
|
import type { PrpEvent } from "../protocol/replay-contract.js";
|
|
import { NativeSessionCloseUnrecoverableError } from "../contracts/native-session-backend.js";
|
|
import type {
|
|
HarnessSession,
|
|
HarnessRuntimeRequestResolution,
|
|
PersistedHarnessProviderIdentity,
|
|
} from "../contracts/harness-driver.js";
|
|
import {
|
|
DurablePrpControlPlane,
|
|
SemanticToolNotDispatchedError,
|
|
durableRecoveryInternals,
|
|
inspectWarmRunTransition,
|
|
spawnRunner,
|
|
waitForProcess,
|
|
type RunnerProcessHandle,
|
|
type RunnerProcessConnection,
|
|
type DurablePrpControlPlaneOptions,
|
|
} from "../control-plane/durable-prp-control-plane.js";
|
|
import {
|
|
type QualifiedAcpxAgent,
|
|
} from "../drivers/acpx/qualified-profiles.js";
|
|
import { ACPX_CREDENTIAL_BINDING_ENV, createAcpxCredentialBinding, createSanitizedAcpxSpawnInput } from "../drivers/acpx/environment.js";
|
|
import {
|
|
createSanitizedAwsAgentCoreEnvironment,
|
|
createSanitizedClaudeManagedEnvironment,
|
|
} from "../drivers/claude-managed/environment.js";
|
|
import { retargetNativeSystemInstructions as retargetComposedInstructions, composeNativeSystemInstructions, type NativeRunRuntimeGrant, type NativeRuntimeContextSnapshot } from "../contracts/runtime-context.js";
|
|
import type {
|
|
NativeAcpxPermissionMode,
|
|
NativeOpenCodePermissionMode,
|
|
} from "../contracts/native-execution.js";
|
|
import { nativeMcpLaunchBinding } from "../drivers/native-mcp.js";
|
|
import {
|
|
prepareIsolatedCodexHome,
|
|
releaseMaterializedNativeRuntimeSkills,
|
|
} from "../drivers/runtime-context-materializer.js";
|
|
import { resolvePackagedRunnerBinary } from "./runner-binary.js";
|
|
import { RUNNERD_CANONICAL_ITEM, RUNNERD_RUNTIME_ACTIVITY } from "../drivers/codex/codex-driver-values.js";
|
|
|
|
// URL directory conversion preserves a trailing separator while path-derived
|
|
// build artifacts do not. Normalize once so a source build cannot be
|
|
// misclassified as an external provider pack by a string-only comparison.
|
|
const packageRoot = resolve(fileURLToPath(new URL("../..", import.meta.url)));
|
|
const executableSuffix = process.platform === "win32" ? ".exe" : "";
|
|
const MAX_NOTIFICATION_COUNT = 2_048;
|
|
const MAX_NOTIFICATION_BYTES = 4 * 1024 * 1024;
|
|
const RUNNER_CLIENT_VERSION = "0.3.0";
|
|
const RUNNER_BOOTSTRAP_TICKET_TTL_MS = 60_000;
|
|
const RUNNERD_MAX_OUTBOX_BYTES = 16 * 1024 * 1024;
|
|
const RUNNERD_P0_RESERVE_BYTES = 1024 * 1024;
|
|
|
|
function readLocalProcessStartedAt(pid: number): string | null {
|
|
if (!Number.isInteger(pid) || pid <= 0) return null;
|
|
try {
|
|
if (process.platform === "linux") {
|
|
return readLinuxProcessStartedAt(pid);
|
|
}
|
|
if (
|
|
["darwin", "freebsd", "openbsd", "aix", "sunos"].includes(
|
|
process.platform,
|
|
)
|
|
) {
|
|
const raw = execFileSync("ps", ["-o", "lstart=", "-p", String(pid)], {
|
|
encoding: "utf8",
|
|
timeout: 1_500,
|
|
windowsHide: true,
|
|
}).trim();
|
|
const parsed = new Date(raw);
|
|
return Number.isNaN(parsed.getTime()) ? null : parsed.toISOString();
|
|
}
|
|
if (process.platform === "win32") {
|
|
const script = [
|
|
`$target = Get-Process -Id ${pid} -ErrorAction Stop`,
|
|
"$target.StartTime.ToUniversalTime().ToString('o')",
|
|
].join("; ");
|
|
for (const command of ["powershell.exe", "pwsh.exe"]) {
|
|
try {
|
|
const raw = execFileSync(
|
|
command,
|
|
["-NoLogo", "-NoProfile", "-NonInteractive", "-Command", script],
|
|
{ encoding: "utf8", timeout: 1_500, windowsHide: true },
|
|
).trim();
|
|
const parsed = new Date(raw);
|
|
if (!Number.isNaN(parsed.getTime())) return parsed.toISOString();
|
|
} catch {
|
|
// Try the other supported PowerShell host.
|
|
}
|
|
}
|
|
}
|
|
} catch {
|
|
// Process exit and restricted process metadata both produce no fingerprint.
|
|
}
|
|
return null;
|
|
}
|
|
|
|
const CONTROL_PLANE_STATE_MAX_BYTES = 256 * 1024 * 1024;
|
|
|
|
function readControlPlaneState(directory: string): Record<string, unknown> {
|
|
const path = resolve(directory, "control-plane-state.json");
|
|
const metadata = lstatSync(path);
|
|
if (
|
|
metadata.isSymbolicLink() ||
|
|
!metadata.isFile() ||
|
|
metadata.size > CONTROL_PLANE_STATE_MAX_BYTES
|
|
) {
|
|
throw new Error("native_runner_control_plane_state_unsafe");
|
|
}
|
|
return record(JSON.parse(readFileSync(path, "utf8")));
|
|
}
|
|
|
|
function readRunnerState(path: string): Record<string, unknown> {
|
|
const metadata = lstatSync(path);
|
|
if (
|
|
metadata.isSymbolicLink() ||
|
|
!metadata.isFile() ||
|
|
metadata.size > 16 * 1024 * 1024
|
|
) {
|
|
throw new Error("native_runner_authority_rotation_state_unsafe");
|
|
}
|
|
return record(JSON.parse(readFileSync(path, "utf8")));
|
|
}
|
|
|
|
function controlPlaneIdentity(
|
|
state: Record<string, unknown>,
|
|
): DurableRecoveryIdentity {
|
|
return structuredClone(
|
|
record(state.identity) as unknown as DurableRecoveryIdentity,
|
|
);
|
|
}
|
|
|
|
function recoveryIdentityMatches(
|
|
value: DurableRecoveryIdentity | Record<string, unknown>,
|
|
expected: DurableRecoveryIdentity,
|
|
): boolean {
|
|
return (
|
|
value.runnerInstanceId === expected.runnerInstanceId &&
|
|
value.environmentLeaseId === expected.environmentLeaseId &&
|
|
value.runId === expected.runId &&
|
|
value.normalizedSessionId === expected.normalizedSessionId &&
|
|
value.turnId === expected.turnId &&
|
|
value.itemId === expected.itemId
|
|
);
|
|
}
|
|
|
|
function assertSuspendedRunnerState(
|
|
state: Record<string, unknown>,
|
|
expected: DurableRecoveryIdentity,
|
|
): void {
|
|
if (
|
|
state.schema !== "paperclip.runner.durable.state.v1" ||
|
|
!recoveryIdentityMatches(state, expected) ||
|
|
state.lifecycle !== "suspended"
|
|
) {
|
|
throw new Error("native_runner_authority_rotation_requires_settled_state");
|
|
}
|
|
}
|
|
|
|
function assertRealDirectory(path: string): void {
|
|
const metadata = lstatSync(path);
|
|
if (metadata.isSymbolicLink() || !metadata.isDirectory()) {
|
|
throw new Error("native_runner_authority_archive_unsafe");
|
|
}
|
|
}
|
|
|
|
function quarantineLocalRuntimeState(root: string, reason: unknown): never {
|
|
assertRealDirectory(root);
|
|
const quarantine = resolve(
|
|
dirname(root),
|
|
`${basename(root)}.quarantine-${randomUUID()}`,
|
|
);
|
|
renameSync(root, quarantine);
|
|
mkdirSync(root, { mode: 0o700 });
|
|
const detail = reason instanceof Error ? reason.message : String(reason);
|
|
throw new Error(
|
|
`native_runner_state_quarantined: ${detail}; the prior state was preserved for operator recovery`,
|
|
);
|
|
}
|
|
|
|
function authorityArchiveDirectory(
|
|
root: string,
|
|
identity: DurableRecoveryIdentity,
|
|
): string {
|
|
const digest = createHash("sha256")
|
|
.update(JSON.stringify(identity))
|
|
.digest("hex")
|
|
.slice(0, 24);
|
|
return resolve(root, "authority-epochs", `epoch-${digest}`);
|
|
}
|
|
|
|
function latestArchivedControlPlaneState(
|
|
root: string,
|
|
desired: DurableRecoveryIdentity,
|
|
): Record<string, unknown> | null {
|
|
const archivesRoot = resolve(root, "authority-epochs");
|
|
if (!existsSync(archivesRoot)) return null;
|
|
assertRealDirectory(archivesRoot);
|
|
const candidates = readdirSync(archivesRoot, { withFileTypes: true })
|
|
.filter((entry) => entry.isDirectory() && !entry.isSymbolicLink())
|
|
.map((entry) => resolve(archivesRoot, entry.name, "control-plane"))
|
|
.filter((directory) =>
|
|
existsSync(resolve(directory, "control-plane-state.json")),
|
|
)
|
|
.map((directory) => ({
|
|
directory,
|
|
modifiedAt: statSync(resolve(directory, "control-plane-state.json"))
|
|
.mtimeMs,
|
|
}))
|
|
.sort((left, right) => right.modifiedAt - left.modifiedAt);
|
|
for (const candidate of candidates) {
|
|
assertRealDirectory(candidate.directory);
|
|
const state = readControlPlaneState(candidate.directory);
|
|
const identity = controlPlaneIdentity(state);
|
|
if (
|
|
identity.runnerInstanceId === desired.runnerInstanceId &&
|
|
identity.environmentLeaseId === desired.environmentLeaseId &&
|
|
identity.normalizedSessionId === desired.normalizedSessionId
|
|
) {
|
|
return state;
|
|
}
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function rotateLocalAuthorityEpoch(
|
|
root: string,
|
|
controlPlaneState: Record<string, unknown>,
|
|
desired: DurableRecoveryIdentity,
|
|
): Record<string, unknown> {
|
|
const priorIdentity = controlPlaneIdentity(controlPlaneState);
|
|
if (
|
|
priorIdentity.runnerInstanceId !== desired.runnerInstanceId ||
|
|
priorIdentity.environmentLeaseId !== desired.environmentLeaseId ||
|
|
priorIdentity.normalizedSessionId !== desired.normalizedSessionId ||
|
|
priorIdentity.runId === desired.runId
|
|
) {
|
|
throw new Error(
|
|
"PRP recovery identity does not match the durable session binding",
|
|
);
|
|
}
|
|
const runnerDirectory = resolve(root, "runner");
|
|
const runnerStatePath = resolve(runnerDirectory, "runner-state.json");
|
|
const archive = authorityArchiveDirectory(root, priorIdentity);
|
|
const archivedControlPlane = resolve(archive, "control-plane");
|
|
const archivedRunnerState = resolve(archive, "runner-state.json");
|
|
const runnerStateSource = existsSync(runnerStatePath)
|
|
? runnerStatePath
|
|
: archivedRunnerState;
|
|
if (!existsSync(runnerStateSource)) {
|
|
throw new Error("native_runner_authority_rotation_state_unavailable");
|
|
}
|
|
assertRealDirectory(runnerDirectory);
|
|
const runnerState = readRunnerState(runnerStateSource);
|
|
if (
|
|
runnerState.runnerInstanceId !== priorIdentity.runnerInstanceId ||
|
|
runnerState.environmentLeaseId !== priorIdentity.environmentLeaseId ||
|
|
runnerState.runId !== priorIdentity.runId ||
|
|
runnerState.normalizedSessionId !== priorIdentity.normalizedSessionId ||
|
|
runnerState.turnId !== priorIdentity.turnId ||
|
|
runnerState.itemId !== priorIdentity.itemId ||
|
|
runnerState.lifecycle !== "suspended"
|
|
) {
|
|
throw new Error("native_runner_authority_rotation_requires_settled_state");
|
|
}
|
|
const archivesRoot = resolve(root, "authority-epochs");
|
|
if (existsSync(archivesRoot)) {
|
|
assertRealDirectory(archivesRoot);
|
|
} else {
|
|
mkdirSync(archivesRoot, { mode: 0o700 });
|
|
}
|
|
if (existsSync(archive)) {
|
|
assertRealDirectory(archive);
|
|
} else {
|
|
mkdirSync(archive, { mode: 0o700 });
|
|
}
|
|
assertRealDirectory(archive);
|
|
const activeControlPlane = resolve(root, "control-plane");
|
|
if (existsSync(activeControlPlane)) {
|
|
assertRealDirectory(activeControlPlane);
|
|
if (existsSync(archivedControlPlane)) {
|
|
throw new Error("native_runner_authority_archive_conflict");
|
|
}
|
|
renameSync(activeControlPlane, archivedControlPlane);
|
|
}
|
|
if (existsSync(runnerStatePath)) {
|
|
if (existsSync(archivedRunnerState)) {
|
|
throw new Error("native_runner_authority_archive_conflict");
|
|
}
|
|
renameSync(runnerStatePath, archivedRunnerState);
|
|
}
|
|
if (!existsSync(archivedControlPlane) || !existsSync(archivedRunnerState)) {
|
|
throw new Error("native_runner_authority_archive_incomplete");
|
|
}
|
|
return controlPlaneState;
|
|
}
|
|
|
|
async function rotateExternalAuthorityEpoch(
|
|
root: string,
|
|
controlPlaneState: Record<string, unknown>,
|
|
desired: DurableRecoveryIdentity,
|
|
readRunnerState: () => Promise<Record<string, unknown>>,
|
|
archiveRunnerState: (input: {
|
|
archiveKey: string;
|
|
priorIdentity: DurableRecoveryIdentity;
|
|
}) => Promise<Record<string, unknown>>,
|
|
): Promise<Record<string, unknown>> {
|
|
const priorIdentity = controlPlaneIdentity(controlPlaneState);
|
|
if (
|
|
priorIdentity.runnerInstanceId !== desired.runnerInstanceId ||
|
|
priorIdentity.environmentLeaseId !== desired.environmentLeaseId ||
|
|
priorIdentity.normalizedSessionId !== desired.normalizedSessionId ||
|
|
priorIdentity.runId === desired.runId
|
|
) {
|
|
throw new Error(
|
|
"PRP recovery identity does not match the durable session binding",
|
|
);
|
|
}
|
|
const archive = authorityArchiveDirectory(root, priorIdentity);
|
|
const archivedControlPlane = resolve(archive, "control-plane");
|
|
const activeControlPlane = resolve(root, "control-plane");
|
|
const archivesRoot = resolve(root, "authority-epochs");
|
|
if (existsSync(archivesRoot)) {
|
|
assertRealDirectory(archivesRoot);
|
|
} else {
|
|
mkdirSync(archivesRoot, { mode: 0o700 });
|
|
}
|
|
if (existsSync(archive)) {
|
|
assertRealDirectory(archive);
|
|
} else {
|
|
mkdirSync(archive, { mode: 0o700 });
|
|
}
|
|
assertRealDirectory(archive);
|
|
if (existsSync(archivedControlPlane)) {
|
|
// This directory is the durable transaction marker. A prior controller
|
|
// may have stopped before or after the remote move, so resume the same
|
|
// idempotent archive instead of starting a new external runner.
|
|
assertRealDirectory(archivedControlPlane);
|
|
if (existsSync(activeControlPlane)) {
|
|
throw new Error("native_runner_authority_archive_conflict");
|
|
}
|
|
const archivedIdentity = controlPlaneIdentity(
|
|
readControlPlaneState(archivedControlPlane),
|
|
);
|
|
if (!recoveryIdentityMatches(archivedIdentity, priorIdentity)) {
|
|
throw new Error("native_runner_authority_archive_conflict");
|
|
}
|
|
} else {
|
|
const runnerState = await readRunnerState();
|
|
assertSuspendedRunnerState(runnerState, priorIdentity);
|
|
assertRealDirectory(activeControlPlane);
|
|
renameSync(activeControlPlane, archivedControlPlane);
|
|
}
|
|
const archivedRunnerState = await archiveRunnerState({
|
|
archiveKey: basename(archive).replace(/^epoch-/, ""),
|
|
priorIdentity,
|
|
});
|
|
assertSuspendedRunnerState(archivedRunnerState, priorIdentity);
|
|
return controlPlaneState;
|
|
}
|
|
|
|
function projectCopilotRunGrant(
|
|
grant: NativeRunRuntimeGrant,
|
|
runnerFilesystemRoot?: string,
|
|
): NativeRunRuntimeGrant {
|
|
const projected = structuredClone(grant);
|
|
const context = projected.runtimeContext;
|
|
if (!runnerFilesystemRoot || !context) return projected;
|
|
const sourceInstructionRoot = context.instructions.bundle.rootPath;
|
|
context.instructions.bundle.rootPath = posix.join(runnerFilesystemRoot, "context", "instructions");
|
|
context.skills.forEach((skill, index) => {
|
|
skill.bundle.rootPath = posix.join(runnerFilesystemRoot, "context", "skills", `${index}-${skill.bundle.digest.slice(0, 12)}`);
|
|
});
|
|
projected.instructions = projected.instructions.replaceAll(sourceInstructionRoot, context.instructions.bundle.rootPath);
|
|
return projected;
|
|
}
|
|
|
|
function rotatedRunAttachPayload(
|
|
state: { commands?: unknown; runAttachTemplate?: unknown },
|
|
desired: DurableRecoveryIdentity,
|
|
authorizedTools: Record<string, unknown> | null,
|
|
completionContract:
|
|
{ revision: string; criterionIds: readonly string[] } | undefined,
|
|
runtimeContext?: NativeRuntimeContextSnapshot | null,
|
|
currentInstructions?: { text: string; context: NativeRuntimeContextSnapshot | null },
|
|
): Record<string, unknown> {
|
|
const commands = Array.isArray(state.commands)
|
|
? state.commands.map(record)
|
|
: [];
|
|
const persistedTemplate =
|
|
state.runAttachTemplate !== null &&
|
|
typeof state.runAttachTemplate === "object" &&
|
|
!Array.isArray(state.runAttachTemplate)
|
|
? (state.runAttachTemplate as Record<string, unknown>)
|
|
: null;
|
|
const commandSeed = [...commands]
|
|
.reverse()
|
|
.find(
|
|
(command) =>
|
|
(command.type === "run.prepare" || command.type === "run.attach") &&
|
|
record(command.payload).provider !== undefined,
|
|
);
|
|
const seed = persistedTemplate ?? record(commandSeed?.payload);
|
|
if (seed.provider === undefined)
|
|
throw new Error("native_runner_authority_rotation_seed_unavailable");
|
|
return retargetRunAttachPayload(
|
|
seed,
|
|
desired,
|
|
authorizedTools,
|
|
completionContract,
|
|
runtimeContext,
|
|
currentInstructions,
|
|
);
|
|
}
|
|
|
|
function retargetRunAttachPayload(
|
|
seedPayload: Record<string, unknown>,
|
|
desired: DurableRecoveryIdentity,
|
|
authorizedTools: Record<string, unknown> | null,
|
|
completionContract:
|
|
{ revision: string; criterionIds: readonly string[] } | undefined,
|
|
runtimeContext?: NativeRuntimeContextSnapshot | null,
|
|
currentInstructions?: { text: string; context: NativeRuntimeContextSnapshot | null },
|
|
): Record<string, unknown> {
|
|
const payload = structuredClone(seedPayload);
|
|
const provider = record(payload.provider);
|
|
if (provider.kind === "acpx" || provider.provider === "acpx") {
|
|
provider.runId = desired.runId;
|
|
provider.normalizedSessionId = desired.normalizedSessionId;
|
|
// A run-scoped registered instruction copy is collected after shutdown.
|
|
// Restore the durable provider identity with the current authenticated
|
|
// context, never the prior run's now-stale filesystem grant.
|
|
if (runtimeContext !== undefined) {
|
|
if (currentInstructions !== undefined) {
|
|
provider.instructions = currentInstructions.context
|
|
? retargetComposedInstructions(currentInstructions.text, currentInstructions.context, runtimeContext)
|
|
: currentInstructions.text;
|
|
} else if (typeof provider.instructions === "string" && provider.runtimeContext) {
|
|
provider.instructions = retargetComposedInstructions(
|
|
provider.instructions,
|
|
provider.runtimeContext as NativeRuntimeContextSnapshot,
|
|
runtimeContext,
|
|
);
|
|
}
|
|
provider.runtimeContext = structuredClone(runtimeContext);
|
|
}
|
|
payload.provider = provider;
|
|
}
|
|
if (authorizedTools !== null) payload.authorizedTools = authorizedTools;
|
|
if (completionContract !== undefined) {
|
|
payload.completionContract = {
|
|
revision: completionContract.revision,
|
|
criterionIds: [...completionContract.criterionIds],
|
|
};
|
|
}
|
|
return payload;
|
|
}
|
|
|
|
function recoveredRunAttachment(state: {
|
|
commands: readonly {
|
|
commandId: string;
|
|
type: string;
|
|
status: string;
|
|
}[];
|
|
committedEvents: readonly { eventType: string }[];
|
|
}): {
|
|
commandId: string;
|
|
status: string;
|
|
providerIdentityEventIndex: number;
|
|
} | null {
|
|
const command = [...state.commands]
|
|
.reverse()
|
|
.find((candidate) => candidate.type === "run.attach");
|
|
if (!command) return null;
|
|
const providerIdentityEventIndex =
|
|
command.status === "completed"
|
|
? latestProviderIdentityEventIndex(state.committedEvents)
|
|
: -1;
|
|
return {
|
|
commandId: command.commandId,
|
|
status: command.status,
|
|
providerIdentityEventIndex,
|
|
};
|
|
}
|
|
|
|
function latestProviderIdentityEventIndex(
|
|
events: readonly { eventType: string }[],
|
|
): number {
|
|
for (let index = events.length - 1; index >= 0; index -= 1) {
|
|
const eventType = events[index]?.eventType;
|
|
if (
|
|
eventType === "harness.ready" ||
|
|
eventType === "session.started" ||
|
|
eventType === "session.resumed"
|
|
) {
|
|
return index;
|
|
}
|
|
}
|
|
return -1;
|
|
}
|
|
|
|
function providerDrainStateFromSnapshot(state: Record<string, unknown>): {
|
|
pendingEventCount: number;
|
|
activeProviderTurnId: string | null;
|
|
providerSettled: boolean;
|
|
} {
|
|
if (
|
|
!Array.isArray(state.pendingEvents) ||
|
|
(state.queuedEvents !== undefined && !Array.isArray(state.queuedEvents)) ||
|
|
[state.activeProviderTurnId, state.activeTurnId].some(
|
|
(value) =>
|
|
value !== undefined &&
|
|
value !== null &&
|
|
(typeof value !== "string" || value.length === 0),
|
|
) ||
|
|
(state.ambiguousTurnStartPending !== undefined &&
|
|
typeof state.ambiguousTurnStartPending !== "boolean") ||
|
|
(state.providerExitUnconfirmed !== undefined &&
|
|
typeof state.providerExitUnconfirmed !== "boolean")
|
|
)
|
|
throw new Error("Provider drain state is malformed.");
|
|
const pending = Array.isArray(state.pendingEvents)
|
|
? state.pendingEvents.length
|
|
: 0;
|
|
const queued = Array.isArray(state.queuedEvents)
|
|
? state.queuedEvents.length
|
|
: 0;
|
|
const activeProviderTurnId =
|
|
[state.activeProviderTurnId, state.activeTurnId].find(
|
|
(value): value is string => typeof value === "string" && value.length > 0,
|
|
) ?? null;
|
|
return {
|
|
pendingEventCount: pending + queued,
|
|
activeProviderTurnId,
|
|
providerSettled:
|
|
activeProviderTurnId === null && state.ambiguousTurnStartPending !== true
|
|
&& state.providerExitUnconfirmed !== true,
|
|
};
|
|
}
|
|
|
|
type ProviderDrainState =
|
|
ReturnType<typeof providerDrainStateFromSnapshot> | "unreadable" | null;
|
|
|
|
async function awaitProviderDrainBarrier(input: {
|
|
readProviderState: () => ProviderDrainState;
|
|
semanticResultsSettled: () => boolean;
|
|
commands: () => readonly {
|
|
commandId: string;
|
|
status: string;
|
|
result?: unknown;
|
|
}[];
|
|
queueDrain: (commandId: string) => void;
|
|
pump: () => void;
|
|
deadline: number;
|
|
pollIntervalMs?: number;
|
|
}): Promise<boolean> {
|
|
let receiptConfirmed = false;
|
|
while (Date.now() < input.deadline) {
|
|
input.pump();
|
|
// A callback is not part of the provider FIFO until its result is durably
|
|
// queued and completed. Never certify a temporarily empty prefix while
|
|
// that admitted old-authority result is still being produced.
|
|
if (!input.semanticResultsSettled()) {
|
|
await new Promise((resolveWait) =>
|
|
setTimeout(resolveWait, input.pollIntervalMs ?? 5),
|
|
);
|
|
continue;
|
|
}
|
|
const state = input.readProviderState();
|
|
// Remote roots still require the exact runner-owned receipt. Their
|
|
// checkpoint separately verifies provider settlement on the remote host.
|
|
if (
|
|
receiptConfirmed &&
|
|
(state === null ||
|
|
(state !== "unreadable" &&
|
|
state.pendingEventCount === 0 &&
|
|
state.providerSettled))
|
|
)
|
|
return true;
|
|
if (state === "unreadable") {
|
|
await new Promise((resolveWait) =>
|
|
setTimeout(resolveWait, input.pollIntervalMs ?? 5),
|
|
);
|
|
continue;
|
|
}
|
|
const commandId = `command_close_drain_${randomUUID().replaceAll("-", "")}`;
|
|
input.queueDrain(commandId);
|
|
while (Date.now() < input.deadline) {
|
|
input.pump();
|
|
const command = input
|
|
.commands()
|
|
.find((candidate) => candidate.commandId === commandId);
|
|
if (command?.status === "completed") {
|
|
const proof = record(
|
|
record(command.result).result,
|
|
).retainedEventsDrained;
|
|
// Older runners and malformed/truthy values cannot certify closure.
|
|
if (typeof proof !== "boolean") return false;
|
|
receiptConfirmed = proof;
|
|
break;
|
|
}
|
|
if (command !== undefined && command.status !== "pending") return false;
|
|
await new Promise((resolveWait) =>
|
|
setTimeout(resolveWait, input.pollIntervalMs ?? 5),
|
|
);
|
|
}
|
|
}
|
|
return false;
|
|
}
|
|
|
|
function providerTurnIsActiveFromCommittedEvents(
|
|
events: readonly { eventType: string }[],
|
|
): boolean {
|
|
let active = false;
|
|
for (const event of events) {
|
|
if (event.eventType === "turn.started") active = true;
|
|
else if (
|
|
event.eventType === "turn.completed" ||
|
|
event.eventType === "turn.failed" ||
|
|
event.eventType === "turn.interrupted" ||
|
|
event.eventType === "turn.cancelled"
|
|
) {
|
|
active = false;
|
|
}
|
|
}
|
|
return active;
|
|
}
|
|
|
|
function turnStartResponseReady(input: {
|
|
responseEpoch: number;
|
|
observedEpoch: number;
|
|
expectedProviderTurnId: string;
|
|
boundTurnId: string;
|
|
}): boolean {
|
|
return (
|
|
input.responseEpoch === input.observedEpoch &&
|
|
input.expectedProviderTurnId.length > 0 &&
|
|
input.boundTurnId === input.expectedProviderTurnId
|
|
);
|
|
}
|
|
|
|
function turnStartNotificationDisposition(input: {
|
|
responsePending: boolean;
|
|
expectedProviderTurnId: string | null;
|
|
observedProviderTurnId: string;
|
|
}): "accept" | "defer" | "reject" {
|
|
if (!input.responsePending) return "accept";
|
|
if (input.expectedProviderTurnId === null) return "defer";
|
|
return input.observedProviderTurnId === input.expectedProviderTurnId
|
|
? "accept"
|
|
: "reject";
|
|
}
|
|
|
|
function turnStartCommandResultValid(input: {
|
|
requestedTurnId: string;
|
|
providerTurnId: string;
|
|
requireRequestedIdentity: boolean;
|
|
}): boolean {
|
|
return (
|
|
input.requestedTurnId.length > 0 &&
|
|
input.providerTurnId.length > 0 &&
|
|
(!input.requireRequestedIdentity ||
|
|
input.providerTurnId === input.requestedTurnId)
|
|
);
|
|
}
|
|
|
|
async function releaseRunnerProcessOwnership(input: {
|
|
runnerSettled: boolean;
|
|
checkpoint:
|
|
((settlement: "settled" | "unsettled") => Promise<void> | void) | null;
|
|
forceKill: () => Promise<void> | void;
|
|
release: (() => Promise<void> | void) | null;
|
|
retireController?: () => Promise<void>;
|
|
}): Promise<void> {
|
|
let releaseFailure: unknown;
|
|
try {
|
|
if (input.release !== null) await input.release();
|
|
} catch (error) {
|
|
releaseFailure = error;
|
|
}
|
|
let checkpointFailure: unknown;
|
|
try {
|
|
// External ingress must stop reconnecting before admitted journal writes
|
|
// can be joined. A failed retirement cannot authorize a new checkpoint.
|
|
await input.retireController?.();
|
|
if (input.checkpoint !== null) {
|
|
// Release only the authenticated control route first. In provider-ingress
|
|
// mode this stops its reconnect loop; it does not release the sandbox
|
|
// lease or the runner process owner. The bounded process wait above may
|
|
// observe the remote exec before this route has fully quiesced, while the
|
|
// exact durable state becomes readable only after it has. Keep the
|
|
// independently verified checkpoint ahead of process containment.
|
|
await input.checkpoint(input.runnerSettled ? "settled" : "unsettled");
|
|
}
|
|
} catch (error) {
|
|
checkpointFailure = error;
|
|
} finally {
|
|
await input.forceKill();
|
|
}
|
|
if (checkpointFailure !== undefined) throw checkpointFailure;
|
|
if (releaseFailure !== undefined) throw releaseFailure;
|
|
}
|
|
|
|
async function awaitRunnerSuspensionBarrier(input: {
|
|
commands: () => readonly {
|
|
commandId: string;
|
|
type: string;
|
|
status: string;
|
|
}[];
|
|
queueSuspend: (commandId: string) => void;
|
|
readRunnerState: () => Promise<Record<string, unknown>>;
|
|
runnerHasExited: () => Promise<boolean>;
|
|
pump: () => void;
|
|
deadline: number;
|
|
pollIntervalMs?: number;
|
|
}): Promise<boolean> {
|
|
if (Date.now() >= input.deadline) return false;
|
|
const readRunnerState = async (): Promise<Record<string, unknown> | undefined> => {
|
|
const remainingMs = input.deadline - Date.now();
|
|
if (remainingMs <= 0) return undefined;
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
try {
|
|
return await Promise.race([
|
|
input.readRunnerState(),
|
|
new Promise<undefined>(resolveDeadline => {
|
|
timer = setTimeout(() => resolveDeadline(undefined), remainingMs);
|
|
}),
|
|
]);
|
|
} finally {
|
|
clearTimeout(timer);
|
|
}
|
|
};
|
|
let existing = [...input.commands()]
|
|
.reverse()
|
|
.find(
|
|
(command) =>
|
|
command.type === "runner.suspend" &&
|
|
(command.status === "pending" || command.status === "completed"),
|
|
);
|
|
if (existing?.status === "completed") {
|
|
// A retained authority may have resumed since this command completed.
|
|
// Reuse its receipt only when the current exact durable state is already
|
|
// suspended; otherwise issue a new command rather than waiting on history.
|
|
try {
|
|
const state = await readRunnerState();
|
|
if (state === undefined) return false;
|
|
if (state.lifecycle === "suspended")
|
|
return Date.now() < input.deadline;
|
|
} catch {
|
|
// The normal bounded barrier below handles unavailable state.
|
|
}
|
|
existing = undefined;
|
|
}
|
|
if (Date.now() >= input.deadline) return false;
|
|
const commandId =
|
|
existing?.commandId ??
|
|
`command_close_suspend_${randomUUID().replaceAll("-", "")}`;
|
|
if (!existing) input.queueSuspend(commandId);
|
|
|
|
while (Date.now() < input.deadline) {
|
|
input.pump();
|
|
const command = input
|
|
.commands()
|
|
.find((candidate) => candidate.commandId === commandId);
|
|
if (
|
|
command !== undefined &&
|
|
command.status !== "pending" &&
|
|
command.status !== "completed"
|
|
) {
|
|
return false;
|
|
}
|
|
let lifecycle: unknown;
|
|
try {
|
|
const state = await readRunnerState();
|
|
if (state === undefined) return false;
|
|
lifecycle = state.lifecycle;
|
|
} catch {
|
|
// A remote filesystem can lag the command-result delivery by a small
|
|
// amount. Keep the single close deadline as the fail-closed bound.
|
|
}
|
|
if (Date.now() >= input.deadline) return false;
|
|
if (
|
|
command?.status === "completed" &&
|
|
lifecycle === "suspended" &&
|
|
Date.now() < input.deadline
|
|
) {
|
|
return true;
|
|
}
|
|
// Process completion alone is not a suspension proof. The durable state
|
|
// write precedes the terminal command result and process exit, so allow
|
|
// either observation to arrive first while staying within the same bound.
|
|
await input.runnerHasExited();
|
|
await new Promise<void>((resolveWait) =>
|
|
setTimeout(resolveWait, input.pollIntervalMs ?? 10),
|
|
);
|
|
}
|
|
return false;
|
|
}
|
|
|
|
function runnerCloseGraceMs(options: Pick<CapabilityRunnerdCodexTransportOptions, "provider" | "acpxAgent" | "closeGraceMs">): number {
|
|
// Native Pi retirement allows thirty seconds for snapshot cleanup and five
|
|
// for its lifetime fence. Keep a drain round trip and suspension reserve too.
|
|
// Explicit caller deadlines remain authoritative; all settlement proofs stay
|
|
// mandatory even when the provider already completed its task.
|
|
return options.closeGraceMs ?? (options.provider === "acpx" && options.acpxAgent === "pi" ? 45_000 : 10_000);
|
|
}
|
|
|
|
function runnerProcessReconnectGraceMs(options: CapabilityRunnerdCodexTransportOptions): number | undefined {
|
|
if (options.runnerProcessReconnectGraceMs === null) return undefined;
|
|
return options.runnerProcessReconnectGraceMs ?? options.runnerReconnectGraceMs;
|
|
}
|
|
|
|
function runnerCloseDeadlines(
|
|
startedAtMs: number,
|
|
graceMs: number,
|
|
): {
|
|
preparationDeadline: number;
|
|
closeDeadline: number;
|
|
} {
|
|
// Stopping a still-finishing provider and draining its suffix are best-effort
|
|
// preparation. Neither may consume the entire budget and enqueue suspend
|
|
// immediately before force-killing the runner. The suspension proof itself
|
|
// must retain a finite opportunity to cross the durable command boundary.
|
|
const suspensionReserveMs = Math.min(2_500, Math.ceil(graceMs / 2));
|
|
return {
|
|
preparationDeadline: startedAtMs + graceMs - suspensionReserveMs,
|
|
closeDeadline: startedAtMs + graceMs,
|
|
};
|
|
}
|
|
|
|
async function awaitAdoptedRunnerAuthentication(input: {
|
|
activeConnectionCount: () => number;
|
|
isAlive: () => Promise<boolean> | boolean;
|
|
throwIfFailed: () => void;
|
|
failure: Promise<never>;
|
|
ready?: () => Promise<void>;
|
|
timeoutMs: number;
|
|
}): Promise<void> {
|
|
if (!Number.isSafeInteger(input.timeoutMs) || input.timeoutMs <= 0) {
|
|
throw new Error("runnerReconnectGraceMs must be a positive safe integer");
|
|
}
|
|
const deadline = Date.now() + input.timeoutMs;
|
|
const timeoutError = () =>
|
|
new Error(
|
|
"native_adopted_runner_authentication_timeout: the existing runner did not authenticate within " +
|
|
`${input.timeoutMs}ms; preserve its process and durable session for operator recovery`,
|
|
);
|
|
let cancelled = false;
|
|
let pollTimer: NodeJS.Timeout | undefined;
|
|
let deadlineTimer: NodeJS.Timeout | undefined;
|
|
const checkDeadline = () => {
|
|
if (Date.now() >= deadline) throw timeoutError();
|
|
};
|
|
const observe = async () => {
|
|
await input.ready?.();
|
|
while (!cancelled) {
|
|
input.throwIfFailed();
|
|
checkDeadline();
|
|
if (input.activeConnectionCount() === 1) return;
|
|
const alive = await input.isAlive();
|
|
if (cancelled) return;
|
|
input.throwIfFailed();
|
|
checkDeadline();
|
|
if (!alive) {
|
|
throw new Error(
|
|
"native_adopted_runner_exited: runner exited before PRP authentication",
|
|
);
|
|
}
|
|
if (input.activeConnectionCount() === 1) return;
|
|
await new Promise<void>((resolveWait) => {
|
|
pollTimer = setTimeout(
|
|
resolveWait,
|
|
Math.min(25, deadline - Date.now()),
|
|
);
|
|
});
|
|
}
|
|
};
|
|
try {
|
|
await Promise.race([
|
|
observe(),
|
|
input.failure,
|
|
new Promise<never>((_resolve, reject) => {
|
|
deadlineTimer = setTimeout(
|
|
() => reject(timeoutError()),
|
|
input.timeoutMs,
|
|
);
|
|
}),
|
|
]);
|
|
} finally {
|
|
cancelled = true;
|
|
clearTimeout(pollTimer);
|
|
clearTimeout(deadlineTimer);
|
|
}
|
|
}
|
|
|
|
/** Preserve the sidecar identity independently of Rust's opaque item ID. */
|
|
export function bridgedAcpxPermissionParams(
|
|
event: Pick<DurableRecoveryCommittedEvent, "eventType" | "envelope">,
|
|
threadId: string,
|
|
turnId: string,
|
|
): Record<string, unknown> | null {
|
|
const request = record(record(record(event.envelope.payload).payload).request);
|
|
if (event.eventType !== "runtime_request.created") return null;
|
|
return acpxPermissionRequestParams(request, threadId, turnId);
|
|
}
|
|
|
|
function acpxPermissionRequestParams(request: Record<string, unknown>, threadId: string, turnId: string): Record<string, unknown> | null {
|
|
if (request.type !== "permission" || request.requestKind !== "permission_approval"
|
|
|| !["session/request_permission", "mcpServer/elicitation/request"].includes(String(record(request.origin).method))) return null;
|
|
if (record(request.origin).method === "mcpServer/elicitation/request") {
|
|
const details = record(request.details);
|
|
return { threadId, turnId, message: request.prompt, choices: request.choices,
|
|
serverName: details.serverName, mode: details.mode, _meta: details._meta, requestedSchema: details.requestedSchema, origin: request.origin };
|
|
}
|
|
const toolCallId = record(request.details).toolCallId;
|
|
// Match the permission adapter's 240-character bound. Never truncate, trim,
|
|
// hash, or substitute itemId: any of those would change the correlation key.
|
|
const validToolCallId = typeof toolCallId === "string"
|
|
&& toolCallId.trim().length > 0 && toolCallId.length <= 240
|
|
&& !/[\u0000-\u001f\u007f-\u009f]/.test(toolCallId);
|
|
return {
|
|
threadId,
|
|
turnId,
|
|
itemId: request.itemId,
|
|
reason: request.prompt,
|
|
choices: request.choices,
|
|
origin: record(request.origin),
|
|
...(validToolCallId ? { toolCallId } : {}),
|
|
};
|
|
}
|
|
|
|
/** Rehydrate only the pending ledger attested by the same live ACP session. */
|
|
export function liveAcpxRuntimeRequests(snapshot: Record<string, unknown>, threadId: string, turnId: string, durableTurnId: string): CodexRpcServerRequest[] {
|
|
if (snapshot.runtimeRequestsLive !== true || snapshot.provider !== "acpx"
|
|
|| snapshot.driverSessionId !== threadId || snapshot.activeProviderTurnId !== turnId
|
|
|| snapshot.runtimeRequestTurnId !== durableTurnId || !durableTurnId
|
|
|| !turnId || !Array.isArray(snapshot.pendingRuntimeRequests)
|
|
|| snapshot.pendingRuntimeRequests.length > 1024) {
|
|
throw new Error("ACPX pending request snapshot binding is invalid");
|
|
}
|
|
const ids = new Set<string>();
|
|
return snapshot.pendingRuntimeRequests.map(value => {
|
|
const request = record(value), origin = record(request.origin);
|
|
const id = request.requestId, method = origin.method;
|
|
if (typeof id !== "string" || !id || id.length > 160 || ids.has(id)
|
|
|| request.schema !== "paperclip.runtime_request.v2" || request.status !== "pending"
|
|
|| request.turnId !== durableTurnId || typeof method !== "string") {
|
|
throw new Error("ACPX pending request snapshot request is invalid");
|
|
}
|
|
ids.add(id);
|
|
const permission = request.type === "permission" && request.requestKind === "permission_approval"
|
|
&& method === "session/request_permission";
|
|
const params = permission
|
|
? acpxPermissionRequestParams(request, threadId, turnId)
|
|
: request.type === "input" && request.requestKind === "runtime" && isAcpxCanonicalInputMethod(method)
|
|
? bridgedCodexQuestionParams(request, method, threadId, turnId) : null;
|
|
if (!params) throw new Error("ACPX pending request snapshot form is invalid");
|
|
return { id, method, params };
|
|
});
|
|
}
|
|
|
|
export function bridgedCodexQuestionParams(
|
|
request: Record<string, unknown>,
|
|
method: string,
|
|
threadId: string,
|
|
turnId: string,
|
|
): Record<string, unknown> | null {
|
|
const questionSet = record(request.input);
|
|
if (
|
|
questionSet.schema !== "paperclip.question_set.v1" ||
|
|
!Array.isArray(questionSet.questions) ||
|
|
questionSet.questions.length === 0
|
|
)
|
|
return null;
|
|
const common = {
|
|
threadId,
|
|
turnId,
|
|
itemId:
|
|
typeof request.itemId === "string"
|
|
? request.itemId
|
|
: String(request.requestId ?? "runtime-input"),
|
|
};
|
|
// ACPX has already normalized and bound these IDs in Rust. Reconstructing a
|
|
// Codex form here would change option IDs and break the answer's return path.
|
|
if (isAcpxCanonicalInputMethod(method)) {
|
|
return { ...common, questionSet, origin: request.origin,
|
|
message: questionSet.description ?? questionSet.title ?? "A tool needs your input" };
|
|
}
|
|
if (method === "mcpServer/elicitation/request") {
|
|
const required: string[] = [];
|
|
const properties = Object.fromEntries(
|
|
questionSet.questions.map((candidate, index) => {
|
|
const question = record(candidate);
|
|
const id =
|
|
typeof question.id === "string"
|
|
? question.id
|
|
: `question-${index + 1}`;
|
|
if (question.required === true) required.push(id);
|
|
const validation = record(question.textValidation);
|
|
const options = Array.isArray(question.options)
|
|
? question.options.map((candidateOption) => {
|
|
const option = record(candidateOption);
|
|
return {
|
|
const: typeof option.id === "string" ? option.id : "option",
|
|
title:
|
|
typeof option.label === "string" ? option.label : "Option",
|
|
...(typeof option.description === "string"
|
|
? { description: option.description }
|
|
: {}),
|
|
};
|
|
})
|
|
: [];
|
|
const isBoolean =
|
|
question.answerMode === "single_select" &&
|
|
options.length === 2 &&
|
|
options[0]?.const === "true" &&
|
|
options[1]?.const === "false";
|
|
const inputType =
|
|
validation.inputType === "integer" ||
|
|
validation.inputType === "number"
|
|
? validation.inputType
|
|
: "string";
|
|
const scalarSchema = isBoolean
|
|
? { type: "boolean" }
|
|
: options.length > 0
|
|
? { type: "string", oneOf: options }
|
|
: {
|
|
type: inputType,
|
|
...(typeof validation.minLength === "number"
|
|
? { minLength: validation.minLength }
|
|
: {}),
|
|
...(typeof validation.maxLength === "number"
|
|
? { maxLength: validation.maxLength }
|
|
: {}),
|
|
...(typeof validation.minimum === "number"
|
|
? { minimum: validation.minimum }
|
|
: {}),
|
|
...(typeof validation.maximum === "number"
|
|
? { maximum: validation.maximum }
|
|
: {}),
|
|
...(typeof validation.pattern === "string"
|
|
? { pattern: validation.pattern }
|
|
: {}),
|
|
};
|
|
return [
|
|
id,
|
|
{
|
|
...(question.answerMode === "multi_select"
|
|
? { type: "array", items: scalarSchema }
|
|
: scalarSchema),
|
|
...(typeof question.header === "string"
|
|
? { title: question.header }
|
|
: typeof question.prompt === "string"
|
|
? { title: question.prompt }
|
|
: {}),
|
|
...(typeof question.helpText === "string"
|
|
? { description: question.helpText }
|
|
: {}),
|
|
},
|
|
];
|
|
}),
|
|
);
|
|
return {
|
|
...common,
|
|
message:
|
|
typeof questionSet.description === "string"
|
|
? questionSet.description
|
|
: typeof questionSet.title === "string"
|
|
? questionSet.title
|
|
: "A tool needs your input",
|
|
requestedSchema: {
|
|
type: "object",
|
|
properties,
|
|
...(required.length > 0 ? { required } : {}),
|
|
},
|
|
};
|
|
}
|
|
return {
|
|
...common,
|
|
...(typeof questionSet.title === "string"
|
|
? { title: questionSet.title }
|
|
: {}),
|
|
...(typeof questionSet.description === "string"
|
|
? { description: questionSet.description }
|
|
: {}),
|
|
...(typeof questionSet.submitLabel === "string"
|
|
? { submitLabel: questionSet.submitLabel }
|
|
: {}),
|
|
questions: questionSet.questions.map((candidate, index) => {
|
|
const question = record(candidate);
|
|
const validation = record(question.textValidation);
|
|
return {
|
|
id:
|
|
typeof question.id === "string"
|
|
? question.id
|
|
: `question-${index + 1}`,
|
|
...(typeof question.header === "string"
|
|
? { header: question.header }
|
|
: {}),
|
|
question:
|
|
typeof question.prompt === "string"
|
|
? question.prompt
|
|
: `Question ${index + 1}`,
|
|
...(typeof question.helpText === "string"
|
|
? { description: question.helpText }
|
|
: {}),
|
|
required: question.required === true,
|
|
...(question.answerMode === "multi_select"
|
|
? { multiSelect: true }
|
|
: {}),
|
|
...(Array.isArray(question.options)
|
|
? {
|
|
options: question.options.map((candidateOption, optionIndex) => {
|
|
const option = record(candidateOption);
|
|
return {
|
|
id:
|
|
typeof option.id === "string"
|
|
? option.id
|
|
: `option-${optionIndex + 1}`,
|
|
label:
|
|
typeof option.label === "string"
|
|
? option.label
|
|
: `Option ${optionIndex + 1}`,
|
|
...(typeof option.description === "string"
|
|
? { description: option.description }
|
|
: {}),
|
|
};
|
|
}),
|
|
}
|
|
: {}),
|
|
...(record(question.customAnswer).enabled === true
|
|
? { isOther: true }
|
|
: {}),
|
|
...(typeof validation.minLength === "number"
|
|
? { minLength: validation.minLength }
|
|
: {}),
|
|
...(typeof validation.maxLength === "number"
|
|
? { maxLength: validation.maxLength }
|
|
: {}),
|
|
};
|
|
}),
|
|
};
|
|
}
|
|
|
|
export interface CapabilityRunnerdProcessEvidence {
|
|
runnerPid: number | null;
|
|
runnerProcessGroupId: number | null;
|
|
providerPid: number | null;
|
|
providerProcessStartedAt: string | null;
|
|
codexPid: number | null;
|
|
codexProcessStartedAt: string | null;
|
|
sidecarPid: number | null;
|
|
sidecarProcessStartedAt: string | null;
|
|
agentPid: number | null;
|
|
agentProcessStartedAt: string | null;
|
|
providerDriver: string | null;
|
|
/** Effective native mode, populated only from the admitted Pi identity. */
|
|
piThinkingLevel?: "off" | "low" | "high" | "max";
|
|
claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max";
|
|
grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh";
|
|
grokDisableWebSearch?: boolean;
|
|
providerVersion: string | null;
|
|
acpxAgent: QualifiedAcpxAgent | null;
|
|
agentServerVersion: string | null;
|
|
agentRuntimeVersion: string | null;
|
|
acpProtocolVersion: number | null;
|
|
providerExecutionKind: "local_process" | "remote_service" | null;
|
|
providerService:
|
|
"anthropic_managed_agents" | "aws_bedrock_agentcore_harness" | null;
|
|
runnerExited: boolean;
|
|
runnerExitCode: number | null;
|
|
runnerSignal: NodeJS.Signals | null;
|
|
childEnvironmentKeys: string[];
|
|
diagnostics: string[];
|
|
}
|
|
|
|
export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionDriver {
|
|
observability?: import("../drivers/agent-observability.js").ProviderObservabilityLaunch;
|
|
homeMode?: "isolated" | "preconfigured";
|
|
provider?: "codex" | "opencode" | "claude_managed" | "aws_agentcore" | "acpx";
|
|
observeCredentials?: boolean;
|
|
onCredentialsUpdated?: (event: PrpEvent) => Promise<void>;
|
|
onAccountingEvent?: (event: PrpEvent) => Promise<void>;
|
|
recoverSemanticToolInput?: DurablePrpControlPlaneOptions["recoverSemanticToolInput"];
|
|
opencodePermissionMode?: NativeOpenCodePermissionMode;
|
|
acpxAgent?: QualifiedAcpxAgent;
|
|
/** Explicit evaluation-only candidate selection, never derived from persisted session input. */
|
|
acpxCandidateProfile?: "pi" | "cursor" | "copilot";
|
|
/** Server/operator authority; never derived from persisted agent or provider input. */
|
|
acpxCandidateAdmission?: AcpxCandidateAdmission;
|
|
acpxPermissionMode?: NativeAcpxPermissionMode;
|
|
acpxMode?: string;
|
|
piThinkingLevel?: "off" | "low" | "high" | "max";
|
|
claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max";
|
|
grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh";
|
|
grokDisableWebSearch?: boolean;
|
|
acpxPermissionModePinned?: boolean;
|
|
acpxSidecarPath?: string;
|
|
/** SHA-256 verified by the provider-pack authority before runner startup. */
|
|
acpxSidecarSha256?: string;
|
|
/** Node executable in the runner filesystem; required for remote JS providers. */
|
|
providerNodeCommand?: string;
|
|
/** SHA-256 verified by the provider-pack authority before runner startup. */
|
|
providerNodeCommandSha256?: string;
|
|
/** Digest of the build-owned provider-pack manifest that authorized remote artifacts. */
|
|
providerPackAuthorityDigest?: string;
|
|
acpxRuntimeDirectory?: string;
|
|
managedProfile?: {
|
|
profileId: string;
|
|
anthropicAgentId: string;
|
|
agentVersion: string;
|
|
environmentId: string;
|
|
betaVersion: "managed-agents-2026-04-01";
|
|
maxSessionListCostUsd: number;
|
|
model: string;
|
|
};
|
|
agentCoreProfile?: {
|
|
profileId: string;
|
|
region: string;
|
|
accountId: string;
|
|
harnessArn: string;
|
|
harnessVersion: string;
|
|
endpointArn: string;
|
|
endpointQualifier: string;
|
|
agentRuntimeArn: string;
|
|
memoryArn: string;
|
|
memoryId: string;
|
|
invocationRoleArn: string;
|
|
contextBucket: string;
|
|
contextPrefix: string;
|
|
contextKmsKeyArn: string;
|
|
qualificationRevision: string;
|
|
eventExpiryDays: 90;
|
|
maxEstimatedSessionCostUsd: number;
|
|
maxIterations: number;
|
|
maxOutputTokens: number;
|
|
timeoutSeconds: number;
|
|
model: string;
|
|
};
|
|
runnerBinary?: string;
|
|
/** Identity reported by an execution environment that owns its Runner artifact. */
|
|
runnerArtifact?: { version: string; digest: string };
|
|
codexCommand?: string;
|
|
codexArgs?: string[];
|
|
disableCodexProjectInstructions?: boolean;
|
|
/** Restrict hosted search without changing the provider default for other runs. */
|
|
disableCodexWebSearch?: boolean;
|
|
/** Controller-visible Codex home used only to seed the isolated runner home. */
|
|
sourceCodexHome?: string | null;
|
|
opencodeCommand?: string;
|
|
/** SHA-256 verified by the provider-pack authority before runner startup. */
|
|
opencodeCommandSha256?: string;
|
|
opencodeProxyPath?: string;
|
|
/** SHA-256 verified by the provider-pack authority before runner startup. */
|
|
opencodeProxySha256?: string;
|
|
opencodeRuntimeDirectory?: string;
|
|
environment?: NodeJS.ProcessEnv;
|
|
/** Provider system instructions supplied by a native execution caller. */
|
|
baseInstructions?: string;
|
|
closeGraceMs?: number;
|
|
/** Bounded provider turn-admission wait; defaults to 30 seconds. */
|
|
turnStartTimeoutMs?: number;
|
|
/** Controller-owned opt-in operation timing. No exporter lives in Runner. */
|
|
measureOperation?<T>(phase: string, work: () => Promise<T>, attributes?: Record<string, number | boolean>): Promise<T>;
|
|
recordTimingReceipt?(receipt: unknown): void;
|
|
onDiagnostic?: (message: string) => void;
|
|
onEvidence?: (evidence: Readonly<CapabilityRunnerdProcessEvidence>) => void;
|
|
/** Persist process ownership immediately after spawn, before waiting for
|
|
* provider bootstrap or activating a deferred PRP registration. */
|
|
onSpawn?: (meta: { pid: number; processGroupId: number | null; startedAt: string }) => Promise<void>;
|
|
stateDirectory?: string;
|
|
lifecyclePolicy?:
|
|
| { mode: "per_turn"; idleTimeoutMs: null }
|
|
| { mode: "warm"; idleTimeoutMs: number };
|
|
runtimeContext?: NativeRuntimeContextSnapshot | null;
|
|
/** Runtime-context paths rewritten for the runner-owned filesystem. */
|
|
runnerRuntimeContext?: NativeRuntimeContextSnapshot | null;
|
|
/** Root path visible to runnerd when it is not on the Paperclip host. */
|
|
runnerFilesystemRoot?: string;
|
|
/** Workspace cwd to retain when a local provider session is reopened. */
|
|
resumeWorkingDirectory?: string;
|
|
/**
|
|
* The provider process is already confined by a sandbox execution target.
|
|
* Codex must use its explicit external-sandbox policy because container
|
|
* runtimes such as Daytona intentionally omit nested namespace privileges.
|
|
*/
|
|
externallySandboxed?: boolean;
|
|
/** Current run's authority catalog, used when a suspended session is rebound. */
|
|
resumeDynamicTools?: readonly Readonly<Record<string, unknown>>[];
|
|
/** Current run's completion authority, rebound without changing provider identity. */
|
|
resumeCompletionContract?: {
|
|
revision: string;
|
|
criterionIds: readonly string[];
|
|
};
|
|
/** Provider turn recorded by the owner checkpoint when restoring an active run. */
|
|
resumeActiveTurnId?: string | null;
|
|
/**
|
|
* Exact provider identity from the database checkpoint. A verified adopted
|
|
* runner may use this when its original identity event has left the bounded
|
|
* PRP replay window. Recovery remains blocked until an authenticated live
|
|
* snapshot or a fresh identity event matches this checkpoint.
|
|
*/
|
|
resumeProviderSession?: {
|
|
driverSessionId: string;
|
|
providerSessionId?: string | null;
|
|
providerIdentity?: PersistedHarnessProviderIdentity;
|
|
};
|
|
/** Explicitly permits ACPX to rotate its provider-native session after a governed wait. */
|
|
providerRecoveryPolicy?:
|
|
| "same_session_only"
|
|
| "allow_replacement_after_resume_failure"
|
|
| "allow_replacement_after_governed_wait";
|
|
prpIdentity?: {
|
|
runnerInstanceId: string;
|
|
environmentLeaseId: string;
|
|
runId: string;
|
|
normalizedSessionId: string;
|
|
turnId: string;
|
|
itemId: string;
|
|
};
|
|
/** Existing server-owned routes only; never provision two ingress owners. */
|
|
warmTransitionRegistrationMode?: "routed_connect";
|
|
/** Rechecks the server-owned recovery claim after each asynchronous boundary. */
|
|
authorizeWarmTransitionRecovery?: (
|
|
stage: "before_bootstrap" | "before_spawn" | "before_authentication",
|
|
) => Promise<void>;
|
|
/** Retires recovery-only caller fences after a fresh exact post-ACK snapshot. */
|
|
onWarmTransitionRecoveryCompleted?: (input: {
|
|
transitionId: string;
|
|
}) => void | Promise<void>;
|
|
}
|
|
|
|
export type RunnerdCodexTransportOptions =
|
|
CapabilityRunnerdCodexTransportOptions;
|
|
|
|
export interface CapabilityRunnerdCodexTransport {
|
|
transport: CodexAppServerTransport;
|
|
evidence(): Readonly<CapabilityRunnerdProcessEvidence>;
|
|
/** Relinquish controller authority without stopping the durable runner. */
|
|
detachControllerForRestart(): Promise<void>;
|
|
/** Setup-only retained cleanup; joins original work and proves physical retirement. */
|
|
retireProbeResources?(timeoutMs?: number): Promise<void>;
|
|
}
|
|
|
|
export type RunnerdCodexTransport = CapabilityRunnerdCodexTransport;
|
|
|
|
export function unwrapRunnerdProviderNotifications(
|
|
input: unknown,
|
|
): Record<string, unknown>[] {
|
|
const payload = record(input);
|
|
if (typeof payload.method === "string") return [payload];
|
|
if (Array.isArray(payload.events)) {
|
|
return payload.events
|
|
.map(record)
|
|
.filter((event) => typeof event.method === "string");
|
|
}
|
|
const latest = record(payload.latest);
|
|
return typeof latest.method === "string" ? [latest] : [];
|
|
}
|
|
|
|
export function unwrapRunnerdProviderNotification(
|
|
input: unknown,
|
|
): Record<string, unknown> {
|
|
const notifications = unwrapRunnerdProviderNotifications(input);
|
|
return notifications.at(-1) ?? record(input);
|
|
}
|
|
|
|
export function latestRunnerdSessionReadiness(
|
|
events: readonly unknown[],
|
|
): Record<string, unknown> | null {
|
|
for (let index = events.length - 1; index >= 0; index -= 1) {
|
|
const event = record(events[index]);
|
|
if (
|
|
event.eventType !== "harness.ready" &&
|
|
event.eventType !== "session.started" &&
|
|
event.eventType !== "session.resumed"
|
|
) continue;
|
|
return record(record(record(event.envelope).payload).payload);
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export function unseenRunnerdCommittedEvents<
|
|
T extends { sourceSeq: number },
|
|
>(events: readonly T[], lastSourceSeq: number): T[] {
|
|
const unseen = events.filter((event) => event.sourceSeq > lastSourceSeq);
|
|
if (unseen.length === 0) return [];
|
|
let expectedSourceSeq = lastSourceSeq + 1;
|
|
for (const event of unseen) {
|
|
if (event.sourceSeq !== expectedSourceSeq) {
|
|
throw new Error(
|
|
`provider_notification_window_exceeded: expected source sequence ${expectedSourceSeq}, received ${event.sourceSeq}`,
|
|
);
|
|
}
|
|
expectedSourceSeq += 1;
|
|
}
|
|
return unseen;
|
|
}
|
|
|
|
export function expandRunnerdCanonicalNotifications(
|
|
method: string,
|
|
input: unknown,
|
|
eventType?: string,
|
|
): Array<{ method: string; params: Record<string, unknown> }> {
|
|
const payload = record(input);
|
|
const events = Array.isArray(payload.events) ? payload.events : [payload];
|
|
return events.map((event) => {
|
|
const params = record(event);
|
|
// A coalesced envelope can contain different event kinds. Classify each
|
|
// payload after expansion, never using the envelope's kind.
|
|
return {
|
|
method: eventType
|
|
? runnerdCanonicalNotificationMethod(eventType, params) ?? method
|
|
: method,
|
|
params,
|
|
};
|
|
});
|
|
}
|
|
|
|
/** Keep durable expiry visible to the facade before its terminal notification. */
|
|
export function runnerdRuntimeRequestExpiryNotification(
|
|
eventType: string, payload: Record<string, unknown>, threadId: string | null, turnId: string | null,
|
|
): CodexRpcNotification | null {
|
|
if (eventType !== "runtime_request.expired") return null;
|
|
return { method: "paperclip/runtimeRequestExpired", params: {
|
|
...(threadId ? { threadId } : {}), ...(turnId ? { turnId } : {}),
|
|
payload, itemId: payload.itemId,
|
|
} };
|
|
}
|
|
|
|
export function runnerdCanonicalNotificationMethod(
|
|
eventType: string,
|
|
payload: Record<string, unknown>,
|
|
): string | undefined {
|
|
// An empty open/resume snapshot is already returned by thread/goal/get and
|
|
// is not a provider-side clear transition. Do not insert a synthetic clear
|
|
// ahead of the first real turn notification.
|
|
if (eventType === "session.goal.snapshot" && payload.goal === null) {
|
|
return undefined;
|
|
}
|
|
// ACPX thoughts retain their reasoning kind through the notification facade.
|
|
// Treating them as assistant deltas exposes them in the task transcript and
|
|
// bypasses the existing reasoning redaction and progress handling.
|
|
if (eventType === "item.delta" && payload.kind === "reasoning") {
|
|
return payload.channel === "detail"
|
|
? "item/reasoning/textDelta"
|
|
: "item/reasoning/summaryTextDelta";
|
|
}
|
|
return (
|
|
{
|
|
"turn.started": "turn/started",
|
|
"item.started": "item/started",
|
|
"item.delta": "item/agentMessage/delta",
|
|
"item.completed": "item/completed",
|
|
"turn.completed": "turn/completed",
|
|
"turn.failed": "turn/completed",
|
|
"turn.interrupted": "turn/completed",
|
|
"turn.cancelled": "turn/completed",
|
|
"usage.reported": "thread/tokenUsage/updated",
|
|
"plan.updated": "turn/plan/updated",
|
|
"workspace.change.updated": "paperclip/workspaceChange/updated",
|
|
"run.result.proposed": "paperclip/runResult",
|
|
"session.goal.snapshot": "thread/goal/updated",
|
|
"session.goal.updated": "thread/goal/updated",
|
|
"session.goal.cleared": "thread/goal/cleared",
|
|
"session.updated":
|
|
payload.status === "budget_reached"
|
|
? "provider/budgetReached"
|
|
: "provider/sessionUpdated",
|
|
} as Record<string, string>
|
|
)[eventType];
|
|
}
|
|
|
|
export function resolveRunnerdSessionIdentity(input: unknown): {
|
|
processId: number | null;
|
|
threadId: string | null;
|
|
sessionId: string | null;
|
|
} {
|
|
const started = record(input);
|
|
const runtimeIdentity = record(started.runtimeIdentity);
|
|
const descriptor = record(started.providerDescriptor);
|
|
const processId =
|
|
runtimeIdentity.processId ??
|
|
runtimeIdentity.process_id ??
|
|
descriptor.processId ??
|
|
started.processId ??
|
|
started.pid;
|
|
const threadId =
|
|
started.threadId ?? started.driverSessionId ?? started.providerSessionId;
|
|
const sessionId =
|
|
started.sessionId ??
|
|
started.providerAccountSessionId ??
|
|
(started.driverSessionId === undefined
|
|
? undefined
|
|
: started.providerSessionId);
|
|
return {
|
|
processId: typeof processId === "number" ? processId : null,
|
|
threadId:
|
|
typeof threadId === "string" && threadId.length > 0 ? threadId : null,
|
|
sessionId:
|
|
typeof sessionId === "string" && sessionId.length > 0 ? sessionId : null,
|
|
};
|
|
}
|
|
|
|
type NotificationQueueEntry =
|
|
| { kind: "notification"; value: CodexRpcNotification; bytes: number }
|
|
| { kind: "control"; dispatch(): void; bytes: number };
|
|
|
|
class NotificationQueue implements AsyncIterable<CodexRpcNotification> {
|
|
#values: NotificationQueueEntry[] = [];
|
|
#waiters: Array<() => void> = [];
|
|
#bytes = 0;
|
|
#closed = false;
|
|
#error: Error | null = null;
|
|
|
|
push(value: CodexRpcNotification): void {
|
|
if (this.#closed) return;
|
|
this.#push({ kind: "notification", value, bytes: Buffer.byteLength(JSON.stringify(value)) });
|
|
}
|
|
|
|
pushControl(value: CodexRpcServerRequest, dispatch: () => void): void {
|
|
if (this.#closed) return;
|
|
// Count the retained request just like notification payloads. The callback
|
|
// is internal; a provider notification cannot create a control entry.
|
|
this.#push({ kind: "control", dispatch, bytes: Buffer.byteLength(JSON.stringify(value)) });
|
|
}
|
|
|
|
#push(entry: NotificationQueueEntry): void {
|
|
if (this.#closed) return;
|
|
if (this.#values.length >= MAX_NOTIFICATION_COUNT || this.#bytes + entry.bytes > MAX_NOTIFICATION_BYTES) {
|
|
throw new Error("PRP provider notification queue bound exceeded");
|
|
}
|
|
this.#values.push(entry);
|
|
this.#bytes += entry.bytes;
|
|
this.#waiters.shift()?.();
|
|
}
|
|
|
|
close(error?: Error): void {
|
|
if (this.#closed) return;
|
|
this.#closed = true;
|
|
this.#error = error ?? null;
|
|
this.#values = [];
|
|
this.#bytes = 0;
|
|
for (const wake of this.#waiters.splice(0)) wake();
|
|
}
|
|
|
|
[Symbol.asyncIterator](): AsyncIterator<CodexRpcNotification> {
|
|
return {
|
|
next: async () => {
|
|
for (;;) {
|
|
if (this.#error !== null) throw this.#error;
|
|
if (this.#closed) return { value: undefined, done: true };
|
|
const queued = this.#values.shift();
|
|
if (queued !== undefined) {
|
|
this.#bytes -= queued.bytes;
|
|
if (queued.kind === "control") {
|
|
// The consumer asks for next only after mapping the previous
|
|
// notification. Start the request here without awaiting the
|
|
// human response, so subsequent provider activity keeps flowing.
|
|
queued.dispatch();
|
|
continue;
|
|
}
|
|
return { value: queued.value, done: false };
|
|
}
|
|
await new Promise<void>(resolve => this.#waiters.push(resolve));
|
|
}
|
|
},
|
|
};
|
|
}
|
|
}
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: {};
|
|
}
|
|
|
|
function appServerThreadGoal(
|
|
value: unknown,
|
|
threadId: string,
|
|
): Record<string, unknown> | null {
|
|
const goal = record(value);
|
|
const objective = typeof goal.objective === "string"
|
|
? goal.objective.trim()
|
|
: "";
|
|
const rawStatus = typeof goal.status === "string" ? goal.status : "";
|
|
const status = rawStatus === "usage_limited"
|
|
? "usageLimited"
|
|
: rawStatus === "budget_limited"
|
|
? "budgetLimited"
|
|
: rawStatus;
|
|
if (
|
|
objective.length === 0 ||
|
|
![
|
|
"active",
|
|
"paused",
|
|
"blocked",
|
|
"limited",
|
|
"usageLimited",
|
|
"budgetLimited",
|
|
"complete",
|
|
].includes(status)
|
|
) return null;
|
|
|
|
const epochSeconds = (timestamp: unknown): number => {
|
|
if (typeof timestamp === "number" && Number.isFinite(timestamp)) {
|
|
return timestamp > 10_000_000_000 ? timestamp / 1_000 : timestamp;
|
|
}
|
|
if (typeof timestamp !== "string") return 0;
|
|
const milliseconds = Date.parse(timestamp);
|
|
return Number.isFinite(milliseconds) ? milliseconds / 1_000 : 0;
|
|
};
|
|
|
|
return {
|
|
threadId,
|
|
objective,
|
|
status,
|
|
tokenBudget:
|
|
typeof goal.tokenBudget === "number" ? goal.tokenBudget : null,
|
|
tokensUsed: typeof goal.tokensUsed === "number" ? goal.tokensUsed : 0,
|
|
timeUsedSeconds:
|
|
typeof goal.timeUsedSeconds === "number"
|
|
? goal.timeUsedSeconds
|
|
: typeof goal.elapsedSeconds === "number"
|
|
? goal.elapsedSeconds
|
|
: 0,
|
|
createdAt: epochSeconds(goal.createdAt),
|
|
updatedAt: epochSeconds(goal.updatedAt),
|
|
};
|
|
}
|
|
|
|
type PendingTraceRehydration = {
|
|
sourceEventId: string;
|
|
eventType: string;
|
|
visibleNotificationCount: number;
|
|
};
|
|
|
|
type PendingDriverTraceInterpretation = CodexTraceInterpretation;
|
|
|
|
function appendRunnerdRehydrationTrace(
|
|
index: RunnerdTraceFrameIndex,
|
|
tracePath: string | undefined,
|
|
sourceEventId: string,
|
|
eventType: string,
|
|
visibleNotificationCount: number,
|
|
debugSequence: number,
|
|
): "written" | "retry" | "not_applicable" {
|
|
if (!tracePath) return "not_applicable";
|
|
if (!existsSync(tracePath)) return "retry";
|
|
try {
|
|
const { frameId, nativeChannelSettled } = index.locate(
|
|
tracePath,
|
|
sourceEventId,
|
|
);
|
|
if (frameId === null)
|
|
return nativeChannelSettled ? "not_applicable" : "retry";
|
|
appendFileSync(
|
|
`${tracePath}.rehydration`,
|
|
`${JSON.stringify({
|
|
kind: "interpretation",
|
|
schema: "paperclip.provider_trace_interpretation.v1",
|
|
debugChannel: "typescript_runnerd_rehydration",
|
|
debugSequence,
|
|
frameId,
|
|
stage: "typescript_runnerd_rehydration",
|
|
ruleId: `runnerd.rehydrate.${eventType}`,
|
|
sourceEventId,
|
|
sourceEventType: eventType,
|
|
disposition: visibleNotificationCount > 0 ? "mapped" : "ignored",
|
|
emittedEventIds: visibleNotificationCount > 0 ? [sourceEventId] : [],
|
|
droppedFields: [],
|
|
fieldMappings: [
|
|
{
|
|
inputPath: "sourceEventId",
|
|
outputPath: "paperclipTrace.sourceEventId",
|
|
action: "copied",
|
|
reason:
|
|
"Preserved the durable event identity while rehydrating the provider notification",
|
|
},
|
|
{
|
|
inputPath: "eventType",
|
|
outputPath: "paperclipTrace.sourceEventType",
|
|
action: "renamed",
|
|
reason:
|
|
"Attached the canonical PRP type to the rehydrated notification",
|
|
},
|
|
],
|
|
reason:
|
|
visibleNotificationCount > 0
|
|
? "Canonical PRP event was rehydrated into the Codex driver notification contract"
|
|
: "Canonical PRP event did not produce a Codex driver notification",
|
|
})}\n`,
|
|
{ encoding: "utf8", mode: 0o600 },
|
|
);
|
|
return "written";
|
|
} catch {
|
|
// Debug delivery is deliberately outside run authority and never fails it.
|
|
return "retry";
|
|
}
|
|
}
|
|
|
|
function appendCodexDriverInterpretationTrace(
|
|
index: RunnerdTraceFrameIndex,
|
|
tracePath: string | undefined,
|
|
input: PendingDriverTraceInterpretation,
|
|
debugSequence: number,
|
|
): "written" | "retry" | "not_applicable" {
|
|
if (!tracePath) return "not_applicable";
|
|
if (!existsSync(tracePath)) return "retry";
|
|
try {
|
|
const { frameId, nativeChannelSettled } = index.locate(
|
|
tracePath,
|
|
input.sourceEventId,
|
|
);
|
|
if (frameId === null) {
|
|
return nativeChannelSettled ? "not_applicable" : "retry";
|
|
}
|
|
appendFileSync(
|
|
`${tracePath}.rehydration`,
|
|
`${JSON.stringify({
|
|
kind: "interpretation",
|
|
schema: "paperclip.provider_trace_interpretation.v1",
|
|
debugChannel: "typescript_runnerd_rehydration",
|
|
debugSequence,
|
|
frameId,
|
|
stage: "typescript_codex_driver_normalization",
|
|
ruleId: `codex_driver.normalize.${input.providerMethod}`,
|
|
sourceEventId: input.sourceEventId,
|
|
sourceEventType: input.sourceEventType,
|
|
disposition: input.disposition,
|
|
emittedEventIds: input.emittedEventIds,
|
|
droppedFields: [],
|
|
fieldMappings: [
|
|
{
|
|
inputPath: "params",
|
|
outputPath: "payload",
|
|
action: "normalized",
|
|
reason:
|
|
"Codex notification fields were normalized into canonical PRP event payloads",
|
|
},
|
|
...input.emittedEventIds.map((eventId) => ({
|
|
outputPath: `event:${eventId}`,
|
|
action: "derived",
|
|
reason:
|
|
"The driver emitted this canonical PRP event from the normalized notification",
|
|
})),
|
|
],
|
|
reason: input.reason,
|
|
})}\n`,
|
|
{ encoding: "utf8", mode: 0o600 },
|
|
);
|
|
return "written";
|
|
} catch {
|
|
// Debug delivery is deliberately outside run authority and never fails it.
|
|
return "retry";
|
|
}
|
|
}
|
|
|
|
function appendRunnerdTraceStatus(
|
|
tracePath: string,
|
|
input: {
|
|
status: "complete" | "incomplete";
|
|
reason: string | null;
|
|
debugSequence: number;
|
|
acknowledgedDebugSequence: number;
|
|
},
|
|
): void {
|
|
appendFileSync(
|
|
`${tracePath}.rehydration`,
|
|
`${JSON.stringify({
|
|
kind: "trace_status",
|
|
debugChannel: "typescript_runnerd_rehydration",
|
|
debugSequence: input.debugSequence,
|
|
status: input.status,
|
|
acknowledgedDebugSequence: input.acknowledgedDebugSequence,
|
|
reason: input.reason,
|
|
})}\n`,
|
|
{ encoding: "utf8", mode: 0o600 },
|
|
);
|
|
}
|
|
|
|
export function rehydrateRunnerdUsageNotification(
|
|
rawParams: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
activeTurnId: string,
|
|
): Record<string, unknown> {
|
|
return {
|
|
...rawParams,
|
|
// The normalized PRP usage event carries the provider session identity
|
|
// rather than replaying the provider's original threadId field. Rehydrate
|
|
// the Codex notification contract so the strict driver does not reject
|
|
// valid accounting as a cross-thread event.
|
|
// The strict facade is opened on the runner-owned harness thread. A
|
|
// providerSessionId can be a distinct backend/agent session (ACPX is one
|
|
// example), so it must remain metadata rather than become a thread bind.
|
|
threadId: openedThreadId,
|
|
turnId: activeTurnId,
|
|
tokenUsage: {
|
|
total: record(rawParams.cumulative),
|
|
runDelta: record(rawParams.runDelta),
|
|
// Normalized counters may contain placeholder zeroes. Keep the runner's
|
|
// authority beside the counters so ACPX/managed driver projections and
|
|
// durable replay cannot certify an incomplete delta as a free receipt.
|
|
runDeltaComplete: rawParams.runDeltaAvailable === true,
|
|
},
|
|
};
|
|
}
|
|
|
|
export function rehydrateRunnerdThreadTokenUsage(
|
|
cumulative: unknown,
|
|
): { total: Record<string, unknown> } | null {
|
|
const total = record(cumulative);
|
|
return Object.keys(total).length === 0 ? null : { total };
|
|
}
|
|
|
|
export function rehydrateRunnerdResultNotification(
|
|
result: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
activeTurnId: string,
|
|
itemId: string,
|
|
): Record<string, unknown> {
|
|
return {
|
|
threadId: openedThreadId,
|
|
// The durable controller turn belongs to the runner envelope, while a
|
|
// protocol facade may expose a different provider-native turn. The result
|
|
// is observed during the latter and must be bound to that active turn.
|
|
turnId: activeTurnId,
|
|
itemId,
|
|
result,
|
|
};
|
|
}
|
|
|
|
export function rehydrateRunnerdTurnNotification(
|
|
rawParams: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
activeTurnId: string,
|
|
method: "turn/started" | "turn/completed",
|
|
): Record<string, unknown> {
|
|
const rawTurn = record(rawParams.turn);
|
|
const providerTurnId =
|
|
typeof rawParams.providerTurnId === "string" &&
|
|
rawParams.providerTurnId.length > 0
|
|
? rawParams.providerTurnId
|
|
: null;
|
|
const rawTurnId =
|
|
providerTurnId ??
|
|
(typeof rawTurn.id === "string" && rawTurn.id.length > 0
|
|
? rawTurn.id
|
|
: typeof rawParams.turnId === "string" && rawParams.turnId.length > 0
|
|
? rawParams.turnId
|
|
: activeTurnId);
|
|
const boundTurnId =
|
|
method === "turn/completed" ? (providerTurnId ?? activeTurnId) : rawTurnId;
|
|
return {
|
|
...rawParams,
|
|
// A canonical runnerd terminal is bound by the authenticated PRP envelope.
|
|
// Its compact `turn.id` is the durable controller turn, not necessarily the
|
|
// provider turn exposed by the Codex facade. Restore both strict bindings.
|
|
threadId: openedThreadId,
|
|
turnId: boundTurnId,
|
|
turn: {
|
|
...rawTurn,
|
|
id: boundTurnId,
|
|
...(rawTurn.status === undefined && rawParams.status !== undefined
|
|
? { status: rawParams.status }
|
|
: {}),
|
|
...(rawTurn.error === undefined && rawParams.error !== undefined
|
|
? { error: rawParams.error }
|
|
: {}),
|
|
},
|
|
};
|
|
}
|
|
|
|
export function rehydrateRunnerdDeltaNotification(
|
|
rawParams: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
activeTurnId: string,
|
|
): Record<string, unknown> {
|
|
// Canonical PRP events already passed runner identity validation. Restore the
|
|
// provider binding just as for item starts/completions; the PRP controller
|
|
// turn is deliberately different from the provider's turn ID.
|
|
return {
|
|
...rawParams,
|
|
threadId: openedThreadId,
|
|
turnId: activeTurnId,
|
|
delta: rawParams.delta ?? rawParams.text,
|
|
};
|
|
}
|
|
|
|
export function rehydrateRunnerdItemNotification(
|
|
rawParams: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
activeTurnId: string,
|
|
): Record<string, unknown> {
|
|
const rawItem = record(rawParams.item);
|
|
const channel = rawItem.channel ?? rawParams.channel;
|
|
const providerPhase = rawItem.phase ?? rawParams.providerPhase;
|
|
const phase =
|
|
providerPhase ??
|
|
(channel === "final"
|
|
? "final_answer"
|
|
: channel === "progress"
|
|
? "commentary"
|
|
: undefined);
|
|
return {
|
|
...rawParams,
|
|
threadId: openedThreadId,
|
|
turnId: activeTurnId,
|
|
item: {
|
|
...rawItem,
|
|
[RUNNERD_CANONICAL_ITEM]: true,
|
|
id: rawItem.id ?? rawParams.itemId,
|
|
type: rawItem.type ?? rawParams.kind,
|
|
status: rawItem.status ?? rawParams.status,
|
|
text: rawItem.text ?? rawParams.text,
|
|
...(phase === undefined ? {} : { phase }),
|
|
...(channel === undefined ? {} : { channel }),
|
|
},
|
|
};
|
|
}
|
|
|
|
export function rehydrateRunnerdPlanNotification(
|
|
rawParams: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
activeTurnId: string,
|
|
): Record<string, unknown> {
|
|
const steps = Array.isArray(rawParams.steps) ? rawParams.steps : [];
|
|
return {
|
|
...rawParams,
|
|
threadId: openedThreadId,
|
|
turnId: activeTurnId,
|
|
// Rust has already normalized the provider's plan entries into PRP's
|
|
// { stepId, body, status } shape. Rebuild Codex's notification contract
|
|
// so the strict TypeScript driver can perform (and expose) its second
|
|
// interpretation stage instead of silently dropping the plan.
|
|
plan: Array.isArray(rawParams.plan)
|
|
? rawParams.plan
|
|
: steps.map((value) => {
|
|
const step = record(value);
|
|
return {
|
|
step: typeof step.body === "string" ? step.body : "",
|
|
status: typeof step.status === "string" ? step.status : "pending",
|
|
};
|
|
}),
|
|
};
|
|
}
|
|
|
|
export function rehydrateRunnerdWorkspaceChangeNotification(
|
|
rawParams: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
activeTurnId: string,
|
|
): Record<string, unknown> {
|
|
return {
|
|
threadId: openedThreadId,
|
|
turnId: activeTurnId,
|
|
// Rust already parsed and bounded the complete Codex turn snapshot. Keep
|
|
// that canonical value intact instead of consulting git or the workspace
|
|
// again in the TypeScript driver.
|
|
workspaceChange: structuredClone(rawParams),
|
|
};
|
|
}
|
|
|
|
export function rehydrateRunnerdGoalNotification(
|
|
rawParams: Record<string, unknown>,
|
|
openedThreadId: string,
|
|
method: "thread/goal/updated" | "thread/goal/cleared",
|
|
): Record<string, unknown> {
|
|
if (method === "thread/goal/cleared") {
|
|
return { ...rawParams, threadId: openedThreadId };
|
|
}
|
|
return {
|
|
...rawParams,
|
|
threadId: openedThreadId,
|
|
goal: appServerThreadGoal(rawParams.goal, openedThreadId),
|
|
};
|
|
}
|
|
|
|
function commandDigest(value: unknown): string {
|
|
return `sha256:${createHash("sha256").update(durableRecoveryInternals.canonicalJson(value)).digest("hex")}`;
|
|
}
|
|
|
|
function approvedRunnerArtifact(runnerBinaryPath: string): {
|
|
version: string;
|
|
digest: string;
|
|
} {
|
|
return {
|
|
version: RUNNER_CLIENT_VERSION,
|
|
digest: `sha256:${createHash("sha256")
|
|
.update(readFileSync(runnerBinaryPath))
|
|
.digest("hex")}`,
|
|
};
|
|
}
|
|
|
|
/** Reads the caller-selected artifact binding; does not approve or execute it. */
|
|
export function readRunnerdArtifactBinding(runnerBinaryPath: string): {
|
|
version: string;
|
|
digest: string;
|
|
} {
|
|
return approvedRunnerArtifact(runnerBinaryPath);
|
|
}
|
|
|
|
export interface RetainedRunnerdCleanupProof {
|
|
readonly binding: Readonly<NativeRunIdentity>;
|
|
readonly identity: Readonly<DurableRecoveryIdentity>;
|
|
readonly backend: Readonly<{ kind: string; name: string }>;
|
|
readonly providerSessionId: string;
|
|
readonly activationDirectory: string;
|
|
readonly sourceFingerprint: string;
|
|
readonly settledFingerprint: string;
|
|
}
|
|
|
|
/** Content-free evidence for this controller's exact child handle. A launch
|
|
* intent without a matching retirement is deliberately not absence proof. */
|
|
export type RetainedRunnerdMaintenanceEpochReceipt = {
|
|
schema: "paperclip.native_cleanup_runner_epoch.v1";
|
|
requestId: string;
|
|
epoch: number;
|
|
launchId: string;
|
|
stateDirectory: string;
|
|
initialFingerprint: string;
|
|
runnerArtifact: { path: string; version: string; digest: string };
|
|
} & (
|
|
| { phase: "launch_intent" }
|
|
| {
|
|
phase: "spawned";
|
|
pid: number;
|
|
processGroupId: number;
|
|
processStartedAt: string;
|
|
spawnedAt: string;
|
|
}
|
|
| {
|
|
phase: "retired";
|
|
pid: number;
|
|
processGroupId: number;
|
|
processStartedAt: string;
|
|
spawnedAt: string;
|
|
exitCode: number | null;
|
|
exitSignal: NodeJS.Signals | null;
|
|
processGroupAbsent: true;
|
|
retiredAt: string;
|
|
finalFingerprint: string;
|
|
}
|
|
);
|
|
|
|
const retainedRunnerdCleanupProofs = new WeakMap<object, readonly number[]>();
|
|
const retainedMaintenanceOperations = new Map<string, Set<Promise<unknown>>>();
|
|
const activeMaintenanceRoots = new Set<string>();
|
|
const maintenanceIdleWaiters = new Map<string, { promise: Promise<void>; resolve: () => void }>();
|
|
|
|
/** No absence claim about an old child; only this controller's joined work. */
|
|
export function retainedRunnerdMaintenanceIsIdle(directory: string): boolean {
|
|
const root = resolve(directory);
|
|
return !activeMaintenanceRoots.has(root) && !retainedMaintenanceOperations.has(root);
|
|
}
|
|
|
|
|
|
|
|
export function waitForRetainedRunnerdMaintenanceIdle(directory: string): Promise<void> {
|
|
const root = resolve(directory);
|
|
if (retainedRunnerdMaintenanceIsIdle(root)) return Promise.resolve();
|
|
const existing = maintenanceIdleWaiters.get(root);
|
|
if (existing) return existing.promise;
|
|
let release!: () => void;
|
|
const promise = new Promise<void>(done => { release = done; });
|
|
maintenanceIdleWaiters.set(root, { promise, resolve: release });
|
|
return promise;
|
|
}
|
|
|
|
function releaseMaintenanceIdleWaiters(root: string) {
|
|
if (!retainedRunnerdMaintenanceIsIdle(root)) return;
|
|
const waiting = maintenanceIdleWaiters.get(root);
|
|
maintenanceIdleWaiters.delete(root);
|
|
waiting?.resolve();
|
|
}
|
|
|
|
/** A timeout revokes cleanup authority, not ownership of an already-started
|
|
* callback. Shutdown must join the original operations, including any that
|
|
* another retained callback registers while the current snapshot settles. */
|
|
export async function drainRetainedRunnerdMaintenanceOperations(): Promise<void> {
|
|
while (activeMaintenanceRoots.size > 0 || retainedMaintenanceOperations.size > 0) {
|
|
const roots = new Set([...activeMaintenanceRoots, ...retainedMaintenanceOperations.keys()]);
|
|
await Promise.all([...roots].map(waitForRetainedRunnerdMaintenanceIdle));
|
|
}
|
|
}
|
|
|
|
const MAINTENANCE_STATE_FILES = [
|
|
"control-plane/control-plane-state.json",
|
|
"runner/runner-state.json",
|
|
"runner/codex-provider-state.json",
|
|
] as const;
|
|
|
|
function maintenanceDenied(): Error {
|
|
return new Error("native_cleanup_maintenance_unproven");
|
|
}
|
|
|
|
function maintenanceProcessAbsent(pid: number): boolean {
|
|
if (!Number.isSafeInteger(pid) || pid <= 0 || process.platform === "win32")
|
|
return false;
|
|
return [pid, -pid].every((target) => {
|
|
try {
|
|
process.kill(target, 0);
|
|
return false;
|
|
} catch (error) {
|
|
return record(error).code === "ESRCH";
|
|
}
|
|
});
|
|
}
|
|
|
|
function readMaintenanceState(root: string) {
|
|
assertRealDirectory(root);
|
|
assertRealDirectory(resolve(root, "runner"));
|
|
assertRealDirectory(resolve(root, "control-plane"));
|
|
const bytes = MAINTENANCE_STATE_FILES.map((file) => {
|
|
const path = resolve(root, file);
|
|
const stat = lstatSync(path);
|
|
if (stat.isSymbolicLink() || !stat.isFile() || stat.size > 32 * 1024 * 1024)
|
|
throw maintenanceDenied();
|
|
return readFileSync(path);
|
|
});
|
|
const [control, runner, provider] = bytes.map((value) =>
|
|
record(JSON.parse(value.toString("utf8"))),
|
|
);
|
|
return {
|
|
control: control!,
|
|
runner: runner!,
|
|
provider: provider!,
|
|
providerFingerprint: createHash("sha256").update(bytes[2]!).digest("hex"),
|
|
fingerprint: createHash("sha256")
|
|
.update(
|
|
JSON.stringify(
|
|
bytes.map((value) =>
|
|
createHash("sha256").update(value).digest("hex"),
|
|
),
|
|
),
|
|
)
|
|
.digest("hex"),
|
|
};
|
|
}
|
|
|
|
/** An exact completed receipt is delivery evidence, never launch authority.
|
|
* The caller must additionally own the preceding retired maintenance epoch. */
|
|
function completedMaintenanceTerminalReceipt(
|
|
state: ReturnType<typeof readMaintenanceState>,
|
|
) {
|
|
const pending = record(state.runner.pendingTerminalDelivery);
|
|
const commands = state.control.commands as Array<Record<string, unknown>>;
|
|
if (
|
|
state.runner.lifecycle !== "suspended" ||
|
|
pending.commandType !== "runner.suspend" ||
|
|
pending.lifecycle !== "suspended" ||
|
|
typeof pending.commandId !== "string" ||
|
|
!pending.commandId ||
|
|
!Number.isSafeInteger(pending.controllerSeq) ||
|
|
Number(pending.controllerSeq) <= 0 ||
|
|
pending.controllerSeq !== state.runner.lastControllerCommandSeq ||
|
|
state.runner.pendingProviderCleanup != null ||
|
|
state.provider.lifecycle !== "prepared" ||
|
|
state.provider.activeProviderTurnId != null ||
|
|
!Array.isArray(commands)
|
|
)
|
|
return null;
|
|
const matching = commands.filter(
|
|
(command) => command.commandId === pending.commandId,
|
|
);
|
|
const command = matching[0];
|
|
const result = record(
|
|
record(state.runner.processedCommands)[pending.commandId],
|
|
);
|
|
if (
|
|
matching.length !== 1 ||
|
|
!command ||
|
|
command.type !== pending.commandType ||
|
|
command.controllerSeq !== pending.controllerSeq ||
|
|
result.commandId !== pending.commandId ||
|
|
result.commandType !== pending.commandType ||
|
|
result.controllerSeq !== pending.controllerSeq ||
|
|
result.status !== "completed" ||
|
|
record(result.result).status !== "completed"
|
|
)
|
|
return null;
|
|
// Match Rust's serialized Command, including nullable defaulted fields.
|
|
const wire = {
|
|
schema: command.schema,
|
|
commandId: command.commandId,
|
|
controllerSeq: command.controllerSeq,
|
|
type: command.type,
|
|
issuedAt: command.issuedAt,
|
|
deadlineAt: command.deadlineAt ?? null,
|
|
precondition: command.precondition ?? null,
|
|
payload: command.payload,
|
|
};
|
|
if (
|
|
record(state.runner.processedCommandFingerprints)[pending.commandId] !==
|
|
commandDigest(wire).slice("sha256:".length)
|
|
)
|
|
return null;
|
|
if (command.status === "pending") {
|
|
// Welcome advertises only the first pending command. Absence from that
|
|
// one-element list cannot prove a later terminal result was delivered.
|
|
if (
|
|
commands.find((entry) => entry.status === "pending") !== command ||
|
|
command.result != null
|
|
)
|
|
return null;
|
|
} else if (
|
|
command.status !== "completed" ||
|
|
commandDigest(command.result) !== commandDigest(result)
|
|
) {
|
|
return null;
|
|
}
|
|
return { commandId: pending.commandId, result };
|
|
}
|
|
|
|
function completedMaintenanceTerminalReplayMatches(
|
|
before: ReturnType<typeof readMaintenanceState>,
|
|
after: ReturnType<typeof readMaintenanceState>,
|
|
) {
|
|
const receipt = completedMaintenanceTerminalReceipt(before);
|
|
if (!receipt) return false;
|
|
const expectedCommands = (
|
|
before.control.commands as Array<Record<string, unknown>>
|
|
).map((command) =>
|
|
command.commandId === receipt.commandId
|
|
? { ...command, status: "completed", result: receipt.result }
|
|
: command,
|
|
);
|
|
return (
|
|
after.runner.lifecycle === "suspended" &&
|
|
after.runner.pendingTerminalDelivery == null &&
|
|
after.runner.pendingProviderCleanup == null &&
|
|
after.providerFingerprint === before.providerFingerprint &&
|
|
commandDigest(after.runner.processedCommands) ===
|
|
commandDigest(before.runner.processedCommands) &&
|
|
commandDigest(after.runner.processedCommandFingerprints) ===
|
|
commandDigest(before.runner.processedCommandFingerprints) &&
|
|
after.runner.lastControllerCommandSeq ===
|
|
before.runner.lastControllerCommandSeq &&
|
|
commandDigest(after.control.commands) === commandDigest(expectedCommands)
|
|
);
|
|
}
|
|
|
|
function assertMaintenanceBinding(
|
|
state: ReturnType<typeof readMaintenanceState>,
|
|
identity: DurableRecoveryIdentity,
|
|
providerSessionId: string,
|
|
allowRestoringOpen = false,
|
|
) {
|
|
if (
|
|
!recoveryIdentityMatches(record(state.control.identity), identity) ||
|
|
!recoveryIdentityMatches(state.runner, identity) ||
|
|
state.runner.schema !== "paperclip.runner.durable.state.v1" ||
|
|
state.provider.schema !== "paperclip.runner.codex-provider-state.v1" ||
|
|
record(state.provider.config).provider !== "codex" ||
|
|
state.provider.threadId !== providerSessionId ||
|
|
!Array.isArray(state.runner.outbox) ||
|
|
!Array.isArray(state.provider.pendingEvents) ||
|
|
!Array.isArray(state.provider.queuedEvents) ||
|
|
!Array.isArray(state.control.commands) ||
|
|
!Array.isArray(state.control.committedEvents) ||
|
|
Object.keys(record(record(state.provider.toolBridge).pending)).length !==
|
|
0 ||
|
|
state.provider.ambiguousTurnStartPending === true ||
|
|
!(
|
|
allowRestoringOpen
|
|
? ["turn_active", "prepared", "session_open"]
|
|
: ["turn_active", "prepared"]
|
|
).includes(String(state.provider.lifecycle))
|
|
)
|
|
throw maintenanceDenied();
|
|
}
|
|
|
|
/** A proof cannot be manufactured by JSON or reused after the activated
|
|
* checkpoint or any of its exact process owners changes. */
|
|
export function retainedRunnerdCleanupProofIsCurrent(
|
|
proof: RetainedRunnerdCleanupProof,
|
|
): boolean {
|
|
const pids = retainedRunnerdCleanupProofs.get(proof);
|
|
if (!pids || !pids.every(maintenanceProcessAbsent)) return false;
|
|
try {
|
|
const state = readMaintenanceState(proof.activationDirectory);
|
|
assertMaintenanceBinding(state, proof.identity, proof.providerSessionId);
|
|
return (
|
|
state.fingerprint === proof.settledFingerprint &&
|
|
state.runner.lifecycle === "suspended" &&
|
|
state.runner.pendingTerminalDelivery == null &&
|
|
state.runner.pendingProviderCleanup == null
|
|
);
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
/** Settle an inventoried COPY of one retained local Codex authority. This is
|
|
* deliberately not a NativeSession: it has no turn-start, tool execution,
|
|
* result selection, or authority-rotation API. The embedding server owns the
|
|
* durable recovery lease, original-source proof, and atomic activation. */
|
|
export interface RetainedRunnerdMaintenanceInput {
|
|
requestId: string;
|
|
binding: NativeRunIdentity;
|
|
identity: DurableRecoveryIdentity;
|
|
backend: { kind: string; name: string };
|
|
stateDirectory: string;
|
|
activationDirectory: string;
|
|
sourceFingerprint: string;
|
|
providerSessionId: string;
|
|
originalRunnerPid: number;
|
|
originalProviderPid: number;
|
|
runnerBinary?: string;
|
|
environment?: NodeJS.ProcessEnv;
|
|
sourceCodexHome?: string | null;
|
|
authorize: () => Promise<void>;
|
|
appendEvent: (event: PrpEvent) => Promise<void>;
|
|
recordEpoch: (
|
|
receipt: RetainedRunnerdMaintenanceEpochReceipt,
|
|
) => Promise<void>;
|
|
signal?: AbortSignal;
|
|
}
|
|
|
|
export async function settleRetainedRunnerdSession(
|
|
input: RetainedRunnerdMaintenanceInput,
|
|
): Promise<RetainedRunnerdCleanupProof> {
|
|
const root = resolve(input.stateDirectory);
|
|
if (!retainedRunnerdMaintenanceIsIdle(root)) throw maintenanceDenied();
|
|
activeMaintenanceRoots.add(root);
|
|
try {
|
|
return await settleRetainedRunnerdSessionOwned(input);
|
|
} finally {
|
|
activeMaintenanceRoots.delete(root);
|
|
releaseMaintenanceIdleWaiters(root);
|
|
}
|
|
}
|
|
|
|
async function settleRetainedRunnerdSessionOwned(
|
|
input: RetainedRunnerdMaintenanceInput,
|
|
): Promise<RetainedRunnerdCleanupProof> {
|
|
const root = resolve(input.stateDirectory);
|
|
if (
|
|
!input.requestId ||
|
|
input.requestId.length > 160 ||
|
|
/[\x00-\x1f]/.test(input.requestId) ||
|
|
retainedMaintenanceOperations.has(root) ||
|
|
root === resolve(input.activationDirectory) ||
|
|
input.binding.runId !== input.identity.runId ||
|
|
input.binding.sessionId !== input.identity.normalizedSessionId ||
|
|
!input.providerSessionId
|
|
)
|
|
throw maintenanceDenied();
|
|
const initial = readMaintenanceState(root);
|
|
assertMaintenanceBinding(initial, input.identity, input.providerSessionId);
|
|
if (initial.fingerprint !== input.sourceFingerprint)
|
|
throw maintenanceDenied();
|
|
const originalCommands = initial.control.commands as Array<
|
|
Record<string, unknown>
|
|
>;
|
|
if (
|
|
originalCommands.some(
|
|
(command) =>
|
|
command.status === "pending" &&
|
|
!["turn.stop", "runner.drain", "runner.suspend"].includes(
|
|
String(command.type),
|
|
),
|
|
)
|
|
)
|
|
throw maintenanceDenied();
|
|
const retainedEvents = [
|
|
...(initial.runner.outbox as unknown[]).map((event) =>
|
|
record(record(record(event).envelope).payload),
|
|
),
|
|
...(initial.provider.pendingEvents as unknown[]).map(record),
|
|
...(initial.provider.queuedEvents as unknown[]).map(record),
|
|
];
|
|
// This bounded recovery does not resolve or redeliver any tool input, even
|
|
// a historical input whose result might already exist in the old journal.
|
|
if (
|
|
retainedEvents.some((event) =>
|
|
[
|
|
"semantic_tool.input",
|
|
"mcp_app.tool_input",
|
|
"runtime.input.requested",
|
|
"runtime_request.created",
|
|
].includes(String(event.eventType)),
|
|
)
|
|
)
|
|
throw maintenanceDenied();
|
|
for (const event of retainedEvents) {
|
|
if (
|
|
!["session.started", "session.resumed", "harness.ready"].includes(
|
|
String(event.eventType),
|
|
)
|
|
)
|
|
continue;
|
|
const identity = resolveRunnerdSessionIdentity(event.payload);
|
|
// Replayed historical identities cannot grant authority to kill a PID
|
|
// that may have since been reused by an unrelated process.
|
|
if (
|
|
identity.threadId !== input.providerSessionId ||
|
|
identity.processId !== input.originalProviderPid
|
|
)
|
|
throw maintenanceDenied();
|
|
}
|
|
const pids = new Set([input.originalRunnerPid, input.originalProviderPid]);
|
|
const providerProofs = new Map<
|
|
number,
|
|
{ sourceEventId: string; digest: string; authenticated: boolean }
|
|
>();
|
|
if (pids.size !== 2 || ![...pids].every(maintenanceProcessAbsent))
|
|
throw maintenanceDenied();
|
|
const runnerBinary = input.runnerBinary ?? defaultCapabilityRunnerdBinary();
|
|
const artifact = approvedRunnerArtifact(runnerBinary);
|
|
const codexHome = resolve(root, "codex-home");
|
|
const deadline = Date.now() + 30_000;
|
|
let failure: unknown;
|
|
const eventCommits = new Set<Promise<void>>();
|
|
const bounded = async <T>(
|
|
operation: Promise<T>,
|
|
expiresAt = deadline,
|
|
cleanup = false,
|
|
): Promise<T> => {
|
|
const retained =
|
|
retainedMaintenanceOperations.get(root) ?? new Set<Promise<unknown>>();
|
|
retainedMaintenanceOperations.set(root, retained);
|
|
retained.add(operation);
|
|
const released = () => {
|
|
retained.delete(operation);
|
|
if (
|
|
!retained.size &&
|
|
retainedMaintenanceOperations.get(root) === retained
|
|
)
|
|
retainedMaintenanceOperations.delete(root);
|
|
releaseMaintenanceIdleWaiters(root);
|
|
};
|
|
void operation.then(released, released);
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
let abort: (() => void) | undefined;
|
|
try {
|
|
return await Promise.race([
|
|
operation,
|
|
new Promise<never>((_resolve, reject) => {
|
|
const fail = () => {
|
|
failure ??= maintenanceDenied();
|
|
reject(failure);
|
|
};
|
|
timer = setTimeout(fail, Math.max(0, expiresAt - Date.now()));
|
|
abort = fail;
|
|
if (!cleanup) {
|
|
input.signal?.addEventListener("abort", abort, { once: true });
|
|
if (input.signal?.aborted) fail();
|
|
}
|
|
}),
|
|
]);
|
|
} finally {
|
|
if (timer) clearTimeout(timer);
|
|
if (abort) input.signal?.removeEventListener("abort", abort);
|
|
}
|
|
};
|
|
const authorize = async () => {
|
|
input.signal?.throwIfAborted();
|
|
if (failure) throw failure;
|
|
if (Date.now() >= deadline) throw maintenanceDenied();
|
|
await bounded(input.authorize());
|
|
// A callback can synchronously revoke/abort and return a resolved promise;
|
|
// that result must not win Promise.race over the already-latched denial.
|
|
input.signal?.throwIfAborted();
|
|
if (failure) throw failure;
|
|
if (Date.now() >= deadline) throw maintenanceDenied();
|
|
};
|
|
await authorize();
|
|
if (!usesPreconfiguredHome(input.environment)) {
|
|
await bounded(
|
|
releaseMaterializedNativeRuntimeSkills(resolve(codexHome, "skills")),
|
|
);
|
|
await bounded(
|
|
prepareIsolatedCodexHome({
|
|
context: null,
|
|
codexHome,
|
|
sourceCodexHome:
|
|
input.sourceCodexHome ?? resolveSourceCodexHome(input.environment),
|
|
apiKey:
|
|
input.environment?.CODEX_API_KEY ?? input.environment?.OPENAI_API_KEY,
|
|
}),
|
|
);
|
|
}
|
|
// A previously journaled suspend must be honored before a later drain.
|
|
// A second exact-authority connection can then drain the retained provider
|
|
// prefix; no old command is removed, reordered, or treated as completed.
|
|
let completedTerminalEpochFingerprint: string | null = null;
|
|
for (let epoch = 0; epoch < 4; epoch++) {
|
|
await authorize();
|
|
if (![...pids].every(maintenanceProcessAbsent)) throw maintenanceDenied();
|
|
const before = readMaintenanceState(root);
|
|
assertMaintenanceBinding(before, input.identity, input.providerSessionId);
|
|
const terminalOnly = before.runner.pendingTerminalDelivery != null;
|
|
const pendingTerminal = record(before.runner.pendingTerminalDelivery);
|
|
const completedTerminalOnly =
|
|
terminalOnly &&
|
|
completedTerminalEpochFingerprint === before.fingerprint &&
|
|
completedMaintenanceTerminalReceipt(before) !== null;
|
|
completedTerminalEpochFingerprint = null;
|
|
if (
|
|
terminalOnly &&
|
|
!completedTerminalOnly &&
|
|
!(before.control.commands as Array<Record<string, unknown>>).some(
|
|
(command) =>
|
|
command.commandId === pendingTerminal.commandId &&
|
|
command.controllerSeq === pendingTerminal.controllerSeq &&
|
|
command.type === "runner.suspend" &&
|
|
pendingTerminal.commandType === "runner.suspend" &&
|
|
pendingTerminal.lifecycle === "suspended" &&
|
|
command.status === "failed",
|
|
)
|
|
)
|
|
throw maintenanceDenied();
|
|
const epochRestoresProvider =
|
|
!terminalOnly && before.provider.lifecycle === "turn_active";
|
|
const epochProviderPids = new Set<number>();
|
|
const epochCompletedCommands = new Set(
|
|
(before.control.commands as Array<Record<string, unknown>>)
|
|
.filter((command) => command.status !== "pending")
|
|
.map((command) => command.commandId),
|
|
);
|
|
if (
|
|
!terminalOnly &&
|
|
!epochRestoresProvider &&
|
|
(before.provider.activeProviderTurnId != null ||
|
|
(before.control.commands as Array<Record<string, unknown>>).some(
|
|
(command) =>
|
|
command.status === "pending" &&
|
|
!["runner.drain", "runner.suspend"].includes(String(command.type)),
|
|
))
|
|
) {
|
|
throw maintenanceDenied();
|
|
}
|
|
let releaseSpawnAdmission!: () => void;
|
|
let rejectSpawnAdmission!: (error: unknown) => void;
|
|
const spawnAdmission = new Promise<void>(
|
|
(resolveAdmission, rejectAdmission) => {
|
|
releaseSpawnAdmission = resolveAdmission;
|
|
rejectSpawnAdmission = rejectAdmission;
|
|
},
|
|
);
|
|
// A launch failure can reject this before any runner reaches authentication.
|
|
void spawnAdmission.catch(() => undefined);
|
|
const core = new DurablePrpControlPlane({
|
|
stateDirectory: resolve(root, "control-plane"),
|
|
identity: input.identity,
|
|
expectedRunnerVersion: artifact.version,
|
|
expectedRunnerDigest: artifact.digest,
|
|
beforeAuthenticatedConnection: async () => {
|
|
await spawnAdmission;
|
|
await authorize();
|
|
},
|
|
onProtocolIntegrityError: (error) => {
|
|
failure = error;
|
|
},
|
|
onSemanticToolInput: async () => {
|
|
// Never delegate a tool from a cleanup connection.
|
|
throw maintenanceDenied();
|
|
},
|
|
onCommittedEvent: (event) => {
|
|
const commit = (async () => {
|
|
try {
|
|
if (
|
|
[
|
|
"semantic_tool.input",
|
|
"mcp_app.tool_input",
|
|
"runtime.input.requested",
|
|
"runtime_request.created",
|
|
].includes(event.eventType)
|
|
) {
|
|
throw maintenanceDenied();
|
|
}
|
|
if (
|
|
["session.started", "session.resumed", "harness.ready"].includes(
|
|
event.eventType,
|
|
)
|
|
) {
|
|
const identity = resolveRunnerdSessionIdentity(event.payload);
|
|
if (
|
|
identity.threadId !== input.providerSessionId ||
|
|
identity.processId === null
|
|
)
|
|
throw maintenanceDenied();
|
|
pids.add(identity.processId);
|
|
if (identity.processId !== input.originalProviderPid) {
|
|
const digest = commandDigest(event.payload);
|
|
const prior = providerProofs.get(identity.processId);
|
|
if (
|
|
prior &&
|
|
(prior.sourceEventId !== event.sourceEventId ||
|
|
prior.digest !== digest)
|
|
)
|
|
throw maintenanceDenied();
|
|
if (!prior) epochProviderPids.add(identity.processId);
|
|
providerProofs.set(identity.processId, {
|
|
sourceEventId: event.sourceEventId,
|
|
digest,
|
|
authenticated: true,
|
|
});
|
|
}
|
|
}
|
|
await authorize();
|
|
await bounded(input.appendEvent(event));
|
|
} catch (error) {
|
|
failure = error;
|
|
throw error;
|
|
}
|
|
})();
|
|
eventCommits.add(commit);
|
|
void commit.then(
|
|
() => eventCommits.delete(commit),
|
|
() => eventCommits.delete(commit),
|
|
);
|
|
return commit;
|
|
},
|
|
});
|
|
let handle: RunnerProcessHandle | null = null;
|
|
let exited = false;
|
|
let epochCompleted = false;
|
|
let retiredFingerprint: string | null = null;
|
|
const epochIdentity = {
|
|
schema: "paperclip.native_cleanup_runner_epoch.v1" as const,
|
|
requestId: input.requestId,
|
|
epoch,
|
|
launchId: randomUUID(),
|
|
stateDirectory: root,
|
|
initialFingerprint: before.fingerprint,
|
|
runnerArtifact: { path: resolve(runnerBinary), ...artifact },
|
|
};
|
|
let spawnedReceipt: Extract<
|
|
RetainedRunnerdMaintenanceEpochReceipt,
|
|
{ phase: "spawned" }
|
|
> | null = null;
|
|
try {
|
|
const pending = core.store.state.commands.filter(
|
|
(command) => command.status === "pending",
|
|
);
|
|
let terminalQueued = pending.some(
|
|
(command) => command.type === "runner.suspend",
|
|
);
|
|
if (
|
|
!terminalOnly &&
|
|
!terminalQueued &&
|
|
before.provider.activeProviderTurnId !== null &&
|
|
before.provider.activeProviderTurnId !== undefined
|
|
) {
|
|
core.queueCommand("turn.stop", {
|
|
reason: "exact retained authority cleanup",
|
|
});
|
|
}
|
|
await core.start();
|
|
await authorize();
|
|
epochIdentity.initialFingerprint = readMaintenanceState(root).fingerprint;
|
|
await bounded(
|
|
input.recordEpoch({ ...epochIdentity, phase: "launch_intent" }),
|
|
);
|
|
await authorize();
|
|
handle = spawnRunner({
|
|
connectUrl: core.connectUrl,
|
|
stateDirectory: resolve(root, "runner"),
|
|
identity: input.identity,
|
|
ticket: core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS),
|
|
maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES,
|
|
p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES,
|
|
maxRuntimeMs: 30_000,
|
|
reconnectGraceMs: 2_000,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
|
runnerBinaryPath: runnerBinary,
|
|
runnerVersion: artifact.version,
|
|
runnerDigest: artifact.digest,
|
|
environment: createCapabilityRunnerdProviderEnvironment({
|
|
provider: "codex",
|
|
options: { environment: input.environment },
|
|
identity: input.identity,
|
|
codexHome,
|
|
runtimeContextPath: resolve(root, "runtime-context.json"),
|
|
hasRuntimeContext: false,
|
|
}),
|
|
});
|
|
if (!handle.child.pid) throw maintenanceDenied();
|
|
pids.add(handle.child.pid);
|
|
void handle.completion.then(
|
|
() => {
|
|
exited = true;
|
|
},
|
|
(error) => {
|
|
exited = true;
|
|
failure = error;
|
|
},
|
|
);
|
|
const processStartedAt = readLocalProcessStartedAt(handle.child.pid);
|
|
if (
|
|
!processStartedAt ||
|
|
!handle.startedAt ||
|
|
handle.processGroupId !== handle.child.pid
|
|
)
|
|
throw maintenanceDenied();
|
|
spawnedReceipt = {
|
|
...epochIdentity,
|
|
phase: "spawned",
|
|
pid: handle.child.pid,
|
|
processGroupId: handle.processGroupId,
|
|
processStartedAt,
|
|
spawnedAt: handle.startedAt,
|
|
};
|
|
await bounded(input.recordEpoch(spawnedReceipt));
|
|
await authorize();
|
|
releaseSpawnAdmission();
|
|
let drainQueued = false;
|
|
while (!exited) {
|
|
await authorize();
|
|
const state = readMaintenanceState(root);
|
|
assertMaintenanceBinding(
|
|
state,
|
|
input.identity,
|
|
input.providerSessionId,
|
|
epochRestoresProvider,
|
|
);
|
|
const provider = providerDrainStateFromSnapshot(state.provider);
|
|
if (!terminalOnly && !terminalQueued && provider.providerSettled) {
|
|
if (!drainQueued) {
|
|
core.queueCommand("runner.drain", {}, undefined, true);
|
|
drainQueued = true;
|
|
} else if (
|
|
provider.pendingEventCount === 0 &&
|
|
(state.runner.outbox as unknown[]).length === 0 &&
|
|
!core.store.state.commands.some(
|
|
(command) => command.status === "pending",
|
|
)
|
|
) {
|
|
core.queueCommand("runner.suspend", {}, undefined, true);
|
|
terminalQueued = true;
|
|
}
|
|
}
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
await handle.completion;
|
|
await authorize();
|
|
epochCompleted = true;
|
|
} finally {
|
|
rejectSpawnAdmission(failure ?? maintenanceDenied());
|
|
if (handle && !exited) {
|
|
await waitForProcess(handle, 250).catch(() => undefined);
|
|
// waitForProcess rejects when it dispatches SIGKILL, before the exact
|
|
// child's exit notification necessarily arrives. Join that existing
|
|
// completion separately; a kill attempt never stands in for proof.
|
|
await bounded(handle.completion, Date.now() + 1_000, true).catch(
|
|
() => undefined,
|
|
);
|
|
}
|
|
await core.stop();
|
|
let processingDrained = false;
|
|
try {
|
|
// Closing sockets does not join already queued JSON/auth callbacks.
|
|
// No retirement fingerprint or new core may race an old store write.
|
|
const normalRetirement = epochCompleted && !failure;
|
|
await bounded(
|
|
core.drainPendingConnectionProcessing(),
|
|
normalRetirement ? deadline : Math.min(deadline, Date.now() + 1_000),
|
|
!normalRetirement,
|
|
);
|
|
processingDrained = true;
|
|
} catch (error) {
|
|
failure ??= error;
|
|
}
|
|
// Do not mistake a bounded wait/kill attempt for retirement. Only the
|
|
// exact child's settled completion plus absence of its entire group can
|
|
// produce this durable receipt. A missing receipt remains unknown.
|
|
if (handle && spawnedReceipt && exited && processingDrained) {
|
|
try {
|
|
const result = await handle.completion;
|
|
if (!maintenanceProcessAbsent(spawnedReceipt.pid))
|
|
throw maintenanceDenied();
|
|
const finalFingerprint = readMaintenanceState(root).fingerprint;
|
|
await bounded(
|
|
input.recordEpoch({
|
|
...spawnedReceipt,
|
|
phase: "retired",
|
|
exitCode: result.code,
|
|
exitSignal: result.signal,
|
|
processGroupAbsent: true,
|
|
retiredAt: new Date().toISOString(),
|
|
finalFingerprint,
|
|
}),
|
|
Date.now() + 1_000,
|
|
true,
|
|
);
|
|
retiredFingerprint = finalFingerprint;
|
|
} catch (error) {
|
|
failure ??= error;
|
|
}
|
|
} else if (handle) {
|
|
failure ??= maintenanceDenied();
|
|
}
|
|
// Timed-out database operations remain observed in the retained map;
|
|
// they cannot authorize another attempt or produce a cleanup proof.
|
|
await bounded(
|
|
Promise.allSettled([...eventCommits]),
|
|
Date.now() + 1_000,
|
|
).catch(() => undefined);
|
|
if (!epochCompleted || failure) {
|
|
// Only a newly authenticated exact provider identity is kill
|
|
// authority. An interrupted startup with no identity stays unknown
|
|
// and cannot produce a settlement proof or clear quarantine.
|
|
const owned = [...providerProofs]
|
|
.filter(([, proof]) => proof.authenticated)
|
|
.map(([pid]) => pid);
|
|
for (const signal of ["SIGTERM", "SIGKILL"] as const) {
|
|
for (const pid of owned) {
|
|
if (!maintenanceProcessAbsent(pid)) {
|
|
try {
|
|
process.kill(-pid, signal);
|
|
} catch {
|
|
/* keep the failed owner retained */
|
|
}
|
|
}
|
|
}
|
|
const cleanupDeadline = Date.now() + 500;
|
|
while (
|
|
!owned.every(maintenanceProcessAbsent) &&
|
|
Date.now() < cleanupDeadline
|
|
) {
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if (failure) throw failure;
|
|
// Retirement itself may await durable authorization callbacks. It proves
|
|
// process exit, not permission to publish a reusable cleanup proof.
|
|
await authorize();
|
|
const settled = readMaintenanceState(root);
|
|
assertMaintenanceBinding(settled, input.identity, input.providerSessionId);
|
|
if (settled.runner.lifecycle !== "suspended") throw maintenanceDenied();
|
|
if (terminalOnly) {
|
|
if (completedTerminalOnly) {
|
|
// The previous epoch durably completed this command but missed its
|
|
// ACK. This epoch may only deliver that exact result and old outbox;
|
|
// it cannot restore a provider or count as a new physical stop.
|
|
if (
|
|
!completedMaintenanceTerminalReplayMatches(before, settled) ||
|
|
settled.fingerprint !== retiredFingerprint ||
|
|
epochProviderPids.size !== 0 ||
|
|
![...pids].every(maintenanceProcessAbsent)
|
|
)
|
|
throw maintenanceDenied();
|
|
continue;
|
|
}
|
|
// This epoch only confirms delivery of a failed old terminal receipt.
|
|
// It cannot count as provider cleanup or create/execute a command. A
|
|
// separate epoch must perform a NEW stop under the persistent marker.
|
|
if (
|
|
settled.providerFingerprint !== before.providerFingerprint ||
|
|
settled.runner.pendingTerminalDelivery != null ||
|
|
commandDigest(settled.runner.pendingProviderCleanup) !==
|
|
commandDigest(pendingTerminal) ||
|
|
commandDigest(settled.control.commands) !==
|
|
commandDigest(before.control.commands) ||
|
|
epochProviderPids.size !== 0 ||
|
|
![...pids].every(maintenanceProcessAbsent)
|
|
)
|
|
throw maintenanceDenied();
|
|
continue;
|
|
}
|
|
// The old suspend can precede the restored process's identity event on
|
|
// the wire. Keep that persisted identity provisional until the next
|
|
// no-launch epoch authenticates the exact source event and payload.
|
|
for (const raw of [
|
|
...(settled.provider.pendingEvents as unknown[]),
|
|
...(settled.provider.queuedEvents as unknown[]),
|
|
]) {
|
|
const event = record(raw);
|
|
if (event.eventType !== "session.resumed") continue;
|
|
const identity = resolveRunnerdSessionIdentity(event.payload);
|
|
if (
|
|
identity.threadId !== input.providerSessionId ||
|
|
identity.processId === null ||
|
|
typeof event.executorEventId !== "string"
|
|
)
|
|
throw maintenanceDenied();
|
|
if (identity.processId === input.originalProviderPid) continue;
|
|
const sourceEventId = `event_executor_${createHash("sha256")
|
|
.update("paperclip.executor-event.v1\0")
|
|
.update(input.identity.runnerInstanceId)
|
|
.update("\0")
|
|
.update(event.executorEventId)
|
|
.digest("hex")}`;
|
|
const digest = commandDigest(event.payload);
|
|
const prior = providerProofs.get(identity.processId);
|
|
if (
|
|
prior &&
|
|
(prior.sourceEventId !== sourceEventId || prior.digest !== digest)
|
|
)
|
|
throw maintenanceDenied();
|
|
if (!prior) {
|
|
epochProviderPids.add(identity.processId);
|
|
providerProofs.set(identity.processId, {
|
|
sourceEventId,
|
|
digest,
|
|
authenticated: false,
|
|
});
|
|
}
|
|
pids.add(identity.processId);
|
|
}
|
|
if (
|
|
!Number.isSafeInteger(before.provider.providerProcessGeneration) ||
|
|
settled.provider.providerProcessGeneration !==
|
|
Number(before.provider.providerProcessGeneration) +
|
|
(epochRestoresProvider ? 1 : 0)
|
|
)
|
|
throw maintenanceDenied();
|
|
const provider = providerDrainStateFromSnapshot(settled.provider);
|
|
const stops = (
|
|
settled.control.commands as Array<Record<string, unknown>>
|
|
).filter(
|
|
(command) =>
|
|
command.type === "turn.stop" &&
|
|
!epochCompletedCommands.has(command.commandId),
|
|
);
|
|
// A successful prior epoch must never cover an unidentified process from
|
|
// a later startup. Prepared drain-only epochs cannot launch a provider;
|
|
// every restoring epoch needs its own authenticated PID and exit proof.
|
|
const stopProven = !epochRestoresProvider
|
|
? epochProviderPids.size === 0
|
|
: epochProviderPids.size === 1 &&
|
|
stops.length > 0 &&
|
|
stops.every(
|
|
(command) =>
|
|
command.status === "completed" &&
|
|
record(record(command.result).result).providerExitConfirmed ===
|
|
true,
|
|
);
|
|
if (!stopProven || ![...pids].every(maintenanceProcessAbsent))
|
|
throw maintenanceDenied();
|
|
if (completedMaintenanceTerminalReceipt(settled)) {
|
|
// Only a joined child whose retirement receipt committed in THIS
|
|
// invocation can enable the next delivery-only epoch. A copied initial
|
|
// terminal marker or an interrupted/unknown child remains quarantined.
|
|
if (!retiredFingerprint || settled.fingerprint !== retiredFingerprint)
|
|
throw maintenanceDenied();
|
|
completedTerminalEpochFingerprint = settled.fingerprint;
|
|
}
|
|
if (
|
|
settled.provider.lifecycle === "prepared" &&
|
|
settled.runner.pendingTerminalDelivery == null &&
|
|
settled.runner.pendingProviderCleanup == null &&
|
|
stopProven &&
|
|
[...providerProofs.values()].every((proof) => proof.authenticated) &&
|
|
provider.providerSettled &&
|
|
provider.pendingEventCount === 0 &&
|
|
(settled.runner.outbox as unknown[]).length === 0 &&
|
|
(settled.control.commands as Array<Record<string, unknown>>).every(
|
|
(command) => command.status !== "pending",
|
|
) &&
|
|
[...pids].every(maintenanceProcessAbsent)
|
|
) {
|
|
const proof = Object.freeze({
|
|
binding: Object.freeze({ ...input.binding }),
|
|
identity: Object.freeze({ ...input.identity }),
|
|
backend: Object.freeze({ ...input.backend }),
|
|
providerSessionId: input.providerSessionId,
|
|
activationDirectory: resolve(input.activationDirectory),
|
|
sourceFingerprint: input.sourceFingerprint,
|
|
settledFingerprint: settled.fingerprint,
|
|
});
|
|
retainedRunnerdCleanupProofs.set(proof, Object.freeze([...pids]));
|
|
return proof;
|
|
}
|
|
}
|
|
throw maintenanceDenied();
|
|
}
|
|
|
|
type BuildOwnedCliArtifact =
|
|
"acpx-runtime-sidecar.cjs" | "opencode-app-server-proxy.cjs";
|
|
|
|
function buildOwnedCliArtifactCandidates(
|
|
artifact: BuildOwnedCliArtifact,
|
|
): readonly string[] {
|
|
return [
|
|
fileURLToPath(new URL(`../cli/${artifact}`, import.meta.url)),
|
|
resolve(packageRoot, "dist", "cli", artifact),
|
|
];
|
|
}
|
|
|
|
function resolveBuildOwnedCliArtifact(
|
|
artifact: BuildOwnedCliArtifact,
|
|
candidates: readonly string[] = buildOwnedCliArtifactCandidates(artifact),
|
|
): string {
|
|
const resolved = candidates.find((candidate) => existsSync(candidate));
|
|
if (resolved) return resolved;
|
|
throw new Error(
|
|
`runner_local_provider_artifact_missing: ${artifact} is absent; build @paperclipai/paperclip-runner TypeScript artifacts with build:typescript before starting a local JS-backed provider`,
|
|
);
|
|
}
|
|
|
|
function acpxProviderPackageAuthority(
|
|
sidecarScript: string,
|
|
ownerPackageRoot = packageRoot,
|
|
agent?: QualifiedAcpxAgent,
|
|
): {
|
|
root: string;
|
|
manifest: string;
|
|
} {
|
|
const cliDirectory = dirname(sidecarScript);
|
|
// Public server packages vendor runner dist directly, without a nested dist
|
|
// directory or a separately published runner package.
|
|
if (basename(sidecarScript) === "acpx-runtime-sidecar.cjs" &&
|
|
basename(cliDirectory) === "cli" && basename(dirname(cliDirectory)) === "paperclip-runner" &&
|
|
basename(resolve(cliDirectory, "../..")) === "vendor" &&
|
|
basename(resolve(cliDirectory, "../../..")) === "dist") {
|
|
const serverRoot = resolve(cliDirectory, "../../../..");
|
|
const workspaceRoot = dirname(serverRoot);
|
|
// Source/Docker builds share the workspace's pnpm store. Published npm
|
|
// bundles and self-rooted pnpm deployments retain their server authority.
|
|
if (agent !== "pi" && agent !== "antigravity" && basename(serverRoot) === "server" &&
|
|
!existsSync(resolve(serverRoot, "node_modules/.pnpm")) &&
|
|
existsSync(resolve(workspaceRoot, "node_modules/.pnpm")) &&
|
|
existsSync(resolve(workspaceRoot, "pnpm-workspace.yaml")) &&
|
|
existsSync(resolve(workspaceRoot, "package.json"))) {
|
|
const workspace = JSON.parse(readFileSync(resolve(workspaceRoot, "package.json"), "utf8"));
|
|
const server = JSON.parse(readFileSync(resolve(serverRoot, "package.json"), "utf8"));
|
|
if (workspace.name === "paperclip" && workspace.private === true && server.name === "@paperclipai/server") {
|
|
return { root: workspaceRoot, manifest: resolve(serverRoot, "package.json") };
|
|
}
|
|
}
|
|
return { root: serverRoot, manifest: resolve(serverRoot, "package.json") };
|
|
}
|
|
if (
|
|
basename(sidecarScript) !== "acpx-runtime-sidecar.cjs" ||
|
|
basename(cliDirectory) !== "cli" ||
|
|
basename(dirname(cliDirectory)) !== "dist"
|
|
) {
|
|
throw new Error(
|
|
"runner_provider_package_root_incompatible: ACPX sidecar must use the provider package dist/cli layout",
|
|
);
|
|
}
|
|
const sidecarPackageRoot = resolve(cliDirectory, "../..");
|
|
// Native provider closures belong to the runner package itself. Their asset
|
|
// guard requires the bound manifest directly inside that same package root;
|
|
// the workspace dependency root used by JS providers is a different owner.
|
|
if (agent === "copilot" || agent === "pi" || agent === "antigravity") {
|
|
return {
|
|
root: sidecarPackageRoot,
|
|
manifest: resolve(sidecarPackageRoot, "package.json"),
|
|
};
|
|
}
|
|
// A local source build lives at <workspace>/packages/paperclip-runner and
|
|
// resolves dependencies from <workspace>/node_modules. `pnpm deploy` makes
|
|
// the package itself the deployment root and owns <deploy>/node_modules/.pnpm.
|
|
// The older npm-installed portable shape nests the scoped package at
|
|
// <deploy>/node_modules/@paperclipai/paperclip-runner. The verifier always
|
|
// receives the directory that owns node_modules, regardless of which
|
|
// portable shape launched the already-authenticated sidecar.
|
|
const sourceDependencyRoot = resolve(ownerPackageRoot, "../..");
|
|
const localDependencyRoot = existsSync(
|
|
resolve(ownerPackageRoot, "node_modules", ".pnpm"),
|
|
)
|
|
? ownerPackageRoot
|
|
: basename(sourceDependencyRoot) === "node_modules"
|
|
? resolve(sourceDependencyRoot, "..")
|
|
: sourceDependencyRoot;
|
|
return sidecarPackageRoot === ownerPackageRoot
|
|
? {
|
|
root: localDependencyRoot,
|
|
manifest: resolve(ownerPackageRoot, "package.json"),
|
|
}
|
|
: {
|
|
root: sidecarPackageRoot,
|
|
manifest: resolve(sidecarPackageRoot, "package.json"),
|
|
};
|
|
}
|
|
|
|
function acpxRunnerLaunchProfile(
|
|
options: CapabilityRunnerdCodexTransportOptions,
|
|
command: string,
|
|
sidecarScript: string,
|
|
identity?: DurableRecoveryIdentity,
|
|
): {
|
|
candidateAdmission?: AcpxCandidateStartupAdmission;
|
|
authorityDigest: string;
|
|
command: string;
|
|
commandSha256: string;
|
|
sidecarScript: string;
|
|
sidecarScriptSha256: string;
|
|
} {
|
|
const localDigest = (path: string) =>
|
|
`sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`;
|
|
let candidateAdmission: AcpxCandidateStartupAdmission | undefined;
|
|
if (options.acpxCandidateAdmission) {
|
|
if (!identity || options.runnerFilesystemRoot) throw new Error("ACPX candidate requires an actual local startup identity");
|
|
candidateAdmission = bindAcpxCandidateAdmission(options.acpxCandidateAdmission, identity);
|
|
}
|
|
if (!options.runnerFilesystemRoot) {
|
|
const buildCommand = process.execPath;
|
|
const buildSidecarCandidates = buildOwnedCliArtifactCandidates(
|
|
"acpx-runtime-sidecar.cjs",
|
|
);
|
|
if (
|
|
options.providerNodeCommand !== undefined ||
|
|
options.providerNodeCommandSha256 !== undefined ||
|
|
options.acpxSidecarPath !== undefined ||
|
|
options.acpxSidecarSha256 !== undefined ||
|
|
options.providerPackAuthorityDigest !== undefined ||
|
|
command !== buildCommand ||
|
|
!buildSidecarCandidates.includes(sidecarScript)
|
|
) {
|
|
throw new Error(
|
|
"runner_local_provider_artifact_incompatible: ACPX local launch must use build-owned artifacts",
|
|
);
|
|
}
|
|
const buildSidecar = resolveBuildOwnedCliArtifact(
|
|
"acpx-runtime-sidecar.cjs",
|
|
buildSidecarCandidates,
|
|
);
|
|
if (sidecarScript !== buildSidecar) {
|
|
throw new Error(
|
|
"runner_local_provider_artifact_incompatible: ACPX local launch must use build-owned artifacts",
|
|
);
|
|
}
|
|
const commandSha256 = localDigest(buildCommand);
|
|
const sidecarScriptSha256 = localDigest(buildSidecar);
|
|
return {
|
|
...(candidateAdmission ? { candidateAdmission } : {}),
|
|
authorityDigest: commandDigest({
|
|
schema: "paperclip.runner.local-acpx-authority.v1",
|
|
commandSha256,
|
|
sidecarScriptSha256,
|
|
}),
|
|
command: buildCommand,
|
|
commandSha256,
|
|
sidecarScript: buildSidecar,
|
|
sidecarScriptSha256,
|
|
};
|
|
}
|
|
const commandSha256 = options.providerNodeCommandSha256;
|
|
const sidecarScriptSha256 = options.acpxSidecarSha256;
|
|
const authorityDigest = options.providerPackAuthorityDigest;
|
|
if (!commandSha256 || !sidecarScriptSha256 || !authorityDigest) {
|
|
throw new Error(
|
|
"runner_remote_provider_artifact_incompatible: ACPX launch profile omitted its provider-pack authority or verified artifact digests",
|
|
);
|
|
}
|
|
return {
|
|
authorityDigest,
|
|
command,
|
|
commandSha256,
|
|
sidecarScript,
|
|
sidecarScriptSha256,
|
|
};
|
|
}
|
|
|
|
function opencodeRunnerLaunchProfile(
|
|
options: CapabilityRunnerdCodexTransportOptions,
|
|
command: string,
|
|
proxyScript: string,
|
|
executable: string,
|
|
): {
|
|
command: string;
|
|
commandSha256: string;
|
|
proxyScript: string;
|
|
proxyScriptSha256: string;
|
|
executable: string;
|
|
executableSha256: string;
|
|
} {
|
|
const localDigest = (path: string) =>
|
|
`sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`;
|
|
const usesLocalBuildOwnedDefaults =
|
|
!options.runnerFilesystemRoot &&
|
|
options.providerNodeCommand === undefined &&
|
|
options.opencodeProxyPath === undefined &&
|
|
options.opencodeCommand === undefined;
|
|
const commandSha256 =
|
|
options.providerNodeCommandSha256 ??
|
|
(usesLocalBuildOwnedDefaults ? localDigest(command) : null);
|
|
const proxyScriptSha256 =
|
|
options.opencodeProxySha256 ??
|
|
(usesLocalBuildOwnedDefaults ? localDigest(proxyScript) : null);
|
|
const executableSha256 =
|
|
options.opencodeCommandSha256 ??
|
|
(usesLocalBuildOwnedDefaults ? localDigest(executable) : null);
|
|
if (!commandSha256 || !proxyScriptSha256 || !executableSha256) {
|
|
throw new Error(
|
|
"runner_remote_provider_artifact_incompatible: OpenCode launch profile omitted verified artifact digests",
|
|
);
|
|
}
|
|
return {
|
|
command,
|
|
commandSha256,
|
|
proxyScript,
|
|
proxyScriptSha256,
|
|
executable,
|
|
executableSha256,
|
|
};
|
|
}
|
|
|
|
function resolveRunnerOpenCodeExecutable(options: RunnerdCodexTransportOptions): string {
|
|
if (options.opencodeCommand !== undefined) return options.opencodeCommand;
|
|
if (options.runnerFilesystemRoot) {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: OpenCode executable is missing from the provider pack");
|
|
}
|
|
return resolvePinnedOpenCodeCommand();
|
|
}
|
|
|
|
function authorizedToolSet(
|
|
tools: readonly Readonly<Record<string, unknown>>[],
|
|
): Record<string, unknown> {
|
|
const operations = tools
|
|
.map((tool) => ({
|
|
operationId: String(tool.name ?? ""),
|
|
version: 1,
|
|
description: String(tool.description ?? ""),
|
|
inputSchema: tool.name === PRP_COMPLETION_TOOL_NAME ? PRP_COMPLETION_RESULT_BRIDGE_INPUT_SCHEMA : record(tool.inputSchema),
|
|
responseSchema: {},
|
|
}))
|
|
.sort((left, right) =>
|
|
left.operationId < right.operationId
|
|
? -1
|
|
: left.operationId > right.operationId
|
|
? 1
|
|
: 0,
|
|
);
|
|
return {
|
|
schema: "paperclip.runner.authorized-tools.v1",
|
|
schemaVersion: 1,
|
|
catalogDigest: commandDigest(operations),
|
|
operations,
|
|
};
|
|
}
|
|
|
|
export function authorizedToolSetForProvider(
|
|
_provider: CapabilityRunnerdCodexTransportOptions["provider"],
|
|
tools: readonly Readonly<Record<string, unknown>>[],
|
|
): Record<string, unknown> {
|
|
// ACPX terminal calls are resolved through the authenticated semantic
|
|
// bridge before the provider receives a result. Keep them in the provider
|
|
// authority catalog so the sidecar can project the call and await
|
|
// server-side completion feedback.
|
|
return authorizedToolSet(tools);
|
|
}
|
|
|
|
/** Include the completion tools that the Codex driver adds at session startup. */
|
|
export function runnerCodexDynamicToolsFit(
|
|
tools: readonly Readonly<Record<string, unknown>>[],
|
|
): boolean {
|
|
const supplied = [...tools, ...codexSemanticToolSpecs()];
|
|
// Keep these bounds aligned with runner-core/provider_bridge.rs. Project
|
|
// large optional catalogs before run.prepare; never raise the protocol caps.
|
|
return supplied.length <= 256 &&
|
|
Buffer.byteLength(JSON.stringify(authorizedToolSet(supplied)), "utf8") <= 768 * 1024;
|
|
}
|
|
|
|
/**
|
|
* Raw provider tracing is consumed by runnerd itself. The provider child still
|
|
* receives the narrower allowlist enforced by Rust's `SupervisedProcess`, so
|
|
* these controller-selected sidecar paths never enter the harness process.
|
|
*/
|
|
function withRunnerdProviderTrace(
|
|
environment: NodeJS.ProcessEnv,
|
|
source: NodeJS.ProcessEnv | undefined,
|
|
): NodeJS.ProcessEnv {
|
|
const result = { ...environment };
|
|
for (const key of [
|
|
"PAPERCLIP_PROVIDER_TRACE_PATH",
|
|
"PAPERCLIP_PROVIDER_TRACE_MAX_BYTES",
|
|
"PAPERCLIP_NATIVE_MCP_STDIO",
|
|
"PAPERCLIP_NATIVE_MCP_PROXY",
|
|
] as const) {
|
|
const value = source?.[key];
|
|
if (value !== undefined) result[key] = value;
|
|
}
|
|
return result;
|
|
}
|
|
function createCapabilityRunnerdProviderEnvironmentBase(input: {
|
|
provider: NonNullable<CapabilityRunnerdCodexTransportOptions["provider"]>;
|
|
options: CapabilityRunnerdCodexTransportOptions;
|
|
identity: DurableRecoveryIdentity;
|
|
codexHome: string;
|
|
runtimeContextPath: string;
|
|
hasRuntimeContext: boolean;
|
|
acpxSidecarPath?: string;
|
|
}): NodeJS.ProcessEnv {
|
|
const commonIdentity = {
|
|
PAPERCLIP_RUNNER_INSTANCE_ID: input.identity.runnerInstanceId,
|
|
PAPERCLIP_RUN_ID: input.identity.runId,
|
|
PAPERCLIP_NORMALIZED_SESSION_ID: input.identity.normalizedSessionId,
|
|
...(input.hasRuntimeContext
|
|
? { PAPERCLIP_NATIVE_RUNTIME_CONTEXT_PATH: input.runtimeContextPath }
|
|
: {}),
|
|
};
|
|
if (input.provider === "opencode") {
|
|
return {
|
|
...createSanitizedOpenCodeRunnerEnvironment(input.options.environment),
|
|
PAPERCLIP_OPENCODE_PERMISSION_MODE:
|
|
input.options.opencodePermissionMode ?? "allow",
|
|
PAPERCLIP_OPENCODE_RUNTIME_DIR:
|
|
input.options.opencodeRuntimeDirectory ??
|
|
resolve(input.options.stateDirectory ?? tmpdir(), "opencode"),
|
|
...commonIdentity,
|
|
};
|
|
}
|
|
if (input.provider === "acpx") {
|
|
const sidecarPath =
|
|
input.acpxSidecarPath ??
|
|
input.options.acpxSidecarPath ??
|
|
resolve(packageRoot, "dist", "cli", "acpx-runtime-sidecar.cjs");
|
|
const providerPackageAuthority = acpxProviderPackageAuthority(
|
|
sidecarPath, packageRoot, input.options.acpxAgent,
|
|
);
|
|
// This is the trusted runner/sidecar boundary. The provider sandbox still
|
|
// uses createSanitizedAcpxSpawnInput and does not inherit gateway tokens.
|
|
const assignedGateway = nativeMcpLaunchBinding(input.options.environment ?? {});
|
|
const credentialBinding = createAcpxCredentialBinding(
|
|
input.options.environment, input.options.acpxAgent ?? "codex", input.identity.normalizedSessionId,
|
|
);
|
|
return {
|
|
...(assignedGateway ? {
|
|
PAPERCLIP_NATIVE_MCP_TOKEN: assignedGateway.token,
|
|
} : {}),
|
|
...createSanitizedAcpxSpawnInput(
|
|
input.options.environment,
|
|
input.options.acpxAgent ?? "codex",
|
|
).env,
|
|
...(credentialBinding === undefined ? {} : { [ACPX_CREDENTIAL_BINDING_ENV]: credentialBinding }),
|
|
...(input.options.acpxAgent === "grok" && input.options.environment?.PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET
|
|
? { PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET: input.options.environment.PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET } : {}),
|
|
...(input.options.acpxAgent === "antigravity" && input.options.environment?.PAPERCLIP_ACPX_ANTIGRAVITY_AUTH_JSON_SECRET
|
|
? { PAPERCLIP_ACPX_ANTIGRAVITY_AUTH_JSON_SECRET: input.options.environment.PAPERCLIP_ACPX_ANTIGRAVITY_AUTH_JSON_SECRET } : {}),
|
|
...commonIdentity,
|
|
// The verified sidecar bundle cannot use import.meta.url while Node
|
|
// executes it through /proc/self/fd. Anchor its closed provider package
|
|
// lookups at the package that owns the already-authenticated bundle.
|
|
PAPERCLIP_ACPX_BUILTIN_ROOT: resolve(dirname(sidecarPath), "../providers"),
|
|
PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT: providerPackageAuthority.root,
|
|
PAPERCLIP_ACPX_PROVIDER_PACKAGE_MANIFEST:
|
|
providerPackageAuthority.manifest,
|
|
...(!input.options.acpxCandidateAdmission && input.options.providerRecoveryPolicy ===
|
|
"allow_replacement_after_governed_wait"
|
|
? {
|
|
PAPERCLIP_ACPX_PROVIDER_RECOVERY_POLICY:
|
|
"allow_replacement_after_governed_wait",
|
|
}
|
|
: {}),
|
|
};
|
|
}
|
|
if (input.provider === "claude_managed") {
|
|
return {
|
|
...createSanitizedClaudeManagedEnvironment(input.options.environment),
|
|
...commonIdentity,
|
|
};
|
|
}
|
|
if (input.provider === "aws_agentcore") {
|
|
return {
|
|
...createSanitizedAwsAgentCoreEnvironment(
|
|
input.options.environment,
|
|
input.codexHome,
|
|
),
|
|
...commonIdentity,
|
|
};
|
|
}
|
|
const environment = createSanitizedCodexEnvironment({
|
|
...input.options.environment,
|
|
HOME: input.options.homeMode === "preconfigured" ? input.options.environment?.HOME : input.codexHome,
|
|
CODEX_HOME: input.codexHome,
|
|
});
|
|
if (input.options.homeMode === "preconfigured") environment.PAPERCLIP_RUNNER_HOME_MODE = "preconfigured";
|
|
for (const key of ["OPENAI_API_KEY", "CODEX_API_KEY"] as const) {
|
|
const apiKey = input.options.environment?.[key];
|
|
if (apiKey?.trim()) environment[key] = apiKey;
|
|
}
|
|
return environment;
|
|
}
|
|
|
|
export function createCapabilityRunnerdProviderEnvironment(input: Parameters<typeof createCapabilityRunnerdProviderEnvironmentBase>[0]): NodeJS.ProcessEnv {
|
|
return applyProviderObservabilityEnvironment(createCapabilityRunnerdProviderEnvironmentBase(input), input.options.observability, input.provider === "acpx");
|
|
}
|
|
|
|
const OPEN_CODE_RUNNER_ENVIRONMENT_KEYS = new Set([
|
|
"PAPERCLIP_AI_PROVIDER_KEY",
|
|
"PAPERCLIP_AI_PROVIDER_URL",
|
|
"PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY",
|
|
"PATH",
|
|
"LANG",
|
|
"LANGUAGE",
|
|
"LC_ALL",
|
|
"LC_CTYPE",
|
|
"TZ",
|
|
"TMPDIR",
|
|
"TEMP",
|
|
"TMP",
|
|
"SSL_CERT_FILE",
|
|
"SSL_CERT_DIR",
|
|
"NODE_EXTRA_CA_CERTS",
|
|
"HTTP_PROXY",
|
|
"HTTPS_PROXY",
|
|
"NO_PROXY",
|
|
"ALL_PROXY",
|
|
"http_proxy",
|
|
"https_proxy",
|
|
"no_proxy",
|
|
"all_proxy",
|
|
"SystemRoot",
|
|
"PATHEXT",
|
|
"WINDIR",
|
|
"RUST_BACKTRACE",
|
|
"OPENROUTER_API_KEY",
|
|
"PAPERCLIP_NATIVE_MCP_NAME",
|
|
"PAPERCLIP_NATIVE_MCP_URL",
|
|
"PAPERCLIP_NATIVE_MCP_TOKEN",
|
|
]);
|
|
|
|
function createSanitizedOpenCodeRunnerEnvironment(
|
|
source: NodeJS.ProcessEnv | undefined,
|
|
): NodeJS.ProcessEnv {
|
|
const candidate = { ...process.env, ...source };
|
|
for (const key of ["PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY"]) {
|
|
delete candidate[key];
|
|
if (source?.[key] !== undefined) candidate[key] = source[key];
|
|
}
|
|
return { ...configuredEnvironment(source), ...githubCredentialEnvironment(source ?? {}), ...Object.fromEntries(
|
|
Object.entries(candidate).filter(
|
|
([key, value]) =>
|
|
typeof value === "string" &&
|
|
(OPEN_CODE_RUNNER_ENVIRONMENT_KEYS.has(key) ||
|
|
/^LC_[A-Z0-9_]{1,32}$/.test(key)),
|
|
),
|
|
) };
|
|
}
|
|
|
|
export function resolveSourceCodexHome(
|
|
environment: NodeJS.ProcessEnv | undefined,
|
|
): string | null {
|
|
const explicit = environment?.CODEX_HOME?.trim();
|
|
if (explicit) return explicit;
|
|
const home = environment?.HOME?.trim();
|
|
return home ? resolve(home, ".codex") : null;
|
|
}
|
|
|
|
export function unwrapToolResponse(response: Record<string, unknown>, preserveEnvelope = false): {
|
|
readonly __paperclipSemanticToolOutcome: true;
|
|
readonly result: unknown;
|
|
readonly isError: boolean;
|
|
} {
|
|
const items = Array.isArray(response.contentItems)
|
|
? response.contentItems
|
|
: [];
|
|
const value = record(items[0]).text;
|
|
let result: unknown = response;
|
|
try {
|
|
if (!preserveEnvelope && typeof value === "string") result = JSON.parse(value);
|
|
} catch {
|
|
result = response;
|
|
}
|
|
return {
|
|
__paperclipSemanticToolOutcome: true as const,
|
|
result,
|
|
isError: response.success === false,
|
|
};
|
|
}
|
|
|
|
/** Only a live, admitted Pi ACP session can enable native turn controls. */
|
|
export function parseAcpxTurnControlCapabilities(
|
|
value: unknown,
|
|
agent: unknown,
|
|
): NativeTurnControlCapabilities {
|
|
const unsupported = { steering: false, queuedFollowUp: false };
|
|
if (value === undefined) return unsupported;
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
|
throw new Error("ACPX turn control capabilities are malformed");
|
|
}
|
|
const controls = value as Record<string, unknown>;
|
|
if (Object.keys(controls).some(key => key !== "steering" && key !== "queuedFollowUp")
|
|
|| typeof controls.steering !== "boolean" || typeof controls.queuedFollowUp !== "boolean") {
|
|
throw new Error("ACPX turn control capabilities are malformed");
|
|
}
|
|
if (agent !== "pi" && (controls.steering || controls.queuedFollowUp)) {
|
|
throw new Error("ACPX profile cannot advertise these turn controls");
|
|
}
|
|
return { steering: controls.steering, queuedFollowUp: controls.queuedFollowUp };
|
|
}
|
|
|
|
class DurablePrpCodexTransport implements CodexAppServerTransport {
|
|
readonly #root: string;
|
|
readonly #ownsRoot: boolean;
|
|
readonly #queue = new NotificationQueue();
|
|
readonly #startedAt = new Date().toISOString();
|
|
readonly #evidence: CapabilityRunnerdProcessEvidence;
|
|
#handler: CodexServerRequestHandler = async () => ({
|
|
success: false,
|
|
contentItems: [
|
|
{
|
|
type: "inputText",
|
|
text: "No Paperclip control-plane tool handler is installed.",
|
|
},
|
|
],
|
|
});
|
|
#handlerInstalled = false;
|
|
readonly #handlerReady: Promise<void>;
|
|
#resolveHandlerReady!: () => void;
|
|
#core: DurablePrpControlPlane | null = null;
|
|
#handle: RunnerProcessHandle | null = null;
|
|
#adoptedRunnerMonitor: NodeJS.Timeout | null = null;
|
|
#adoptedRunnerAuthenticated = false;
|
|
#pump: NodeJS.Timeout | null = null;
|
|
#eventSourceSeq = 0;
|
|
#eventIdentity: DurableRecoveryIdentity | null = null;
|
|
#pendingWarmRecoveryCompletion: {
|
|
transitionId: string;
|
|
identity: DurableRecoveryIdentity;
|
|
} | null = null;
|
|
#warmRecoveryCompletionInFlight: Promise<void> | null = null;
|
|
#deferredTurnStartEvents: DurableRecoveryCommittedEvent[] = [];
|
|
#recoveryTurnBindingPending = false;
|
|
#threadId = "";
|
|
#observedProviderModel: string | undefined;
|
|
#sessionId: string | null = null;
|
|
#providerIdentity: Record<string, unknown> | null = null;
|
|
#turnControls: NativeTurnControlCapabilities = { steering: false, queuedFollowUp: false };
|
|
|
|
supportsTurnReasoning(): boolean {
|
|
if (this.options.provider !== "opencode") return false;
|
|
// Use the durable selected model on both fresh and recovered sessions.
|
|
// Older states may retain the provider only in the preparation command.
|
|
const state = this.#core?.store.state;
|
|
const template = this.#runAttachTemplate ?? state?.runAttachTemplate
|
|
?? state?.commands.findLast(command =>
|
|
(command.type === "run.prepare" || command.type === "run.attach")
|
|
&& record(command.payload).provider !== undefined,
|
|
)?.payload;
|
|
const model = record(record(template).provider).model;
|
|
return typeof model === "string" && model.startsWith("openrouter/");
|
|
}
|
|
|
|
turnControlCapabilities(): NativeTurnControlCapabilities | null {
|
|
if (this.options.provider !== "acpx") return null;
|
|
if (this.#closed || this.#failure) return { steering: false, queuedFollowUp: false };
|
|
return { ...this.#turnControls };
|
|
}
|
|
#providerIdentityEventType:
|
|
"harness.ready" | "session.started" | "session.resumed" | null = null;
|
|
#checkpointProviderIdentityExpectation: {
|
|
driverSessionId: string;
|
|
providerSessionId: string;
|
|
providerIdentity: Record<string, unknown> | null;
|
|
} | null = null;
|
|
#checkpointProviderIdentityConfirmed = false;
|
|
#turnId = "";
|
|
#turnStartResponsePending = false;
|
|
#turnStartResponseSettled: Promise<void> = Promise.resolve();
|
|
#turnStartResponseEpoch = 0;
|
|
#observedTurnStartEpoch = 0;
|
|
#expectedProviderTurnId: string | null = null;
|
|
#turnStartAdmission: {
|
|
settled: Promise<boolean>;
|
|
resolve: (accepted: boolean) => void;
|
|
} | null = null;
|
|
#durableTurnId = "";
|
|
#authorizedTools: Record<string, unknown> | null = null;
|
|
#runAttachTemplate: Record<string, unknown> | null = null;
|
|
#preparedWorkingDirectory: string | null = null;
|
|
#closed = false;
|
|
readonly #closedSignal: Promise<void>;
|
|
#resolveClosedSignal!: () => void;
|
|
#closePromise: Promise<void> | null = null;
|
|
#probeRetirement: Promise<void> | null = null;
|
|
readonly prepareConversationResume?: CodexAppServerTransport["prepareConversationResume"];
|
|
#detachPromise: Promise<void> | null = null;
|
|
#startupOperation: Promise<unknown> | null = null;
|
|
#attachmentOperation: Promise<void> | null = null;
|
|
#startupDrainTimedOut = false;
|
|
#controllerDetachedForRestart = false;
|
|
#failure: Error | null = null;
|
|
readonly #failureSignal: Promise<never>;
|
|
#rejectFailureSignal!: (error: Error) => void;
|
|
#runnerRecoveryInProgress = false;
|
|
#startupComplete = false;
|
|
#startupFailureCode = "native_runner_process_exited";
|
|
#controlPlaneCheckpoint:
|
|
((settlement: "settled" | "unsettled") => Promise<void> | void) | null =
|
|
null;
|
|
#controlPlaneRelease: ((reason?: "detach") => Promise<void> | void) | null = null;
|
|
#nextTraceDebugSequence = 1;
|
|
#traceRehydrationSpoolOverflow = false;
|
|
readonly #traceFrameIndex = new RunnerdTraceFrameIndex();
|
|
#pendingTraceRehydrations: PendingTraceRehydration[] = [];
|
|
#pendingDriverTraceInterpretations: PendingDriverTraceInterpretation[] = [];
|
|
#restoredRuntimeRequests: CodexRpcServerRequest[] = [];
|
|
readonly #bridgedRuntimeInputs = new Map<string, { durableTurnId: string; permission?: boolean }>();
|
|
|
|
constructor(readonly options: CapabilityRunnerdCodexTransportOptions) {
|
|
if (options.adoptExistingRunner && options.authorizeRunnerAdoption && !options.prepareLaunch) {
|
|
throw new Error("runner_adoption_authorization_requires_launch_preparation");
|
|
}
|
|
if (options.prepareLaunch && (!options.runnerProcessLauncher || !options.controlPlaneRegistration || !options.runnerArtifact)) {
|
|
throw new Error("runner_launch_preparation_requires_process_owner_and_artifact");
|
|
}
|
|
resolveClaudeEffort(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.claudeEffort);
|
|
resolveGrokEffort(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.grokEffort);
|
|
resolveGrokDisableWebSearch(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.grokDisableWebSearch);
|
|
resolvePiThinkingLevel(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.piThinkingLevel);
|
|
if (options.acpxMode !== undefined && options.provider !== "acpx") {
|
|
throw new Error("acpxMode requires the ACPX provider");
|
|
}
|
|
if (options.provider === "acpx") resolveAcpxProviderMode(options.acpxAgent ?? "codex", options.acpxMode);
|
|
if (options.homeMode === "preconfigured") {
|
|
if ((options.provider ?? "codex") !== "codex" && !(options.provider === "acpx" && ["claude", "codex"].includes(options.acpxAgent ?? "codex"))) {
|
|
throw new Error("preconfigured_home_unsupported_provider");
|
|
}
|
|
const source = options.environment ?? process.env;
|
|
const homes = options.runnerFilesystemRoot ? Object.fromEntries(
|
|
["HOME", "CODEX_HOME", "CLAUDE_CONFIG_DIR", "XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME"]
|
|
.flatMap(key => source[key] === undefined ? [] : [[key, source[key]!]]),
|
|
) : preconfiguredHomeEnvironment(source);
|
|
if (!homes.HOME) throw new Error("preconfigured_home_unavailable");
|
|
this.options = { ...options, environment: { ...source, ...homes,
|
|
PAPERCLIP_RUNNER_HOME_MODE: "preconfigured",
|
|
...(options.acpxAgent === "claude" ? { ACPX_CLAUDE_INCLUDE_USER_SETTINGS: "1" } : {}),
|
|
} };
|
|
}
|
|
if (options.homeMode !== "preconfigured" && options.environment?.PAPERCLIP_RUNNER_HOME_MODE !== undefined) {
|
|
const environment = { ...options.environment };
|
|
delete environment.PAPERCLIP_RUNNER_HOME_MODE;
|
|
this.options = { ...options, environment };
|
|
}
|
|
if (options.adoptExistingRunner && !options.stateDirectory?.trim()) {
|
|
throw new Error("native_adopted_runner_state_directory_required");
|
|
}
|
|
if (options.acpxCandidateAdmission !== undefined) {
|
|
const candidate = options.acpxCandidateAdmission;
|
|
const admitted = parseAcpxCandidateAdmission(candidate, { companyId: candidate.companyId, agent: options.acpxAgent ?? "", model: candidate.model });
|
|
this.options = { ...this.options, acpxCandidateAdmission: Object.freeze(admitted), providerRecoveryPolicy: "same_session_only" };
|
|
this.prepareConversationResume = params => this.#prepareCandidateConversationResume(params);
|
|
if (options.provider !== "acpx" || process.platform !== "darwin" || process.arch !== "arm64"
|
|
|| options.runnerFilesystemRoot || options.prepareLaunch || options.runnerProcessLauncher
|
|
|| options.homeMode === "preconfigured" || options.lifecyclePolicy?.mode === "warm"
|
|
|| options.adoptExistingRunner || options.resumeActiveTurnId
|
|
) {
|
|
throw new Error("ACPX candidate admission supports only a fresh local isolated per-turn execution");
|
|
}
|
|
}
|
|
if (options.acpxCandidateAdmission && options.resumeProviderSession) {
|
|
const prior = options.resumeProviderSession;
|
|
const identity = record(prior.providerIdentity);
|
|
if (!prior.driverSessionId?.trim() || !prior.providerSessionId?.trim()
|
|
|| identity.kind !== "acpx" || identity.normalizedSessionId !== options.prpIdentity?.normalizedSessionId
|
|
|| identity.requestedModel !== options.acpxCandidateAdmission.model || identity.effectiveModel !== options.acpxCandidateAdmission.model
|
|
|| typeof identity.profileDigest !== "string" || !/^sha256:[a-f0-9]{64}$/.test(identity.profileDigest)) {
|
|
throw new Error("ACPX candidate continuation requires the exact retained native conversation identity");
|
|
}
|
|
}
|
|
if (options.provider === "acpx" && options.acpxAgent !== undefined
|
|
&& ACPX_CAPABILITY_PROFILES[options.acpxAgent].qualification === "pending"
|
|
&& options.acpxCandidateProfile !== options.acpxAgent
|
|
&& !(options.acpxAgent === "antigravity" && options.acpxCandidateAdmission)) {
|
|
throw new Error("The candidate ACPX profile requires explicit evaluation opt-in");
|
|
}
|
|
this.#failureSignal = new Promise<never>((_resolve, reject) => {
|
|
this.#rejectFailureSignal = reject;
|
|
});
|
|
this.#handlerReady = new Promise<void>(resolve => {
|
|
this.#resolveHandlerReady = resolve;
|
|
});
|
|
this.#closedSignal = new Promise<void>(resolve => {
|
|
this.#resolveClosedSignal = resolve;
|
|
});
|
|
// Failure is also observed by request/notification paths. Register an
|
|
// internal handler so a process exit after the owner has closed the
|
|
// session cannot become an unhandled process-level rejection.
|
|
void this.#failureSignal.catch(() => undefined);
|
|
this.#ownsRoot = options.stateDirectory === undefined;
|
|
this.#turnId = options.resumeActiveTurnId ?? "";
|
|
this.#root =
|
|
options.stateDirectory ??
|
|
mkdtempSync(resolve(tmpdir(), "paperclip-runner-lab-prp-"));
|
|
if (options.resumeDynamicTools !== undefined) {
|
|
this.#authorizedTools = authorizedToolSetForProvider(options.provider, [
|
|
...options.resumeDynamicTools,
|
|
...codexSemanticToolSpecs(),
|
|
]);
|
|
}
|
|
mkdirSync(this.#root, { recursive: true, mode: 0o700 });
|
|
this.#evidence = {
|
|
runnerPid: null,
|
|
runnerProcessGroupId: null,
|
|
providerPid: null,
|
|
providerProcessStartedAt: null,
|
|
codexPid: null,
|
|
codexProcessStartedAt: null,
|
|
sidecarPid: null,
|
|
sidecarProcessStartedAt: null,
|
|
agentPid: null,
|
|
agentProcessStartedAt: null,
|
|
providerDriver: null,
|
|
providerVersion: null,
|
|
acpxAgent: null,
|
|
agentServerVersion: null,
|
|
agentRuntimeVersion: null,
|
|
acpProtocolVersion: null,
|
|
providerExecutionKind: null,
|
|
providerService: null,
|
|
runnerExited: false,
|
|
runnerExitCode: null,
|
|
runnerSignal: null,
|
|
childEnvironmentKeys: Object.keys(
|
|
options.provider === "acpx"
|
|
? createSanitizedAcpxSpawnInput(
|
|
options.environment,
|
|
options.acpxAgent ?? "codex",
|
|
).env
|
|
: options.provider === "opencode"
|
|
? createSanitizedOpenCodeRunnerEnvironment(options.environment)
|
|
: options.provider === "claude_managed"
|
|
? createSanitizedClaudeManagedEnvironment(options.environment)
|
|
: options.provider === "aws_agentcore"
|
|
? createSanitizedAwsAgentCoreEnvironment(
|
|
options.environment,
|
|
resolve(this.#root, "codex-home"),
|
|
)
|
|
: createSanitizedCodexEnvironment(options.environment),
|
|
).sort(),
|
|
diagnostics: ["lab transport selected authenticated durable PRP"],
|
|
};
|
|
}
|
|
|
|
evidence(): CapabilityRunnerdProcessEvidence {
|
|
return structuredClone(this.#evidence);
|
|
}
|
|
|
|
async request(
|
|
method: string,
|
|
params: Record<string, unknown>,
|
|
): Promise<Record<string, unknown>> {
|
|
if (this.#closed) throw new Error("PRP Codex transport is closed");
|
|
this.#throwIfFailed();
|
|
if (
|
|
this.#pendingWarmRecoveryCompletion !== null &&
|
|
!["thread/read", "initialize", "collaborationMode/list"].includes(method)
|
|
) {
|
|
throw new Error("native_runner_warm_transition_completion_pending");
|
|
}
|
|
if (method === "initialize") return { user: {} };
|
|
if (method === "thread/start") return this.#trackStartup(() => this.#start(params));
|
|
if (method === "collaborationMode/list") {
|
|
// runnerd negotiates the real Codex preset or the provider-proxy-owned
|
|
// planning contract during session.open. This transport-level mask
|
|
// confirms that closed boundary; turn/start remains runner-managed and
|
|
// never forwards this sentinel to the outer TypeScript driver.
|
|
return this.options.provider === undefined ||
|
|
this.options.provider === "codex" ||
|
|
this.options.provider === "opencode" ||
|
|
this.options.provider === "acpx"
|
|
? {
|
|
data: [
|
|
{
|
|
name: "Plan",
|
|
mode: "plan",
|
|
model: "runner-managed",
|
|
reasoning_effort: null,
|
|
},
|
|
],
|
|
}
|
|
: { data: [] };
|
|
}
|
|
if (method === "turn/start") return this.#startTurn(params);
|
|
if (method === "turn/steer") {
|
|
const input = Array.isArray(params.input) ? params.input.map(record) : [];
|
|
const text = input
|
|
.map((item) => (typeof item.text === "string" ? item.text : ""))
|
|
.join("\n");
|
|
const expectedTurnId =
|
|
typeof params.expectedTurnId === "string"
|
|
? params.expectedTurnId
|
|
: this.#turnId;
|
|
if (!text.trim()) throw new Error("turn/steer requires a message");
|
|
if (params.mode !== undefined && params.mode !== "steer" && params.mode !== "follow_up") throw new Error("turn/steer mode is invalid");
|
|
if (expectedTurnId !== this.#turnId)
|
|
throw new Error("turn/steer named a stale turn");
|
|
const correlationId =
|
|
typeof params.correlationId === "string"
|
|
? params.correlationId
|
|
: undefined;
|
|
await this.#command(
|
|
"turn.steer",
|
|
{
|
|
text,
|
|
turnId: this.#durableTurnId,
|
|
providerTurnId: expectedTurnId,
|
|
...(params.mode === "follow_up" ? { mode: "follow_up" } : {}),
|
|
...(correlationId ? { correlationId } : {}),
|
|
},
|
|
correlationId,
|
|
);
|
|
return {};
|
|
}
|
|
if (method === "turn/interrupt") {
|
|
await this.#command("turn.interrupt", params);
|
|
return {};
|
|
}
|
|
if (method === "thread/turns/list" || method === "thread/items/list") {
|
|
if (params.threadId !== this.#threadId) throw new Error("codex_history_identity_mismatch");
|
|
const snapshot = await this.request("thread/read", { threadId: this.#threadId, includeTurns: false });
|
|
const turns = record(snapshot.thread).turns as Array<Record<string, unknown>>;
|
|
let data: Array<Record<string, unknown>>;
|
|
if (method === "thread/turns/list") {
|
|
data = turns.map(turn => ({ ...turn, items: [], itemsView: "notLoaded" }));
|
|
} else {
|
|
if (!turns.some(turn => turn.id === params.turnId)) throw new Error("codex_history_unavailable: requested turn is outside the retained runner event window");
|
|
const items = new Map<string, Record<string, unknown>>();
|
|
let semanticResultItem: Record<string, unknown> | null = null;
|
|
let observedTurn = "";
|
|
let observedStart = false;
|
|
for (const event of this.#core?.store.state.committedEvents ?? []) {
|
|
if (event.envelope.runId !== this.#core?.store.state.identity.runId) continue;
|
|
const payload = record(record(event.envelope.payload).payload);
|
|
if (event.eventType === "turn.started") observedTurn = String(payload.providerTurnId ?? payload.turnId ?? record(payload.turn).id ?? "");
|
|
if (event.eventType === "turn.started" && observedTurn === params.turnId) observedStart = true;
|
|
if (event.eventType === "run.result.proposed" && observedTurn === params.turnId) {
|
|
// Reconciliation can precede notification delivery. Recover the
|
|
// runner's authoritative result with the exact retained turn,
|
|
// rather than launching work again just to obtain a disposition.
|
|
const id = `runner-result-${event.sourceSeq}`;
|
|
semanticResultItem = { turnId: observedTurn, item: { id, type: "agentMessage", text: JSON.stringify(payload) } };
|
|
}
|
|
if (event.eventType !== "item.completed" || observedTurn !== params.turnId) continue;
|
|
const item = record(rehydrateRunnerdItemNotification(payload, this.#threadId, observedTurn).item);
|
|
if (typeof item.id === "string") items.set(item.id, { turnId: observedTurn, item });
|
|
}
|
|
if (!observedStart) throw new Error("codex_history_incomplete: requested turn start is outside the retained runner event window");
|
|
data = [...items.values()];
|
|
// Runner authority wins over schema-shaped prose in an assistant item.
|
|
if (semanticResultItem) data.push(semanticResultItem);
|
|
}
|
|
if (params.sortDirection === "desc") data.reverse();
|
|
const offset = params.cursor == null ? 0 : Number(params.cursor);
|
|
if (!Number.isSafeInteger(offset) || offset < 0 || offset > data.length) throw new Error("codex_history_invalid_cursor");
|
|
const limit = typeof params.limit === "number" ? Math.max(1, Math.min(100, params.limit)) : 100;
|
|
return { data: data.slice(offset, offset + limit), nextCursor: offset + limit < data.length ? String(offset + limit) : null };
|
|
}
|
|
if (method === "thread/read") {
|
|
if (this.#core === null) {
|
|
await this.#trackStartup(() => {
|
|
this.#recoveryTurnBindingPending = true;
|
|
return this.#resume();
|
|
});
|
|
}
|
|
// Ask the authenticated runner for its live provider snapshot rather
|
|
// than reading its filesystem. This both supports remote process owners
|
|
// and proves any identity restored after PRP event compaction before the
|
|
// checkpoint-backed thread is exposed to the driver.
|
|
const restoreRuntimeRequests = this.#recoveryTurnBindingPending
|
|
&& this.options.adoptExistingRunner !== undefined && this.options.provider === "acpx";
|
|
const snapshot = await this.#commandResult("session.snapshot", restoreRuntimeRequests
|
|
? { includePendingRuntimeRequests: true } : {});
|
|
this.#confirmCheckpointProviderIdentity(
|
|
snapshot,
|
|
"authenticated session.snapshot",
|
|
);
|
|
const activeProviderTurnId =
|
|
typeof snapshot.activeProviderTurnId === "string" &&
|
|
snapshot.activeProviderTurnId.length > 0
|
|
? snapshot.activeProviderTurnId
|
|
: null;
|
|
if (activeProviderTurnId !== null) this.#turnId = activeProviderTurnId;
|
|
const recoveredTurns: Array<Record<string, unknown>> =
|
|
activeProviderTurnId === null
|
|
? []
|
|
: [{ id: activeProviderTurnId, status: "inProgress" }];
|
|
if (activeProviderTurnId === null && this.#turnId.length > 0) {
|
|
const recoveryDeadline = Date.now() + 5_000;
|
|
let terminal = (this.#core?.store.state.committedEvents ?? []).find(
|
|
() => false,
|
|
);
|
|
while (Date.now() < recoveryDeadline) {
|
|
this.#throwIfFailed();
|
|
this.#pumpEvents();
|
|
terminal = [...(this.#core?.store.state.committedEvents ?? [])]
|
|
.reverse()
|
|
.find((event) => {
|
|
if (
|
|
event.eventType !== "turn.completed" &&
|
|
event.eventType !== "turn.failed" &&
|
|
event.eventType !== "turn.interrupted" &&
|
|
event.eventType !== "turn.cancelled"
|
|
)
|
|
return false;
|
|
const payload = record(record(event.envelope.payload).payload);
|
|
const providerTurnId =
|
|
payload.providerTurnId ??
|
|
payload.turnId ??
|
|
record(payload.turn).id;
|
|
return providerTurnId === this.#turnId;
|
|
});
|
|
if (terminal !== undefined) break;
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
if (terminal !== undefined) {
|
|
const payload = record(record(terminal.envelope.payload).payload);
|
|
recoveredTurns.push({
|
|
id: this.#turnId,
|
|
// Reconciliation must retain the cause, including the runner's
|
|
// explicit process-loss marker, rather than inventing error:null.
|
|
error: payload.error ?? record(payload.turn).error ?? null,
|
|
status:
|
|
terminal.eventType === "turn.completed"
|
|
? "completed"
|
|
: terminal.eventType === "turn.interrupted"
|
|
? "interrupted"
|
|
: terminal.eventType === "turn.cancelled"
|
|
? "cancelled"
|
|
: "failed",
|
|
});
|
|
}
|
|
}
|
|
if (restoreRuntimeRequests && activeProviderTurnId !== null) {
|
|
this.#restoredRuntimeRequests = liveAcpxRuntimeRequests(snapshot, this.#threadId, activeProviderTurnId, this.#durableTurnId);
|
|
for (const request of this.#restoredRuntimeRequests) {
|
|
this.#bridgedRuntimeInputs.set(String(request.id), {
|
|
durableTurnId: this.#durableTurnId,
|
|
permission: request.method === "session/request_permission",
|
|
});
|
|
}
|
|
}
|
|
// A controller can lose the checkpoint after a continuation is accepted.
|
|
// Keep its prior terminal as the history anchor, so driver recovery can
|
|
// adopt the later accepted turn instead of submitting it again. Items
|
|
// remain lazy and fail closed if their start left the retained window.
|
|
const priorTerminals = new Map<string, Record<string, unknown>>();
|
|
for (const event of this.#core?.store.state.committedEvents ?? []) {
|
|
if (event.envelope.runId !== this.#core?.store.state.identity.runId ||
|
|
!["turn.completed", "turn.failed", "turn.interrupted", "turn.cancelled"].includes(event.eventType)) continue;
|
|
const payload = record(record(event.envelope.payload).payload);
|
|
const turnId = payload.providerTurnId ?? payload.turnId ?? record(payload.turn).id;
|
|
if (typeof turnId !== "string" || !turnId || turnId === this.#turnId) continue;
|
|
priorTerminals.set(turnId, {
|
|
id: turnId,
|
|
status: event.eventType.slice("turn.".length),
|
|
error: payload.error ?? record(payload.turn).error ?? null,
|
|
});
|
|
}
|
|
recoveredTurns.unshift(...priorTerminals.values());
|
|
this.#recoveryTurnBindingPending = false;
|
|
this.#pumpEvents();
|
|
const prepared = this.#preparedThreadWorkingDirectory();
|
|
if (prepared.cwd !== undefined && typeof snapshot.cwd === "string"
|
|
&& snapshot.cwd.length > 0 && snapshot.cwd !== prepared.cwd) {
|
|
throw new Error("runner_prepared_launch_working_directory_mismatch");
|
|
}
|
|
return {
|
|
thread: {
|
|
id: this.#threadId,
|
|
sessionId: this.#sessionId,
|
|
...(this.#providerIdentity === null
|
|
? {}
|
|
: { providerIdentity: structuredClone(this.#providerIdentity) }),
|
|
// Some providers omit cwd from their snapshot. A prepared launch
|
|
// supplies it; never fall back to a controller-local directory.
|
|
cwd: prepared.cwd ?? (
|
|
typeof snapshot.cwd === "string" && snapshot.cwd.length > 0
|
|
? snapshot.cwd
|
|
: (this.options.environment?.PAPERCLIP_WORKSPACE_CWD ??
|
|
this.options.runnerFilesystemRoot ??
|
|
tmpdir())),
|
|
turns: recoveredTurns,
|
|
},
|
|
};
|
|
}
|
|
if (method === "thread/goal/get") {
|
|
const result = await this.#commandResult("session.goal.get", params);
|
|
return {
|
|
goal: appServerThreadGoal(result.goal, this.#threadId),
|
|
};
|
|
}
|
|
if (method === "thread/goal/set") {
|
|
const result = await this.#commandResult("session.goal.set", params);
|
|
const snapshot = record(result.snapshot);
|
|
return {
|
|
goal: appServerThreadGoal(
|
|
snapshot.goal ?? result.goal,
|
|
this.#threadId,
|
|
),
|
|
};
|
|
}
|
|
if (method === "thread/goal/clear") {
|
|
await this.#commandResult("session.goal.clear", params);
|
|
return {};
|
|
}
|
|
if (method === "session/budget/increase") {
|
|
await this.#command("session.budget.increase", params);
|
|
return {};
|
|
}
|
|
if (method === "session/destroy") {
|
|
await this.#command("session.destroy", params);
|
|
return {};
|
|
}
|
|
if (method === "thread/resume") {
|
|
if (
|
|
this.#checkpointProviderIdentityExpectation !== null &&
|
|
!this.#checkpointProviderIdentityConfirmed
|
|
) {
|
|
const snapshot = await this.#commandResult("session.snapshot", {});
|
|
this.#confirmCheckpointProviderIdentity(
|
|
snapshot,
|
|
"authenticated session.snapshot",
|
|
);
|
|
}
|
|
return {
|
|
...this.#preparedThreadWorkingDirectory(),
|
|
thread: {
|
|
id: this.#threadId,
|
|
sessionId: this.#sessionId,
|
|
...(this.#providerIdentity === null
|
|
? {}
|
|
: { providerIdentity: structuredClone(this.#providerIdentity) }),
|
|
},
|
|
};
|
|
}
|
|
throw new Error(
|
|
`PRP Codex transport does not expose provider method ${method}`,
|
|
);
|
|
}
|
|
|
|
notify(_method: string, _params?: Record<string, unknown>): void {}
|
|
|
|
notifications(): AsyncIterable<CodexRpcNotification> {
|
|
return this.#queue;
|
|
}
|
|
|
|
takeRestoredRuntimeRequests(): CodexRpcServerRequest[] {
|
|
this.#throwIfFailed();
|
|
const requests = this.#restoredRuntimeRequests;
|
|
this.#restoredRuntimeRequests = [];
|
|
return requests.filter(request => this.#bridgedRuntimeInputs.has(String(request.id))
|
|
&& request.params.threadId === this.#threadId && request.params.turnId === this.#turnId);
|
|
}
|
|
|
|
setServerRequestHandler(handler: CodexServerRequestHandler): void {
|
|
this.#handler = handler;
|
|
this.#handlerInstalled = true;
|
|
this.#resolveHandlerReady();
|
|
this.#recoverPendingSemanticInputsWhenReady();
|
|
}
|
|
|
|
#recoverPendingSemanticInputsWhenReady(): void {
|
|
if (this.#startupComplete && this.#handlerInstalled) {
|
|
void Promise.resolve().then(() => this.#core?.recoverPendingSemanticInputs())
|
|
.catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error))));
|
|
}
|
|
}
|
|
|
|
async #awaitWarmRunAttachmentReady(): Promise<void> {
|
|
// Remote runner ingress already has a bounded reconnect budget. Reuse the
|
|
// same budget here so a transient tunnel reconnect cannot trip the shorter
|
|
// generic command timeout and replace an otherwise healthy warm runner.
|
|
const reconnectGraceMs = this.options.runnerReconnectGraceMs ?? 5_000;
|
|
await waitForWarmAttachmentReadiness({
|
|
graceMs: reconnectGraceMs,
|
|
waitForConnection: (deadline) => this.#awaitWarmRunnerConnection(deadline),
|
|
snapshot: (deadline) => this.#commandResult(
|
|
"session.snapshot",
|
|
{
|
|
quiesceForWarmAttach: true,
|
|
},
|
|
deadline,
|
|
),
|
|
onBlocked: (blockers) => this.#diagnostic(`warm attachment awaiting quiescence: ${JSON.stringify(blockers)}`),
|
|
});
|
|
}
|
|
|
|
async #awaitWarmRunnerConnection(deadline: number): Promise<void> {
|
|
const core = this.#core;
|
|
if (core === null) throw new Error("native_runner_authority_unavailable");
|
|
let reportedReconnectWait = false;
|
|
while (Date.now() < deadline) {
|
|
this.#throwIfFailed();
|
|
const connectionCount = core.activeRunnerConnectionCount();
|
|
if (connectionCount === 1) {
|
|
if (reportedReconnectWait) {
|
|
this.#diagnostic(
|
|
"warm runner re-authenticated before authority rotation",
|
|
);
|
|
}
|
|
return;
|
|
}
|
|
if (connectionCount > 1) {
|
|
throw new Error(
|
|
`native_runner_warm_attachment_ambiguous: expected one authenticated runner, found ${connectionCount}`,
|
|
);
|
|
}
|
|
if (!reportedReconnectWait) {
|
|
reportedReconnectWait = true;
|
|
this.#diagnostic(
|
|
"warm runner connection interrupted; waiting for re-authentication before authority rotation",
|
|
);
|
|
}
|
|
if (await this.#runnerHasExited()) {
|
|
throw new Error(
|
|
"native_runner_warm_attachment_runner_exited: runner exited before authority rotation",
|
|
);
|
|
}
|
|
await Promise.race([
|
|
new Promise<void>((resolveWait) => setTimeout(resolveWait, 25)),
|
|
this.#failureSignal,
|
|
]);
|
|
}
|
|
throw new Error(
|
|
`provider_transport_failed: warm runner did not re-authenticate within ${this.options.runnerReconnectGraceMs ?? 5_000}ms`,
|
|
);
|
|
}
|
|
|
|
attachRun(input: {
|
|
runId: string;
|
|
turnId: string;
|
|
itemId: string;
|
|
currentRunGrant?: NativeRunRuntimeGrant;
|
|
}): Promise<void> {
|
|
if (this.#attachmentOperation !== null) {
|
|
return Promise.reject(new Error("native_runner_run_attachment_in_progress"));
|
|
}
|
|
const operation = Promise.resolve().then(() => {
|
|
this.#assertStartupOpen();
|
|
return this.#attachRun(input);
|
|
}).finally(() => { this.#attachmentOperation = null; });
|
|
this.#attachmentOperation = operation;
|
|
return operation;
|
|
}
|
|
|
|
async #attachRun(input: {
|
|
runId: string;
|
|
turnId: string;
|
|
itemId: string;
|
|
currentRunGrant?: NativeRunRuntimeGrant;
|
|
}): Promise<void> {
|
|
const previous = record(this.#runAttachTemplate?.provider).runtimeContext;
|
|
const current = input.currentRunGrant?.runtimeContext;
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.attach", () => this.#attachRunWork(input),
|
|
previous === undefined ? undefined : {contextReused:current === undefined || commandDigest(current) === commandDigest(previous)}) : this.#attachRunWork(input);
|
|
}
|
|
|
|
async #attachRunWork(input: {
|
|
runId: string;
|
|
turnId: string;
|
|
itemId: string;
|
|
currentRunGrant?: NativeRunRuntimeGrant;
|
|
}): Promise<void> {
|
|
if (this.#pendingWarmRecoveryCompletion !== null) {
|
|
throw new Error("native_runner_warm_transition_completion_pending");
|
|
}
|
|
const core = this.#core;
|
|
if (!core || !this.#startupComplete) {
|
|
throw new Error("native_runner_prp_run_rotation_unavailable");
|
|
}
|
|
await this.#awaitWarmRunAttachmentReady();
|
|
this.#assertStartupOpen();
|
|
const currentRunGrant = input.currentRunGrant
|
|
? projectCopilotRunGrant(input.currentRunGrant, this.options.runnerFilesystemRoot)
|
|
: undefined;
|
|
if (this.options.provider === "acpx" && this.options.acpxAgent === "copilot" && !currentRunGrant) {
|
|
throw new Error("native_runner_current_instruction_grant_unavailable");
|
|
}
|
|
const prior = core.store.state.identity;
|
|
const desired: DurableRecoveryIdentity = {
|
|
...prior,
|
|
runId: input.runId,
|
|
turnId: input.turnId,
|
|
itemId: input.itemId,
|
|
};
|
|
const registration = this.options.controlPlaneRegistration
|
|
? await this.options.controlPlaneRegistration(core, desired)
|
|
: null;
|
|
const previousRelease = this.#controlPlaneRelease;
|
|
let previousReleased = false;
|
|
let activationStarted = false;
|
|
try {
|
|
this.#assertStartupOpen();
|
|
const connection: RunnerProcessConnection =
|
|
registration?.connection ??
|
|
(registration?.connectUrl
|
|
? { mode: "connect", connectUrl: registration.connectUrl }
|
|
: { mode: "connect", connectUrl: core.connectUrl });
|
|
const commandId = `command_attach_${createHash("sha256")
|
|
.update(`${prior.runId}:${desired.runId}:${desired.turnId}`)
|
|
.digest("hex")
|
|
.slice(0, 32)}`;
|
|
const runAttachTemplate = this.#runAttachTemplate
|
|
? retargetRunAttachPayload(
|
|
this.#runAttachTemplate,
|
|
desired,
|
|
this.#authorizedTools,
|
|
this.options.resumeCompletionContract,
|
|
currentRunGrant?.runtimeContext,
|
|
currentRunGrant ? { text: currentRunGrant.instructions, context: currentRunGrant.runtimeContext } : undefined,
|
|
)
|
|
: rotatedRunAttachPayload(
|
|
core.store.state,
|
|
desired,
|
|
this.#authorizedTools,
|
|
this.options.resumeCompletionContract,
|
|
currentRunGrant?.runtimeContext,
|
|
currentRunGrant ? { text: currentRunGrant.instructions, context: currentRunGrant.runtimeContext } : undefined,
|
|
);
|
|
if (this.options.prepareLaunch) {
|
|
const priorProvider = record(runAttachTemplate.provider);
|
|
const priorRequirements = new RunnerLaunchPreparationStore(this.#root).load(prior)?.requirements;
|
|
await this.#prepareOwnedProvider(runAttachTemplate, desired,
|
|
currentRunGrant ? currentRunGrant.runtimeContext : priorRequirements?.runtimeContext
|
|
?? priorProvider.runtimeContext as NativeRuntimeContextSnapshot | null,
|
|
currentRunGrant?.instructions ?? priorRequirements?.instructions ?? String(priorProvider.instructions ?? ""));
|
|
}
|
|
this.#assertStartupOpen();
|
|
this.#runAttachTemplate = structuredClone(runAttachTemplate);
|
|
const payload = {
|
|
...runAttachTemplate,
|
|
paperclipNextAuthority: { identity: desired, connection },
|
|
};
|
|
core.queueCommand("run.attach", payload, commandId, true);
|
|
// ACPX run attachment checkpoints the old sidecar and starts a fresh
|
|
// provider process. Pi must use the same absolute cold-admission budget
|
|
// as startup/recovery even though its Runner authority remains warm.
|
|
// Other providers retain the ordinary command bound.
|
|
await this.#waitCommand(
|
|
"run.attach",
|
|
commandId,
|
|
this.#coldAdmissionDeadline(),
|
|
true,
|
|
);
|
|
const attached = core.getCommand(commandId);
|
|
if (attached?.status !== "completed") {
|
|
throw new Error("native_runner_prp_run_rotation_failed");
|
|
}
|
|
|
|
activationStarted = true;
|
|
core.rotateRunIdentity(desired, runAttachTemplate);
|
|
await registration?.activate?.();
|
|
if (registration?.failure) {
|
|
void registration.failure.catch((error: unknown) => {
|
|
this.#failTransport(
|
|
error instanceof Error ? error : new Error(String(error)),
|
|
);
|
|
});
|
|
}
|
|
await this.#awaitRegistrationReady(registration?.ready);
|
|
const activationDeadline =
|
|
Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000);
|
|
while (
|
|
!recoveryIdentityMatches(core.store.state.identity, desired) ||
|
|
core.store.state.warmTransition !== undefined ||
|
|
core.activeRunnerConnectionCount() === 0
|
|
) {
|
|
if (
|
|
Date.now() >= activationDeadline ||
|
|
(await this.#runnerHasExited())
|
|
) {
|
|
throw new Error("native_runner_warm_transition_activation_pending");
|
|
}
|
|
await new Promise<void>((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
this.#eventIdentity = structuredClone(desired);
|
|
this.#eventSourceSeq = 0;
|
|
this.#deferredTurnStartEvents = [];
|
|
this.#durableTurnId = desired.turnId;
|
|
await previousRelease?.();
|
|
previousReleased = true;
|
|
this.#controlPlaneRelease = registration?.release ?? null;
|
|
} catch (error) {
|
|
const failure = error instanceof Error ? error : new Error(String(error));
|
|
// The future route is ours from registration onward, including failures
|
|
// in template construction, capability admission, and result waiting.
|
|
// Keep the prior release owned by close until its handoff is confirmed.
|
|
if (!this.#closed) this.#controlPlaneRelease = previousReleased ? null : previousRelease;
|
|
await Promise.resolve()
|
|
.then(() => registration?.release())
|
|
.catch(() => undefined);
|
|
if (activationStarted) {
|
|
// Once the completed handoff is exposed, an activation failure makes
|
|
// both routes unavailable; neither remains an ordinary-work owner.
|
|
this.#controlPlaneRelease = null;
|
|
if (!previousReleased) {
|
|
await Promise.resolve()
|
|
.then(() => previousRelease?.())
|
|
.catch(() => undefined);
|
|
}
|
|
this.#failTransport(failure);
|
|
}
|
|
throw failure;
|
|
}
|
|
|
|
}
|
|
|
|
async resolveRuntimeRequest(input: {
|
|
requestId: string;
|
|
turnId: string;
|
|
resolution: HarnessRuntimeRequestResolution;
|
|
}): Promise<void> {
|
|
if (this.#pendingWarmRecoveryCompletion !== null) {
|
|
throw new Error("native_runner_warm_transition_completion_pending");
|
|
}
|
|
const pending = this.#bridgedRuntimeInputs.get(input.requestId);
|
|
if (!pending)
|
|
throw new Error(
|
|
`PRP runtime request ${input.requestId} is no longer pending`,
|
|
);
|
|
if (input.turnId !== this.#turnId) throw new Error("native_runner_runtime_response_stale_turn");
|
|
if (!pending.permission && !("response" in input.resolution) && input.resolution.action !== "cancel" && input.resolution.action !== "decline") {
|
|
throw new Error(
|
|
"runnerd-native runtime requests require a canonical question response",
|
|
);
|
|
}
|
|
const commandId = `command_runtime_input_${createHash("sha256")
|
|
.update(`${input.requestId}:${pending.durableTurnId}`)
|
|
.digest("hex")
|
|
.slice(0, 24)}`;
|
|
await this.#command(
|
|
"request.resolve",
|
|
{
|
|
requestId: input.requestId,
|
|
turnId: pending.durableTurnId,
|
|
...("response" in input.resolution ? { response: input.resolution.response } : { resolution: input.resolution }),
|
|
},
|
|
commandId,
|
|
);
|
|
}
|
|
|
|
recordTraceInterpretation(input: CodexTraceInterpretation): void {
|
|
const tracePath = this.options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH;
|
|
if (!tracePath) return;
|
|
const traceResult = appendCodexDriverInterpretationTrace(
|
|
this.#traceFrameIndex,
|
|
tracePath,
|
|
input,
|
|
this.#nextTraceDebugSequence,
|
|
);
|
|
if (traceResult === "written") {
|
|
this.#nextTraceDebugSequence += 1;
|
|
} else if (
|
|
traceResult === "retry" &&
|
|
this.#pendingDriverTraceInterpretations.length < 4_096
|
|
) {
|
|
this.#pendingDriverTraceInterpretations.push(structuredClone(input));
|
|
} else if (traceResult === "retry") {
|
|
this.#traceRehydrationSpoolOverflow = true;
|
|
}
|
|
}
|
|
|
|
processInfo(): CodexTransportProcessInfo {
|
|
return {
|
|
pid: this.#evidence.runnerPid,
|
|
processGroupId: this.#evidence.runnerProcessGroupId,
|
|
startedAt: this.options.adoptExistingRunner?.startedAt ?? this.#handle?.startedAt ?? this.#startedAt,
|
|
exited: this.#evidence.runnerExited,
|
|
exitCode: this.#evidence.runnerExitCode,
|
|
signal: this.#evidence.runnerSignal,
|
|
};
|
|
}
|
|
|
|
async #publishSpawnedProcess(handle: RunnerProcessHandle): Promise<void> {
|
|
this.#evidence.runnerPid = handle.child.pid ?? null;
|
|
this.#evidence.runnerProcessGroupId = handle.processGroupId ?? null;
|
|
this.#publish();
|
|
if (handle.child.pid !== undefined) {
|
|
await this.options.onSpawn?.({
|
|
pid: handle.child.pid,
|
|
processGroupId: handle.processGroupId ?? null,
|
|
startedAt: handle.startedAt ?? this.#startedAt,
|
|
});
|
|
}
|
|
}
|
|
|
|
async #readDurableRunnerState(): Promise<Record<string, unknown>> {
|
|
if (this.options.readRunnerState) return this.options.readRunnerState();
|
|
return record(
|
|
JSON.parse(
|
|
readFileSync(
|
|
resolve(this.#root, "runner", "runner-state.json"),
|
|
"utf8",
|
|
),
|
|
),
|
|
);
|
|
}
|
|
|
|
#providerDrainState():
|
|
| {
|
|
pendingEventCount: number;
|
|
activeProviderTurnId: string | null;
|
|
providerSettled: boolean;
|
|
}
|
|
| "unreadable"
|
|
| null {
|
|
if (this.options.runnerFilesystemRoot !== undefined || this.options.prepareLaunch) return null;
|
|
const provider = this.options.provider ?? "codex";
|
|
const filename =
|
|
provider === "acpx"
|
|
? "acpx-provider-state.json"
|
|
: provider === "claude_managed" || provider === "aws_agentcore"
|
|
? "managed-provider-state.json"
|
|
: "codex-provider-state.json";
|
|
const stateDirectory =
|
|
this.options.runnerStateDirectory ?? resolve(this.#root, "runner");
|
|
const statePath = resolve(stateDirectory, filename);
|
|
if (!existsSync(statePath)) {
|
|
if (this.#startupComplete) return "unreadable";
|
|
return {
|
|
pendingEventCount: 0,
|
|
activeProviderTurnId: null,
|
|
providerSettled: true,
|
|
};
|
|
}
|
|
try {
|
|
const state = record(JSON.parse(readFileSync(statePath, "utf8")));
|
|
return providerDrainStateFromSnapshot(state);
|
|
} catch {
|
|
return "unreadable";
|
|
}
|
|
}
|
|
|
|
async #stopProviderBeforeSuspend(deadline: number): Promise<void> {
|
|
const state = this.#providerDrainState();
|
|
const core = this.#core;
|
|
const inferredActiveProviderTurnId =
|
|
state === null &&
|
|
core !== null &&
|
|
providerTurnIsActiveFromCommittedEvents(core.store.state.committedEvents)
|
|
? this.#turnId || this.#durableTurnId
|
|
: null;
|
|
const activeProviderTurnId =
|
|
state !== null && state !== "unreadable"
|
|
? state.activeProviderTurnId
|
|
: inferredActiveProviderTurnId;
|
|
// Pi's idle RPC close can consume the entire suspension reserve. Retire
|
|
// its already-settled process during preparation instead. Native turn.stop
|
|
// independently checks the idle ledger and inherited lifetime fence;
|
|
// drain and runner.suspend still prove exact durable settlement afterward.
|
|
const stopIdlePi = this.options.provider === "acpx"
|
|
&& this.options.acpxAgent === "pi"
|
|
&& state !== "unreadable"
|
|
&& (state === null || (state.activeProviderTurnId === null && state.providerSettled));
|
|
if (
|
|
state === "unreadable" ||
|
|
(activeProviderTurnId === null && !stopIdlePi) ||
|
|
core === null
|
|
) {
|
|
return;
|
|
}
|
|
const commandId = `command_close_stop_${randomUUID().replaceAll("-", "")}`;
|
|
core.queueCommand(
|
|
"turn.stop",
|
|
{ reason: "transport closing after durable run terminal" },
|
|
commandId,
|
|
true,
|
|
);
|
|
while (Date.now() < deadline) {
|
|
this.#pumpEventsSafely();
|
|
const command = core.store.state.commands.find(
|
|
(candidate) => candidate.commandId === commandId,
|
|
);
|
|
if (command?.status === "completed") {
|
|
this.#diagnostic(
|
|
activeProviderTurnId === null
|
|
? "stopped idle Pi provider before runner suspension"
|
|
: `stopped active provider turn ${activeProviderTurnId} before runner suspension`,
|
|
);
|
|
return;
|
|
}
|
|
if (command !== undefined && command.status !== "pending") {
|
|
this.#diagnostic(
|
|
`provider turn stop ${command.status} before runner suspension`,
|
|
);
|
|
return;
|
|
}
|
|
if (await this.#runnerHasExited()) return;
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 5));
|
|
}
|
|
this.#diagnostic("provider turn stop timed out before runner suspension");
|
|
}
|
|
|
|
async #drainSettledProviderEventsBeforeSuspend(
|
|
timeoutMs = 1_000,
|
|
): Promise<boolean> {
|
|
const core = this.#core;
|
|
if (!core) return false;
|
|
try {
|
|
const drained = await awaitProviderDrainBarrier({
|
|
readProviderState: () => this.#providerDrainState(),
|
|
semanticResultsSettled: () => core.semanticToolResultsSettled(),
|
|
commands: () => core.store.state.commands,
|
|
queueDrain: (commandId) => {
|
|
core.queueCommand("runner.drain", {}, commandId, true);
|
|
},
|
|
pump: () => this.#pumpEventsSafely(),
|
|
deadline: Date.now() + timeoutMs,
|
|
});
|
|
if (drained) return true;
|
|
} catch {
|
|
// A failed drain still proceeds through bounded suspension/containment,
|
|
// but can never authorize a reusable checkpoint or deletion of evidence.
|
|
}
|
|
const lastDrain = [...core.store.state.commands]
|
|
.reverse()
|
|
.find((command) => command.type === "runner.drain");
|
|
this.#diagnostic(
|
|
"provider suffix did not prove durable drain before bounded runner suspension: " +
|
|
JSON.stringify({
|
|
providerState: this.#providerDrainState(),
|
|
semanticResultsSettled: core.semanticToolResultsSettled(),
|
|
drainStatus: lastDrain?.status ?? null,
|
|
retainedEventsDrained:
|
|
record(record(lastDrain?.result).result).retainedEventsDrained ?? null,
|
|
}),
|
|
);
|
|
return false;
|
|
}
|
|
|
|
close(reason?: string): Promise<void> {
|
|
// Detachment relinquishes process ownership. A late execution finalizer
|
|
// must not suspend or signal the runner now owned by the next controller.
|
|
if (this.#controllerDetachedForRestart)
|
|
return this.#detachPromise ?? Promise.resolve();
|
|
if (reason) {
|
|
this.#diagnostic(
|
|
`runner transport close requested: ${reason.replaceAll(/[\r\n]/g, " ").slice(0, 1_000)}`,
|
|
);
|
|
}
|
|
this.#closePromise ??= this.#closeOnce();
|
|
return this.#closePromise;
|
|
}
|
|
|
|
retireProbeResources(timeoutMs?: number): Promise<void> {
|
|
if (this.#probeRetirement) return this.#probeRetirement;
|
|
const operation = (async () => {
|
|
const deadline = Date.now() + probeCleanupBudget(timeoutMs);
|
|
// Cancellation/timeouts revoke admission but cannot cancel an in-flight
|
|
// filesystem request, registration, or child spawn. Join those originals.
|
|
await awaitProbeCleanup(Promise.allSettled([this.#startupOperation, this.#attachmentOperation]), deadline, "joining the original launch");
|
|
if (this.#closePromise) await awaitProbeCleanup(this.#closePromise.catch(() => undefined), deadline, "joining the original close");
|
|
const handle = this.#handle;
|
|
if (handle) {
|
|
// Close already exercised the original owner's signal authority. Proof
|
|
// must never dispatch another signal using a possibly reused PID/group.
|
|
try { await awaitProbeCleanup(handle.completion, deadline, "waiting for the original child exit"); }
|
|
catch (error) { if (handle.child.pid !== undefined) throw error; }
|
|
await awaitProbeProof(() => handle.isExitConfirmed?.() !== false, deadline, "confirming the original owner exit");
|
|
}
|
|
// A remote owner must explicitly prove its own process retirement. An
|
|
// absent local handle or a dispatched remote signal is not that proof.
|
|
if (this.options.prepareLaunch || this.options.runnerFilesystemRoot || this.options.runnerProcessLauncher || this.options.adoptExistingRunner) {
|
|
if (handle?.isExitConfirmed?.() !== true && this.#evidence.runnerPid !== null) throw new Error("Native probe remote retirement remains unconfirmed");
|
|
}
|
|
this.#pumpEventsSafely();
|
|
const pids = [...new Set([this.#evidence.runnerPid, this.#evidence.sidecarPid, this.#evidence.agentPid, this.#evidence.providerPid])].filter((pid): pid is number => pid !== null);
|
|
if (pids.length && process.platform === "win32") throw new Error("Native probe platform process retirement remains unconfirmed");
|
|
await awaitProbeProof(() => pids.every(maintenanceProcessAbsent), deadline, "confirming recorded process absence");
|
|
if (this.options.provider === "acpx" && this.options.acpxAgent === "antigravity") {
|
|
const sessionId = this.#eventIdentity?.normalizedSessionId ?? this.options.prpIdentity?.normalizedSessionId;
|
|
if (sessionId) await awaitProbeCleanup(confirmAntigravityProbeNativeRetirement(this.options.acpxRuntimeDirectory ?? resolve(this.#root, "acpx"), sessionId), deadline, "confirming the original Google owner retirement");
|
|
else if (handle?.child.pid !== undefined) throw new Error("Native probe credential lifetime is unidentified");
|
|
}
|
|
})();
|
|
this.#probeRetirement = operation;
|
|
void operation.catch(() => { if (this.#probeRetirement === operation) this.#probeRetirement = null; });
|
|
return operation;
|
|
}
|
|
|
|
detachControllerForRestart(): Promise<void> {
|
|
this.#detachPromise ??= this.#detachControllerOnce();
|
|
return this.#detachPromise;
|
|
}
|
|
|
|
async #stopAndDrainAuthority(deadline: number): Promise<void> {
|
|
const core = this.#core;
|
|
if (!core) return;
|
|
await core.stop();
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
try {
|
|
await Promise.race([
|
|
core.drainPendingConnectionProcessing(),
|
|
new Promise<never>((_resolve, reject) => {
|
|
timer = setTimeout(() => reject(new Error(
|
|
"Controller journal drain timed out; preserve the session before replacing its writer.",
|
|
)), Math.max(0, deadline - Date.now()));
|
|
}),
|
|
]);
|
|
} finally {
|
|
clearTimeout(timer);
|
|
}
|
|
}
|
|
|
|
async #detachControllerOnce(): Promise<void> {
|
|
if (this.#closed) return;
|
|
const deadline = Date.now() + runnerCloseGraceMs(this.options);
|
|
this.#controllerDetachedForRestart = true;
|
|
this.#closed = true;
|
|
this.#resolveClosedSignal();
|
|
this.#turnStartAdmission?.resolve(false);
|
|
await this.#retireControllerForRecovery(deadline);
|
|
this.#handle = null;
|
|
this.#diagnostic(
|
|
"controller authority detached for restart; durable runner left alive",
|
|
);
|
|
}
|
|
|
|
async #retireControllerForRecovery(deadline: number): Promise<void> {
|
|
if (this.#pump !== null) clearInterval(this.#pump);
|
|
this.#pump = null;
|
|
if (this.#adoptedRunnerMonitor !== null)
|
|
clearInterval(this.#adoptedRunnerMonitor);
|
|
this.#adoptedRunnerMonitor = null;
|
|
this.#core?.disconnectActiveRunner();
|
|
const release = this.#controlPlaneRelease;
|
|
this.#controlPlaneRelease = null;
|
|
await Promise.resolve(release?.("detach")).catch((error: unknown) => {
|
|
this.#diagnostic(
|
|
`controller route release failed during retirement: ${String(error)}`,
|
|
);
|
|
});
|
|
try {
|
|
await this.#stopAndDrainAuthority(deadline);
|
|
} finally {
|
|
this.#queue.close();
|
|
}
|
|
}
|
|
|
|
#trackStartup<T>(start: () => Promise<T>): Promise<T> {
|
|
if (this.#startupOperation !== null) {
|
|
return Promise.reject(new Error("PRP provider thread is already started"));
|
|
}
|
|
const operation = Promise.resolve().then(() => {
|
|
this.#assertStartupOpen();
|
|
return start();
|
|
});
|
|
this.#startupOperation = operation;
|
|
return operation;
|
|
}
|
|
|
|
#assertStartupOpen(): void {
|
|
if (this.#closed) throw new Error("runnerd transport closed during provider startup");
|
|
}
|
|
|
|
async #closeOnce(): Promise<void> {
|
|
const deadline = Date.now() + runnerCloseGraceMs(this.options);
|
|
try {
|
|
await this.#closeAndSettle();
|
|
} catch (error) {
|
|
this.#queue.close();
|
|
try { await this.#retireControllerForRecovery(deadline); }
|
|
catch (retirementError) {
|
|
this.#diagnostic(`controller retirement after failed close is unconfirmed: ${String(retirementError)}`);
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
async #closeAndSettle(): Promise<void> {
|
|
this.#closed = true;
|
|
this.#resolveClosedSignal();
|
|
const { preparationDeadline, closeDeadline } = runnerCloseDeadlines(Date.now(), runnerCloseGraceMs(this.options));
|
|
let preparationConfirmed = true;
|
|
const attachmentPending = this.#attachmentOperation !== null;
|
|
if (this.#startupOperation !== null || attachmentPending) {
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
try {
|
|
await Promise.race([
|
|
Promise.allSettled([this.#startupOperation, this.#attachmentOperation]),
|
|
new Promise<never>((_resolve, reject) => {
|
|
timer = setTimeout(() => reject(new Error("Native preparation drain timed out")), Math.max(0, preparationDeadline - Date.now()));
|
|
}),
|
|
]);
|
|
} catch {
|
|
preparationConfirmed = false;
|
|
this.#startupDrainTimedOut = true;
|
|
} finally { clearTimeout(timer); }
|
|
}
|
|
this.#turnStartAdmission?.resolve(false);
|
|
const adoptedRunner = this.options.adoptExistingRunner;
|
|
// `settled` is a durable-state assertion, not merely the absence of a
|
|
// process handle. Registration can install a remote checkpoint callback
|
|
// before process launch; a synchronous launch failure must therefore stay
|
|
// `unsettled` and preserve its original bootstrap diagnostic.
|
|
let runnerSuspended = false;
|
|
let providerDrained = false;
|
|
let suspensionRequired = false;
|
|
let lastSuspensionState: Record<string, unknown> | null = null;
|
|
let suspensionState: {
|
|
commandStatus: string | null;
|
|
runnerLifecycle: string | null;
|
|
runnerIdentityMatches: boolean | null;
|
|
} | null = null;
|
|
if (
|
|
this.#core !== null &&
|
|
(this.#handle !== null || adoptedRunner !== undefined) &&
|
|
(adoptedRunner === undefined || this.#adoptedRunnerAuthenticated) &&
|
|
(this.#failure === null || this.#startupComplete)
|
|
) {
|
|
// A terminal provider frame can become visible one control loop before
|
|
// its durable provider suffix is ACKed. Drain it before suspension so a
|
|
// fresh run authority never inherits the prior run's pending events.
|
|
|
|
if (!(await this.#runnerHasExited())) {
|
|
// Let an already-admitted tool result reach its original provider
|
|
// before turn.stop can retire that tool-call identity. This shares the
|
|
// close preparation deadline; a stuck callback never stalls cleanup.
|
|
while (
|
|
!this.#core.semanticToolResultsSettled() &&
|
|
Date.now() < preparationDeadline
|
|
) {
|
|
this.#pumpEventsSafely();
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 5));
|
|
}
|
|
// The drain always needs one real command round trip to the runner
|
|
// process, whether or not a turn was active: stopping an active
|
|
// turn only changes how much trailing event traffic that round
|
|
// trip may need to carry. Give both cases the same budget so a
|
|
// slow-but-idle runner is not held to a tighter deadline than a
|
|
// runner that just stopped a turn.
|
|
await this.#stopProviderBeforeSuspend(preparationDeadline);
|
|
providerDrained = await this.#drainSettledProviderEventsBeforeSuspend(
|
|
Math.min(5_000, Math.max(0, preparationDeadline - Date.now())),
|
|
);
|
|
}
|
|
// Local durable roots are reused too. Process exit alone cannot prove
|
|
// their authority is safe to rotate; require the same exact suspension
|
|
// barrier even when there is no remote checkpoint callback.
|
|
suspensionRequired = true;
|
|
runnerSuspended = await awaitRunnerSuspensionBarrier({
|
|
commands: () => this.#core?.store.state.commands ?? [],
|
|
queueSuspend: (commandId) => {
|
|
this.#core?.queueCommand("runner.suspend", {}, commandId, true);
|
|
},
|
|
readRunnerState: async () => {
|
|
const state = await this.#readDurableRunnerState();
|
|
lastSuspensionState = state;
|
|
assertSuspendedRunnerState(state, this.#core!.store.state.identity);
|
|
return state;
|
|
},
|
|
runnerHasExited: () => this.#runnerHasExited(),
|
|
pump: () => this.#pumpEventsSafely(),
|
|
deadline: closeDeadline,
|
|
});
|
|
if (!runnerSuspended) {
|
|
const command = [...this.#core.store.state.commands].reverse().find(
|
|
(candidate) => candidate.type === "runner.suspend",
|
|
);
|
|
// Reuse the barrier's last observation. Diagnostic reads must not
|
|
// extend the close deadline or depend on a now-unreachable remote root.
|
|
const state = lastSuspensionState as Record<string, unknown> | null;
|
|
suspensionState = {
|
|
commandStatus: command?.status ?? null,
|
|
runnerLifecycle: state !== null && [
|
|
"ready", "suspended", "closed", "recoverable_failure",
|
|
].includes(String(state.lifecycle)) ? String(state.lifecycle) : null,
|
|
runnerIdentityMatches: state === null ? null
|
|
: state.schema === "paperclip.runner.durable.state.v1"
|
|
&& recoveryIdentityMatches(state, this.#core.store.state.identity),
|
|
};
|
|
this.#diagnostic(
|
|
"runner did not prove durable suspension before checkpoint",
|
|
);
|
|
}
|
|
try {
|
|
if (this.#handle) {
|
|
const result = await waitForProcess(
|
|
this.#handle,
|
|
Math.max(0, closeDeadline - Date.now()),
|
|
);
|
|
this.#evidence.runnerExited = true;
|
|
this.#evidence.runnerExitCode = result.code;
|
|
this.#evidence.runnerSignal = result.signal as NodeJS.Signals | null;
|
|
if (result.stderr.trim())
|
|
this.#diagnostic(result.stderr.trim().slice(-4_096));
|
|
} else if (adoptedRunner) {
|
|
while (
|
|
(await adoptedRunner.isAlive()) &&
|
|
Date.now() < closeDeadline
|
|
) {
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 25));
|
|
}
|
|
if (await adoptedRunner.isAlive()) {
|
|
await adoptedRunner.signal?.("SIGKILL");
|
|
}
|
|
this.#evidence.runnerExited = !(await adoptedRunner.isAlive());
|
|
}
|
|
} catch (error) {
|
|
this.#diagnostic(`runner shutdown failed: ${String(error)}`);
|
|
}
|
|
}
|
|
this.#flushPendingTraceRehydrations();
|
|
const tracePath = this.options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH;
|
|
if (tracePath) {
|
|
const incomplete =
|
|
this.#traceRehydrationSpoolOverflow ||
|
|
this.#pendingTraceRehydrations.length > 0 ||
|
|
this.#pendingDriverTraceInterpretations.length > 0;
|
|
const debugSequence = this.#nextTraceDebugSequence++;
|
|
try {
|
|
appendRunnerdTraceStatus(tracePath, {
|
|
status: incomplete ? "incomplete" : "complete",
|
|
reason: incomplete
|
|
? this.#traceRehydrationSpoolOverflow
|
|
? "typescript_rehydration_spool_full"
|
|
: "typescript_rehydration_correlation_incomplete"
|
|
: null,
|
|
debugSequence,
|
|
acknowledgedDebugSequence: debugSequence - 1,
|
|
});
|
|
} catch {
|
|
// Raw trace failure is intentionally independent of run authority.
|
|
}
|
|
this.#pendingTraceRehydrations = [];
|
|
this.#pendingDriverTraceInterpretations = [];
|
|
this.#traceFrameIndex.clear();
|
|
}
|
|
if (this.#pump !== null) clearInterval(this.#pump);
|
|
this.#pump = null;
|
|
if (this.#adoptedRunnerMonitor !== null)
|
|
clearInterval(this.#adoptedRunnerMonitor);
|
|
this.#adoptedRunnerMonitor = null;
|
|
this.#queue.close();
|
|
// A suspended remote runner still owns the only readable copy of its
|
|
// provider state. Quiesce the authenticated route, then probe its
|
|
// independently verified durable state before releasing the process owner;
|
|
// an incomplete or identity-conflicting state remains fail-closed.
|
|
const finalProviderState = this.#providerDrainState();
|
|
const runnerSettled =
|
|
preparationConfirmed && runnerSuspended &&
|
|
providerDrained &&
|
|
this.#core?.semanticToolResultsSettled() === true &&
|
|
(finalProviderState === null ||
|
|
(finalProviderState !== "unreadable" &&
|
|
finalProviderState.pendingEventCount === 0 &&
|
|
finalProviderState.providerSettled));
|
|
try {
|
|
await releaseRunnerProcessOwnership({
|
|
runnerSettled,
|
|
// An alive PID does not authorize controlling or replacing a runner
|
|
// that never authenticated to this controller (for example after an
|
|
// executable upgrade). Retain its prior checkpoint without rewriting it.
|
|
checkpoint:
|
|
adoptedRunner && !this.#adoptedRunnerAuthenticated
|
|
? null
|
|
: this.#controlPlaneCheckpoint,
|
|
forceKill: async () => {
|
|
const handle = this.#handle;
|
|
if (!handle || this.#evidence.runnerExited) return;
|
|
handle.child.kill("SIGKILL");
|
|
// A remote kill dispatches an asynchronous, ownership-fenced RPC.
|
|
// Do not release the session for reuse until its process monitor has
|
|
// settled: the successor would otherwise overwrite that ownership
|
|
// marker while the previous runner still holds the fixed listener.
|
|
const result = await waitForProcess(handle, 15_000);
|
|
this.#evidence.runnerExited = true;
|
|
this.#evidence.runnerExitCode = result.code;
|
|
this.#evidence.runnerSignal = result.signal as NodeJS.Signals | null;
|
|
},
|
|
release: this.#controlPlaneRelease,
|
|
retireController: () => this.#stopAndDrainAuthority(closeDeadline),
|
|
});
|
|
} finally {
|
|
await this.#core?.stop();
|
|
this.#controlPlaneCheckpoint = null;
|
|
this.#controlPlaneRelease = null;
|
|
}
|
|
if (suspensionRequired && !runnerSettled) {
|
|
const settlement = {
|
|
runnerSuspended,
|
|
providerDrained,
|
|
suspensionState,
|
|
semanticTools: this.#core?.semanticToolSettlementDiagnostics(),
|
|
finalProviderState,
|
|
};
|
|
try {
|
|
this.options.onDiagnostic?.(`native_session_settlement_incomplete ${JSON.stringify(settlement)}`);
|
|
} catch {
|
|
// Keep the settlement failure authoritative if its observer fails.
|
|
}
|
|
// Sample after ingress retirement and its drain. A result observed before
|
|
// that barrier cannot rule out one last admitted tool callback.
|
|
throw new NativeSessionCloseUnrecoverableError(
|
|
settlement, this.#core?.semanticToolResultsSettled() === true,
|
|
);
|
|
}
|
|
if (!preparationConfirmed) {
|
|
throw new Error(`Native runner ${attachmentPending ? "attachment" : "startup"} cleanup is unconfirmed; recovery state retained.`);
|
|
}
|
|
if (this.#ownsRoot && !adoptedRunner) {
|
|
rmSync(this.#root, { recursive: true, force: true });
|
|
}
|
|
this.#publish();
|
|
}
|
|
|
|
async #prepareOwnedProvider(
|
|
template: Record<string, unknown>, identity: DurableRecoveryIdentity,
|
|
runtimeContext: NativeRuntimeContextSnapshot | null, instructions: string,
|
|
recovery?: PreparedRunnerProviderLaunch,
|
|
recordedRequirements?: RunnerProviderLaunchRequirements,
|
|
): Promise<PreparedRunnerProviderLaunch> {
|
|
const prior = record(template.provider);
|
|
const requirements = recordedRequirements ?? runnerProviderLaunchRequirements(this.options, identity, {
|
|
model: prior.model, approvalPolicy: prior.approvalPolicy,
|
|
permissions: prior.readOnly === true || record(prior.providerPolicy).readOnly === true || prior.collaborationMode === "plan"
|
|
? "paperclip-runner-workspace-read-only" : "paperclip-runner-workspace-only",
|
|
conversationMode: prior.conversationMode,
|
|
config: { include_collaboration_mode_instructions: prior.includeCollaborationModeInstructions !== false },
|
|
}, runtimeContext, instructions);
|
|
if (requirements.schema !== "paperclip.runner.launch-requirements.v1"
|
|
|| commandDigest(requirements.identity) !== commandDigest(identity)) {
|
|
throw new Error("runner_launch_recovery_requirements_unavailable");
|
|
}
|
|
this.#assertStartupOpen();
|
|
const prepared = await this.options.prepareLaunch!(requirements, recovery);
|
|
validatePreparedRunnerLaunch(requirements, prepared);
|
|
this.#recordPreparedWorkingDirectory(prepared);
|
|
this.#assertStartupOpen();
|
|
template.provider = prepared.provider;
|
|
new RunnerLaunchPreparationStore(this.#root).save(requirements, prepared, this.#core!.store.state.identity);
|
|
this.#core!.persistPreparedLaunchConfiguration(identity, prepared.configurationId);
|
|
return prepared;
|
|
}
|
|
|
|
// The prepared launch is the only source of the provider directory. Core
|
|
// records it in session metadata; the session cannot move to another one.
|
|
#recordPreparedWorkingDirectory(prepared: Pick<PreparedRunnerProviderLaunch, "provider">): void {
|
|
const workingDirectory = preparedRunnerLaunchWorkingDirectory(prepared);
|
|
if (workingDirectory === null) {
|
|
throw new Error("runner_prepared_launch_working_directory_unavailable");
|
|
}
|
|
if (this.#preparedWorkingDirectory !== null && this.#preparedWorkingDirectory !== workingDirectory) {
|
|
throw new Error("runner_prepared_launch_working_directory_changed");
|
|
}
|
|
this.#preparedWorkingDirectory = workingDirectory;
|
|
}
|
|
|
|
#preparedThreadWorkingDirectory(): { cwd?: string } {
|
|
if (!this.options.prepareLaunch) return {};
|
|
if (this.#preparedWorkingDirectory === null) {
|
|
throw new Error("runner_prepared_launch_working_directory_unavailable");
|
|
}
|
|
return { cwd: this.#preparedWorkingDirectory };
|
|
}
|
|
|
|
async #prepareLaunch(
|
|
params: Record<string, unknown>, identity: DurableRecoveryIdentity,
|
|
dynamicTools: Record<string, unknown>[],
|
|
) {
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.prepare", () => this.#prepareLaunchWork(params, identity, dynamicTools), {toolCount:dynamicTools.length, toolSchemaBytes:Buffer.byteLength(JSON.stringify(dynamicTools))}) : this.#prepareLaunchWork(params, identity, dynamicTools);
|
|
}
|
|
|
|
async #prepareLaunchWork(
|
|
params: Record<string, unknown>, identity: DurableRecoveryIdentity,
|
|
dynamicTools: Record<string, unknown>[],
|
|
) {
|
|
if (this.options.acpxCandidateAdmission) {
|
|
const candidate = this.options.acpxCandidateAdmission;
|
|
parseAcpxCandidateAdmission(candidate, { companyId: candidate.companyId, agent: this.options.acpxAgent ?? "", model: String(params.model ?? "") });
|
|
}
|
|
if (this.options.prepareLaunch) {
|
|
const provider = this.options.provider ?? "codex";
|
|
this.#authorizedTools = authorizedToolSetForProvider(provider, dynamicTools);
|
|
const requirements = runnerProviderLaunchRequirements(this.options, identity, params,
|
|
this.options.runtimeContext ?? null,
|
|
String(params.developerInstructions ?? params.baseInstructions ?? "You are a Paperclip agent."));
|
|
const prepared = await this.options.prepareLaunch(requirements);
|
|
validatePreparedRunnerLaunch(requirements, prepared);
|
|
this.#recordPreparedWorkingDirectory(prepared);
|
|
this.#assertStartupOpen();
|
|
new RunnerLaunchPreparationStore(this.#root).save(requirements, prepared, this.#core!.store.state.identity);
|
|
this.#core!.persistPreparedLaunchConfiguration(identity, prepared.configurationId);
|
|
const completionContract = record(params.completionContract);
|
|
return {
|
|
runAttachTemplate: { observeCredentials: this.options.observeCredentials !== false,
|
|
authorizedTools: this.#authorizedTools,
|
|
...(completionContract.revision && Array.isArray(completionContract.criterionIds) ? { completionContract } : {}),
|
|
provider: prepared.provider },
|
|
runnerAcpxLaunchProfile: undefined, runnerOpenCodeLaunchProfile: undefined,
|
|
environment: {},
|
|
};
|
|
}
|
|
const provider = this.options.provider ?? "codex";
|
|
const sourceRuntimeContext = this.options.runtimeContext ?? null;
|
|
if (provider === "codex" && this.options.runnerFilesystemRoot && !this.options.codexCommand) {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: remote Codex omitted its qualified guest executable; verify the selected environment's Codex artifacts");
|
|
}
|
|
const runtimeContext =
|
|
this.options.runnerRuntimeContext ?? sourceRuntimeContext;
|
|
const localRuntimeContextPath = resolve(this.#root, "runtime-context.json");
|
|
const runtimeContextPath = this.options.runnerFilesystemRoot
|
|
? resolve(this.options.runnerFilesystemRoot, "runtime-context.json")
|
|
: localRuntimeContextPath;
|
|
if (runtimeContext !== null) {
|
|
writeFileSync(
|
|
localRuntimeContextPath,
|
|
`${JSON.stringify(runtimeContext)}\n`,
|
|
{ mode: 0o600 },
|
|
);
|
|
}
|
|
const localCodexHome = resolve(this.#root, "codex-home");
|
|
const codexHome = this.options.homeMode === "preconfigured"
|
|
? preconfiguredCodexHome(this.options.environment)
|
|
: this.options.runnerFilesystemRoot
|
|
? resolve(this.options.runnerFilesystemRoot, "codex-home")
|
|
: localCodexHome;
|
|
if (provider === "aws_agentcore") {
|
|
mkdirSync(codexHome, { recursive: true, mode: 0o700 });
|
|
}
|
|
if (provider === "codex" && this.options.homeMode !== "preconfigured") {
|
|
await prepareIsolatedCodexHome({
|
|
context: sourceRuntimeContext,
|
|
codexHome: localCodexHome,
|
|
observability: this.options.observability,
|
|
sourceCodexHome:
|
|
this.options.sourceCodexHome ??
|
|
resolveSourceCodexHome(this.options.environment),
|
|
apiKey:
|
|
this.options.environment?.CODEX_API_KEY ??
|
|
this.options.environment?.OPENAI_API_KEY,
|
|
nativeMcp: nativeMcpLaunchBinding(this.options.environment),
|
|
});
|
|
}
|
|
this.#assertStartupOpen();
|
|
const opencodeProxyPath =
|
|
this.options.opencodeProxyPath ??
|
|
(provider === "opencode" && !this.options.runnerFilesystemRoot
|
|
? resolveBuildOwnedCliArtifact("opencode-app-server-proxy.cjs")
|
|
: fileURLToPath(
|
|
new URL("../cli/opencode-app-server-proxy.cjs", import.meta.url),
|
|
));
|
|
const acpxSidecarPath =
|
|
this.options.acpxSidecarPath ??
|
|
(provider === "acpx" && !this.options.runnerFilesystemRoot
|
|
? resolveBuildOwnedCliArtifact("acpx-runtime-sidecar.cjs")
|
|
: fileURLToPath(
|
|
new URL("../cli/acpx-runtime-sidecar.cjs", import.meta.url),
|
|
));
|
|
const providerNodeCommand =
|
|
this.options.providerNodeCommand ?? process.execPath;
|
|
const effectiveCodexCommand = provider === "codex"
|
|
? this.options.codexCommand ?? resolveCodexCommand(undefined, this.options.environment, String(params.cwd ?? tmpdir()))
|
|
: undefined;
|
|
const opencodeExecutable =
|
|
provider === "opencode"
|
|
? resolveRunnerOpenCodeExecutable(this.options)
|
|
: null;
|
|
const runnerAcpxLaunchProfile =
|
|
provider === "acpx"
|
|
? acpxRunnerLaunchProfile(
|
|
this.options,
|
|
providerNodeCommand,
|
|
acpxSidecarPath,
|
|
identity,
|
|
)
|
|
: undefined;
|
|
const runnerOpenCodeLaunchProfile =
|
|
provider === "opencode"
|
|
? opencodeRunnerLaunchProfile(
|
|
this.options,
|
|
providerNodeCommand,
|
|
opencodeProxyPath,
|
|
opencodeExecutable!,
|
|
)
|
|
: undefined;
|
|
if (
|
|
this.options.runnerFilesystemRoot &&
|
|
(provider === "opencode" || provider === "acpx")
|
|
) {
|
|
const providerPaths = [
|
|
["provider Node", providerNodeCommand],
|
|
...(provider === "opencode"
|
|
? [["OpenCode proxy", opencodeProxyPath], ["OpenCode executable", opencodeExecutable!]] as const
|
|
: [["ACPX sidecar", acpxSidecarPath]] as const),
|
|
] as const;
|
|
for (const [label, candidate] of providerPaths) {
|
|
if (
|
|
candidate.startsWith("/Users/") ||
|
|
/^[A-Za-z]:\\\\Users\\\\/.test(candidate)
|
|
) {
|
|
throw new Error(
|
|
`runner_remote_provider_artifact_incompatible: ${label} path belongs to the controller host`,
|
|
);
|
|
}
|
|
}
|
|
if (!this.options.providerNodeCommand) {
|
|
throw new Error(
|
|
"runner_remote_provider_artifact_incompatible: remote JS provider omitted its provider-pack Node executable",
|
|
);
|
|
}
|
|
if (provider === "opencode" && !this.options.opencodeProxyPath) {
|
|
throw new Error(
|
|
"runner_remote_provider_artifact_incompatible: remote OpenCode omitted its packaged proxy",
|
|
);
|
|
}
|
|
if (provider === "acpx" && !this.options.acpxSidecarPath) {
|
|
throw new Error(
|
|
"runner_remote_provider_artifact_incompatible: remote ACPX omitted its packaged sidecar",
|
|
);
|
|
}
|
|
}
|
|
this.#authorizedTools = authorizedToolSetForProvider(
|
|
provider,
|
|
dynamicTools,
|
|
);
|
|
const acpxAgent =
|
|
provider === "acpx" ? (this.options.acpxAgent ?? "codex") : null;
|
|
const unboundBaseInstructions = String(
|
|
params.developerInstructions ?? params.baseInstructions ?? "You are a Paperclip agent.",
|
|
);
|
|
const baseInstructions =
|
|
sourceRuntimeContext && runtimeContext
|
|
? unboundBaseInstructions.replaceAll(
|
|
sourceRuntimeContext.instructions.bundle.rootPath,
|
|
runtimeContext.instructions.bundle.rootPath,
|
|
)
|
|
: unboundBaseInstructions;
|
|
const completionContract = record(params.completionContract);
|
|
const runAttachTemplate = {
|
|
observeCredentials: this.options.observeCredentials !== false,
|
|
authorizedTools: this.#authorizedTools,
|
|
...(completionContract.revision && Array.isArray(completionContract.criterionIds)
|
|
? { completionContract } : {}),
|
|
provider: createRunnerProviderLaunch(runnerProviderLaunchRequirements(
|
|
this.options, identity, params, runtimeContext, baseInstructions,
|
|
), {
|
|
options: this.options, root: this.#root, cwd: String(params.cwd ?? tmpdir()),
|
|
codexHome, providerNodeCommand, codexCommand: effectiveCodexCommand,
|
|
opencodeProxyPath, acpxSidecarPath,
|
|
}),
|
|
};
|
|
return { runAttachTemplate, runnerAcpxLaunchProfile, runnerOpenCodeLaunchProfile,
|
|
environment: withRunnerdProviderTrace(
|
|
createCapabilityRunnerdProviderEnvironment({
|
|
provider,
|
|
options: {
|
|
...this.options,
|
|
stateDirectory: this.#root,
|
|
acpxAgent: acpxAgent ?? undefined,
|
|
},
|
|
identity,
|
|
codexHome,
|
|
runtimeContextPath,
|
|
hasRuntimeContext: runtimeContext !== null,
|
|
acpxSidecarPath,
|
|
}),
|
|
this.options.environment,
|
|
),
|
|
};
|
|
|
|
}
|
|
|
|
async #start(
|
|
params: Record<string, unknown>,
|
|
): Promise<Record<string, unknown>> {
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.cold_start", () => this.#startWork(params)) : this.#startWork(params);
|
|
}
|
|
|
|
async #startWork(
|
|
params: Record<string, unknown>,
|
|
): Promise<Record<string, unknown>> {
|
|
if (this.#core !== null)
|
|
throw new Error("PRP provider thread is already started");
|
|
if (this.options.adoptExistingRunner) {
|
|
// A crash can precede the first driver checkpoint even though runnerd
|
|
// already opened the provider. Exact process adoption must reuse that
|
|
// authority instead of enqueueing another run.prepare/session.open pair.
|
|
await this.#resume();
|
|
return this.#openedThreadResponse(params);
|
|
}
|
|
const token = randomUUID().replaceAll("-", "");
|
|
const identity = this.options.prpIdentity ?? {
|
|
runnerInstanceId: `runner_lab_${token}`,
|
|
environmentLeaseId: `lease_lab_${token}`,
|
|
runId: `run_lab_${token}`,
|
|
normalizedSessionId: `session_lab_${token}`,
|
|
turnId: `turn_lab_${token}`,
|
|
itemId: `item_lab_${token}`,
|
|
};
|
|
const runnerBinaryPath =
|
|
this.options.runnerBinary ?? (this.options.prepareLaunch ? "paperclip-runnerd" : defaultCapabilityRunnerdBinary());
|
|
const runnerArtifact = this.options.runnerArtifact ?? approvedRunnerArtifact(runnerBinaryPath);
|
|
this.#durableTurnId = identity.turnId;
|
|
this.#eventIdentity = structuredClone(identity);
|
|
const dynamicTools = Array.isArray(params.dynamicTools)
|
|
? params.dynamicTools.map(record)
|
|
: [];
|
|
const core = new DurablePrpControlPlane({
|
|
stateDirectory: resolve(this.#root, "control-plane"),
|
|
identity,
|
|
expectedRunnerVersion: runnerArtifact.version,
|
|
expectedRunnerDigest: runnerArtifact.digest,
|
|
onProtocolIntegrityError: (error) => this.#failTransport(error),
|
|
onSemanticToolInput: (call) => this.#handleSemanticToolInput(call),
|
|
recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call),
|
|
onCommittedEvent: async (event) => {
|
|
await this.options.onAccountingEvent?.(event);
|
|
if (event.eventType === "auth.credentials.updated") {
|
|
await this.options.onCredentialsUpdated?.(event);
|
|
}
|
|
},
|
|
connectionLeaseTtlMs: 60 * 60 * 1_000,
|
|
});
|
|
this.#core = core;
|
|
// Externally launched runners own their state directory (for example in a
|
|
// Daytona sandbox). Do not create an empty controller-side placeholder:
|
|
// prior-run authority checks must be able to distinguish absent remote
|
|
// state from malformed direct state.
|
|
if (this.options.runnerStateDirectory === undefined && !this.options.prepareLaunch) {
|
|
mkdirSync(resolve(this.#root, "runner"), {
|
|
recursive: true,
|
|
mode: 0o700,
|
|
});
|
|
}
|
|
const prepared = await this.#prepareLaunch(params, identity, dynamicTools);
|
|
const { runAttachTemplate } = prepared;
|
|
// Preserve the first generation's provider attachment seed independently
|
|
// of bounded command history. The in-memory copy serves a live warm
|
|
// continuation; the control-plane copy serves a controller/runner resume.
|
|
this.#runAttachTemplate = structuredClone(runAttachTemplate);
|
|
core.persistRunAttachTemplate(runAttachTemplate);
|
|
core.queueCommand("run.prepare", runAttachTemplate);
|
|
core.queueCommand("session.open", { reuse: "same_session" });
|
|
const registration = this.options.controlPlaneRegistration
|
|
? await this.options.controlPlaneRegistration(core)
|
|
: null;
|
|
this.#startupFailureCode =
|
|
registration?.startupFailureCode ?? "runner_local_connect_failed";
|
|
if (registration === null) await core.start();
|
|
else {
|
|
this.#controlPlaneCheckpoint = registration.checkpoint ?? null;
|
|
this.#controlPlaneRelease = registration.release;
|
|
}
|
|
if (this.#closed && this.#startupDrainTimedOut) {
|
|
const release = this.#controlPlaneRelease;
|
|
this.#controlPlaneRelease = null;
|
|
await release?.();
|
|
}
|
|
this.#assertStartupOpen();
|
|
const admissionDeadline = this.#coldAdmissionDeadline();
|
|
const bootstrapTicket = core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS);
|
|
await core.flushCheckpoint();
|
|
this.#assertStartupOpen();
|
|
const handle = spawnRunner({
|
|
connection: registration?.connection ?? {
|
|
mode: "connect",
|
|
connectUrl: registration?.connectUrl ?? core.connectUrl,
|
|
},
|
|
stateDirectory:
|
|
this.options.runnerStateDirectory ?? resolve(this.#root, "runner"),
|
|
identity,
|
|
ticket: bootstrapTicket,
|
|
maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES,
|
|
p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES,
|
|
maxRuntimeMs: 0,
|
|
...(this.options.acpxCandidateAdmission ? { maxLifetimeMs: Math.max(1, this.options.acpxCandidateAdmission.expiresAt - Date.now()) } : {}),
|
|
reconnectGraceMs: runnerProcessReconnectGraceMs(this.options),
|
|
lifecyclePolicy: this.options.lifecyclePolicy,
|
|
runnerBinaryPath,
|
|
runnerVersion: runnerArtifact.version,
|
|
runnerDigest: runnerArtifact.digest,
|
|
acpxLaunchProfile: prepared.runnerAcpxLaunchProfile,
|
|
opencodeLaunchProfile: prepared.runnerOpenCodeLaunchProfile,
|
|
environment: prepared.environment,
|
|
diagnosticsDirectory: resolve(this.#root, "diagnostics"),
|
|
processLauncher: this.options.runnerProcessLauncher,
|
|
});
|
|
this.#handle = handle;
|
|
this.#watchRunner(handle);
|
|
await this.#publishSpawnedProcess(handle);
|
|
this.#assertStartupOpen();
|
|
await registration?.activate?.();
|
|
this.#assertStartupOpen();
|
|
if (registration?.failure) {
|
|
void registration.failure.catch((error: unknown) => {
|
|
this.#failTransport(
|
|
error instanceof Error ? error : new Error(String(error)),
|
|
);
|
|
});
|
|
}
|
|
await this.#awaitRegistrationReady(registration?.ready);
|
|
this.#assertStartupOpen();
|
|
this.#evidence.runnerPid = handle.child.pid ?? null;
|
|
this.#evidence.runnerProcessGroupId = handle.processGroupId ?? null;
|
|
this.#publish();
|
|
this.#pump = setInterval(() => this.#pumpEventsSafely(), 5);
|
|
await this.#waitCommand("run.prepare", undefined, undefined, true);
|
|
await this.#waitCommand("session.open", undefined, admissionDeadline, true);
|
|
await this.#waitForProviderIdentity(undefined, admissionDeadline);
|
|
this.#startupComplete = true;
|
|
this.#diagnostic("runnerd authenticated to the durable PRP control plane");
|
|
return this.#openedThreadResponse(params);
|
|
|
|
}
|
|
|
|
#openedThreadResponse(params: Record<string, unknown>): Record<string, unknown> {
|
|
const provider = this.options.provider ?? "codex";
|
|
const acpxAgent = this.options.acpxAgent ?? "codex";
|
|
return {
|
|
model: this.#observedProviderModel,
|
|
...this.#preparedThreadWorkingDirectory(),
|
|
thread: {
|
|
id: this.#threadId,
|
|
sessionId: this.#sessionId,
|
|
...(this.#providerIdentity === null
|
|
? {}
|
|
: { providerIdentity: structuredClone(this.#providerIdentity) }),
|
|
model: params.model,
|
|
modelProvider: openedThreadModelProvider(provider, acpxAgent, params.model),
|
|
},
|
|
};
|
|
}
|
|
|
|
async #prepareCandidateConversationResume(params: Record<string, unknown>): Promise<void> {
|
|
const candidate = this.options.acpxCandidateAdmission;
|
|
const prior = this.options.resumeProviderSession;
|
|
if (!candidate || !prior || this.#core || this.#startupOperation
|
|
|| params.threadId !== prior.driverSessionId || params.model !== candidate.model
|
|
|| typeof params.cwd !== "string" || params.cwd !== this.options.resumeWorkingDirectory
|
|
|| existsSync(resolve(this.#root, "control-plane", "control-plane-state.json"))
|
|
|| existsSync(resolve(this.#root, "runner", "runner-state.json"))) {
|
|
throw new Error("ACPX candidate continuation requires a fresh daemon and the exact current conversation selection");
|
|
}
|
|
parseAcpxCandidateAdmission(candidate, { companyId: candidate.companyId, agent: "antigravity", model: String(params.model) });
|
|
this.#checkpointProviderIdentityExpectation = {
|
|
driverSessionId: prior.driverSessionId, providerSessionId: prior.providerSessionId!,
|
|
providerIdentity: structuredClone(record(prior.providerIdentity)),
|
|
};
|
|
await this.#trackStartup(() => this.#start(params));
|
|
if (!this.#checkpointProviderIdentityConfirmed) throw new Error("ACPX candidate continuation has no verified native conversation receipt");
|
|
}
|
|
|
|
async #resume(): Promise<void> {
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.resume", () => this.#resumeWork()) : this.#resumeWork();
|
|
}
|
|
|
|
async #resumeWork(): Promise<void> {
|
|
if (this.options.acpxCandidateAdmission) throw new Error("ACPX candidate durable runner recovery is not admitted");
|
|
const desiredIdentity = this.options.prpIdentity;
|
|
if (desiredIdentity === undefined) {
|
|
throw new Error("PRP provider resume state is unavailable");
|
|
}
|
|
const controlPlaneDirectory = resolve(this.#root, "control-plane");
|
|
const controlPlaneStatePath = resolve(
|
|
controlPlaneDirectory,
|
|
"control-plane-state.json",
|
|
);
|
|
const localProvider =
|
|
this.options.provider === undefined ||
|
|
this.options.provider === "codex" ||
|
|
this.options.provider === "opencode" ||
|
|
this.options.provider === "acpx";
|
|
const localStateOwner =
|
|
this.options.readRunnerState === undefined &&
|
|
this.options.runnerStateDirectory === undefined &&
|
|
this.options.runnerFilesystemRoot === undefined && !this.options.prepareLaunch;
|
|
const localRunnerStatePath = resolve(
|
|
this.#root,
|
|
"runner",
|
|
"runner-state.json",
|
|
);
|
|
let candidateRunnerState =
|
|
localStateOwner && existsSync(localRunnerStatePath)
|
|
? readRunnerState(localRunnerStatePath)
|
|
: null;
|
|
const hasRunnerWarmBoundary =
|
|
candidateRunnerState?.warmTransition !== undefined ||
|
|
candidateRunnerState?.schema ===
|
|
"paperclip.runner.durable.state.warm-transition.v1";
|
|
let controlPlaneState: Record<string, unknown> | null;
|
|
try {
|
|
controlPlaneState = existsSync(controlPlaneStatePath)
|
|
? readControlPlaneState(controlPlaneDirectory)
|
|
: null;
|
|
} catch (error) {
|
|
if (localProvider && localStateOwner && !hasRunnerWarmBoundary) {
|
|
quarantineLocalRuntimeState(this.#root, error);
|
|
}
|
|
throw error;
|
|
}
|
|
let identity = controlPlaneState
|
|
? controlPlaneIdentity(controlPlaneState)
|
|
: desiredIdentity;
|
|
if (
|
|
this.options.readRunnerState &&
|
|
controlPlaneState &&
|
|
(controlPlaneState.warmTransition !== undefined ||
|
|
controlPlaneState.completedWarmTransition !== undefined ||
|
|
controlPlaneState.schema ===
|
|
"paperclip.runner.durable.control-plane-state.warm-transition.v1" ||
|
|
(Array.isArray(controlPlaneState.commands) &&
|
|
controlPlaneState.commands.some((entry) => {
|
|
const command = record(entry);
|
|
return (
|
|
command.type === "run.attach" &&
|
|
command.status === "pending" &&
|
|
record(command.payload).paperclipNextAuthority !== undefined
|
|
);
|
|
})))
|
|
) {
|
|
candidateRunnerState = await this.options.readRunnerState();
|
|
}
|
|
let warmRecovery: {
|
|
runnerState: Record<string, unknown>;
|
|
runnerIdentity: DurableRecoveryIdentity;
|
|
transitionId: string;
|
|
commandId: string;
|
|
proof: NonNullable<ReturnType<typeof inspectWarmRunTransition>>;
|
|
port?: number;
|
|
} | null = null;
|
|
const hasWarmBoundary =
|
|
controlPlaneState?.warmTransition !== undefined ||
|
|
controlPlaneState?.schema ===
|
|
"paperclip.runner.durable.control-plane-state.warm-transition.v1" ||
|
|
hasRunnerWarmBoundary ||
|
|
candidateRunnerState?.warmTransition !== undefined ||
|
|
candidateRunnerState?.schema ===
|
|
"paperclip.runner.durable.state.warm-transition.v1";
|
|
if (hasWarmBoundary) {
|
|
// This lane must precede ordinary archive/identity-rebinding recovery.
|
|
// No malformed or unsupported transition is reinterpreted as legacy.
|
|
if (
|
|
(!localStateOwner && !this.options.readRunnerState) ||
|
|
!localProvider ||
|
|
!controlPlaneState ||
|
|
!candidateRunnerState ||
|
|
(this.options.controlPlaneRegistration !== undefined &&
|
|
this.options.warmTransitionRegistrationMode !== "routed_connect") ||
|
|
this.options.adoptExistingRunner !== undefined
|
|
) {
|
|
throw new Error(
|
|
"native_runner_warm_transition_requires_exact_owned_endpoint",
|
|
);
|
|
}
|
|
const artifact = this.options.runnerArtifact ?? approvedRunnerArtifact(
|
|
this.options.runnerBinary ?? defaultCapabilityRunnerdBinary(),
|
|
);
|
|
const proof = inspectWarmRunTransition({
|
|
controlPlaneState,
|
|
runnerState: candidateRunnerState,
|
|
expectedNewIdentity: desiredIdentity,
|
|
expectedRunnerVersion: artifact.version,
|
|
expectedRunnerDigest: artifact.digest,
|
|
});
|
|
if (!proof)
|
|
throw new Error("native_runner_warm_transition_snapshot_mismatch");
|
|
const receipt = proof.receipt;
|
|
const connection = receipt.connection;
|
|
const endpoint =
|
|
typeof connection.connectUrl === "string"
|
|
? new URL(connection.connectUrl)
|
|
: null;
|
|
if (
|
|
connection.mode !== "connect" ||
|
|
endpoint === null ||
|
|
endpoint.search ||
|
|
endpoint.hash ||
|
|
endpoint.username ||
|
|
endpoint.password ||
|
|
(!this.options.controlPlaneRegistration &&
|
|
(endpoint.protocol !== "ws:" ||
|
|
endpoint.hostname !== "127.0.0.1" ||
|
|
endpoint.pathname !== "/durableRecovery/connect" ||
|
|
!endpoint.port))
|
|
) {
|
|
throw new Error("native_runner_warm_transition_snapshot_mismatch");
|
|
}
|
|
warmRecovery = {
|
|
runnerState: candidateRunnerState,
|
|
runnerIdentity: proof.runnerIdentity,
|
|
proof,
|
|
transitionId: receipt.transitionId,
|
|
commandId: receipt.commandId,
|
|
...(!this.options.controlPlaneRegistration
|
|
? { port: Number(endpoint.port) }
|
|
: {}),
|
|
};
|
|
// Fence concurrent public work before exposing the core or awaiting
|
|
// route/materialization hooks, not merely after activation finishes.
|
|
this.#pendingWarmRecoveryCompletion = {
|
|
transitionId: receipt.transitionId,
|
|
identity: structuredClone(desiredIdentity),
|
|
};
|
|
}
|
|
const exactAuthority =
|
|
controlPlaneState !== null &&
|
|
identity.runnerInstanceId === desiredIdentity.runnerInstanceId &&
|
|
identity.environmentLeaseId === desiredIdentity.environmentLeaseId &&
|
|
identity.runId === desiredIdentity.runId &&
|
|
identity.normalizedSessionId === desiredIdentity.normalizedSessionId &&
|
|
identity.turnId === desiredIdentity.turnId &&
|
|
identity.itemId === desiredIdentity.itemId;
|
|
let rotatedAuthority = false;
|
|
if (controlPlaneState === null) {
|
|
if (!localProvider) {
|
|
throw new Error("PRP provider resume state is unavailable");
|
|
}
|
|
const archivedState = latestArchivedControlPlaneState(
|
|
this.#root,
|
|
desiredIdentity,
|
|
);
|
|
if (!archivedState) {
|
|
throw new Error("PRP provider resume state is unavailable");
|
|
}
|
|
if (localStateOwner) {
|
|
try {
|
|
controlPlaneState = rotateLocalAuthorityEpoch(
|
|
this.#root,
|
|
archivedState,
|
|
desiredIdentity,
|
|
);
|
|
} catch (error) {
|
|
quarantineLocalRuntimeState(this.#root, error);
|
|
}
|
|
} else {
|
|
if (
|
|
this.options.readRunnerState === undefined ||
|
|
this.options.archiveExternalRunnerState === undefined
|
|
) {
|
|
throw new Error("native_runner_prp_run_rotation_unavailable");
|
|
}
|
|
controlPlaneState = await rotateExternalAuthorityEpoch(
|
|
this.#root,
|
|
archivedState,
|
|
desiredIdentity,
|
|
this.options.readRunnerState,
|
|
this.options.archiveExternalRunnerState,
|
|
);
|
|
}
|
|
identity = desiredIdentity;
|
|
rotatedAuthority = true;
|
|
} else if (!exactAuthority && warmRecovery === null) {
|
|
if (!localProvider || controlPlaneState === null) {
|
|
throw new Error("native_runner_prp_run_rotation_unavailable");
|
|
}
|
|
if (localStateOwner) {
|
|
try {
|
|
controlPlaneState = rotateLocalAuthorityEpoch(
|
|
this.#root,
|
|
controlPlaneState,
|
|
desiredIdentity,
|
|
);
|
|
} catch (error) {
|
|
quarantineLocalRuntimeState(this.#root, error);
|
|
}
|
|
} else {
|
|
if (
|
|
this.options.readRunnerState === undefined ||
|
|
this.options.prepareExternalRunnerState === undefined ||
|
|
this.options.archiveExternalRunnerState === undefined
|
|
) {
|
|
throw new Error("native_runner_prp_run_rotation_unavailable");
|
|
}
|
|
await this.options.prepareExternalRunnerState();
|
|
controlPlaneState = await rotateExternalAuthorityEpoch(
|
|
this.#root,
|
|
controlPlaneState,
|
|
desiredIdentity,
|
|
this.options.readRunnerState,
|
|
this.options.archiveExternalRunnerState,
|
|
);
|
|
}
|
|
identity = desiredIdentity;
|
|
rotatedAuthority = true;
|
|
} else if (
|
|
localStateOwner &&
|
|
!existsSync(resolve(this.#root, "runner", "runner-state.json"))
|
|
) {
|
|
quarantineLocalRuntimeState(
|
|
this.#root,
|
|
new Error("PRP provider resume state is unavailable"),
|
|
);
|
|
}
|
|
const runnerBinaryPath =
|
|
this.options.runnerBinary ?? (this.options.prepareLaunch ? "paperclip-runnerd" : defaultCapabilityRunnerdBinary());
|
|
const runnerArtifact = this.options.runnerArtifact ?? approvedRunnerArtifact(runnerBinaryPath);
|
|
this.#durableTurnId = identity.turnId;
|
|
this.#eventIdentity = structuredClone(identity);
|
|
const provider = this.options.provider ?? "codex";
|
|
// Adoption authenticates the original artifact and process. Launch profiles
|
|
// belong only to a new spawn and may reference a retired controller build.
|
|
const preparesProviderLaunch = !this.options.prepareLaunch && !this.options.adoptExistingRunner;
|
|
if (!this.options.prepareLaunch && provider === "codex" && (this.options.homeMode === "preconfigured" || record(record(controlPlaneState!.runAttachTemplate).provider).preconfiguredHome !== undefined)) {
|
|
const seed = rotatedRunAttachPayload(controlPlaneState!, desiredIdentity, this.#authorizedTools, this.options.resumeCompletionContract);
|
|
const priorHome = record(seed.provider).preconfiguredHome;
|
|
const expectedHome = this.options.homeMode === "preconfigured"
|
|
? { HOME: this.options.environment!.HOME, CODEX_HOME: preconfiguredCodexHome(this.options.environment) } : undefined;
|
|
if (JSON.stringify(priorHome) !== JSON.stringify(expectedHome)) throw new Error("runner_preconfigured_home_binding_changed");
|
|
}
|
|
const sourceRuntimeContext = this.options.runtimeContext ?? null;
|
|
const runtimeContext =
|
|
this.options.runnerRuntimeContext ?? sourceRuntimeContext;
|
|
const localRuntimeContextPath = resolve(this.#root, "runtime-context.json");
|
|
const runtimeContextPath = this.options.runnerFilesystemRoot
|
|
? resolve(this.options.runnerFilesystemRoot, "runtime-context.json")
|
|
: localRuntimeContextPath;
|
|
const localCodexHome = resolve(this.#root, "codex-home");
|
|
const codexHome = this.options.homeMode === "preconfigured"
|
|
? preconfiguredCodexHome(this.options.environment)
|
|
: this.options.runnerFilesystemRoot
|
|
? resolve(this.options.runnerFilesystemRoot, "codex-home")
|
|
: localCodexHome;
|
|
const opencodeProxyPath =
|
|
(preparesProviderLaunch ? undefined : "") ?? this.options.opencodeProxyPath ??
|
|
(provider === "opencode" && !this.options.runnerFilesystemRoot
|
|
? resolveBuildOwnedCliArtifact("opencode-app-server-proxy.cjs")
|
|
: fileURLToPath(
|
|
new URL("../cli/opencode-app-server-proxy.cjs", import.meta.url),
|
|
));
|
|
const acpxSidecarPath =
|
|
(preparesProviderLaunch ? undefined : "") ?? this.options.acpxSidecarPath ??
|
|
(provider === "acpx" && !this.options.runnerFilesystemRoot
|
|
? resolveBuildOwnedCliArtifact("acpx-runtime-sidecar.cjs")
|
|
: fileURLToPath(
|
|
new URL("../cli/acpx-runtime-sidecar.cjs", import.meta.url),
|
|
));
|
|
const providerNodeCommand =
|
|
this.options.providerNodeCommand ?? process.execPath;
|
|
const opencodeExecutable =
|
|
provider === "opencode" && preparesProviderLaunch
|
|
? resolveRunnerOpenCodeExecutable(this.options)
|
|
: null;
|
|
const runnerAcpxLaunchProfile =
|
|
provider === "acpx" && preparesProviderLaunch
|
|
? acpxRunnerLaunchProfile(
|
|
this.options,
|
|
providerNodeCommand,
|
|
acpxSidecarPath,
|
|
identity,
|
|
)
|
|
: undefined;
|
|
const runnerOpenCodeLaunchProfile =
|
|
provider === "opencode" && preparesProviderLaunch
|
|
? opencodeRunnerLaunchProfile(
|
|
this.options,
|
|
providerNodeCommand,
|
|
opencodeProxyPath,
|
|
opencodeExecutable!,
|
|
)
|
|
: undefined;
|
|
const core = new DurablePrpControlPlane({
|
|
stateDirectory: controlPlaneDirectory,
|
|
identity,
|
|
expectedRunnerVersion: runnerArtifact.version,
|
|
expectedRunnerDigest: runnerArtifact.digest,
|
|
onProtocolIntegrityError: (error) => this.#failTransport(error),
|
|
onSemanticToolInput: (call) => this.#handleSemanticToolInput(call),
|
|
recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call),
|
|
onCommittedEvent: async (event) => {
|
|
await this.options.onAccountingEvent?.(event);
|
|
if (event.eventType === "auth.credentials.updated") {
|
|
await this.options.onCredentialsUpdated?.(event);
|
|
}
|
|
},
|
|
...(warmRecovery
|
|
? {
|
|
beforeAuthenticatedConnection: async (admission) => {
|
|
// Core policy already validated the current lease and tuple.
|
|
// Receipt replay still needs the recovery claim, including the
|
|
// completed-core/lost-final-ACK window. Ordinary post-ACK lease
|
|
// reconnects no longer depend on that historical claim.
|
|
if (
|
|
core.store.state.warmTransition?.receipt.transitionId ===
|
|
warmRecovery.transitionId ||
|
|
admission.warmTransitionId === warmRecovery.transitionId
|
|
) {
|
|
await this.options.authorizeWarmTransitionRecovery?.(
|
|
"before_authentication",
|
|
);
|
|
}
|
|
},
|
|
}
|
|
: {}),
|
|
connectionLeaseTtlMs: 60 * 60 * 1_000,
|
|
});
|
|
this.#core = core;
|
|
// Reusing resources still requires current environment authorization. The
|
|
// optional hook avoids re-preparation after controller storage is lost.
|
|
// Existing consumers retain the preparation callback's authorization path.
|
|
const authorizeAdoption = this.options.adoptExistingRunner && this.options.authorizeRunnerAdoption;
|
|
if (this.options.prepareLaunch && authorizeAdoption) {
|
|
const configurationId = core.preparedLaunchConfigurationId(desiredIdentity);
|
|
if (!configurationId) throw new Error("runner_launch_recovery_configuration_unavailable");
|
|
await authorizeAdoption(configurationId, structuredClone(desiredIdentity));
|
|
this.#assertStartupOpen();
|
|
this.#recordPreparedWorkingDirectory({ provider: record(record(controlPlaneState!.runAttachTemplate).provider) });
|
|
}
|
|
let environmentPrepared: PreparedRunnerProviderLaunch | undefined;
|
|
if (this.options.prepareLaunch && !authorizeAdoption) {
|
|
const preparations = new RunnerLaunchPreparationStore(this.#root);
|
|
const saved = preparations.load(desiredIdentity);
|
|
const previous = preparations.load(controlPlaneIdentity(controlPlaneState!));
|
|
if ((exactAuthority || warmRecovery !== null) && !saved) {
|
|
throw new Error("runner_launch_recovery_requirements_unavailable");
|
|
}
|
|
const { paperclipNextAuthority: _authority, ...savedAttachment } =
|
|
warmRecovery ? core.getCommand(warmRecovery.commandId)!.payload : {};
|
|
const template = warmRecovery ? structuredClone(savedAttachment)
|
|
: rotatedRunAttachPayload(controlPlaneState!, desiredIdentity, this.#authorizedTools,
|
|
this.options.resumeCompletionContract);
|
|
if (saved) {
|
|
const recordedProvider = warmRecovery
|
|
? core.getCommand(warmRecovery.commandId)?.payload.provider
|
|
: exactAuthority ? record(controlPlaneState!.runAttachTemplate).provider : undefined;
|
|
if ((exactAuthority || warmRecovery !== null)
|
|
&& (recordedProvider === undefined || commandDigest(recordedProvider) !== commandDigest(saved.prepared.provider))) {
|
|
throw new Error("runner_prepared_launch_recovery_changed");
|
|
}
|
|
template.provider = saved.prepared.provider;
|
|
}
|
|
const preparedContext = this.options.runtimeContext === undefined
|
|
? previous?.requirements.runtimeContext ?? null : this.options.runtimeContext;
|
|
const preparedInstructions = this.options.baseInstructions
|
|
?? (previous?.requirements.runtimeContext
|
|
? retargetComposedInstructions(previous.requirements.instructions, previous.requirements.runtimeContext, preparedContext)
|
|
: previous?.requirements.instructions ?? String(record(template.provider).instructions ?? ""));
|
|
environmentPrepared = await this.#prepareOwnedProvider(template, desiredIdentity,
|
|
preparedContext, preparedInstructions, saved?.prepared, saved?.requirements);
|
|
this.#runAttachTemplate = structuredClone(template);
|
|
}
|
|
if (rotatedAuthority) {
|
|
if (provider === "acpx" && this.options.acpxAgent === "copilot"
|
|
&& typeof this.options.baseInstructions !== "string") {
|
|
throw new Error("native_runner_current_instruction_grant_unavailable");
|
|
}
|
|
const runAttachTemplate = rotatedRunAttachPayload(
|
|
controlPlaneState,
|
|
desiredIdentity,
|
|
this.#authorizedTools,
|
|
this.options.resumeCompletionContract,
|
|
runtimeContext,
|
|
this.options.baseInstructions === undefined
|
|
? undefined
|
|
: { text: this.options.baseInstructions, context: sourceRuntimeContext },
|
|
);
|
|
if (!this.options.prepareLaunch && provider === "codex") {
|
|
// These controller-owned, token-free paths belong to the new run.
|
|
// Keep the durable provider profile and thread identity unchanged.
|
|
const recordedCommand = record(runAttachTemplate.provider).command;
|
|
runAttachTemplate.runtimeLaunchArgs =
|
|
createRunnerdCodexAppServerArgs({
|
|
args: this.options.codexArgs,
|
|
disableWebSearch: this.options.disableCodexWebSearch,
|
|
environment: this.options.environment,
|
|
codexHome,
|
|
workingDirectory: this.options.resumeWorkingDirectory,
|
|
runtimeStateRoot: this.options.runnerFilesystemRoot ?? this.#root,
|
|
codexCommand: typeof recordedCommand === "string" ? recordedCommand : this.options.codexCommand,
|
|
instructionWorkingCopyRoot: runtimeContext?.instructions.workingCopy?.rootPath,
|
|
disableProjectInstructions: this.options.disableCodexProjectInstructions,
|
|
readOnlyRoots: [
|
|
...trustedRuntimeReadOnlyRoots(this.options.environment),
|
|
...(runtimeContext
|
|
? [
|
|
resolve(codexHome, "skills"),
|
|
runtimeContext.instructions.bundle.rootPath,
|
|
...runtimeContext.skills.map(
|
|
(skill) => skill.bundle.rootPath,
|
|
),
|
|
]
|
|
: []),
|
|
],
|
|
});
|
|
}
|
|
if (environmentPrepared) {
|
|
runAttachTemplate.provider = environmentPrepared.provider;
|
|
}
|
|
this.#runAttachTemplate = structuredClone(runAttachTemplate);
|
|
core.persistRunAttachTemplate(runAttachTemplate);
|
|
core.queueCommand("run.attach", runAttachTemplate);
|
|
}
|
|
const committedEvents = core.store.state.committedEvents;
|
|
const runAttachment = recoveredRunAttachment(core.store.state);
|
|
// Reconnecting the exact run authority has no run.attach command to wake
|
|
// provider restoration. Queue a unique, side-effect-free barrier before
|
|
// runnerd starts so every provider backend restores its durable session.
|
|
const recoveryProbeCommandId =
|
|
warmRecovery === null && exactAuthority && runAttachment === null
|
|
? `command_resume_probe_${randomUUID().replaceAll("-", "")}`
|
|
: null;
|
|
if (recoveryProbeCommandId !== null) {
|
|
core.queueCommand("runner.drain", {}, recoveryProbeCommandId);
|
|
}
|
|
// A controller retry can open the exact authority after run.attach has
|
|
// already reached a durable outcome. Re-observe that command instead of
|
|
// silently waiting for an identity that a failed command can never emit.
|
|
// If attachment completed, replay only its latest identity event into the
|
|
// transport's in-memory evidence; session events are consumed internally
|
|
// and are not duplicated onto the provider notification stream.
|
|
this.#eventSourceSeq =
|
|
runAttachment !== null && runAttachment.providerIdentityEventIndex >= 0
|
|
? committedEvents[runAttachment.providerIdentityEventIndex]!.sourceSeq -
|
|
1
|
|
: core.store.state.ackedSourceSeq;
|
|
const adoptedProviderIdentityIndex =
|
|
latestProviderIdentityEventIndex(committedEvents);
|
|
if (
|
|
this.options.adoptExistingRunner &&
|
|
exactAuthority &&
|
|
adoptedProviderIdentityIndex >= 0
|
|
) {
|
|
this.#applyProviderIdentityEvent(
|
|
committedEvents[adoptedProviderIdentityIndex]!,
|
|
);
|
|
} else if (
|
|
(exactAuthority || warmRecovery !== null) &&
|
|
this.options.resumeProviderSession?.driverSessionId.trim() &&
|
|
this.options.resumeProviderSession.providerSessionId?.trim()
|
|
) {
|
|
this.#threadId = this.options.resumeProviderSession.driverSessionId;
|
|
this.#sessionId = this.options.resumeProviderSession.providerSessionId;
|
|
if (this.options.resumeProviderSession.providerIdentity !== undefined) {
|
|
this.#providerIdentity = record(
|
|
structuredClone(this.options.resumeProviderSession.providerIdentity),
|
|
);
|
|
}
|
|
this.#checkpointProviderIdentityExpectation = {
|
|
driverSessionId: this.#threadId,
|
|
providerSessionId: this.#sessionId,
|
|
providerIdentity:
|
|
this.#providerIdentity === null
|
|
? null
|
|
: structuredClone(this.#providerIdentity),
|
|
};
|
|
this.#diagnostic(
|
|
this.options.adoptExistingRunner && adoptedProviderIdentityIndex < 0
|
|
? "restored adopted provider identity from the exact durable checkpoint after PRP event compaction; awaiting live confirmation"
|
|
: "restored provider identity from the exact durable checkpoint; awaiting live confirmation",
|
|
);
|
|
}
|
|
type RecoveryRegistration = Awaited<
|
|
ReturnType<
|
|
NonNullable<
|
|
CapabilityRunnerdCodexTransportOptions["controlPlaneRegistration"]
|
|
>
|
|
>
|
|
>;
|
|
let oldTransitionRegistration: RecoveryRegistration | null = null;
|
|
let newTransitionRegistration: RecoveryRegistration | null = null;
|
|
this.#assertStartupOpen();
|
|
if (warmRecovery && this.options.controlPlaneRegistration) {
|
|
try {
|
|
oldTransitionRegistration = await this.options.controlPlaneRegistration(
|
|
core,
|
|
warmRecovery.proof.receipt.oldIdentity,
|
|
);
|
|
const oldConnection = oldTransitionRegistration.connection ?? {
|
|
mode: "connect",
|
|
connectUrl: oldTransitionRegistration.connectUrl,
|
|
};
|
|
if (
|
|
oldConnection.mode !== "connect" ||
|
|
typeof oldConnection.connectUrl !== "string"
|
|
) {
|
|
throw new Error(
|
|
"native_runner_warm_transition_registered_endpoint_mismatch",
|
|
);
|
|
}
|
|
this.#assertStartupOpen();
|
|
newTransitionRegistration = await this.options.controlPlaneRegistration(
|
|
core,
|
|
warmRecovery.proof.receipt.newIdentity,
|
|
);
|
|
const newConnection = newTransitionRegistration.connection ?? {
|
|
mode: "connect",
|
|
connectUrl: newTransitionRegistration.connectUrl,
|
|
};
|
|
if (
|
|
newConnection.mode !== "connect" ||
|
|
durableRecoveryInternals.canonicalJson(newConnection) !==
|
|
durableRecoveryInternals.canonicalJson(
|
|
warmRecovery.proof.receipt.connection,
|
|
)
|
|
) {
|
|
throw new Error(
|
|
"native_runner_warm_transition_registered_endpoint_mismatch",
|
|
);
|
|
}
|
|
} catch (error) {
|
|
await Promise.allSettled(
|
|
[oldTransitionRegistration, newTransitionRegistration].map((entry) =>
|
|
Promise.resolve().then(() => entry?.release()),
|
|
),
|
|
);
|
|
throw error;
|
|
}
|
|
}
|
|
const registration =
|
|
warmRecovery && oldTransitionRegistration && newTransitionRegistration
|
|
? recoveryIdentityMatches(
|
|
warmRecovery.runnerIdentity,
|
|
warmRecovery.proof.receipt.oldIdentity,
|
|
)
|
|
? oldTransitionRegistration
|
|
: newTransitionRegistration
|
|
: this.options.controlPlaneRegistration
|
|
? await this.options.controlPlaneRegistration(core)
|
|
: null;
|
|
this.#startupFailureCode =
|
|
registration?.startupFailureCode ?? "runner_local_connect_failed";
|
|
if (registration === null) await core.start(warmRecovery?.port);
|
|
else {
|
|
this.#controlPlaneCheckpoint = registration.checkpoint ?? null;
|
|
this.#controlPlaneRelease =
|
|
oldTransitionRegistration && newTransitionRegistration
|
|
? async () => {
|
|
await Promise.all([
|
|
oldTransitionRegistration!.release(),
|
|
newTransitionRegistration!.release(),
|
|
]);
|
|
}
|
|
: registration.release;
|
|
}
|
|
if (this.#closed && this.#startupDrainTimedOut) {
|
|
const release = this.#controlPlaneRelease;
|
|
this.#controlPlaneRelease = null;
|
|
await release?.();
|
|
}
|
|
this.#assertStartupOpen();
|
|
// Route and immutable endpoint admission precedes every launch-material
|
|
// write. A refused transition leaves its retained home/context untouched.
|
|
if (runtimeContext !== null && !this.options.prepareLaunch) {
|
|
writeFileSync(
|
|
localRuntimeContextPath,
|
|
`${JSON.stringify(runtimeContext)}\n`,
|
|
{ mode: 0o600 },
|
|
);
|
|
}
|
|
if (provider === "aws_agentcore" && !this.options.prepareLaunch)
|
|
mkdirSync(codexHome, { recursive: true, mode: 0o700 });
|
|
if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured") {
|
|
await releaseMaterializedNativeRuntimeSkills(
|
|
resolve(localCodexHome, "skills"),
|
|
);
|
|
this.#assertStartupOpen();
|
|
await prepareIsolatedCodexHome({
|
|
context: sourceRuntimeContext,
|
|
codexHome: localCodexHome,
|
|
observability: this.options.observability,
|
|
sourceCodexHome:
|
|
this.options.sourceCodexHome ??
|
|
resolveSourceCodexHome(this.options.environment),
|
|
apiKey:
|
|
this.options.environment?.CODEX_API_KEY ??
|
|
this.options.environment?.OPENAI_API_KEY,
|
|
nativeMcp: nativeMcpLaunchBinding(this.options.environment),
|
|
});
|
|
}
|
|
const adoptedRunner = this.options.adoptExistingRunner;
|
|
if (warmRecovery) {
|
|
await this.options.authorizeWarmTransitionRecovery?.("before_bootstrap");
|
|
}
|
|
this.#assertStartupOpen();
|
|
const bootstrapTicket = adoptedRunner
|
|
? null
|
|
: warmRecovery
|
|
? core.issueWarmTransitionBootstrapTicket(
|
|
{
|
|
transitionId: warmRecovery.transitionId,
|
|
runnerState: warmRecovery.runnerState,
|
|
},
|
|
RUNNER_BOOTSTRAP_TICKET_TTL_MS,
|
|
)
|
|
: core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS);
|
|
await core.flushCheckpoint();
|
|
if (warmRecovery) {
|
|
await this.options.authorizeWarmTransitionRecovery?.("before_spawn");
|
|
}
|
|
// A replacement executor restores its cold provider before acknowledging recovery.
|
|
this.#assertStartupOpen();
|
|
const admissionDeadline = this.#coldAdmissionDeadline(adoptedRunner === undefined);
|
|
const handle = adoptedRunner
|
|
? null
|
|
: spawnRunner({
|
|
connection: registration?.connection ?? {
|
|
mode: "connect",
|
|
connectUrl: registration?.connectUrl ?? core.connectUrl,
|
|
},
|
|
stateDirectory:
|
|
this.options.runnerStateDirectory ?? resolve(this.#root, "runner"),
|
|
identity: warmRecovery?.runnerIdentity ?? identity,
|
|
ticket: bootstrapTicket!,
|
|
maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES,
|
|
p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES,
|
|
maxRuntimeMs: 0,
|
|
reconnectGraceMs: runnerProcessReconnectGraceMs(this.options),
|
|
lifecyclePolicy: this.options.lifecyclePolicy,
|
|
runnerBinaryPath,
|
|
runnerVersion: runnerArtifact.version,
|
|
runnerDigest: runnerArtifact.digest,
|
|
acpxLaunchProfile: runnerAcpxLaunchProfile,
|
|
opencodeLaunchProfile: runnerOpenCodeLaunchProfile,
|
|
environment: this.options.prepareLaunch ? {} : withRunnerdProviderTrace(
|
|
createCapabilityRunnerdProviderEnvironment({
|
|
provider,
|
|
options: {
|
|
...this.options,
|
|
stateDirectory: this.#root,
|
|
},
|
|
identity,
|
|
codexHome,
|
|
runtimeContextPath,
|
|
hasRuntimeContext: runtimeContext !== null,
|
|
acpxSidecarPath,
|
|
}),
|
|
this.options.environment,
|
|
),
|
|
diagnosticsDirectory: resolve(this.#root, "diagnostics"),
|
|
processLauncher: this.options.runnerProcessLauncher,
|
|
});
|
|
if (handle) {
|
|
this.#handle = handle;
|
|
this.#watchRunner(handle);
|
|
await this.#publishSpawnedProcess(handle);
|
|
}
|
|
this.#assertStartupOpen();
|
|
if (oldTransitionRegistration && newTransitionRegistration) {
|
|
await oldTransitionRegistration.activate?.();
|
|
this.#assertStartupOpen();
|
|
await newTransitionRegistration.activate?.();
|
|
} else await registration?.activate?.();
|
|
this.#assertStartupOpen();
|
|
if (registration?.failure) {
|
|
void registration.failure.catch((error: unknown) => {
|
|
this.#failTransport(
|
|
error instanceof Error ? error : new Error(String(error)),
|
|
);
|
|
});
|
|
}
|
|
if (!adoptedRunner) await this.#awaitRegistrationReady(registration?.ready);
|
|
this.#assertStartupOpen();
|
|
if (adoptedRunner) {
|
|
this.#evidence.runnerPid = adoptedRunner.pid;
|
|
this.#evidence.runnerProcessGroupId = adoptedRunner.processGroupId;
|
|
this.#watchAdoptedRunner(adoptedRunner);
|
|
} else {
|
|
this.#evidence.runnerPid = handle?.child.pid ?? null;
|
|
this.#evidence.runnerProcessGroupId = handle?.processGroupId ?? null;
|
|
}
|
|
this.#publish();
|
|
if (warmRecovery) {
|
|
const deadline =
|
|
Date.now() + (this.options.runnerReconnectGraceMs ?? 5_000);
|
|
while (
|
|
!recoveryIdentityMatches(core.store.state.identity, desiredIdentity) ||
|
|
core.store.state.warmTransition !== undefined
|
|
) {
|
|
this.#assertStartupOpen();
|
|
if (Date.now() >= deadline || (await this.#runnerHasExited())) {
|
|
throw new Error("native_runner_warm_transition_recovery_pending");
|
|
}
|
|
await new Promise<void>((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
if (core.getCommand(warmRecovery.commandId)?.status !== "completed") {
|
|
throw new Error("native_runner_warm_transition_result_unproven");
|
|
}
|
|
this.#durableTurnId = desiredIdentity.turnId;
|
|
this.#eventIdentity = structuredClone(desiredIdentity);
|
|
this.#eventSourceSeq = 0;
|
|
this.#deferredTurnStartEvents = [];
|
|
if (oldTransitionRegistration && newTransitionRegistration) {
|
|
await oldTransitionRegistration.release();
|
|
this.#assertStartupOpen();
|
|
this.#controlPlaneRelease = newTransitionRegistration.release;
|
|
this.#controlPlaneCheckpoint =
|
|
newTransitionRegistration.checkpoint ?? null;
|
|
}
|
|
}
|
|
this.#assertStartupOpen();
|
|
this.#pump = setInterval(() => this.#pumpEventsSafely(), 5);
|
|
if (adoptedRunner) {
|
|
await this.#awaitAdoptedRunnerConnection(
|
|
adoptedRunner,
|
|
registration?.ready,
|
|
);
|
|
}
|
|
// Only a replacement executor can be restoring a cold Pi process. Share
|
|
// one deadline across every recovery barrier; live adoption stays at 30 s.
|
|
if (runAttachment) {
|
|
await this.#waitCommand("run.attach", runAttachment.commandId, admissionDeadline, true);
|
|
}
|
|
if (recoveryProbeCommandId !== null) {
|
|
await this.#waitCommand("runner.drain", recoveryProbeCommandId, admissionDeadline, true);
|
|
if (this.#checkpointProviderIdentityExpectation !== null) {
|
|
// A replacement runner can restore the exact provider while its fresh
|
|
// session.resumed event is compacted or delayed behind the completed
|
|
// recovery barrier. Confirm the live session directly instead of
|
|
// waiting only on the bounded event replay. The authenticated command
|
|
// result is still checked against the exact database checkpoint, so a
|
|
// missing or changed provider identity continues to fail closed.
|
|
const snapshot = await this.#commandResult("session.snapshot", {}, admissionDeadline, true);
|
|
this.#confirmCheckpointProviderIdentity(
|
|
snapshot,
|
|
"authenticated recovery session.snapshot",
|
|
);
|
|
}
|
|
}
|
|
// A relaunched executor proves its restored provider with either its fresh
|
|
// resume identity or the authenticated snapshot above. An adopted live
|
|
// executor keeps the already-verified provider session, so its
|
|
// authenticated drain plus the committed identity are the corresponding
|
|
// continuity proof.
|
|
await this.#waitForProviderIdentity(
|
|
recoveryProbeCommandId !== null &&
|
|
adoptedRunner === undefined &&
|
|
!this.#checkpointProviderIdentityConfirmed
|
|
? "session.resumed"
|
|
: undefined,
|
|
admissionDeadline,
|
|
);
|
|
this.#startupComplete = true;
|
|
this.#diagnostic(
|
|
rotatedAuthority
|
|
? "runnerd attached the durable provider session to a fresh PRP run authority"
|
|
: "runnerd restored its durable PRP session and provider thread",
|
|
);
|
|
this.#recoverPendingSemanticInputsWhenReady();
|
|
|
|
}
|
|
|
|
async #recoverSemanticToolInput(call: Parameters<NonNullable<DurablePrpControlPlaneOptions["onSemanticToolInput"]>>[0]) {
|
|
if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") {
|
|
return this.#handleSemanticToolInput(call);
|
|
}
|
|
const outcome = await this.options.recoverSemanticToolInput?.(call);
|
|
return outcome ? this.#bindSemanticToolOutcome(call, outcome) : null;
|
|
}
|
|
|
|
async #handleSemanticToolInput(
|
|
call: Parameters<
|
|
NonNullable<DurablePrpControlPlaneOptions["onSemanticToolInput"]>
|
|
>[0],
|
|
) {
|
|
try {
|
|
this.#throwIfFailed();
|
|
} catch {
|
|
throw new SemanticToolNotDispatchedError();
|
|
}
|
|
const core = this.#core;
|
|
const threadId = this.#threadId;
|
|
const epoch = this.#turnStartResponseEpoch;
|
|
const admission = this.#turnStartAdmission;
|
|
// This callback runs independently of the notification pump. Do not copy
|
|
// its provisional turn_lab identity into a provider request before the
|
|
// exact durable command result and matching turn/started bind that turn.
|
|
const accepted =
|
|
admission === null
|
|
? true
|
|
: await Promise.race([admission.settled, this.#failureSignal]).catch(() => {
|
|
throw new SemanticToolNotDispatchedError();
|
|
});
|
|
// An adopted Runner can authenticate and deliver queued tools before the
|
|
// driver has finished reconstructing its session. Handler publication is
|
|
// the dispatch barrier; the default RPC diagnostic is not a tool outcome.
|
|
// The owning startup/transport lifetime bounds this wait, and the original
|
|
// authority/epoch checks below still apply after it becomes ready.
|
|
if (!this.#handlerInstalled) {
|
|
await Promise.race([
|
|
this.#handlerReady,
|
|
this.#failureSignal,
|
|
this.#closedSignal.then(() => { throw new SemanticToolNotDispatchedError(); }),
|
|
]).catch(() => { throw new SemanticToolNotDispatchedError(); });
|
|
}
|
|
try {
|
|
this.#throwIfFailed();
|
|
} catch {
|
|
throw new SemanticToolNotDispatchedError();
|
|
}
|
|
if (
|
|
!accepted ||
|
|
this.#closed ||
|
|
epoch !== this.#turnStartResponseEpoch ||
|
|
threadId !== this.#threadId ||
|
|
core === null ||
|
|
core !== this.#core ||
|
|
call.correlation.runId !== core.store.state.identity.runId ||
|
|
call.correlation.normalizedSessionId !==
|
|
core.store.state.identity.normalizedSessionId ||
|
|
call.correlation.turnId !== core.store.state.identity.turnId
|
|
) {
|
|
throw new SemanticToolNotDispatchedError();
|
|
}
|
|
const outcome = unwrapToolResponse(
|
|
await this.#handler({
|
|
id: call.callId,
|
|
method: "item/tool/call",
|
|
params: {
|
|
threadId,
|
|
turnId: this.#turnId,
|
|
callId: call.callId,
|
|
tool: call.operationId,
|
|
arguments: call.input,
|
|
},
|
|
...(call.sourceEventId && call.sourceEventType
|
|
? {
|
|
paperclipTrace: {
|
|
sourceEventId: call.sourceEventId,
|
|
sourceEventType: call.sourceEventType,
|
|
},
|
|
}
|
|
: {}),
|
|
}),
|
|
this.options.provider === "acpx" &&
|
|
(call.operationId === "paperclip_finish" || call.operationId === "paperclip_block"),
|
|
);
|
|
return this.#bindSemanticToolOutcome(call, outcome);
|
|
}
|
|
|
|
#bindSemanticToolOutcome(call: { operationId: string; callId: string }, outcome: { result: unknown; isError?: boolean }) {
|
|
if (call.operationId === "call_api") {
|
|
const result = record(outcome.result);
|
|
if (
|
|
result.operationId !== call.operationId ||
|
|
result.callId !== call.callId
|
|
) {
|
|
return {
|
|
...outcome,
|
|
result: {
|
|
ok: !outcome.isError,
|
|
operationId: call.operationId,
|
|
callId: call.callId,
|
|
...(outcome.isError
|
|
? { error: outcome.result }
|
|
: { result: outcome.result }),
|
|
},
|
|
};
|
|
}
|
|
}
|
|
return outcome;
|
|
}
|
|
|
|
async #startTurn(
|
|
params: Record<string, unknown>,
|
|
): Promise<Record<string, unknown>> {
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.turn_start", () => this.#startTurnWork(params), {inputBytes:Buffer.byteLength(JSON.stringify(params)), tokenDataAvailable:false}) : this.#startTurnWork(params);
|
|
}
|
|
|
|
async #startTurnWork(
|
|
params: Record<string, unknown>,
|
|
): Promise<Record<string, unknown>> {
|
|
const turnStartTimeoutMs = this.options.turnStartTimeoutMs ?? 30_000;
|
|
if (!Number.isSafeInteger(turnStartTimeoutMs) || turnStartTimeoutMs <= 0) {
|
|
throw new Error("turnStartTimeoutMs must be a positive safe integer");
|
|
}
|
|
const commandDeadline = Date.now() + turnStartTimeoutMs;
|
|
const input = Array.isArray(params.input) ? params.input.map(record) : [];
|
|
const message = input
|
|
.filter((item) => item.type !== "skill")
|
|
.map((item) => (typeof item.text === "string" ? item.text : ""))
|
|
.join("\n");
|
|
const reasoningMode = params.reasoningMode;
|
|
if (reasoningMode !== undefined && (
|
|
!this.supportsTurnReasoning()
|
|
|| (reasoningMode !== "default" && reasoningMode !== "disabled")
|
|
)) throw new Error("turn.reasoningMode requires an OpenCode/OpenRouter model and must be default or disabled");
|
|
const skills = resolveRunnerdCodexSkillInputs(
|
|
input.filter((item) => item.type === "skill"),
|
|
this.options.runtimeContext ?? null,
|
|
this.options.homeMode === "preconfigured" ? preconfiguredCodexHome(this.options.environment) : resolve(this.options.runnerFilesystemRoot ?? this.#root, "codex-home"),
|
|
this.options.prepareLaunch
|
|
? record((this.#runAttachTemplate ?? this.#core?.store.state.runAttachTemplate)?.provider).runtimeContext as NativeRuntimeContextSnapshot | undefined
|
|
: undefined,
|
|
);
|
|
if (skills.length && (this.options.provider ?? "codex") !== "codex") {
|
|
throw new Error("Explicit skill inputs are supported only by Codex");
|
|
}
|
|
const pendingTurnId = `turn_lab_${randomUUID().replaceAll("-", "")}`;
|
|
this.#turnId = pendingTurnId;
|
|
const responseEpoch = ++this.#turnStartResponseEpoch;
|
|
this.#turnStartAdmission?.resolve(false);
|
|
let resolveAdmission!: (accepted: boolean) => void;
|
|
this.#turnStartAdmission = {
|
|
settled: new Promise((resolve) => {
|
|
resolveAdmission = resolve;
|
|
}),
|
|
resolve: (accepted) => resolveAdmission(accepted),
|
|
};
|
|
this.#turnStartResponsePending = true;
|
|
let releaseStartResponse!: () => void;
|
|
this.#turnStartResponseSettled = new Promise<void>(resolve => { releaseStartResponse = resolve; });
|
|
this.#expectedProviderTurnId = null;
|
|
let responseReady = false;
|
|
try {
|
|
// Persist a fresh requested identity with the durable command. ACPX uses
|
|
// it as its provider request identity, so a same-run recovery turn cannot
|
|
// alias an already-settled request from the retained provider session.
|
|
// Codex and OpenCode continue to return their provider-assigned identity.
|
|
const commandStarted = this.options.recordTimingReceipt ? performance.now() : 0;
|
|
const startResult = await this.#commandResult(
|
|
"turn.start",
|
|
{
|
|
text: message,
|
|
...(this.options.recordTimingReceipt ? {performance:true} : {}),
|
|
...((this.options.provider ?? "codex") === "codex" && params.effort !== undefined ? { effort: params.effort } : {}),
|
|
...(reasoningMode === undefined ? {} : { reasoningMode }),
|
|
...(skills.length ? { skills } : {}),
|
|
turnId: pendingTurnId,
|
|
},
|
|
commandDeadline,
|
|
);
|
|
try { this.options.recordTimingReceipt?.({...record(startResult.timing), roundTripMs:performance.now()-commandStarted}); } catch {}
|
|
const expectedProviderTurnId =
|
|
typeof startResult.providerTurnId === "string" &&
|
|
startResult.providerTurnId.length > 0
|
|
? startResult.providerTurnId
|
|
: null;
|
|
if (expectedProviderTurnId === null) {
|
|
const error = new Error(
|
|
"runnerd turn.start omitted its provider turn identity",
|
|
);
|
|
this.#failTransport(error);
|
|
throw error;
|
|
}
|
|
if (
|
|
!turnStartCommandResultValid({
|
|
requestedTurnId: pendingTurnId,
|
|
providerTurnId: expectedProviderTurnId,
|
|
requireRequestedIdentity: this.options.provider === "acpx",
|
|
})
|
|
) {
|
|
const error = new Error(
|
|
"runnerd ACPX turn.start changed its requested provider turn identity",
|
|
);
|
|
this.#failTransport(error);
|
|
throw error;
|
|
}
|
|
this.#expectedProviderTurnId = expectedProviderTurnId;
|
|
// Command completion only means runnerd accepted the command. Bind the
|
|
// provider turn from the subsequent turn/started event before answering
|
|
// the strict driver. Correlate that event with the exact identity in this
|
|
// command's durable result so a delayed prior-turn event cannot satisfy
|
|
// the new response fence. ACPX echoes the requested identity, while
|
|
// Codex and OpenCode return their provider-assigned identity.
|
|
const deadline =
|
|
this.options.turnStartTimeoutMs === undefined
|
|
? Date.now() + 30_000
|
|
: commandDeadline;
|
|
const providerTurnStarted = () =>
|
|
turnStartResponseReady({
|
|
responseEpoch,
|
|
observedEpoch: this.#observedTurnStartEpoch,
|
|
expectedProviderTurnId,
|
|
boundTurnId: this.#turnId,
|
|
});
|
|
const waitForStarted = async () => {
|
|
while (!providerTurnStarted() && Date.now() < deadline) {
|
|
this.#throwIfFailed();
|
|
this.#pumpEvents();
|
|
if (providerTurnStarted()) break;
|
|
if (await this.#runnerHasExited())
|
|
throw new Error("runnerd exited before provider turn startup");
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
if (!providerTurnStarted())
|
|
throw new Error("runnerd did not report the provider turn identity");
|
|
};
|
|
if (this.options.measureOperation) await this.options.measureOperation("runner.transport.started_event", waitForStarted);
|
|
else await waitForStarted();
|
|
responseReady = true;
|
|
return { turn: { id: this.#turnId, status: "inProgress" } };
|
|
} finally {
|
|
if (!responseReady) {
|
|
resolveAdmission(false);
|
|
if (this.#turnStartResponseEpoch === responseEpoch) {
|
|
this.#turnStartResponsePending = false;
|
|
this.#expectedProviderTurnId = null;
|
|
}
|
|
} else {
|
|
// Resolving this async method schedules the strict driver's response
|
|
// continuation as a microtask. Keep terminal frames held until the
|
|
// following task so the driver can bind and emit turn.accepted first.
|
|
// The epoch prevents a late release from clearing a newer turn fence.
|
|
const release = setTimeout(() => {
|
|
resolveAdmission(
|
|
!this.#closed && this.#turnStartResponseEpoch === responseEpoch,
|
|
);
|
|
if (this.#turnStartResponseEpoch !== responseEpoch) return;
|
|
this.#turnStartResponsePending = false;
|
|
this.#expectedProviderTurnId = null;
|
|
if (!this.#closed) this.#pumpEventsSafely();
|
|
}, 0);
|
|
release.unref();
|
|
}
|
|
}
|
|
|
|
}
|
|
|
|
async #command(
|
|
type: string,
|
|
payload: Record<string, unknown>,
|
|
correlationId?: string,
|
|
): Promise<void> {
|
|
const core = this.#core;
|
|
if (core === null) throw new Error("PRP provider thread is not started");
|
|
const commandId = correlationId
|
|
? `command_steer_${createHash("sha256")
|
|
.update(`${this.#durableTurnId}:${correlationId}`)
|
|
.digest("hex")
|
|
.slice(0, 32)}`
|
|
: `command_lab_${randomUUID().replaceAll("-", "")}`;
|
|
const existing = core.store.state.commands.find(
|
|
(command) => command.commandId === commandId,
|
|
);
|
|
if (existing) {
|
|
if (
|
|
existing.type !== type ||
|
|
durableRecoveryInternals.canonicalJson(existing.payload) !==
|
|
durableRecoveryInternals.canonicalJson(payload)
|
|
) {
|
|
throw new Error(
|
|
`PRP steering correlation ${correlationId} was reused with different content`,
|
|
);
|
|
}
|
|
} else {
|
|
core.queueCommand(type, payload, commandId, true);
|
|
}
|
|
await this.#waitCommand(type, commandId);
|
|
}
|
|
|
|
async #commandResult(
|
|
type: string,
|
|
payload: Record<string, unknown>,
|
|
deadline?: number,
|
|
abortOnClose = false,
|
|
): Promise<Record<string, unknown>> {
|
|
const core = this.#core;
|
|
if (core === null) throw new Error("PRP provider thread is not started");
|
|
const commandId = `command_lab_${randomUUID().replaceAll("-", "")}`;
|
|
core.queueCommand(type, payload, commandId, true);
|
|
await this.#waitCommand(type, commandId, deadline, abortOnClose);
|
|
const command = core.getCommand(commandId);
|
|
if (command?.status !== "completed" || command.type !== type) {
|
|
throw new Error(`PRP command ${type} omitted its durable result`);
|
|
}
|
|
const result = record(record(command.result).result);
|
|
const completion = this.#pendingWarmRecoveryCompletion;
|
|
if (completion && type === "session.snapshot") {
|
|
const expectation = this.#checkpointProviderIdentityExpectation;
|
|
const providerIdentity = resolveRunnerdSessionIdentity(result);
|
|
if (
|
|
command.type !== "session.snapshot" ||
|
|
core.store.state.warmTransition !== undefined ||
|
|
!recoveryIdentityMatches(
|
|
core.store.state.identity,
|
|
completion.identity,
|
|
) ||
|
|
core.store.state.completedWarmTransition?.receipt.transitionId !==
|
|
completion.transitionId ||
|
|
expectation === null ||
|
|
providerIdentity.threadId !== expectation.driverSessionId ||
|
|
providerIdentity.sessionId !== expectation.providerSessionId ||
|
|
!["prepared", "session_open", "turn_active"].includes(
|
|
String(result.status),
|
|
)
|
|
) {
|
|
throw new Error("native_runner_warm_transition_completion_unproven");
|
|
}
|
|
this.#confirmCheckpointProviderIdentity(
|
|
result,
|
|
"fresh post-activation session.snapshot",
|
|
);
|
|
// This newly queued command completed only after runner consumed the
|
|
// final activation ACK. Callback failure retains the completion gate;
|
|
// retry observes a fresh snapshot, never repeats a provider turn.
|
|
const completing = (this.#warmRecoveryCompletionInFlight ??=
|
|
Promise.resolve().then(() =>
|
|
this.options.onWarmTransitionRecoveryCompleted?.({
|
|
transitionId: completion.transitionId,
|
|
}),
|
|
));
|
|
try {
|
|
await completing;
|
|
if (this.#pendingWarmRecoveryCompletion === completion) {
|
|
this.#pendingWarmRecoveryCompletion = null;
|
|
}
|
|
} finally {
|
|
if (this.#warmRecoveryCompletionInFlight === completing) {
|
|
this.#warmRecoveryCompletionInFlight = null;
|
|
}
|
|
}
|
|
}
|
|
return result;
|
|
}
|
|
|
|
#coldAdmissionDeadline(coldProcess = true): number | undefined {
|
|
const timeout = coldAdmissionTimeoutMs(this.options.provider, this.options.acpxAgent, coldProcess);
|
|
// Preserve the existing independent waits for other providers and live adoption.
|
|
return timeout === 60_000 ? Date.now() + timeout : undefined;
|
|
}
|
|
|
|
async #waitForProviderIdentity(
|
|
expectedEventType?: "harness.ready" | "session.started" | "session.resumed",
|
|
deadline = Date.now() + 30_000,
|
|
): Promise<void> {
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.provider_identity", () => this.#waitForProviderIdentityWork(expectedEventType, deadline)) : this.#waitForProviderIdentityWork(expectedEventType, deadline);
|
|
}
|
|
|
|
async #waitForProviderIdentityWork(
|
|
expectedEventType?: "harness.ready" | "session.started" | "session.resumed",
|
|
deadline = Date.now() + 30_000,
|
|
): Promise<void> {
|
|
while (Date.now() < deadline) {
|
|
this.#throwIfFailed();
|
|
if (this.#closed) throw new Error("runnerd transport closed during provider startup");
|
|
this.#pumpEvents();
|
|
if (
|
|
this.#threadId.length > 0 &&
|
|
(this.#checkpointProviderIdentityExpectation !== null ||
|
|
this.#evidence.providerExecutionKind === "remote_service" ||
|
|
this.#evidence.providerPid !== null) &&
|
|
(expectedEventType === undefined ||
|
|
this.#providerIdentityEventType === expectedEventType)
|
|
)
|
|
return;
|
|
if (await this.#runnerHasExited())
|
|
throw new Error("runnerd exited before provider startup");
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
throw new Error("runnerd did not report its provider identity");
|
|
|
|
}
|
|
|
|
async #waitCommand(
|
|
type: string,
|
|
commandId?: string,
|
|
deadline = Date.now() + 30_000,
|
|
abortOnClose = false,
|
|
): Promise<void> {
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.command_ack", () => this.#waitCommandWork(type, commandId, deadline, abortOnClose)) : this.#waitCommandWork(type, commandId, deadline, abortOnClose);
|
|
}
|
|
|
|
async #waitCommandWork(
|
|
type: string,
|
|
commandId?: string,
|
|
deadline = Date.now() + 30_000,
|
|
abortOnClose = false,
|
|
): Promise<void> {
|
|
await waitForRunnerCommand({
|
|
type,
|
|
deadline,
|
|
abortOnClose,
|
|
isClosed: () => this.#closed,
|
|
throwIfFailed: () => this.#throwIfFailed(),
|
|
command: () => commandId === undefined
|
|
? this.#core?.store.state.commands.find((candidate) => candidate.type === type)
|
|
: this.#core?.getCommand(commandId),
|
|
runnerHasExited: () => this.#runnerHasExited(),
|
|
failureCode: () => this.#startupComplete ? "provider_transport_failed" : this.#startupFailureCode,
|
|
});
|
|
|
|
}
|
|
|
|
|
|
#pumpEvents(): void {
|
|
const core = this.#core;
|
|
// The controller may activate its new epoch before attachRun observes the
|
|
// confirmed handoff. Never consume either epoch with the other's cursor.
|
|
if (
|
|
!core ||
|
|
!this.#eventIdentity ||
|
|
!recoveryIdentityMatches(core.store.state.identity, this.#eventIdentity)
|
|
)
|
|
return;
|
|
this.#flushPendingTraceRehydrations();
|
|
const events = core.store.state.committedEvents;
|
|
for (;;) {
|
|
const deferredEvent =
|
|
!this.#recoveryTurnBindingPending &&
|
|
(!this.#turnStartResponsePending || this.#expectedProviderTurnId !== null)
|
|
? this.#deferredTurnStartEvents[0]
|
|
: undefined;
|
|
const event =
|
|
deferredEvent ??
|
|
events.find((candidate) => candidate.sourceSeq > this.#eventSourceSeq);
|
|
if (event === undefined) {
|
|
this.#eventSourceSeq = Math.max(this.#eventSourceSeq, core.recoveredSourceSeq);
|
|
return;
|
|
}
|
|
const fromDeferredQueue = deferredEvent !== undefined;
|
|
if (!fromDeferredQueue && event.sourceSeq > this.#eventSourceSeq + 1) {
|
|
this.#eventSourceSeq = Math.max(this.#eventSourceSeq, Math.min(event.sourceSeq - 1, core.recoveredSourceSeq));
|
|
}
|
|
if (!fromDeferredQueue && event.sourceSeq !== this.#eventSourceSeq + 1) {
|
|
throw new Error(
|
|
`PRP provider event window advanced past source sequence ${this.#eventSourceSeq + 1}`,
|
|
);
|
|
}
|
|
if (this.#recoveryTurnBindingPending && ![
|
|
"harness.ready", "session.started", "session.resumed",
|
|
].includes(event.eventType)) {
|
|
// Reconnection can deliver mid-turn items before thread/read obtains
|
|
// the authenticated active provider turn. Retain canonical events,
|
|
// not notifications rehydrated with an empty/stale turn identity.
|
|
// Identity events still advance startup; command results are consumed
|
|
// independently, so session.snapshot cannot deadlock behind this gate.
|
|
if (this.#deferredTurnStartEvents.length >= 4_096) {
|
|
throw new Error("recovery produced too many events before its turn binding");
|
|
}
|
|
this.#eventSourceSeq = event.sourceSeq;
|
|
this.#deferredTurnStartEvents.push(structuredClone(event));
|
|
continue;
|
|
}
|
|
const eventPayload = record(event.envelope.payload).payload;
|
|
const turnStartWhileCommandResultPending =
|
|
this.#turnStartResponsePending &&
|
|
this.#expectedProviderTurnId === null &&
|
|
(event.eventType === "turn.started" ||
|
|
(event.eventType === "provider.event" &&
|
|
unwrapRunnerdProviderNotifications(eventPayload).some(
|
|
(notification) => notification.method === "turn/started",
|
|
)));
|
|
// The durable command result is the only correlation authority for a
|
|
// provider-assigned turn id. Copy an early start and its following
|
|
// events out of the control plane's sliding window until that exact
|
|
// expected identity is installed.
|
|
if (
|
|
!fromDeferredQueue &&
|
|
(turnStartWhileCommandResultPending ||
|
|
(this.#turnStartResponsePending &&
|
|
this.#expectedProviderTurnId === null &&
|
|
this.#deferredTurnStartEvents.length > 0))
|
|
) {
|
|
if (this.#deferredTurnStartEvents.length >= 4_096) {
|
|
throw new Error(
|
|
"turn/start produced too many events before its durable command result",
|
|
);
|
|
}
|
|
this.#eventSourceSeq = event.sourceSeq;
|
|
this.#deferredTurnStartEvents.push(structuredClone(event));
|
|
continue;
|
|
}
|
|
const terminalWhileTurnStartPending =
|
|
this.#turnStartResponsePending &&
|
|
([
|
|
"turn.completed",
|
|
"turn.failed",
|
|
"turn.interrupted",
|
|
"turn.cancelled",
|
|
].includes(event.eventType) ||
|
|
(event.eventType === "provider.event" &&
|
|
unwrapRunnerdProviderNotifications(eventPayload).some(
|
|
(notification) => notification.method === "turn/completed",
|
|
)));
|
|
if (terminalWhileTurnStartPending) {
|
|
if (!fromDeferredQueue) {
|
|
this.#eventSourceSeq = event.sourceSeq;
|
|
this.#deferredTurnStartEvents.push(structuredClone(event));
|
|
}
|
|
return;
|
|
}
|
|
// The control plane retains a sliding committed-event window. Track its
|
|
// durable protocol cursor rather than an array index: once that array is
|
|
// full, new events replace its prefix without increasing its length.
|
|
if (fromDeferredQueue) this.#deferredTurnStartEvents.shift();
|
|
else this.#eventSourceSeq = event.sourceSeq;
|
|
if (event.eventType === "auth.credentials.updated" && this.options.onCredentialsUpdated) continue;
|
|
if (event.eventType === "runtime.activity.recorded") {
|
|
this.#queue.push({ method: "paperclip/runtimeActivityRecorded", params: {
|
|
[RUNNERD_RUNTIME_ACTIVITY]: true,
|
|
runId: event.envelope.runId,
|
|
payload: eventPayload,
|
|
} });
|
|
continue;
|
|
}
|
|
if (
|
|
event.eventType === "harness.ready" ||
|
|
event.eventType === "session.started" ||
|
|
event.eventType === "session.resumed"
|
|
) {
|
|
this.#applyProviderIdentityEvent(event);
|
|
continue;
|
|
}
|
|
if (event.eventType === "session.capabilities.updated" && this.options.provider === "acpx") {
|
|
const capabilities = record(record(event.envelope.payload).payload);
|
|
if (capabilities.turnControls !== undefined) {
|
|
this.#turnControls = parseAcpxTurnControlCapabilities(capabilities.turnControls, this.#evidence.acpxAgent);
|
|
}
|
|
}
|
|
if (event.eventType === "harness.diagnostic") {
|
|
const diagnostic = record(record(event.envelope.payload).payload);
|
|
if (
|
|
diagnostic.providerMethod === "acpx/process" &&
|
|
diagnostic.role === "acp_agent" &&
|
|
typeof diagnostic.pid === "number"
|
|
) {
|
|
this.#evidence.agentPid = diagnostic.pid;
|
|
this.#evidence.agentProcessStartedAt = readLocalProcessStartedAt(
|
|
diagnostic.pid,
|
|
);
|
|
this.#publish();
|
|
}
|
|
}
|
|
if (event.eventType === "runtime_request.created") {
|
|
const request = record(record(event.envelope.payload).payload).request;
|
|
const normalizedRequest = record(request);
|
|
const requestId =
|
|
typeof normalizedRequest.requestId === "string"
|
|
? normalizedRequest.requestId
|
|
: "";
|
|
const origin = record(normalizedRequest.origin);
|
|
const method = typeof origin.method === "string" ? origin.method : "";
|
|
const permission = normalizedRequest.type === "permission" && normalizedRequest.requestKind === "permission_approval"
|
|
&& ["session/request_permission", "mcpServer/elicitation/request"].includes(method);
|
|
const params = permission ? bridgedAcpxPermissionParams(
|
|
event,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
) : bridgedCodexQuestionParams(
|
|
normalizedRequest,
|
|
method,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
);
|
|
if (
|
|
requestId &&
|
|
params &&
|
|
(method === "item/tool/requestUserInput" ||
|
|
method === "tool/requestUserInput" ||
|
|
method === "mcpServer/elicitation/request" ||
|
|
isAcpxCanonicalInputMethod(method) || permission) &&
|
|
!this.#bridgedRuntimeInputs.has(requestId)
|
|
) {
|
|
const binding = {
|
|
permission,
|
|
durableTurnId:
|
|
typeof event.envelope.turnId === "string"
|
|
? event.envelope.turnId
|
|
: this.#durableTurnId,
|
|
};
|
|
this.#bridgedRuntimeInputs.set(requestId, binding);
|
|
const request: CodexRpcServerRequest = {
|
|
id: requestId,
|
|
method,
|
|
params,
|
|
paperclipTrace: {
|
|
sourceEventId: event.sourceEventId,
|
|
sourceEventType: event.eventType,
|
|
},
|
|
};
|
|
const threadId = this.#threadId, turnId = this.#turnId, handler = this.#handler;
|
|
this.#queue.pushControl(request, () => {
|
|
// A detached controller, replaced turn or settled durable request
|
|
// cannot acquire approval authority when a slow consumer resumes.
|
|
if (this.#closed || this.#failure || this.#core !== core || this.#handler !== handler
|
|
|| this.#threadId !== threadId || this.#turnId !== turnId
|
|
|| this.#bridgedRuntimeInputs.get(requestId) !== binding) return;
|
|
try {
|
|
void handler(request).catch((error) => {
|
|
this.#failTransport(error instanceof Error ? error : new Error(String(error)));
|
|
});
|
|
} catch (error) {
|
|
this.#failTransport(error instanceof Error ? error : new Error(String(error)));
|
|
}
|
|
});
|
|
}
|
|
continue;
|
|
}
|
|
if (
|
|
event.eventType === "runtime_request.resolved" ||
|
|
event.eventType === "runtime_request.cancelled" ||
|
|
event.eventType === "runtime_request.expired"
|
|
) {
|
|
const requestId = record(
|
|
record(event.envelope.payload).payload,
|
|
).requestId;
|
|
if (typeof requestId === "string")
|
|
this.#bridgedRuntimeInputs.delete(requestId);
|
|
// Operator resolution/Stop keep their existing acknowledgement path.
|
|
const expiry = runnerdRuntimeRequestExpiryNotification(event.eventType, record(eventPayload), this.#threadId, this.#turnId);
|
|
if (expiry) this.#queue.push({ ...expiry,
|
|
...(this.options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH ? { paperclipTrace: {
|
|
sourceEventId: event.sourceEventId, sourceEventType: event.eventType,
|
|
} } : {}),
|
|
});
|
|
continue;
|
|
}
|
|
const sessionUpdatePayload = record(eventPayload);
|
|
const canonicalMethod = runnerdCanonicalNotificationMethod(
|
|
event.eventType,
|
|
sessionUpdatePayload,
|
|
);
|
|
const notifications =
|
|
isCanonicalProviderEventType(event.eventType) && !canonicalMethod
|
|
? [{ method: "paperclip/canonicalProviderEvent", params: {
|
|
...(this.#threadId ? { threadId: this.#threadId } : {}),
|
|
...(this.#turnId ? { turnId: this.#turnId } : {}),
|
|
eventType: event.eventType, payload: eventPayload,
|
|
itemId: event.envelope.itemId,
|
|
} }]
|
|
: event.eventType === "provider.event"
|
|
? unwrapRunnerdProviderNotifications(eventPayload)
|
|
: canonicalMethod
|
|
? expandRunnerdCanonicalNotifications(canonicalMethod, eventPayload, event.eventType)
|
|
: [];
|
|
for (const payload of notifications) {
|
|
const method = payload.method;
|
|
if (typeof method !== "string") continue;
|
|
const rawParams = record(payload.params);
|
|
const rawTurn = record(rawParams.turn);
|
|
// Correlate starts only with an explicit provider-native identity.
|
|
// The later rehydration fallback may use the durable controller turn,
|
|
// which is not evidence that the provider accepted this command.
|
|
const explicitProviderTurnId =
|
|
method === "turn/started"
|
|
? typeof rawParams.providerTurnId === "string" &&
|
|
rawParams.providerTurnId.length > 0
|
|
? rawParams.providerTurnId
|
|
: typeof rawTurn.id === "string" && rawTurn.id.length > 0
|
|
? rawTurn.id
|
|
: event.eventType === "provider.event" &&
|
|
typeof rawParams.turnId === "string" &&
|
|
rawParams.turnId.length > 0
|
|
? rawParams.turnId
|
|
: null
|
|
: null;
|
|
const params =
|
|
method === "thread/tokenUsage/updated"
|
|
? rehydrateRunnerdUsageNotification(
|
|
rawParams,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
)
|
|
: method === "turn/plan/updated"
|
|
? rehydrateRunnerdPlanNotification(
|
|
rawParams,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
)
|
|
: method === "paperclip/workspaceChange/updated"
|
|
? rehydrateRunnerdWorkspaceChangeNotification(
|
|
rawParams,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
)
|
|
: method === "paperclip/runResult"
|
|
? rehydrateRunnerdResultNotification(
|
|
rawParams,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
typeof event.envelope.itemId === "string"
|
|
? event.envelope.itemId
|
|
: "semantic-result",
|
|
)
|
|
: method === "thread/goal/updated" ||
|
|
method === "thread/goal/cleared"
|
|
? rehydrateRunnerdGoalNotification(
|
|
rawParams,
|
|
this.#threadId,
|
|
method,
|
|
)
|
|
: event.eventType === "item.delta"
|
|
? rehydrateRunnerdDeltaNotification(rawParams, this.#threadId, this.#turnId)
|
|
: event.eventType !== "provider.event" &&
|
|
(method === "item/started" || method === "item/completed")
|
|
? rehydrateRunnerdItemNotification(
|
|
rawParams,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
)
|
|
: event.eventType !== "provider.event" &&
|
|
(method === "turn/started" || method === "turn/completed")
|
|
? rehydrateRunnerdTurnNotification(
|
|
rawParams,
|
|
this.#threadId,
|
|
this.#turnId,
|
|
method,
|
|
)
|
|
: rawParams;
|
|
if (
|
|
params.turnId === undefined &&
|
|
typeof event.envelope.turnId === "string" && event.envelope.turnId.length > 0
|
|
) {
|
|
params.turnId = event.envelope.turnId;
|
|
}
|
|
// Only the canonical start establishes provider turn identity. Other
|
|
// normalized events inherit the durable controller turn from the PRP
|
|
// envelope; allowing one of them to update this binding would replace
|
|
// the provider-native id before its terminal is rehydrated.
|
|
if (method === "turn/started") {
|
|
const providerTurnId = record(params.turn).id ?? params.turnId;
|
|
const disposition = turnStartNotificationDisposition({
|
|
responsePending: this.#turnStartResponsePending,
|
|
expectedProviderTurnId: this.#expectedProviderTurnId,
|
|
observedProviderTurnId: explicitProviderTurnId ?? "",
|
|
});
|
|
if (disposition === "defer") {
|
|
throw new Error(
|
|
"turn/started advanced before its durable command result",
|
|
);
|
|
}
|
|
if (disposition === "reject") {
|
|
const error = new Error(
|
|
"turn/started identity disagreed with its durable command result",
|
|
);
|
|
this.#failTransport(error);
|
|
throw error;
|
|
}
|
|
if (typeof providerTurnId === "string" && providerTurnId.length > 0) {
|
|
this.#turnId = providerTurnId;
|
|
if (this.#turnStartResponsePending) {
|
|
this.#observedTurnStartEpoch = this.#turnStartResponseEpoch;
|
|
}
|
|
}
|
|
}
|
|
this.#queue.push({
|
|
method,
|
|
params,
|
|
...(this.options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH
|
|
? {
|
|
paperclipTrace: {
|
|
sourceEventId: event.sourceEventId,
|
|
sourceEventType: event.eventType,
|
|
},
|
|
}
|
|
: {}),
|
|
});
|
|
}
|
|
if (this.options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH) {
|
|
const pending = {
|
|
sourceEventId: event.sourceEventId,
|
|
eventType: event.eventType,
|
|
visibleNotificationCount: notifications.length,
|
|
};
|
|
const traceResult = appendRunnerdRehydrationTrace(
|
|
this.#traceFrameIndex,
|
|
this.options.environment.PAPERCLIP_PROVIDER_TRACE_PATH,
|
|
pending.sourceEventId,
|
|
pending.eventType,
|
|
pending.visibleNotificationCount,
|
|
this.#nextTraceDebugSequence,
|
|
);
|
|
if (traceResult === "written") {
|
|
this.#nextTraceDebugSequence += 1;
|
|
} else if (
|
|
traceResult === "retry" &&
|
|
this.#pendingTraceRehydrations.length < 4_096
|
|
) {
|
|
this.#pendingTraceRehydrations.push(pending);
|
|
} else if (traceResult === "retry") {
|
|
this.#traceRehydrationSpoolOverflow = true;
|
|
}
|
|
}
|
|
// The strict Codex driver must bind the provider turn from the response
|
|
// before it observes terminal notifications. A fast provider can commit
|
|
// start and terminal events in one durable batch, so stop after exposing
|
|
// turn/started while the request is pending. The regular pump drains the
|
|
// remaining events after the promise resolves.
|
|
if (
|
|
this.#turnStartResponsePending &&
|
|
notifications.some(
|
|
(notification) => notification.method === "turn/started",
|
|
)
|
|
) {
|
|
return;
|
|
}
|
|
}
|
|
}
|
|
|
|
#applyProviderIdentityEvent(event: DurableRecoveryCommittedEvent): void {
|
|
if (
|
|
event.eventType !== "harness.ready" &&
|
|
event.eventType !== "session.started" &&
|
|
event.eventType !== "session.resumed"
|
|
) {
|
|
return;
|
|
}
|
|
this.#providerIdentityEventType = event.eventType;
|
|
const started = record(record(event.envelope.payload).payload);
|
|
const runtimeIdentity = record(started.runtimeIdentity);
|
|
const descriptor = record(started.providerDescriptor);
|
|
const {
|
|
processId: pid,
|
|
threadId,
|
|
sessionId,
|
|
} = resolveRunnerdSessionIdentity(started);
|
|
const providerIdentity = record(started.providerIdentity);
|
|
const observedModel = this.options.provider === "acpx" ? providerIdentity.effectiveModel : descriptor.effectiveModel;
|
|
if (typeof observedModel === "string" && observedModel.trim()) this.#observedProviderModel = observedModel;
|
|
this.#confirmCheckpointProviderIdentity(started, event.eventType);
|
|
if (this.options.provider === "acpx") {
|
|
if (descriptor.turnControls !== undefined && descriptor.agent !== (this.options.acpxAgent ?? "codex")) {
|
|
throw new Error("ACPX capability identity differs from the admitted profile");
|
|
}
|
|
const claudeEffort = admittedClaudeEffort(this.options.acpxAgent ?? "codex", this.options.claudeEffort, descriptor, providerIdentity);
|
|
const grokEffort = admittedGrokEffort(this.options.acpxAgent ?? "codex", this.options.grokEffort, descriptor, providerIdentity);
|
|
admittedGrokWebSearchPolicy(this.options.acpxAgent ?? "codex", this.options.grokDisableWebSearch, descriptor, providerIdentity);
|
|
if (claudeEffort !== undefined) this.#evidence.claudeEffort = claudeEffort;
|
|
if (grokEffort !== undefined) this.#evidence.grokEffort = grokEffort;
|
|
if (this.options.grokDisableWebSearch !== undefined) this.#evidence.grokDisableWebSearch = this.options.grokDisableWebSearch;
|
|
const piThinkingLevel = admittedPiThinkingLevel(this.options.acpxAgent ?? "codex", this.options.piThinkingLevel, descriptor, providerIdentity);
|
|
if (piThinkingLevel !== undefined) this.#evidence.piThinkingLevel = piThinkingLevel;
|
|
this.#turnControls = parseAcpxTurnControlCapabilities(descriptor.turnControls, descriptor.agent);
|
|
}
|
|
if (pid !== null) {
|
|
this.#evidence.providerPid = pid;
|
|
this.#evidence.providerProcessStartedAt = readLocalProcessStartedAt(pid);
|
|
if (descriptor.driver === "acpx_runtime") {
|
|
this.#evidence.sidecarPid = pid;
|
|
this.#evidence.sidecarProcessStartedAt =
|
|
this.#evidence.providerProcessStartedAt;
|
|
} else {
|
|
this.#evidence.codexPid = pid;
|
|
this.#evidence.codexProcessStartedAt =
|
|
this.#evidence.providerProcessStartedAt;
|
|
}
|
|
}
|
|
if (typeof descriptor.driver === "string")
|
|
this.#evidence.providerDriver = descriptor.driver;
|
|
if (typeof descriptor.providerVersion === "string")
|
|
this.#evidence.providerVersion = descriptor.providerVersion;
|
|
if (
|
|
descriptor.agent === "pi" ||
|
|
descriptor.agent === "claude" ||
|
|
descriptor.agent === "codex" ||
|
|
descriptor.agent === "cursor" ||
|
|
descriptor.agent === "copilot" ||
|
|
descriptor.agent === "antigravity"
|
|
)
|
|
this.#evidence.acpxAgent = descriptor.agent;
|
|
if (typeof descriptor.agentServerVersion === "string")
|
|
this.#evidence.agentServerVersion = descriptor.agentServerVersion;
|
|
if (typeof descriptor.agentRuntimeVersion === "string")
|
|
this.#evidence.agentRuntimeVersion = descriptor.agentRuntimeVersion;
|
|
if (typeof descriptor.acpProtocolVersion === "number")
|
|
this.#evidence.acpProtocolVersion = descriptor.acpProtocolVersion;
|
|
if (typeof descriptor.agentProcessId === "number") {
|
|
this.#evidence.agentPid = descriptor.agentProcessId;
|
|
this.#evidence.agentProcessStartedAt = readLocalProcessStartedAt(
|
|
descriptor.agentProcessId,
|
|
);
|
|
}
|
|
if (
|
|
runtimeIdentity.executionKind === "local_process" ||
|
|
runtimeIdentity.executionKind === "remote_service"
|
|
) {
|
|
this.#evidence.providerExecutionKind = runtimeIdentity.executionKind;
|
|
}
|
|
if (runtimeIdentity.service === "anthropic_managed_agents") {
|
|
this.#evidence.providerService = "anthropic_managed_agents";
|
|
} else if (runtimeIdentity.service === "aws_bedrock_agentcore_harness") {
|
|
this.#evidence.providerService = "aws_bedrock_agentcore_harness";
|
|
}
|
|
if (threadId !== null) this.#threadId = threadId;
|
|
if (sessionId !== null) this.#sessionId = sessionId;
|
|
if (typeof providerIdentity.kind === "string") {
|
|
this.#providerIdentity = structuredClone(providerIdentity);
|
|
}
|
|
this.#publish();
|
|
}
|
|
|
|
#confirmCheckpointProviderIdentity(input: unknown, source: string): void {
|
|
const checkpointExpectation = this.#checkpointProviderIdentityExpectation;
|
|
if (checkpointExpectation === null) return;
|
|
const { threadId, sessionId } = resolveRunnerdSessionIdentity(input);
|
|
const providerIdentity = record(record(input).providerIdentity);
|
|
if (
|
|
threadId === null &&
|
|
sessionId === null &&
|
|
typeof providerIdentity.kind !== "string"
|
|
) {
|
|
return;
|
|
}
|
|
const providerIdentityMatches =
|
|
checkpointExpectation.providerIdentity === null ||
|
|
(typeof providerIdentity.kind === "string" &&
|
|
durableRecoveryInternals.canonicalJson(providerIdentity) ===
|
|
durableRecoveryInternals.canonicalJson(
|
|
checkpointExpectation.providerIdentity,
|
|
));
|
|
const mismatchFields = [
|
|
...(threadId === checkpointExpectation.driverSessionId
|
|
? []
|
|
: ["driverSessionId"]),
|
|
...(sessionId === checkpointExpectation.providerSessionId
|
|
? []
|
|
: ["providerSessionId"]),
|
|
...(providerIdentityMatches ? [] : ["providerIdentity"]),
|
|
];
|
|
if (mismatchFields.length > 0) {
|
|
throw new Error(
|
|
`native_adopted_provider_identity_mismatch: ${source} did not match the exact durable checkpoint (${mismatchFields.join(", ")})`,
|
|
);
|
|
}
|
|
if (this.#checkpointProviderIdentityConfirmed) return;
|
|
this.#checkpointProviderIdentityConfirmed = true;
|
|
this.#diagnostic(`confirmed adopted provider identity against ${source}`);
|
|
}
|
|
|
|
#flushPendingTraceRehydrations(): void {
|
|
const tracePath = this.options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH;
|
|
if (!tracePath) return;
|
|
const retry: PendingTraceRehydration[] = [];
|
|
for (const pending of this.#pendingTraceRehydrations) {
|
|
const traceResult = appendRunnerdRehydrationTrace(
|
|
this.#traceFrameIndex,
|
|
tracePath,
|
|
pending.sourceEventId,
|
|
pending.eventType,
|
|
pending.visibleNotificationCount,
|
|
this.#nextTraceDebugSequence,
|
|
);
|
|
if (traceResult === "written") this.#nextTraceDebugSequence += 1;
|
|
else if (traceResult === "retry") retry.push(pending);
|
|
}
|
|
this.#pendingTraceRehydrations = retry;
|
|
|
|
const driverRetry: PendingDriverTraceInterpretation[] = [];
|
|
for (const pending of this.#pendingDriverTraceInterpretations) {
|
|
const traceResult = appendCodexDriverInterpretationTrace(
|
|
this.#traceFrameIndex,
|
|
tracePath,
|
|
pending,
|
|
this.#nextTraceDebugSequence,
|
|
);
|
|
if (traceResult === "written") this.#nextTraceDebugSequence += 1;
|
|
else if (traceResult === "retry") driverRetry.push(pending);
|
|
}
|
|
this.#pendingDriverTraceInterpretations = driverRetry;
|
|
}
|
|
|
|
#pumpEventsSafely(): void {
|
|
try {
|
|
this.#pumpEvents();
|
|
} catch (error) {
|
|
this.#failTransport(
|
|
new Error(
|
|
`provider_transport_failed: ${error instanceof Error ? error.message : String(error)}`,
|
|
),
|
|
);
|
|
}
|
|
}
|
|
|
|
async #runnerHasExited(): Promise<boolean> {
|
|
if (this.#handle) return this.#handle.isExitConfirmed?.() ?? this.#handle.child.exitCode !== null;
|
|
const adoptedRunner = this.options.adoptExistingRunner;
|
|
if (!adoptedRunner) return true;
|
|
return !(await adoptedRunner.isAlive());
|
|
}
|
|
|
|
#watchAdoptedRunner(
|
|
adoptedRunner: NonNullable<
|
|
CapabilityRunnerdCodexTransportOptions["adoptExistingRunner"]
|
|
>,
|
|
): void {
|
|
let checking = false;
|
|
this.#adoptedRunnerMonitor = setInterval(() => {
|
|
if (checking || this.#closed) return;
|
|
checking = true;
|
|
void Promise.resolve(adoptedRunner.isAlive())
|
|
.then((alive) => {
|
|
if (alive || this.#closed) return;
|
|
this.#evidence.runnerExited = true;
|
|
this.#publish();
|
|
this.#failTransport(
|
|
new Error(
|
|
"native_adopted_runner_exited: the verified runner exited while its durable authority was active",
|
|
),
|
|
);
|
|
})
|
|
.catch(() => {
|
|
if (!this.#closed) {
|
|
this.#failTransport(
|
|
new Error(
|
|
"native_adopted_runner_identity_unverifiable: runner liveness could not be revalidated",
|
|
),
|
|
);
|
|
}
|
|
})
|
|
.finally(() => {
|
|
checking = false;
|
|
});
|
|
}, 250);
|
|
this.#adoptedRunnerMonitor.unref?.();
|
|
}
|
|
|
|
async #awaitAdoptedRunnerConnection(
|
|
adoptedRunner: NonNullable<
|
|
CapabilityRunnerdCodexTransportOptions["adoptExistingRunner"]
|
|
>,
|
|
ready?: () => Promise<void>,
|
|
): Promise<void> {
|
|
const core = this.#core;
|
|
if (!core) throw new Error("native_runner_authority_unavailable");
|
|
this.#diagnostic(
|
|
`waiting for adopted runner ${adoptedRunner.pid} to authenticate to its durable PRP authority`,
|
|
);
|
|
try {
|
|
await awaitAdoptedRunnerAuthentication({
|
|
activeConnectionCount: () => core.activeRunnerConnectionCount(),
|
|
isAlive: () => adoptedRunner.isAlive(),
|
|
throwIfFailed: () => { this.#assertStartupOpen(); this.#throwIfFailed(); },
|
|
failure: Promise.race([
|
|
this.#failureSignal,
|
|
this.#closedSignal.then(() => { throw new Error("runnerd transport closed during provider startup"); }),
|
|
]),
|
|
ready,
|
|
timeoutMs: this.options.runnerReconnectGraceMs ?? 30_000,
|
|
});
|
|
} catch (error) {
|
|
const failure = error instanceof Error ? error : new Error(String(error));
|
|
this.#failTransport(failure);
|
|
throw failure;
|
|
}
|
|
this.#assertStartupOpen();
|
|
this.#adoptedRunnerAuthenticated = true;
|
|
this.#diagnostic(
|
|
`adopted runner ${adoptedRunner.pid} authenticated to its durable PRP authority`,
|
|
);
|
|
}
|
|
|
|
#watchRunner(handle: RunnerProcessHandle): void {
|
|
void handle.completion.then(
|
|
(result) => {
|
|
const exitConfirmed = handle.isExitConfirmed?.() ?? true;
|
|
this.#evidence.runnerExited = exitConfirmed;
|
|
this.#evidence.runnerExitCode = result.code;
|
|
this.#evidence.runnerSignal = result.signal as NodeJS.Signals | null;
|
|
const detail = result.stderr.trim() || result.stdout.trim();
|
|
if (detail) this.#diagnostic(detail.slice(-4_096));
|
|
this.#publish();
|
|
if (this.#closed || this.#failure !== null || this.#handle !== handle)
|
|
return;
|
|
// A per-turn runner exits after its terminal suffix is durably ACKed.
|
|
// Drain that suffix into the provider-facing queue before classifying
|
|
// process completion; clean terminal exit is the expected lifecycle.
|
|
this.#pumpEventsSafely();
|
|
const expectedPerTurnExit =
|
|
exitConfirmed &&
|
|
result.code === 0 &&
|
|
(this.options.lifecyclePolicy?.mode ?? "per_turn") === "per_turn" &&
|
|
(this.#core?.store.state.committedEvents.some(
|
|
(event) => event.eventType === "runner.suspending",
|
|
) ??
|
|
false);
|
|
if (expectedPerTurnExit) return;
|
|
const code = !exitConfirmed
|
|
? "runner_process_termination_unconfirmed"
|
|
: /provider_frame_too_large|stdout frame exceeded/i.test(detail)
|
|
? "provider_frame_too_large"
|
|
: /runner_ingress_bind_conflict/i.test(detail)
|
|
? "runner_ingress_bind_conflict"
|
|
: /transport[_ ]reconnect[_ ]grace[_ ]exceeded/i.test(detail)
|
|
? "transport_reconnect_grace_exceeded"
|
|
: this.#startupComplete
|
|
? "native_runner_process_exited"
|
|
: this.#startupFailureCode;
|
|
if (
|
|
code === "native_runner_process_exited" &&
|
|
exitConfirmed &&
|
|
this.options.runnerReconnectGraceMs !== undefined &&
|
|
handle.restart
|
|
) {
|
|
void this.#recoverRunnerProcess(handle, detail);
|
|
return;
|
|
}
|
|
this.#failTransport(
|
|
new Error(
|
|
`${code}: runnerd ${exitConfirmed ? "exited unexpectedly" : "process supervision ended"}${result.code === null ? "" : ` with code ${result.code}`}${detail ? `: ${detail.slice(-1_000)}` : ""}`,
|
|
),
|
|
);
|
|
},
|
|
(error) => {
|
|
if (this.#closed || this.#failure !== null || this.#handle !== handle)
|
|
return;
|
|
if (
|
|
this.#startupComplete &&
|
|
handle.isExitConfirmed?.() !== false &&
|
|
this.options.runnerReconnectGraceMs !== undefined &&
|
|
handle.restart
|
|
) {
|
|
void this.#recoverRunnerProcess(
|
|
handle,
|
|
error instanceof Error ? error.message : String(error),
|
|
);
|
|
return;
|
|
}
|
|
const code = handle.isExitConfirmed?.() === false
|
|
? "runner_process_termination_unconfirmed"
|
|
: this.#startupComplete
|
|
? "native_runner_process_exited"
|
|
: this.#startupFailureCode;
|
|
this.#failTransport(
|
|
new Error(
|
|
`${code}: runnerd ${handle.isExitConfirmed?.() === false ? "process supervision" : "process"} failed: ${error instanceof Error ? error.message : String(error)}`,
|
|
{ cause: error },
|
|
),
|
|
);
|
|
},
|
|
);
|
|
}
|
|
|
|
async #recoverRunnerProcess(
|
|
failedHandle: RunnerProcessHandle,
|
|
initialDetail: string,
|
|
): Promise<void> {
|
|
if (
|
|
this.#runnerRecoveryInProgress ||
|
|
this.#closed ||
|
|
this.#failure !== null ||
|
|
this.#handle !== failedHandle ||
|
|
this.#core === null ||
|
|
!failedHandle.restart
|
|
)
|
|
return;
|
|
this.#runnerRecoveryInProgress = true;
|
|
const graceMs = this.options.runnerReconnectGraceMs ?? 0;
|
|
const deadline = Date.now() + graceMs;
|
|
const delays = [250, 500, 1_000, 2_000, 5_000] as const;
|
|
let attempt = 0;
|
|
let restart = failedHandle.restart;
|
|
let lastDetail = initialDetail;
|
|
this.#diagnostic(
|
|
`runner process disconnected; recovery is allowed for ${graceMs}ms`,
|
|
);
|
|
try {
|
|
while (!this.#closed && this.#failure === null && Date.now() < deadline) {
|
|
if (attempt > 0) {
|
|
const base = delays[Math.min(attempt - 1, delays.length - 1)]!;
|
|
const jittered = Math.max(
|
|
1,
|
|
Math.round(base * (0.75 + Math.random() * 0.5)),
|
|
);
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, jittered));
|
|
}
|
|
if (this.#closed || this.#failure !== null) return;
|
|
if (Date.now() >= deadline) break;
|
|
const priorConnectionCount = this.#core.store.state.connectionCount;
|
|
let recoveredHandle: RunnerProcessHandle;
|
|
try {
|
|
const ticket = this.#core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS);
|
|
await this.#core.flushCheckpoint();
|
|
if (this.#closed || this.#failure !== null) return;
|
|
recoveredHandle = restart(ticket);
|
|
} catch (error) {
|
|
lastDetail = error instanceof Error ? error.message : String(error);
|
|
attempt += 1;
|
|
continue;
|
|
}
|
|
this.#handle = recoveredHandle;
|
|
if (recoveredHandle.restart) restart = recoveredHandle.restart;
|
|
this.#evidence.runnerExited = false;
|
|
this.#evidence.runnerExitCode = null;
|
|
this.#evidence.runnerSignal = null;
|
|
this.#evidence.runnerPid = recoveredHandle.child.pid ?? null;
|
|
this.#evidence.runnerProcessGroupId =
|
|
recoveredHandle.processGroupId ?? null;
|
|
this.#publish();
|
|
|
|
let processSettled = false;
|
|
let observationError: unknown;
|
|
let observationDetail = "";
|
|
const completion = recoveredHandle.completion.then(
|
|
(result) => {
|
|
processSettled = true;
|
|
observationDetail = result.stderr.trim() || result.stdout.trim();
|
|
lastDetail = observationDetail;
|
|
return false;
|
|
},
|
|
(error) => {
|
|
processSettled = true;
|
|
observationError = error;
|
|
observationDetail = error instanceof Error ? error.message : String(error);
|
|
lastDetail = observationDetail;
|
|
return false;
|
|
},
|
|
);
|
|
const authenticated = (async () => {
|
|
while (
|
|
!processSettled &&
|
|
!this.#closed &&
|
|
this.#failure === null &&
|
|
Date.now() < deadline
|
|
) {
|
|
if (
|
|
this.#core !== null &&
|
|
this.#core.store.state.connectionCount > priorConnectionCount &&
|
|
this.#core.activeRunnerConnectionCount() === 1
|
|
)
|
|
return true;
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 25));
|
|
}
|
|
return false;
|
|
})();
|
|
if (await Promise.race([completion, authenticated])) {
|
|
this.#diagnostic("runner process restored its durable PRP session");
|
|
this.#watchRunner(recoveredHandle);
|
|
return;
|
|
}
|
|
if (recoveredHandle.isExitConfirmed?.() === false) {
|
|
this.#failTransport(
|
|
new Error(
|
|
`runner_process_termination_unconfirmed: replacement runner health could not be verified${observationDetail ? `: ${observationDetail.slice(-1_000)}` : ""}`,
|
|
{ cause: observationError },
|
|
),
|
|
);
|
|
return;
|
|
}
|
|
attempt += 1;
|
|
}
|
|
if (!this.#closed) {
|
|
this.#failTransport(
|
|
new Error(
|
|
`transport_reconnect_grace_exceeded: runner process recovery exceeded ${graceMs}ms${lastDetail ? `: ${lastDetail.slice(-1_000)}` : ""}`,
|
|
),
|
|
);
|
|
}
|
|
} finally {
|
|
this.#runnerRecoveryInProgress = false;
|
|
}
|
|
}
|
|
|
|
#failTransport(error: Error): void {
|
|
if (this.#failure !== null || this.#closed) return;
|
|
this.#failure = error;
|
|
this.#rejectFailureSignal(error);
|
|
if (this.#pump !== null) clearInterval(this.#pump);
|
|
this.#pump = null;
|
|
try {
|
|
this.#diagnostic(error.message);
|
|
} finally {
|
|
// An observer is not allowed to leave notification consumers waiting
|
|
// after the request path has already received this terminal failure.
|
|
this.#queue.close(error);
|
|
}
|
|
}
|
|
|
|
#throwIfFailed(): void {
|
|
if (this.#failure !== null) throw this.#failure;
|
|
}
|
|
|
|
async #awaitRegistrationReady(
|
|
ready: (() => Promise<void>) | undefined,
|
|
): Promise<void> {
|
|
return this.options.measureOperation ? this.options.measureOperation("runner.transport.registration", () => this.#awaitRegistrationReadyWork(ready)) : this.#awaitRegistrationReadyWork(ready);
|
|
}
|
|
|
|
async #awaitRegistrationReadyWork(
|
|
ready: (() => Promise<void>) | undefined,
|
|
): Promise<void> {
|
|
this.#assertStartupOpen();
|
|
if (ready === undefined) {
|
|
this.#throwIfFailed();
|
|
return;
|
|
}
|
|
await Promise.race([
|
|
ready(), this.#failureSignal,
|
|
this.#closedSignal.then(() => { this.#assertStartupOpen(); }),
|
|
]);
|
|
this.#assertStartupOpen();
|
|
this.#throwIfFailed();
|
|
|
|
}
|
|
|
|
#diagnostic(message: string): void {
|
|
message = redactObservabilityDiagnostic(message, this.options.environment, this.options.observability?.otelConfig);
|
|
this.#evidence.diagnostics.push(message);
|
|
if (this.#evidence.diagnostics.length > 64)
|
|
this.#evidence.diagnostics.shift();
|
|
this.options.onDiagnostic?.(message);
|
|
this.#publish();
|
|
}
|
|
|
|
#publish(): void {
|
|
this.options.onEvidence?.(this.evidence());
|
|
}
|
|
}
|
|
|
|
export function defaultCapabilityRunnerdBinary(platform = process.platform, architecture = process.arch): string {
|
|
// Standalone builds use dist/live; the server vendors that compiled tree
|
|
// directly under vendor/paperclip-runner. Resolve beside our own live module.
|
|
const outputRoot = resolve(fileURLToPath(new URL("../", import.meta.url)));
|
|
const staged = resolvePackagedRunnerBinary(outputRoot, platform, architecture)
|
|
?? resolvePackagedRunnerBinary(resolve(packageRoot, "dist"), platform, architecture);
|
|
if (staged) return staged;
|
|
if (platform !== process.platform || architecture !== process.arch) {
|
|
throw new Error(`runner_remote_artifact_unavailable: install the Paperclip release containing the ${platform}-${architecture} daemon or configure PAPERCLIP_RUNNER_REMOTE_BINARY_PATH`);
|
|
}
|
|
return resolve(
|
|
packageRoot,
|
|
`runner/target/debug/paperclip-runnerd${executableSuffix}`,
|
|
);
|
|
}
|
|
|
|
/** Starts an authenticated durable PRP authority, runnerd, and Codex provider transport. */
|
|
export function createCapabilityRunnerdCodexTransport(
|
|
options: CapabilityRunnerdCodexTransportOptions = {},
|
|
): CapabilityRunnerdCodexTransport {
|
|
const transport = new DurablePrpCodexTransport(options);
|
|
return {
|
|
transport,
|
|
evidence: () => transport.evidence(),
|
|
detachControllerForRestart: () => transport.detachControllerForRestart(),
|
|
retireProbeResources: timeoutMs => transport.retireProbeResources(timeoutMs),
|
|
};
|
|
}
|
|
|
|
export const createRunnerdCodexTransport =
|
|
createCapabilityRunnerdCodexTransport;
|
|
|
|
const retainedNativeProbeCleanups = new Set<{ run: () => Promise<void> }>();
|
|
|
|
function probeCleanupBudget(timeoutMs = 10_000): number {
|
|
return Number.isFinite(timeoutMs) && timeoutMs > 0 ? Math.min(timeoutMs, 10_000) : 10_000;
|
|
}
|
|
async function awaitProbeCleanup<T>(pending: Promise<T>, deadline: number, detail: string): Promise<T> {
|
|
let timeout: ReturnType<typeof setTimeout> | undefined;
|
|
try {
|
|
return await Promise.race([pending, new Promise<never>((_resolve, reject) => {
|
|
timeout = setTimeout(() => reject(new Error(`Native probe retirement timed out while ${detail}; recovery state retained.`)), Math.max(0, deadline - Date.now()));
|
|
})]);
|
|
} finally { clearTimeout(timeout); }
|
|
}
|
|
async function awaitProbeProof(proven: () => boolean, deadline: number, detail: string): Promise<void> {
|
|
while (!proven()) {
|
|
if (Date.now() >= deadline) throw new Error(`Native probe retirement remains unconfirmed while ${detail}; recovery state retained.`);
|
|
await new Promise(resolve => setTimeout(resolve, Math.min(100, Math.max(1, deadline - Date.now()))));
|
|
}
|
|
}
|
|
|
|
export class NativeEnvironmentProbeCleanupError extends AggregateError {
|
|
readonly code = "NATIVE_ENVIRONMENT_PROBE_CLEANUP_PENDING";
|
|
constructor(
|
|
readonly retired: Promise<void>,
|
|
readonly operationError: unknown,
|
|
readonly cleanupErrors: readonly unknown[],
|
|
readonly retryCleanup?: () => Promise<void>,
|
|
) {
|
|
const errors = [...(operationError === undefined ? [] : [operationError]), ...cleanupErrors];
|
|
super(errors, errors.map(error => error instanceof Error ? error.message : "Native probe cleanup failed.").join("; "), { cause: operationError ?? cleanupErrors[0] });
|
|
this.name = "NativeEnvironmentProbeCleanupError";
|
|
}
|
|
}
|
|
|
|
function retainNativeProbeCleanup(cleanup: (retry: boolean) => Promise<void>) {
|
|
let operation: Promise<void> | undefined;
|
|
let attempts = 0;
|
|
const owner = { run: (): Promise<void> => {
|
|
if (operation) return operation;
|
|
const retry = attempts++ > 0;
|
|
operation = Promise.resolve().then(() => cleanup(retry)).then(() => {
|
|
retainedNativeProbeCleanups.delete(owner);
|
|
}, error => { operation = undefined; throw error; });
|
|
// The HTTP caller may retain only the error, without awaiting retirement.
|
|
// Keep awaited failures observable while preventing detached rejections.
|
|
void operation.catch(() => undefined);
|
|
return operation;
|
|
} };
|
|
retainedNativeProbeCleanups.add(owner);
|
|
return { retired: owner.run(), retryCleanup: owner.run };
|
|
}
|
|
|
|
export async function probeNativeRunnerEnvironment(options: {
|
|
runtimeDirectory: string;
|
|
provider: "codex" | "opencode" | "acpx";
|
|
model: string | null;
|
|
reasoningEffort?: string;
|
|
/** Verify native session identity/model and cleanup without a billable turn. */
|
|
admissionOnly?: boolean;
|
|
environment: NodeJS.ProcessEnv;
|
|
timeoutMs?: number;
|
|
/** A finite cleanup budget, capped at the ordinary ten-second boundary. */
|
|
cleanupTimeoutMs?: number;
|
|
workingDirectory?: string;
|
|
transportOptions?: CapabilityRunnerdCodexTransportOptions;
|
|
onCodexCredentialRefresh?: (filename: string) => Promise<void>;
|
|
onCleanupConfirmed?: () => Promise<void>;
|
|
}): Promise<{
|
|
provider: "codex" | "opencode" | "acpx";
|
|
providerDriver: string;
|
|
effectiveModel: string;
|
|
helloProbePassed: boolean;
|
|
admissionOnly?: true;
|
|
sessionAdmissionPassed?: true;
|
|
}> {
|
|
const environment = { ...options.environment };
|
|
const remoteFilesystemRoot = options.transportOptions?.runnerFilesystemRoot;
|
|
const ownsPreparation = options.transportOptions?.prepareLaunch !== undefined;
|
|
if (options.workingDirectory !== undefined && !remoteFilesystemRoot && !ownsPreparation) throw new Error("Native setup remote working directory requires a remote runner filesystem.");
|
|
const workingDirectory = options.workingDirectory ?? resolve(options.runtimeDirectory, "workspace");
|
|
if (!remoteFilesystemRoot && !ownsPreparation) mkdirSync(workingDirectory, { mode: 0o700 });
|
|
const controller = new AbortController();
|
|
let registrationTeardownError: unknown;
|
|
let registrationOperation: Promise<unknown> | undefined;
|
|
let registrationReleaseOperation: Promise<void> | undefined;
|
|
let registrationReleaseFailed = false;
|
|
let releaseRegistration: (() => Promise<void>) | undefined;
|
|
let cleanupCallbackOperation: Promise<void> | undefined;
|
|
let cleanupCallbackFailed = false;
|
|
let credentialRefreshOperation: Promise<void> | undefined;
|
|
let credentialRefreshFailed = false;
|
|
const preserveCredentialRefresh = (retry = false) => {
|
|
if (retry && credentialRefreshFailed) { credentialRefreshOperation = undefined; credentialRefreshFailed = false; }
|
|
return credentialRefreshOperation ??= Promise.resolve()
|
|
.then(() => options.onCodexCredentialRefresh?.(resolve(remoteFilesystemRoot ?? options.runtimeDirectory, "codex-home", "auth.json")))
|
|
.catch(error => { credentialRefreshFailed = true; throw error; });
|
|
};
|
|
let registrationCleanupConfirmed = true;
|
|
const cleanupBudget = probeCleanupBudget(options.cleanupTimeoutMs);
|
|
const sourceCodexHome = ownsPreparation ? undefined : options.transportOptions?.sourceCodexHome ?? resolveSourceCodexHome(environment);
|
|
delete environment.PAPERCLIP_RUNNER_EXTERNAL_SANDBOX;
|
|
if (!ownsPreparation) {
|
|
environment.HOME = resolve(remoteFilesystemRoot ?? options.runtimeDirectory, "probe-home");
|
|
environment.CODEX_HOME = resolve(remoteFilesystemRoot ?? options.runtimeDirectory, "codex-home");
|
|
}
|
|
// A prepared launch reports its own directory; Core does not assign one.
|
|
if (!ownsPreparation) environment.PAPERCLIP_WORKSPACE_CWD = workingDirectory;
|
|
const bundle = createRunnerdCodexTransport({
|
|
...options.transportOptions,
|
|
...(options.transportOptions?.controlPlaneRegistration ? { controlPlaneRegistration: (authority, identity) => {
|
|
const operation = (async () => {
|
|
controller.signal.throwIfAborted();
|
|
registrationCleanupConfirmed = false;
|
|
const registration = await options.transportOptions!.controlPlaneRegistration!(authority, identity, controller.signal);
|
|
const release = () => registrationReleaseOperation ??= Promise.resolve()
|
|
.then(() => registration.release())
|
|
.then(() => { registrationCleanupConfirmed = true; }, error => {
|
|
registrationReleaseFailed = true;
|
|
registrationTeardownError = error;
|
|
throw error;
|
|
});
|
|
releaseRegistration = release;
|
|
if (controller.signal.aborted) {
|
|
await release();
|
|
controller.signal.throwIfAborted();
|
|
}
|
|
return { ...registration, release };
|
|
})();
|
|
registrationOperation = operation;
|
|
return operation;
|
|
} } : {}),
|
|
...(options.transportOptions?.runnerProcessLauncher ? { runnerProcessLauncher: spec => {
|
|
controller.signal.throwIfAborted();
|
|
return options.transportOptions!.runnerProcessLauncher!(spec);
|
|
} } : {}),
|
|
provider: options.provider,
|
|
sourceCodexHome: sourceCodexHome ?? "",
|
|
stateDirectory: options.runtimeDirectory,
|
|
environment,
|
|
externallySandboxed: options.transportOptions?.externallySandboxed === true,
|
|
disableCodexProjectInstructions: true,
|
|
disableCodexWebSearch: options.provider === "codex",
|
|
closeGraceMs: cleanupBudget,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
|
});
|
|
const driver = new CodexAppServerDriver({
|
|
taskEnvelope: {
|
|
schema: "paperclip.skillless_task.v1", objective: "Verify the selected native account and model.",
|
|
completionContract: { revision: "environment-probe-v1", criteria: [] },
|
|
constraints: [], expectedResultSchema: "paperclip.run_result.v1",
|
|
},
|
|
...(options.model ? { model: options.model } : {}),
|
|
...(options.reasoningEffort ? { reasoningEffort: options.reasoningEffort } : {}),
|
|
conversationMode: "direct",
|
|
requestedCollaborationMode: "plan",
|
|
approvalPolicy: "never",
|
|
includeCollaborationModeInstructions: false,
|
|
dynamicTools: [],
|
|
environment,
|
|
transportFactory: () => bundle.transport,
|
|
requireProviderSessionIdentity: true,
|
|
...(ownsPreparation ? { workingDirectoryAuthority: "runner_prepared_launch" as const }
|
|
: remoteFilesystemRoot ? { workingDirectoryAuthority: "remote_runner" as const } : {}),
|
|
});
|
|
let rejectAborted!: (error: unknown) => void;
|
|
const aborted = new Promise<never>((_resolve, reject) => { rejectAborted = reject; });
|
|
const timer = setTimeout(() => {
|
|
const error = new Error("Native provider hello probe timed out.");
|
|
controller.abort(error);
|
|
rejectAborted(error);
|
|
}, Math.max(1, Math.min(options.timeoutMs ?? 45_000, 120_000)));
|
|
let session: HarnessSession | undefined;
|
|
let completed = false;
|
|
let effectiveModel: string | undefined;
|
|
let failure: unknown;
|
|
const operation = (async () => {
|
|
session = await driver.openSession({ runId: options.transportOptions?.prpIdentity?.runId ?? "environment-probe", normalizedSessionId: options.transportOptions?.prpIdentity?.normalizedSessionId ?? "environment-probe",
|
|
workingDirectory, signal: controller.signal });
|
|
controller.signal.throwIfAborted();
|
|
const ids = session.ids();
|
|
const expectedDriver = { codex: "codex_app_server", opencode: "opencode_server", acpx: "acpx_runtime" }[options.provider];
|
|
if (!ids.driverSessionId || !ids.providerSessionId || bundle.evidence().providerDriver !== expectedDriver) {
|
|
throw new Error(`The native environment probe returned no matching provider identity (${bundle.evidence().providerDriver ?? "missing driver"}).`);
|
|
}
|
|
const events = session.events()[Symbol.asyncIterator]();
|
|
const first = await events.next();
|
|
const context = record(first.value?.payload.context);
|
|
if (first.value?.eventType !== "session.started" || !context.model || record(record(context.sandbox).permissionProfile).id !== "paperclip-runner-workspace-read-only") {
|
|
throw new Error("The native environment probe returned no observed model or read-only runtime identity.");
|
|
}
|
|
{
|
|
effectiveModel = typeof context.model === "string" && context.model.trim() && context.model !== "unknown" ? context.model : undefined;
|
|
if (!effectiveModel || (options.model !== null && effectiveModel !== options.model)) throw new Error("The selected native runtime returned a missing or mismatched provider model.");
|
|
}
|
|
controller.signal.throwIfAborted();
|
|
if (options.admissionOnly) { completed = true; return; }
|
|
const { turnId } = await session.startTurn({ message: { role: "user", text: "Respond only with hello. Do not use tools or inspect files." } });
|
|
let responseObserved = false;
|
|
while (true) {
|
|
const next = await events.next();
|
|
if (next.done) break;
|
|
const event = next.value;
|
|
if (event.turnId !== turnId) continue;
|
|
if (event.eventType === "item.completed" && event.payload.kind === "agentMessage"
|
|
&& typeof event.payload.text === "string" && event.payload.text.trim()) {
|
|
responseObserved = true;
|
|
}
|
|
if (event.eventType === "turn.completed") {
|
|
if (!responseObserved || !effectiveModel) throw new Error("The native provider hello probe returned no response or observed model.");
|
|
completed = true;
|
|
return;
|
|
}
|
|
if (event.eventType === "turn.failed" || event.eventType === "turn.interrupted" || event.eventType === "turn.cancelled") {
|
|
const detail = record(event.payload.error).message;
|
|
throw new Error(typeof detail === "string" ? detail : "The native provider hello probe failed.");
|
|
}
|
|
}
|
|
throw new Error("The native provider hello probe ended without a successful turn.");
|
|
})();
|
|
try {
|
|
await Promise.race([operation, aborted]);
|
|
} catch (error) {
|
|
failure = error;
|
|
} finally {
|
|
clearTimeout(timer);
|
|
controller.abort(failure ?? new Error("Native provider hello probe complete."));
|
|
const teardownErrors: unknown[] = [];
|
|
const teardownDeadline = Date.now() + cleanupBudget;
|
|
const awaitTeardown = async (pending: Promise<unknown>, detail: string): Promise<void> => {
|
|
let timeout: ReturnType<typeof setTimeout> | undefined;
|
|
try {
|
|
await Promise.race([
|
|
pending,
|
|
new Promise<never>((_resolve, reject) => {
|
|
timeout = setTimeout(() => reject(new Error(`Native setup probe cleanup timed out while ${detail}; recovery state retained.`)), Math.max(0, teardownDeadline - Date.now()));
|
|
}),
|
|
]);
|
|
} finally { clearTimeout(timeout); }
|
|
};
|
|
const closingOperation = session ? session.close({ reason: "environment probe complete" }) : bundle.transport.close("environment probe complete");
|
|
try {
|
|
await awaitTeardown(closingOperation, "closing the selected runtime");
|
|
if (bundle.evidence().runnerPid !== null && !bundle.evidence().runnerExited) throw new Error("Native setup probe cleanup is incomplete.");
|
|
} catch (error) { teardownErrors.push(error); }
|
|
try { await awaitTeardown(operation.catch(() => undefined), "draining native setup"); }
|
|
catch (error) { teardownErrors.push(error); }
|
|
if (registrationOperation) {
|
|
try { await awaitTeardown(registrationOperation.catch(() => undefined), "draining authority registration"); }
|
|
catch (error) { teardownErrors.push(error); }
|
|
}
|
|
if (registrationTeardownError !== undefined && !teardownErrors.includes(registrationTeardownError)) teardownErrors.push(registrationTeardownError);
|
|
if (!registrationCleanupConfirmed) teardownErrors.push(new Error("Native setup probe authority release is unconfirmed; recovery state retained."));
|
|
if (bundle.evidence().runnerPid !== null && bundle.evidence().runnerExited && options.provider === "codex" && options.onCodexCredentialRefresh) {
|
|
try {
|
|
await awaitTeardown(preserveCredentialRefresh(), "preserving credential refresh");
|
|
}
|
|
catch (error) { teardownErrors.push(error); }
|
|
}
|
|
const confirmCleanup = (retry = false) => {
|
|
if (retry && cleanupCallbackFailed) { cleanupCallbackOperation = undefined; cleanupCallbackFailed = false; }
|
|
return cleanupCallbackOperation ??= Promise.resolve().then(() => options.onCleanupConfirmed?.())
|
|
.catch(error => { cleanupCallbackFailed = true; throw error; });
|
|
};
|
|
if (!teardownErrors.length && options.onCleanupConfirmed) {
|
|
try { await awaitTeardown(confirmCleanup(), "confirming credential handoff"); }
|
|
catch (error) { teardownErrors.push(error); }
|
|
}
|
|
if (teardownErrors.length) {
|
|
const retained = retainNativeProbeCleanup(async retry => {
|
|
const deadline = Date.now() + cleanupBudget;
|
|
await awaitProbeCleanup(Promise.allSettled([operation, closingOperation, registrationOperation]), deadline, "joining original probe operations");
|
|
if (!bundle.retireProbeResources) throw new Error("Native probe physical retirement proof is unavailable");
|
|
await awaitProbeCleanup(bundle.retireProbeResources(Math.max(1, deadline - Date.now())), deadline, "confirming physical retirement");
|
|
if (retry && registrationReleaseFailed) { registrationReleaseOperation = undefined; registrationReleaseFailed = false; }
|
|
if (!registrationCleanupConfirmed && releaseRegistration) await awaitProbeCleanup(releaseRegistration(), deadline, "releasing original probe authority");
|
|
if (bundle.evidence().runnerPid !== null && options.provider === "codex" && options.onCodexCredentialRefresh) {
|
|
await awaitProbeCleanup(preserveCredentialRefresh(retry), deadline, "preserving credential refresh");
|
|
}
|
|
if (!registrationCleanupConfirmed) throw new Error("Native probe authority release remains unconfirmed");
|
|
await awaitProbeCleanup(confirmCleanup(retry), deadline, "confirming credential handoff");
|
|
});
|
|
throw new NativeEnvironmentProbeCleanupError(retained.retired, failure, teardownErrors, retained.retryCleanup);
|
|
}
|
|
}
|
|
if (failure !== undefined) throw failure;
|
|
if (!completed) throw new Error("The native provider hello probe did not complete.");
|
|
return Object.freeze({ provider: options.provider, providerDriver: bundle.evidence().providerDriver!, effectiveModel: effectiveModel!,
|
|
...(options.admissionOnly ? { helloProbePassed: false, admissionOnly: true as const, sessionAdmissionPassed: true as const } : { helloProbePassed: true }) });
|
|
}
|
|
|
|
/** The transport provider identifies the biller, independently of model family. */
|
|
function openedThreadModelProvider(
|
|
provider: CapabilityRunnerdCodexTransportOptions["provider"],
|
|
acpxAgent: QualifiedAcpxAgent,
|
|
model: unknown,
|
|
): string {
|
|
if (provider === "opencode" && typeof model === "string") return model.split("/", 1)[0]!;
|
|
if (provider === "claude_managed") return "anthropic";
|
|
if (provider === "aws_agentcore") return "aws";
|
|
if (provider === "acpx") {
|
|
if (acpxAgent === "pi") return "openrouter";
|
|
if (acpxAgent === "grok") return "xai";
|
|
if (acpxAgent === "claude") return "anthropic";
|
|
if (acpxAgent === "cursor") return "cursor";
|
|
if (acpxAgent === "copilot") return "github";
|
|
if (acpxAgent === "antigravity") return "google";
|
|
}
|
|
return "openai";
|
|
}
|
|
|
|
export const runnerdLaunchProfileInternals = Object.freeze({
|
|
openedThreadModelProvider,
|
|
acpxProviderPackageAuthority,
|
|
acpxRunnerLaunchProfile,
|
|
resolveBuildOwnedCliArtifact,
|
|
resolveRunnerOpenCodeExecutable,
|
|
maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES,
|
|
p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES,
|
|
});
|
|
|
|
export const runnerdRecoveryInternals = Object.freeze({
|
|
projectCopilotRunGrant,
|
|
completedMaintenanceTerminalReceipt,
|
|
completedMaintenanceTerminalReplayMatches,
|
|
readControlPlaneState,
|
|
awaitProviderDrainBarrier,
|
|
awaitAdoptedRunnerAuthentication,
|
|
awaitRunnerSuspensionBarrier,
|
|
providerDrainStateFromSnapshot,
|
|
providerTurnIsActiveFromCommittedEvents,
|
|
recoveredRunAttachment,
|
|
releaseRunnerProcessOwnership,
|
|
runnerCloseDeadlines,
|
|
runnerCloseGraceMs,
|
|
rotatedRunAttachPayload,
|
|
rotateExternalAuthorityEpoch,
|
|
turnStartCommandResultValid,
|
|
turnStartNotificationDisposition,
|
|
turnStartResponseReady,
|
|
});
|
|
|
|
|
|
/** Map controller asset paths to the assigned copy on the provider filesystem. */
|
|
export function resolveRunnerdCodexSkillInputs(
|
|
inputs: Record<string, unknown>[],
|
|
context: NativeRuntimeContextSnapshot | null,
|
|
codexHome: string,
|
|
preparedContext?: NativeRuntimeContextSnapshot,
|
|
): Array<{ type: "skill"; name: string; path: string }> {
|
|
if (inputs.length > 64) throw new Error("Too many explicit skill inputs");
|
|
const seen = new Set<string>();
|
|
return inputs.map((input) => {
|
|
const assigned = context?.skills.find((skill) => skill.runtimeName === input.name);
|
|
if (
|
|
!assigned || !/^[a-zA-Z0-9_-]+$/.test(assigned.runtimeName)
|
|
|| input.path !== resolve(assigned.bundle.rootPath, "SKILL.md")
|
|
|| seen.has(assigned.runtimeName)
|
|
) {
|
|
throw new Error("Explicit skill input must reference a unique assigned runtime skill");
|
|
}
|
|
seen.add(assigned.runtimeName);
|
|
const preparedSkill = preparedContext?.skills.find(skill => skill.key === assigned.key && skill.runtimeName === assigned.runtimeName);
|
|
if (preparedContext && !preparedSkill) throw new Error("Prepared launch omitted an assigned explicit skill");
|
|
return {
|
|
type: "skill",
|
|
name: assigned.runtimeName,
|
|
path: preparedSkill ? resolve(preparedSkill.bundle.rootPath, "SKILL.md")
|
|
: resolve(codexHome, "skills", assigned.runtimeName, "SKILL.md"),
|
|
};
|
|
});
|
|
}
|
|
|
|
/** Controller admission budget; ordinary command and turn deadlines are independent. */
|
|
export function coldAdmissionTimeoutMs(provider: string | undefined, agent: string | undefined, coldProcess: boolean): number {
|
|
return coldProcess && provider === "acpx" && agent === "pi" ? 60_000 : 30_000;
|
|
}
|
|
|
|
export interface RunnerCommandFailure extends Error {
|
|
/** Private diagnostic value copied from the authenticated command result. */
|
|
readonly runnerCommandResult: unknown;
|
|
}
|
|
|
|
/** Shared polling path keeps startup cancellation separate from owned cleanup commands. */
|
|
export async function waitForRunnerCommand(input: {
|
|
type: string;
|
|
deadline: number;
|
|
abortOnClose: boolean;
|
|
isClosed: () => boolean;
|
|
throwIfFailed: () => void;
|
|
command: () => { status: string; result?: unknown } | undefined;
|
|
runnerHasExited: () => Promise<boolean>;
|
|
failureCode: () => string;
|
|
}): Promise<void> {
|
|
while (Date.now() < input.deadline) {
|
|
input.throwIfFailed();
|
|
if (input.abortOnClose && input.isClosed()) {
|
|
throw new Error(`runnerd transport closed while waiting for ${input.type}`);
|
|
}
|
|
const command = input.command();
|
|
if (command?.status === "completed") return;
|
|
if (command !== undefined && command.status !== "pending") {
|
|
const error = new Error(`PRP command ${input.type} ${command.status}: ${JSON.stringify(command.result)}`) as RunnerCommandFailure;
|
|
Object.defineProperty(error, "runnerCommandResult", { value: structuredClone(command.result), enumerable: false });
|
|
throw error;
|
|
}
|
|
if (await input.runnerHasExited()) {
|
|
throw new Error(`runnerd exited while waiting for ${input.type}`);
|
|
}
|
|
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
|
}
|
|
throw new Error(`${input.failureCode()}: PRP command ${input.type} timed out`);
|
|
}
|