mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 04:25:52 +01:00
perf(runner): reuse compatible protected provider packs (#17749)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts provider runtimes in remote execution environments. > - Sandbox images already contain qualified provider packs. > - Startup required the image pack to match the entire current app release manifest. > - An unrelated app commit could therefore cause pack verification, transfer, and installation again. > - This pull request selects protected packs by their provider interface and platform. > - The benefit is no extra pack probe, directory scan, version command, link, or upload on the compatible image path. ## Linked Issues or Issue Description **What happened?** Starting a remote task can reject a compatible image pack because its source revision or unrelated provider differs from the controller. Pack replacement added about 16 seconds in the investigated startup. Startup also repeated full content checks. **Expected behavior** Use an already qualified, protected pack when its selected provider interface and target platform are compatible. Keep the selected artifact identity for process launch and recovery. **Steps to reproduce** 1. Install a qualified provider pack in a sandbox image. 2. Start a controller built from another app commit. 3. Start a remote OpenCode task. The old complete-manifest comparison rejects the image pack even when the provider interface still works. Related repairs: #16493 reduced transfer cost. #17512 retained exact artifact authority during reattachment. This change preserves that recovery rule and changes fresh-launch pack selection. ## What Changed - Add explicit OpenCode and ACPX interface revisions to produced pack manifests. App Git SHAs and dependency pins remain provenance. - Read bounded protected-installation metadata in the existing final runner metadata query. Accept only root-owned paths that the execution user cannot write, including ancestors and manifest. Reject writable aliases. - Select and prepare the pack in server-owned environment setup before constructing the existing transport. Bind provider launch paths and digests from the selected installation. Save that identity for exact reattachment. - Keep full content verification for uploads, writable installations, and legacy packs. Reuse successful verification within one startup. Keep older attached runners compatible with the optional metadata request. - Seal image packs and document the one-time image refresh that enables reuse across later compatible releases. Keep selected ACPX agent qualification checks. ## Verification - Red/green regression: the old code rejects a protected pack from another app revision with `manifest mismatch`; the fixed path selects it without executing verification commands. - Native startup regression: select the installed pack's own Node digest with one existing runner metadata request and no pack verification or version commands. - Targeted pack, artifact, recovery, execution-driver, and Rust metadata tests pass. Incompatible interfaces/platforms, malformed identities, writable roots, path aliases, and exact reattachment are covered. - Final revision `91dc1676ff67bdbe5866c8586daac10240530cca`: workspace typecheck and build pass. All 53 current-head checks are successful, with no pending or failed checks. Greptile is 5/5 with no open findings. - The broad local `pnpm test:run` attempt reported a workspace-runtime conflict test failure; its isolated rerun passed. The broad run was stopped after that failure. Full CI now passes, including the server test shards. This is not a claim that the full local run passed. - Linux filesystem evidence: the production metadata functions read a real root-owned preinstalled pack as a non-root user in 16 microseconds median and 20 microseconds p95 across 100 warm reads. This measures metadata admission, not model or hosted response latency. - All 910 affected server tests and 57 Copilot attestation tests pass locally on the final implementation. Full CI covers the unchanged Runner transport. No provider qualification hash was refreshed. - Built the actual Linux daemon at `416e7ec7425d0d3565b4a5e1808002d650a60414` and ran its metadata command as UID 65534. The protected installation returned its real manifest with `immutable: true`; writable alias and missing-pack cases returned `immutable: false`. This is metadata/permission evidence, not provider execution. - Hosted rollout and a hosted end-to-end latency measurement have not run. ## Risks - Existing images need one rebuild to publish the compatibility declaration. Packs without it keep the legacy content-verification path. - Fast admission relies on image publication producing a qualified protected tree. Root execution and task-writable installations retain full verification. - Bump an interface revision when required runner/pack behavior changes incompatibly. Deploy a new image to roll out compatible runtime fixes or upgrades. - This removes provider-pack preparation delay. Other runner startup, context, model, and finalization costs remain separate. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol`, reasoning effort `xhigh`, with tool use and code execution. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
f1922d95e1
commit
9ae4c25e7e
20 files changed
+570
-72
No files matched your search
@@ -1,5 +1,5 @@
|
||||
import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
|
||||
import { verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js";
|
||||
import { type ProviderPackInstallation, verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js";
|
||||
import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import { writeExecutionControl } from "../execution-control-notifications.js";
|
||||
@@ -23,7 +23,7 @@ import { isCloudManagedInstance } from "../cloud-instance.js";
|
||||
import { dotRunnerBroker } from "../dot-runner-broker.js";
|
||||
import { CURSOR_DISTRIBUTION_PINS, QUALIFIED_ACPX_PROFILES, QUALIFIED_ACPX_VERSION } from "../../vendor/paperclip-runner/index.js";
|
||||
import { isProviderMode } from "../../vendor/paperclip-runner/index.js";
|
||||
import { bundledRemoteProviderPackManifestPath, bundledRemoteRunnerBinary, runnerBinaryTarget } from "../../vendor/paperclip-runner/index.js";
|
||||
import { PROVIDER_PACK_COMPATIBILITY, bundledRemoteProviderPackManifestPath, bundledRemoteRunnerBinary, runnerBinaryTarget } from "../../vendor/paperclip-runner/index.js";
|
||||
import { nativeRetryCancellationEligible, rethrowNativeCancellationLockConflict, assertCancellationRequest, cancellationIntentId as callerCancellationIntentId, cancellationRequestId } from "./native-cancellation-request.js";
|
||||
import { readNativePlanWait } from "./native-plan-wait.js";
|
||||
import { nativeProviderLifecycle } from "./provider-lifecycle.js";
|
||||
@@ -133,7 +133,7 @@ import { createNativeSshCommandRunner } from "./native-ssh-command-runner.js";
|
||||
import { discoverRemoteExecutable, parseRemoteExecutableCandidate } from "./codex-model-fallback.js";
|
||||
export { parseRemoteExecutableCandidate } from "./codex-model-fallback.js";
|
||||
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
||||
import { createNativeRunnerExecutionDriver, resolveRemoteRunnerTransportMode } from "./native-runner-execution-driver.js";
|
||||
import { prepareNativeRemoteRunner, createNativeRunnerExecutionDriver, resolveRemoteRunnerTransportMode } from "./native-runner-execution-driver.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
and,
|
||||
@@ -10316,6 +10316,7 @@ type RemoteProviderPackManifest = {
|
||||
schema: typeof REMOTE_PROVIDER_PACK_SCHEMA;
|
||||
digest: string;
|
||||
payload: {
|
||||
compatibility?: { opencode: number; acpx: number };
|
||||
pins: typeof REMOTE_PROVIDER_PACK_PINS;
|
||||
target: { platform: string; architecture: string };
|
||||
runnerSourceRevision: string;
|
||||
@@ -10427,6 +10428,7 @@ function readRemoteProviderPackIdentity(packRoot: string, verifyControllerFiles:
|
||||
if (
|
||||
manifest?.schema !== REMOTE_PROVIDER_PACK_SCHEMA ||
|
||||
!payload ||
|
||||
(payload.compatibility !== undefined && canonicalJson(payload.compatibility) !== canonicalJson(PROVIDER_PACK_COMPATIBILITY)) ||
|
||||
canonicalJson(payload.pins) !== canonicalJson(REMOTE_PROVIDER_PACK_PINS) ||
|
||||
canonicalJson(payload.acpxProfileDigests) !==
|
||||
canonicalJson(REMOTE_PROVIDER_PACK_PROFILE_DIGESTS) ||
|
||||
@@ -12141,7 +12143,9 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
|| !lstatSync(configuredProviderPackRoot).isDirectory()) {
|
||||
throw new Error("runner_remote_provider_artifact_incompatible: configure PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH with the build-owned provider pack");
|
||||
}
|
||||
return readRemoteProviderPackManifest(configuredProviderPackRoot);
|
||||
// Selection reads identity only. Content verification belongs to the
|
||||
// install path; a compatible image pack needs no controller tree scan.
|
||||
return readRemoteProviderPackIdentity(configuredProviderPackRoot, false);
|
||||
})();
|
||||
return currentProviderPackManifest;
|
||||
};
|
||||
@@ -12261,6 +12265,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
: Promise.resolve();
|
||||
|
||||
const verifyRemoteProviderPack = async (packRoot: string) => {
|
||||
if (providerPackVerified && packRoot === activeRemoteProviderPackRoot) return;
|
||||
if (!remoteTarget || !remoteCommandRunner || !expectedProviderPackManifest)
|
||||
return;
|
||||
await verifyRemoteProviderPackIdentity({ runner: remoteCommandRunner,
|
||||
@@ -12293,6 +12298,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
model: input.execution.provider.kind === "codex" ? input.execution.provider.model : null,
|
||||
reuseRetainedRunner: sandboxLeaseAcquisition?.outcome === "resumed", targetPlatformArtifactVerified: remotePiCompanion !== null || useBundledRemoteImageAssets || targetMatchedCodexArtifact || remoteClaude,
|
||||
trace: input.trace, onLog: input.onLog,
|
||||
providerPackRoots: requiresRemoteProviderPack && stagedRemoteProviderPackRoot
|
||||
? [stagedRemoteProviderPackRoot, "/opt/paperclip-runner/provider-pack"] : undefined,
|
||||
})
|
||||
: null;
|
||||
const verifyRemoteRunner = async (mode: "dial_wss" | "listen_ws", executable?: string | null) => {
|
||||
@@ -12300,6 +12307,33 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
};
|
||||
const verifyRemoteCodex = async () => { await remoteNativeArtifacts?.verifyRemoteCodex(); };
|
||||
|
||||
// A custom operator-selected pack is an explicit runtime selection. The
|
||||
// standard app-image path is the default transport source, not an upgrade request.
|
||||
const explicitProviderPackSelection = Boolean(input.runnerRemoteProviderPackPath?.trim()
|
||||
&& input.runnerRemoteProviderPackPath.trim() !== "/opt/paperclip-runner/provider-pack");
|
||||
const useInstalledProviderPack = async () => {
|
||||
const expected = expectedProviderPackManifest;
|
||||
if (!expected || !remoteTarget || !remoteCommandRunner) return false;
|
||||
for (const installation of remoteNativeArtifacts?.providerPackInventory() ?? []) {
|
||||
try {
|
||||
const manifest = await measureNativeRunnerSpan(input.trace, "provider_pack.compatibility", () => verifyRemoteProviderPackIdentity({
|
||||
runner: remoteCommandRunner, cwd: remoteTarget.remoteCwd, packRoot: installation.root,
|
||||
manifest: expected, installation,
|
||||
provider: input.execution.provider as { kind: "opencode" } | { kind: "acpx"; agent: string },
|
||||
requireExactIdentity: reattachProviderPack || explicitProviderPackSelection,
|
||||
}));
|
||||
if (!reattachProviderPack) providerPackAuthority!.select(manifest);
|
||||
expectedProviderPackManifest = manifest;
|
||||
activeRemoteProviderPackRoot = installation.root;
|
||||
providerPackVerified = true;
|
||||
return true;
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.name === "AbortError") throw error;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
const prepareRemoteRunner = async (
|
||||
requiredMode: "dial_wss" | "listen_ws",
|
||||
) => {
|
||||
@@ -12312,7 +12346,11 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
return;
|
||||
selectedRemoteMode = requiredMode;
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner" || input.remoteWarmRecovery) {
|
||||
await verifyRemoteRunner(requiredMode);
|
||||
await remoteNativeArtifacts?.verifyRemoteRunner(requiredMode, undefined, true);
|
||||
if (requiresRemoteProviderPack && await useInstalledProviderPack()) {
|
||||
remotePrepared = true;
|
||||
return;
|
||||
}
|
||||
if (requiresRemoteProviderPack && stagedRemoteProviderPackRoot) {
|
||||
await verifyRemoteProviderPack(stagedRemoteProviderPackRoot);
|
||||
activeRemoteProviderPackRoot = stagedRemoteProviderPackRoot;
|
||||
@@ -12321,6 +12359,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
return;
|
||||
}
|
||||
await remoteNativeArtifacts?.prepare(requiredMode);
|
||||
if (requiresRemoteProviderPack && await useInstalledProviderPack()) {
|
||||
remotePrepared = true;
|
||||
return;
|
||||
}
|
||||
if (
|
||||
requiresRemoteProviderPack &&
|
||||
expectedProviderPackManifest &&
|
||||
@@ -12383,8 +12425,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
"runner_remote_provider_artifact_incompatible: this remote transport cannot stage a provider pack; preinstall the exact manifest-matched pack",
|
||||
);
|
||||
}
|
||||
const archive = await measureNativeRunnerSpan(input.trace, "provider_pack.archive.verify", () =>
|
||||
readProviderPackArchive(configuredProviderPackRoot, expectedProviderPackManifest!.digest),
|
||||
const archive = await measureNativeRunnerSpan(input.trace, "provider_pack.archive.verify", () => {
|
||||
readRemoteProviderPackManifest(configuredProviderPackRoot);
|
||||
return readProviderPackArchive(configuredProviderPackRoot, expectedProviderPackManifest!.digest);
|
||||
},
|
||||
);
|
||||
const escapedPackRoot = stagedRemoteProviderPackRoot.replaceAll(
|
||||
"'",
|
||||
@@ -13365,6 +13409,11 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
if (!current) throw new Error("native_session_tool_authority_unavailable");
|
||||
return current.execute(call);
|
||||
};
|
||||
if (requiresRemoteProviderPack) {
|
||||
await prepareNativeRemoteRunner({ target, runnerIngressAuthorized: input.runnerIngressAuthorized === true,
|
||||
runnerPublicUrl: input.runnerPublicUrl, runnerCaBundlePath: input.runnerCaBundlePath,
|
||||
runId: input.execution.binding.runId, prepareRemote: prepareRemoteRunner });
|
||||
}
|
||||
const backend = createNativeSessionBackend(runnerExecution, {
|
||||
runnerInstanceId:
|
||||
retainedTransition?.runnerInstanceId ?? input.runnerInstanceId,
|
||||
@@ -13476,10 +13525,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
...(expectedProviderPackManifest && stagedRemoteProviderPackRoot
|
||||
...(expectedProviderPackManifest && activeRemoteProviderPackRoot
|
||||
? {
|
||||
providerNodeCommand: posix.join(
|
||||
stagedRemoteProviderPackRoot,
|
||||
activeRemoteProviderPackRoot,
|
||||
expectedProviderPackManifest.payload.artifacts.nodeCommand.path,
|
||||
),
|
||||
providerNodeCommandSha256:
|
||||
@@ -13487,7 +13536,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
.sha256,
|
||||
providerPackAuthorityDigest: expectedProviderPackManifest.digest,
|
||||
opencodeCommand: posix.join(
|
||||
stagedRemoteProviderPackRoot,
|
||||
activeRemoteProviderPackRoot,
|
||||
expectedProviderPackManifest.payload.artifacts
|
||||
.opencodeExecutable.path,
|
||||
),
|
||||
@@ -13495,7 +13544,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
expectedProviderPackManifest.payload.artifacts
|
||||
.opencodeExecutable.sha256,
|
||||
opencodeProxyPath: posix.join(
|
||||
stagedRemoteProviderPackRoot,
|
||||
activeRemoteProviderPackRoot,
|
||||
expectedProviderPackManifest.payload.artifacts.opencodeProxy
|
||||
.path,
|
||||
),
|
||||
@@ -13503,7 +13552,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
expectedProviderPackManifest.payload.artifacts.opencodeProxy
|
||||
.sha256,
|
||||
acpxSidecarPath: posix.join(
|
||||
stagedRemoteProviderPackRoot,
|
||||
activeRemoteProviderPackRoot,
|
||||
expectedProviderPackManifest.payload.artifacts.acpxSidecar.path,
|
||||
),
|
||||
acpxSidecarSha256:
|
||||
@@ -13768,6 +13817,7 @@ export function createRemoteNativeArtifactPreparation(input: {
|
||||
model?: string | null;
|
||||
reuseRetainedRunner?: boolean;
|
||||
targetPlatformArtifactVerified?: boolean;
|
||||
providerPackRoots?: readonly string[];
|
||||
trace?: NativeRunTrace;
|
||||
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
|
||||
}) {
|
||||
@@ -13778,18 +13828,25 @@ export function createRemoteNativeArtifactPreparation(input: {
|
||||
const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null;
|
||||
const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null;
|
||||
if (explicitRemoteCodex && remoteCodexNpmSpec) throw new Error("runner_remote_codex_source_conflict");
|
||||
let providerPackInventory: ProviderPackInstallation[] = [];
|
||||
const verifyRemoteRunner = async (
|
||||
requiredMode: "dial_wss" | "listen_ws",
|
||||
executable = remoteBinary,
|
||||
inspectProviderPacks = false,
|
||||
) => {
|
||||
if (!remoteTarget || !remoteCommandRunner || !executable) return;
|
||||
const metadataResult = await remoteCommandRunner.execute({
|
||||
let metadataResult = await remoteCommandRunner.execute({
|
||||
command: executable,
|
||||
args: ["--build-metadata"],
|
||||
args: ["--build-metadata", ...(inspectProviderPacks ? (input.providerPackRoots ?? []).flatMap(root => ["--provider-pack-root", root]) : [])],
|
||||
cwd: remoteTarget.remoteCwd,
|
||||
bypassSession: true,
|
||||
timeoutMs: 30_000,
|
||||
});
|
||||
if (inspectProviderPacks && input.providerPackRoots?.length && metadataResult.exitCode !== null
|
||||
&& metadataResult.exitCode !== 0 && !metadataResult.timedOut && !metadataResult.signal) {
|
||||
metadataResult = await remoteCommandRunner.execute({ command: executable, args: ["--build-metadata"],
|
||||
cwd: remoteTarget.remoteCwd, bypassSession: true, timeoutMs: 30_000 });
|
||||
}
|
||||
if (metadataResult.exitCode !== 0 || metadataResult.timedOut) {
|
||||
throw new Error("runner_remote_artifact_verification_failed");
|
||||
}
|
||||
@@ -13802,6 +13859,12 @@ export function createRemoteNativeArtifactPreparation(input: {
|
||||
});
|
||||
}
|
||||
assertRemoteRunnerBuildMetadata(metadata, requiredMode);
|
||||
if (inspectProviderPacks) {
|
||||
providerPackInventory = Array.isArray(metadata.providerPacks)
|
||||
? metadata.providerPacks.filter((pack): pack is ProviderPackInstallation =>
|
||||
typeof pack?.root === "string" && input.providerPackRoots?.includes(pack.root) === true
|
||||
&& typeof pack.immutable === "boolean") : [];
|
||||
}
|
||||
};
|
||||
|
||||
const reportedCodexVersions = new Set<string | null>();
|
||||
@@ -13982,7 +14045,7 @@ export function createRemoteNativeArtifactPreparation(input: {
|
||||
});
|
||||
}
|
||||
await measureNativeRunnerSpan(input.trace, "runner.artifact.verify", () =>
|
||||
verifyRemoteRunner(requiredMode),
|
||||
verifyRemoteRunner(requiredMode, undefined, true),
|
||||
);
|
||||
};
|
||||
const prepareHarness = async () => {
|
||||
@@ -14097,5 +14160,5 @@ export function createRemoteNativeArtifactPreparation(input: {
|
||||
await prepareHarness();
|
||||
}
|
||||
};
|
||||
return { prepare, verifyRemoteRunner, verifyRemoteCodex };
|
||||
return { prepare, verifyRemoteRunner, verifyRemoteCodex, providerPackInventory: () => providerPackInventory };
|
||||
}
|
||||
Reference in new issue
Block a user