Files
PaperClipAI/server/src/services/native-runtime/remote-provider-pack-verification.ts
T
DottaandPaperclip 9ae4c25e7e perf(runner): reuse compatible protected provider packs (#17749)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner starts provider runtimes in remote execution
environments.
> - Sandbox images already contain qualified provider packs.
> - Startup required the image pack to match the entire current app
release manifest.
> - An unrelated app commit could therefore cause pack verification,
transfer, and installation again.
> - This pull request selects protected packs by their provider
interface and platform.
> - The benefit is no extra pack probe, directory scan, version command,
link, or upload on the compatible image path.

## Linked Issues or Issue Description

**What happened?**

Starting a remote task can reject a compatible image pack because its
source revision or unrelated provider differs from the controller. Pack
replacement added about 16 seconds in the investigated startup. Startup
also repeated full content checks.

**Expected behavior**

Use an already qualified, protected pack when its selected provider
interface and target platform are compatible. Keep the selected artifact
identity for process launch and recovery.

**Steps to reproduce**

1. Install a qualified provider pack in a sandbox image.
2. Start a controller built from another app commit.
3. Start a remote OpenCode task. The old complete-manifest comparison
rejects the image pack even when the provider interface still works.

Related repairs: #16493 reduced transfer cost. #17512 retained exact
artifact authority during reattachment. This change preserves that
recovery rule and changes fresh-launch pack selection.

## What Changed

- Add explicit OpenCode and ACPX interface revisions to produced pack
manifests. App Git SHAs and dependency pins remain provenance.
- Read bounded protected-installation metadata in the existing final
runner metadata query. Accept only root-owned paths that the execution
user cannot write, including ancestors and manifest. Reject writable
aliases.
- Select and prepare the pack in server-owned environment setup before
constructing the existing transport. Bind provider launch paths and
digests from the selected installation. Save that identity for exact
reattachment.
- Keep full content verification for uploads, writable installations,
and legacy packs. Reuse successful verification within one startup. Keep
older attached runners compatible with the optional metadata request.
- Seal image packs and document the one-time image refresh that enables
reuse across later compatible releases. Keep selected ACPX agent
qualification checks.

## Verification

- Red/green regression: the old code rejects a protected pack from
another app revision with `manifest mismatch`; the fixed path selects it
without executing verification commands.
- Native startup regression: select the installed pack's own Node digest
with one existing runner metadata request and no pack verification or
version commands.
- Targeted pack, artifact, recovery, execution-driver, and Rust metadata
tests pass. Incompatible interfaces/platforms, malformed identities,
writable roots, path aliases, and exact reattachment are covered.
- Final revision `91dc1676ff67bdbe5866c8586daac10240530cca`: workspace
typecheck and build pass. All 53 current-head checks are successful,
with no pending or failed checks. Greptile is 5/5 with no open findings.
- The broad local `pnpm test:run` attempt reported a workspace-runtime
conflict test failure; its isolated rerun passed. The broad run was
stopped after that failure. Full CI now passes, including the server
test shards. This is not a claim that the full local run passed.
- Linux filesystem evidence: the production metadata functions read a
real root-owned preinstalled pack as a non-root user in 16 microseconds
median and 20 microseconds p95 across 100 warm reads. This measures
metadata admission, not model or hosted response latency.
- All 910 affected server tests and 57 Copilot attestation tests pass
locally on the final implementation. Full CI covers the unchanged Runner
transport. No provider qualification hash was refreshed.
- Built the actual Linux daemon at
`416e7ec7425d0d3565b4a5e1808002d650a60414` and ran its metadata command
as UID 65534. The protected installation returned its real manifest with
`immutable: true`; writable alias and missing-pack cases returned
`immutable: false`. This is metadata/permission evidence, not provider
execution.
- Hosted rollout and a hosted end-to-end latency measurement have not
run.

## Risks

- Existing images need one rebuild to publish the compatibility
declaration. Packs without it keep the legacy content-verification path.
- Fast admission relies on image publication producing a qualified
protected tree. Root execution and task-writable installations retain
full verification.
- Bump an interface revision when required runner/pack behavior changes
incompatibly. Deploy a new image to roll out compatible runtime fixes or
upgrades.
- This removes provider-pack preparation delay. Other runner startup,
context, model, and finalization costs remain separate.

## Model Used

OpenAI Codex, configured model `gpt-6.1-sol`, reasoning effort `xhigh`,
with tool use and code execution. The context-window size is not exposed
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 14:43:12 -05:00

146 lines
8.9 KiB
TypeScript

import { createHash } from "node:crypto";
import { posix } from "node:path";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
import { runRemoteProviderVerification } from "./remote-provider-pack.js";
export function canonicalProviderPackJson(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(canonicalProviderPackJson).join(",")}]`;
if (value && typeof value === "object") return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b))
.map(([key, entry]) => `${JSON.stringify(key)}:${canonicalProviderPackJson(entry)}`).join(",")}}`;
return JSON.stringify(value) ?? "null";
}
export type ProviderPackIdentity = {
schema: string;
digest: string;
payload: {
compatibility?: { opencode: number; acpx: number };
acpxProfileDigests?: Record<string, string>;
providers?: Record<string, unknown>;
candidateProviders?: Record<string, unknown>;
pins: { opencode: string; [key: string]: string | null };
target: { platform: string; architecture: string };
runnerSourceRevision: string;
artifacts: { nodeCommand: { path: string; sha256: string }; opencodeCommand: { path: string; sha256: string }; [key: string]: { path: string; sha256: string } };
};
};
export type ProviderPackInstallation = { root: string; manifest: unknown; immutable: boolean };
export type PackProvider = { kind: "opencode" } | { kind: "acpx"; agent: string };
/** Admission of a build-qualified, OS-protected installation. Source revisions,
* dependency pins and unrelated provider changes are provenance, not an interface.
* The metadata comes from the trusted runner, never from a provider tool result.
*/
function compatibleInstallation<T extends ProviderPackIdentity>(
installation: ProviderPackInstallation, expected: T, packRoot: string,
provider: PackProvider, requireExactIdentity: boolean,
): T {
const actual = installation.manifest as T | null;
const payload = actual?.payload;
const contract = expected.payload.compatibility?.[provider.kind];
if (!installation.immutable || installation.root !== packRoot || !payload
|| actual.schema !== expected.schema
|| actual.digest !== `sha256:${createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex")}`
|| canonicalProviderPackJson(payload.target) !== canonicalProviderPackJson(expected.payload.target)
|| (requireExactIdentity ? actual.digest !== expected.digest
: !Number.isSafeInteger(contract) || contract! < 1 || payload.compatibility?.[provider.kind] !== contract)
|| typeof payload.pins?.opencode !== "string") {
throw new Error("runner_remote_provider_artifact_incompatible: installed provider pack interface or installation is incompatible");
}
// Launch profiles and recovery receipts bind the selected installation's own
// bytes, never the current controller's artifact hashes.
for (const [name, expectedArtifact] of Object.entries(expected.payload.artifacts)) {
const artifact = payload.artifacts?.[name];
if (artifact?.path !== expectedArtifact.path || !/^sha256:[a-f0-9]{64}$/.test(artifact.sha256)) {
throw new Error("runner_remote_provider_artifact_incompatible: installed provider pack artifact identity is invalid");
}
}
if (provider.kind === "acpx") {
const agent = provider.agent;
const currentProfile = expected.payload.acpxProfileDigests?.[agent];
const selected = expected.payload.providers?.[agent] ?? expected.payload.candidateProviders?.[agent];
const installed = payload.providers?.[agent] ?? payload.candidateProviders?.[agent];
// ACPX's existing qualified launch profiles remain authoritative for the
// selected agent. Other providers in the same pack do not invalidate it.
if (currentProfile ? payload.acpxProfileDigests?.[agent] !== currentProfile
: !selected || !installed || ["version", "profileDigest", "closureDigest", "qualification", "path"].some(key =>
(installed as Record<string, unknown>)[key] !== (selected as Record<string, unknown>)[key])) {
throw new Error("runner_remote_provider_artifact_incompatible: selected ACPX provider profile is incompatible");
}
}
return structuredClone(actual);
}
export async function verifyRemoteProviderPack<T extends ProviderPackIdentity>({ runner, cwd, packRoot, manifest, installation, provider, requireExactIdentity = false }: {
runner: CommandManagedRuntimeRunner;
cwd: string;
packRoot: string;
manifest: T;
installation?: ProviderPackInstallation;
provider?: PackProvider;
requireExactIdentity?: boolean;
}): Promise<T> {
if (installation) {
if (!provider) throw new Error("runner_remote_provider_artifact_incompatible: provider interface is required");
return compatibleInstallation(installation, manifest, packRoot, provider, requireExactIdentity);
}
const expected = Buffer.from(
canonicalProviderPackJson(manifest),
"utf8",
).toString("base64");
const providerNodeCommand = posix.join(
packRoot,
manifest.payload.artifacts.nodeCommand.path,
);
const verifyScript = [
"const fs=require('node:fs')",
"const crypto=require('node:crypto')",
"const path=require('node:path')",
"const root=process.argv[1]",
"const expected=Buffer.from(process.argv[2],'base64').toString('utf8')",
"const actual=fs.readFileSync(path.join(root,'provider-pack.json'),'utf8').trim()",
"const canonical=(v)=>Array.isArray(v)?'['+v.map(canonical).join(',')+']':v&&typeof v==='object'?'{'+Object.keys(v).sort().map(k=>JSON.stringify(k)+':'+canonical(v[k])).join(',')+'}':JSON.stringify(v)",
"const manifest=JSON.parse(actual)",
"if(canonical(manifest)!==expected)throw new Error('manifest mismatch')",
"const hash=(p)=>'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(path.join(root,p))).digest('hex')",
"const tree=(treeRoot)=>{const digest=crypto.createHash('sha256');const visit=(directory,prefix='')=>{for(const entry of fs.readdirSync(directory,{withFileTypes:true}).sort((a,b)=>a.name.localeCompare(b.name))){const relative=prefix?prefix+'/'+entry.name:entry.name;const absolute=path.join(directory,entry.name);if(entry.isDirectory()){digest.update('directory\\0'+relative+'\\n');visit(absolute,relative)}else if(entry.isFile()){digest.update('file\\0'+relative+'\\0'+'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(absolute)).digest('hex')+'\\n')}else if(entry.isSymbolicLink()){digest.update('symlink\\0'+relative+'\\0'+fs.readlinkSync(absolute)+'\\n')}else throw new Error('unsupported dist entry '+relative)}};visit(treeRoot);return 'sha256:'+digest.digest('hex')}",
"for(const name of ['nodeCommand','productionLock','opencodeCommand','opencodeExecutable','opencodeProxy','acpxSidecar','grokLauncher']){const artifact=manifest.payload.artifacts[name];if(hash(artifact.path)!==artifact.sha256)throw new Error(name+' digest mismatch')}",
"if(tree(path.join(root,'dist'))!==manifest.payload.distDigest)throw new Error('dist tree digest mismatch')",
"for(const candidate of Object.values({...manifest.payload.providers,...manifest.payload.candidateProviders})){if(tree(path.join(root,candidate.path))!==candidate.sha256)throw new Error('candidate asset tree digest mismatch')}",
"const version=process.versions.node.split('.').map(Number)",
"const minimum=manifest.payload.pins.nodeMinimum.split('.').map(Number)",
"if(version[0]<minimum[0]||(version[0]===minimum[0]&&(version[1]<minimum[1]||(version[1]===minimum[1]&&version[2]<minimum[2]))))throw new Error('Node version incompatible')",
"if(process.platform!==manifest.payload.target.platform||process.arch!==manifest.payload.target.architecture)throw new Error('provider pack target mismatch')",
"const packageVersion=(pkg)=>JSON.parse(fs.readFileSync(path.join(root,'node_modules',...pkg.split('/'),'package.json'),'utf8')).version",
"const expectedPackages={acpx:manifest.payload.pins.acpx,'@agentclientprotocol/claude-agent-acp':manifest.payload.pins.claudeAcp,'@agentclientprotocol/codex-acp':manifest.payload.pins.codexAcp,'opencode-ai':manifest.payload.pins.opencode}",
"for(const [pkg,version] of Object.entries(expectedPackages))if(packageVersion(pkg)!==version)throw new Error(pkg+' version mismatch')",
].join(";");
await runRemoteProviderVerification(() => runner.execute({
command: providerNodeCommand,
args: ["-e", verifyScript, packRoot, expected],
cwd: cwd,
bypassSession: true,
timeoutMs: 30_000,
}), "provider pack verification");
const opencodeCommand = posix.join(
packRoot,
manifest.payload.artifacts.opencodeCommand.path,
);
const opencodeVersion = await runRemoteProviderVerification(() => runner.execute({
command: opencodeCommand,
args: ["--version"],
cwd: cwd,
bypassSession: true,
timeoutMs: 30_000,
}), "OpenCode version verification");
if (
opencodeVersion.stdout.trim() !== manifest.payload.pins.opencode
) {
throw new Error(
"runner_remote_provider_artifact_incompatible: OpenCode version mismatch",
);
}
return structuredClone(manifest);
}