mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 14:16:39 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts provider runtimes in remote execution environments. > - Sandbox images already contain qualified provider packs. > - Startup required the image pack to match the entire current app release manifest. > - An unrelated app commit could therefore cause pack verification, transfer, and installation again. > - This pull request selects protected packs by their provider interface and platform. > - The benefit is no extra pack probe, directory scan, version command, link, or upload on the compatible image path. ## Linked Issues or Issue Description **What happened?** Starting a remote task can reject a compatible image pack because its source revision or unrelated provider differs from the controller. Pack replacement added about 16 seconds in the investigated startup. Startup also repeated full content checks. **Expected behavior** Use an already qualified, protected pack when its selected provider interface and target platform are compatible. Keep the selected artifact identity for process launch and recovery. **Steps to reproduce** 1. Install a qualified provider pack in a sandbox image. 2. Start a controller built from another app commit. 3. Start a remote OpenCode task. The old complete-manifest comparison rejects the image pack even when the provider interface still works. Related repairs: #16493 reduced transfer cost. #17512 retained exact artifact authority during reattachment. This change preserves that recovery rule and changes fresh-launch pack selection. ## What Changed - Add explicit OpenCode and ACPX interface revisions to produced pack manifests. App Git SHAs and dependency pins remain provenance. - Read bounded protected-installation metadata in the existing final runner metadata query. Accept only root-owned paths that the execution user cannot write, including ancestors and manifest. Reject writable aliases. - Select and prepare the pack in server-owned environment setup before constructing the existing transport. Bind provider launch paths and digests from the selected installation. Save that identity for exact reattachment. - Keep full content verification for uploads, writable installations, and legacy packs. Reuse successful verification within one startup. Keep older attached runners compatible with the optional metadata request. - Seal image packs and document the one-time image refresh that enables reuse across later compatible releases. Keep selected ACPX agent qualification checks. ## Verification - Red/green regression: the old code rejects a protected pack from another app revision with `manifest mismatch`; the fixed path selects it without executing verification commands. - Native startup regression: select the installed pack's own Node digest with one existing runner metadata request and no pack verification or version commands. - Targeted pack, artifact, recovery, execution-driver, and Rust metadata tests pass. Incompatible interfaces/platforms, malformed identities, writable roots, path aliases, and exact reattachment are covered. - Final revision `91dc1676ff67bdbe5866c8586daac10240530cca`: workspace typecheck and build pass. All 53 current-head checks are successful, with no pending or failed checks. Greptile is 5/5 with no open findings. - The broad local `pnpm test:run` attempt reported a workspace-runtime conflict test failure; its isolated rerun passed. The broad run was stopped after that failure. Full CI now passes, including the server test shards. This is not a claim that the full local run passed. - Linux filesystem evidence: the production metadata functions read a real root-owned preinstalled pack as a non-root user in 16 microseconds median and 20 microseconds p95 across 100 warm reads. This measures metadata admission, not model or hosted response latency. - All 910 affected server tests and 57 Copilot attestation tests pass locally on the final implementation. Full CI covers the unchanged Runner transport. No provider qualification hash was refreshed. - Built the actual Linux daemon at `416e7ec7425d0d3565b4a5e1808002d650a60414` and ran its metadata command as UID 65534. The protected installation returned its real manifest with `immutable: true`; writable alias and missing-pack cases returned `immutable: false`. This is metadata/permission evidence, not provider execution. - Hosted rollout and a hosted end-to-end latency measurement have not run. ## Risks - Existing images need one rebuild to publish the compatibility declaration. Packs without it keep the legacy content-verification path. - Fast admission relies on image publication producing a qualified protected tree. Root execution and task-writable installations retain full verification. - Bump an interface revision when required runner/pack behavior changes incompatibly. Deploy a new image to roll out compatible runtime fixes or upgrades. - This removes provider-pack preparation delay. Other runner startup, context, model, and finalization costs remain separate. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol`, reasoning effort `xhigh`, with tool use and code execution. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
146 lines
8.9 KiB
TypeScript
146 lines
8.9 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { posix } from "node:path";
|
|
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
|
import { runRemoteProviderVerification } from "./remote-provider-pack.js";
|
|
|
|
export function canonicalProviderPackJson(value: unknown): string {
|
|
if (Array.isArray(value)) return `[${value.map(canonicalProviderPackJson).join(",")}]`;
|
|
if (value && typeof value === "object") return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b))
|
|
.map(([key, entry]) => `${JSON.stringify(key)}:${canonicalProviderPackJson(entry)}`).join(",")}}`;
|
|
return JSON.stringify(value) ?? "null";
|
|
}
|
|
|
|
export type ProviderPackIdentity = {
|
|
schema: string;
|
|
digest: string;
|
|
payload: {
|
|
compatibility?: { opencode: number; acpx: number };
|
|
acpxProfileDigests?: Record<string, string>;
|
|
providers?: Record<string, unknown>;
|
|
candidateProviders?: Record<string, unknown>;
|
|
pins: { opencode: string; [key: string]: string | null };
|
|
target: { platform: string; architecture: string };
|
|
runnerSourceRevision: string;
|
|
artifacts: { nodeCommand: { path: string; sha256: string }; opencodeCommand: { path: string; sha256: string }; [key: string]: { path: string; sha256: string } };
|
|
};
|
|
};
|
|
|
|
export type ProviderPackInstallation = { root: string; manifest: unknown; immutable: boolean };
|
|
export type PackProvider = { kind: "opencode" } | { kind: "acpx"; agent: string };
|
|
|
|
/** Admission of a build-qualified, OS-protected installation. Source revisions,
|
|
* dependency pins and unrelated provider changes are provenance, not an interface.
|
|
* The metadata comes from the trusted runner, never from a provider tool result.
|
|
*/
|
|
function compatibleInstallation<T extends ProviderPackIdentity>(
|
|
installation: ProviderPackInstallation, expected: T, packRoot: string,
|
|
provider: PackProvider, requireExactIdentity: boolean,
|
|
): T {
|
|
const actual = installation.manifest as T | null;
|
|
const payload = actual?.payload;
|
|
const contract = expected.payload.compatibility?.[provider.kind];
|
|
if (!installation.immutable || installation.root !== packRoot || !payload
|
|
|| actual.schema !== expected.schema
|
|
|| actual.digest !== `sha256:${createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex")}`
|
|
|| canonicalProviderPackJson(payload.target) !== canonicalProviderPackJson(expected.payload.target)
|
|
|| (requireExactIdentity ? actual.digest !== expected.digest
|
|
: !Number.isSafeInteger(contract) || contract! < 1 || payload.compatibility?.[provider.kind] !== contract)
|
|
|| typeof payload.pins?.opencode !== "string") {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: installed provider pack interface or installation is incompatible");
|
|
}
|
|
// Launch profiles and recovery receipts bind the selected installation's own
|
|
// bytes, never the current controller's artifact hashes.
|
|
for (const [name, expectedArtifact] of Object.entries(expected.payload.artifacts)) {
|
|
const artifact = payload.artifacts?.[name];
|
|
if (artifact?.path !== expectedArtifact.path || !/^sha256:[a-f0-9]{64}$/.test(artifact.sha256)) {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: installed provider pack artifact identity is invalid");
|
|
}
|
|
}
|
|
if (provider.kind === "acpx") {
|
|
const agent = provider.agent;
|
|
const currentProfile = expected.payload.acpxProfileDigests?.[agent];
|
|
const selected = expected.payload.providers?.[agent] ?? expected.payload.candidateProviders?.[agent];
|
|
const installed = payload.providers?.[agent] ?? payload.candidateProviders?.[agent];
|
|
// ACPX's existing qualified launch profiles remain authoritative for the
|
|
// selected agent. Other providers in the same pack do not invalidate it.
|
|
if (currentProfile ? payload.acpxProfileDigests?.[agent] !== currentProfile
|
|
: !selected || !installed || ["version", "profileDigest", "closureDigest", "qualification", "path"].some(key =>
|
|
(installed as Record<string, unknown>)[key] !== (selected as Record<string, unknown>)[key])) {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: selected ACPX provider profile is incompatible");
|
|
}
|
|
}
|
|
return structuredClone(actual);
|
|
}
|
|
|
|
export async function verifyRemoteProviderPack<T extends ProviderPackIdentity>({ runner, cwd, packRoot, manifest, installation, provider, requireExactIdentity = false }: {
|
|
runner: CommandManagedRuntimeRunner;
|
|
cwd: string;
|
|
packRoot: string;
|
|
manifest: T;
|
|
installation?: ProviderPackInstallation;
|
|
provider?: PackProvider;
|
|
requireExactIdentity?: boolean;
|
|
}): Promise<T> {
|
|
if (installation) {
|
|
if (!provider) throw new Error("runner_remote_provider_artifact_incompatible: provider interface is required");
|
|
return compatibleInstallation(installation, manifest, packRoot, provider, requireExactIdentity);
|
|
}
|
|
const expected = Buffer.from(
|
|
canonicalProviderPackJson(manifest),
|
|
"utf8",
|
|
).toString("base64");
|
|
const providerNodeCommand = posix.join(
|
|
packRoot,
|
|
manifest.payload.artifacts.nodeCommand.path,
|
|
);
|
|
const verifyScript = [
|
|
"const fs=require('node:fs')",
|
|
"const crypto=require('node:crypto')",
|
|
"const path=require('node:path')",
|
|
"const root=process.argv[1]",
|
|
"const expected=Buffer.from(process.argv[2],'base64').toString('utf8')",
|
|
"const actual=fs.readFileSync(path.join(root,'provider-pack.json'),'utf8').trim()",
|
|
"const canonical=(v)=>Array.isArray(v)?'['+v.map(canonical).join(',')+']':v&&typeof v==='object'?'{'+Object.keys(v).sort().map(k=>JSON.stringify(k)+':'+canonical(v[k])).join(',')+'}':JSON.stringify(v)",
|
|
"const manifest=JSON.parse(actual)",
|
|
"if(canonical(manifest)!==expected)throw new Error('manifest mismatch')",
|
|
"const hash=(p)=>'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(path.join(root,p))).digest('hex')",
|
|
"const tree=(treeRoot)=>{const digest=crypto.createHash('sha256');const visit=(directory,prefix='')=>{for(const entry of fs.readdirSync(directory,{withFileTypes:true}).sort((a,b)=>a.name.localeCompare(b.name))){const relative=prefix?prefix+'/'+entry.name:entry.name;const absolute=path.join(directory,entry.name);if(entry.isDirectory()){digest.update('directory\\0'+relative+'\\n');visit(absolute,relative)}else if(entry.isFile()){digest.update('file\\0'+relative+'\\0'+'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(absolute)).digest('hex')+'\\n')}else if(entry.isSymbolicLink()){digest.update('symlink\\0'+relative+'\\0'+fs.readlinkSync(absolute)+'\\n')}else throw new Error('unsupported dist entry '+relative)}};visit(treeRoot);return 'sha256:'+digest.digest('hex')}",
|
|
"for(const name of ['nodeCommand','productionLock','opencodeCommand','opencodeExecutable','opencodeProxy','acpxSidecar','grokLauncher']){const artifact=manifest.payload.artifacts[name];if(hash(artifact.path)!==artifact.sha256)throw new Error(name+' digest mismatch')}",
|
|
"if(tree(path.join(root,'dist'))!==manifest.payload.distDigest)throw new Error('dist tree digest mismatch')",
|
|
"for(const candidate of Object.values({...manifest.payload.providers,...manifest.payload.candidateProviders})){if(tree(path.join(root,candidate.path))!==candidate.sha256)throw new Error('candidate asset tree digest mismatch')}",
|
|
"const version=process.versions.node.split('.').map(Number)",
|
|
"const minimum=manifest.payload.pins.nodeMinimum.split('.').map(Number)",
|
|
"if(version[0]<minimum[0]||(version[0]===minimum[0]&&(version[1]<minimum[1]||(version[1]===minimum[1]&&version[2]<minimum[2]))))throw new Error('Node version incompatible')",
|
|
"if(process.platform!==manifest.payload.target.platform||process.arch!==manifest.payload.target.architecture)throw new Error('provider pack target mismatch')",
|
|
"const packageVersion=(pkg)=>JSON.parse(fs.readFileSync(path.join(root,'node_modules',...pkg.split('/'),'package.json'),'utf8')).version",
|
|
"const expectedPackages={acpx:manifest.payload.pins.acpx,'@agentclientprotocol/claude-agent-acp':manifest.payload.pins.claudeAcp,'@agentclientprotocol/codex-acp':manifest.payload.pins.codexAcp,'opencode-ai':manifest.payload.pins.opencode}",
|
|
"for(const [pkg,version] of Object.entries(expectedPackages))if(packageVersion(pkg)!==version)throw new Error(pkg+' version mismatch')",
|
|
].join(";");
|
|
await runRemoteProviderVerification(() => runner.execute({
|
|
command: providerNodeCommand,
|
|
args: ["-e", verifyScript, packRoot, expected],
|
|
cwd: cwd,
|
|
bypassSession: true,
|
|
timeoutMs: 30_000,
|
|
}), "provider pack verification");
|
|
const opencodeCommand = posix.join(
|
|
packRoot,
|
|
manifest.payload.artifacts.opencodeCommand.path,
|
|
);
|
|
const opencodeVersion = await runRemoteProviderVerification(() => runner.execute({
|
|
command: opencodeCommand,
|
|
args: ["--version"],
|
|
cwd: cwd,
|
|
bypassSession: true,
|
|
timeoutMs: 30_000,
|
|
}), "OpenCode version verification");
|
|
if (
|
|
opencodeVersion.stdout.trim() !== manifest.payload.pins.opencode
|
|
) {
|
|
throw new Error(
|
|
"runner_remote_provider_artifact_incompatible: OpenCode version mismatch",
|
|
);
|
|
}
|
|
return structuredClone(manifest);
|
|
}
|