mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 19:35:13 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts provider runtimes in remote execution environments. > - Sandbox images already contain qualified provider packs. > - Startup required the image pack to match the entire current app release manifest. > - An unrelated app commit could therefore cause pack verification, transfer, and installation again. > - This pull request selects protected packs by their provider interface and platform. > - The benefit is no extra pack probe, directory scan, version command, link, or upload on the compatible image path. ## Linked Issues or Issue Description **What happened?** Starting a remote task can reject a compatible image pack because its source revision or unrelated provider differs from the controller. Pack replacement added about 16 seconds in the investigated startup. Startup also repeated full content checks. **Expected behavior** Use an already qualified, protected pack when its selected provider interface and target platform are compatible. Keep the selected artifact identity for process launch and recovery. **Steps to reproduce** 1. Install a qualified provider pack in a sandbox image. 2. Start a controller built from another app commit. 3. Start a remote OpenCode task. The old complete-manifest comparison rejects the image pack even when the provider interface still works. Related repairs: #16493 reduced transfer cost. #17512 retained exact artifact authority during reattachment. This change preserves that recovery rule and changes fresh-launch pack selection. ## What Changed - Add explicit OpenCode and ACPX interface revisions to produced pack manifests. App Git SHAs and dependency pins remain provenance. - Read bounded protected-installation metadata in the existing final runner metadata query. Accept only root-owned paths that the execution user cannot write, including ancestors and manifest. Reject writable aliases. - Select and prepare the pack in server-owned environment setup before constructing the existing transport. Bind provider launch paths and digests from the selected installation. Save that identity for exact reattachment. - Keep full content verification for uploads, writable installations, and legacy packs. Reuse successful verification within one startup. Keep older attached runners compatible with the optional metadata request. - Seal image packs and document the one-time image refresh that enables reuse across later compatible releases. Keep selected ACPX agent qualification checks. ## Verification - Red/green regression: the old code rejects a protected pack from another app revision with `manifest mismatch`; the fixed path selects it without executing verification commands. - Native startup regression: select the installed pack's own Node digest with one existing runner metadata request and no pack verification or version commands. - Targeted pack, artifact, recovery, execution-driver, and Rust metadata tests pass. Incompatible interfaces/platforms, malformed identities, writable roots, path aliases, and exact reattachment are covered. - Final revision `91dc1676ff67bdbe5866c8586daac10240530cca`: workspace typecheck and build pass. All 53 current-head checks are successful, with no pending or failed checks. Greptile is 5/5 with no open findings. - The broad local `pnpm test:run` attempt reported a workspace-runtime conflict test failure; its isolated rerun passed. The broad run was stopped after that failure. Full CI now passes, including the server test shards. This is not a claim that the full local run passed. - Linux filesystem evidence: the production metadata functions read a real root-owned preinstalled pack as a non-root user in 16 microseconds median and 20 microseconds p95 across 100 warm reads. This measures metadata admission, not model or hosted response latency. - All 910 affected server tests and 57 Copilot attestation tests pass locally on the final implementation. Full CI covers the unchanged Runner transport. No provider qualification hash was refreshed. - Built the actual Linux daemon at `416e7ec7425d0d3565b4a5e1808002d650a60414` and ran its metadata command as UID 65534. The protected installation returned its real manifest with `immutable: true`; writable alias and missing-pack cases returned `immutable: false`. This is metadata/permission evidence, not provider execution. - Hosted rollout and a hosted end-to-end latency measurement have not run. ## Risks - Existing images need one rebuild to publish the compatibility declaration. Packs without it keep the legacy content-verification path. - Fast admission relies on image publication producing a qualified protected tree. Root execution and task-writable installations retain full verification. - Bump an interface revision when required runner/pack behavior changes incompatibly. Deploy a new image to roll out compatible runtime fixes or upgrades. - This removes provider-pack preparation delay. Other runner startup, context, model, and finalization costs remain separate. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol`, reasoning effort `xhigh`, with tool use and code execution. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
119 lines
5.0 KiB
TypeScript
119 lines
5.0 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { closeSync, constants, fstatSync, lstatSync, openSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
import { join } from "node:path";
|
|
import { redactSensitiveText } from "../../redaction.js";
|
|
import { canonicalProviderPackJson, type ProviderPackIdentity } from "./remote-provider-pack-verification.js";
|
|
|
|
export type ProviderPackBinding = {
|
|
companyId: string;
|
|
agentId: string;
|
|
workspaceId: string;
|
|
sessionId: string;
|
|
runnerInstanceId: string;
|
|
environmentLeaseId: string;
|
|
remoteCwd: string;
|
|
providerDigest: string;
|
|
};
|
|
export type RunnerArtifactIdentity = { version: string; digest: string };
|
|
|
|
const schema = "paperclip.remote-provider-pack-authority.v1";
|
|
const filename = "provider-pack-authority.json";
|
|
const maximumBytes = 512 * 1024;
|
|
|
|
function assertManifest(manifest: ProviderPackIdentity): void {
|
|
if (manifest?.schema !== "paperclip-runner/remote-provider-pack/v1" || !manifest.payload
|
|
|| manifest.digest !== `sha256:${createHash("sha256").update(canonicalProviderPackJson(manifest.payload)).digest("hex")}`) {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: saved provider pack authority is invalid");
|
|
}
|
|
}
|
|
|
|
export function resolveRemoteProviderPackAuthority<T extends ProviderPackIdentity>(input: {
|
|
stateDirectory: string;
|
|
binding: ProviderPackBinding;
|
|
reattach: boolean;
|
|
current: () => T;
|
|
runnerArtifact: () => RunnerArtifactIdentity;
|
|
}): { manifest: T; runnerArtifact: RunnerArtifactIdentity; select: (manifest: T) => void; verified: () => void } {
|
|
const root = lstatSync(input.stateDirectory);
|
|
if (!root.isDirectory() || root.isSymbolicLink()) {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: provider pack authority directory is unsafe");
|
|
}
|
|
const path = join(input.stateDirectory, filename);
|
|
let manifest: T | undefined;
|
|
let runnerArtifact: RunnerArtifactIdentity | undefined;
|
|
if (input.reattach) {
|
|
let fd: number | undefined;
|
|
try {
|
|
fd = openSync(path, constants.O_RDONLY | constants.O_NOFOLLOW);
|
|
const stat = fstatSync(fd);
|
|
if (!stat.isFile() || stat.size > maximumBytes) throw new Error("invalid authority file");
|
|
const saved = JSON.parse(readFileSync(fd, "utf8"));
|
|
if (saved.schema !== schema || canonicalProviderPackJson(saved.binding) !== canonicalProviderPackJson(input.binding)) {
|
|
throw new Error("authority binding mismatch");
|
|
}
|
|
assertManifest(saved.manifest);
|
|
if (typeof saved.runnerArtifact?.version !== "string" || !saved.runnerArtifact.version
|
|
|| !/^sha256:[a-f0-9]{64}$/.test(saved.runnerArtifact.digest)) throw new Error("invalid runner artifact identity");
|
|
manifest = saved.manifest as T;
|
|
runnerArtifact = saved.runnerArtifact;
|
|
} catch (error) {
|
|
if ((error as NodeJS.ErrnoException).code !== "ENOENT") {
|
|
throw new Error("runner_remote_provider_artifact_incompatible: saved provider pack authority is invalid or does not match this execution", { cause: error });
|
|
}
|
|
} finally {
|
|
if (fd !== undefined) closeSync(fd);
|
|
}
|
|
}
|
|
manifest ??= input.current();
|
|
assertManifest(manifest);
|
|
runnerArtifact ??= input.runnerArtifact();
|
|
const selectedRunner = structuredClone(runnerArtifact);
|
|
let selected = structuredClone(manifest);
|
|
const serialize = (manifest: T) => {
|
|
const receipt = JSON.stringify({ schema, binding: input.binding, manifest, runnerArtifact: selectedRunner });
|
|
if (Buffer.byteLength(receipt) > maximumBytes) throw new Error("runner_remote_provider_artifact_incompatible: provider pack authority exceeds its size limit");
|
|
return receipt;
|
|
};
|
|
let receipt = serialize(selected);
|
|
return {
|
|
get manifest() { return selected; },
|
|
select: (manifest: T) => {
|
|
if (input.reattach) throw new Error("runner_remote_provider_artifact_incompatible: cannot replace an attached pack authority");
|
|
assertManifest(manifest);
|
|
const next = structuredClone(manifest);
|
|
receipt = serialize(next);
|
|
selected = next;
|
|
},
|
|
runnerArtifact: selectedRunner,
|
|
verified: () => {
|
|
const temporary = `${path}.${randomUUID()}.tmp`;
|
|
try {
|
|
writeFileSync(temporary, receipt, { flag: "wx", mode: 0o600 });
|
|
renameSync(temporary, path);
|
|
} finally {
|
|
rmSync(temporary, { force: true });
|
|
}
|
|
},
|
|
};
|
|
}
|
|
|
|
export function providerPackWorkspaceId(workspaceId: string, runId: string, environmentLeaseId: string): string {
|
|
return workspaceId === runId ? environmentLeaseId : workspaceId;
|
|
}
|
|
|
|
export async function recordAdmittedRemoteProviderPack(
|
|
authority: { verified: () => void },
|
|
onWarning: (message: string) => void | Promise<void>,
|
|
): Promise<boolean> {
|
|
try {
|
|
authority.verified();
|
|
return true;
|
|
} catch (error) {
|
|
const reason = redactSensitiveText(error instanceof Error ? error.message : String(error)).slice(-1_024);
|
|
try {
|
|
await onWarning(`[paperclip-runner] could not save remote release authority; recovery still requires a verified receipt or current release artifacts: ${reason}\n`);
|
|
} catch {}
|
|
return false;
|
|
}
|
|
}
|