mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts provider runtimes in remote execution environments. > - Sandbox images already contain qualified provider packs. > - Startup required the image pack to match the entire current app release manifest. > - An unrelated app commit could therefore cause pack verification, transfer, and installation again. > - This pull request selects protected packs by their provider interface and platform. > - The benefit is no extra pack probe, directory scan, version command, link, or upload on the compatible image path. ## Linked Issues or Issue Description **What happened?** Starting a remote task can reject a compatible image pack because its source revision or unrelated provider differs from the controller. Pack replacement added about 16 seconds in the investigated startup. Startup also repeated full content checks. **Expected behavior** Use an already qualified, protected pack when its selected provider interface and target platform are compatible. Keep the selected artifact identity for process launch and recovery. **Steps to reproduce** 1. Install a qualified provider pack in a sandbox image. 2. Start a controller built from another app commit. 3. Start a remote OpenCode task. The old complete-manifest comparison rejects the image pack even when the provider interface still works. Related repairs: #16493 reduced transfer cost. #17512 retained exact artifact authority during reattachment. This change preserves that recovery rule and changes fresh-launch pack selection. ## What Changed - Add explicit OpenCode and ACPX interface revisions to produced pack manifests. App Git SHAs and dependency pins remain provenance. - Read bounded protected-installation metadata in the existing final runner metadata query. Accept only root-owned paths that the execution user cannot write, including ancestors and manifest. Reject writable aliases. - Select and prepare the pack in server-owned environment setup before constructing the existing transport. Bind provider launch paths and digests from the selected installation. Save that identity for exact reattachment. - Keep full content verification for uploads, writable installations, and legacy packs. Reuse successful verification within one startup. Keep older attached runners compatible with the optional metadata request. - Seal image packs and document the one-time image refresh that enables reuse across later compatible releases. Keep selected ACPX agent qualification checks. ## Verification - Red/green regression: the old code rejects a protected pack from another app revision with `manifest mismatch`; the fixed path selects it without executing verification commands. - Native startup regression: select the installed pack's own Node digest with one existing runner metadata request and no pack verification or version commands. - Targeted pack, artifact, recovery, execution-driver, and Rust metadata tests pass. Incompatible interfaces/platforms, malformed identities, writable roots, path aliases, and exact reattachment are covered. - Final revision `91dc1676ff67bdbe5866c8586daac10240530cca`: workspace typecheck and build pass. All 53 current-head checks are successful, with no pending or failed checks. Greptile is 5/5 with no open findings. - The broad local `pnpm test:run` attempt reported a workspace-runtime conflict test failure; its isolated rerun passed. The broad run was stopped after that failure. Full CI now passes, including the server test shards. This is not a claim that the full local run passed. - Linux filesystem evidence: the production metadata functions read a real root-owned preinstalled pack as a non-root user in 16 microseconds median and 20 microseconds p95 across 100 warm reads. This measures metadata admission, not model or hosted response latency. - All 910 affected server tests and 57 Copilot attestation tests pass locally on the final implementation. Full CI covers the unchanged Runner transport. No provider qualification hash was refreshed. - Built the actual Linux daemon at `416e7ec7425d0d3565b4a5e1808002d650a60414` and ran its metadata command as UID 65534. The protected installation returned its real manifest with `immutable: true`; writable alias and missing-pack cases returned `immutable: false`. This is metadata/permission evidence, not provider execution. - Hosted rollout and a hosted end-to-end latency measurement have not run. ## Risks - Existing images need one rebuild to publish the compatibility declaration. Packs without it keep the legacy content-verification path. - Fast admission relies on image publication producing a qualified protected tree. Root execution and task-writable installations retain full verification. - Bump an interface revision when required runner/pack behavior changes incompatibly. Deploy a new image to roll out compatible runtime fixes or upgrades. - This removes provider-pack preparation delay. Other runner startup, context, model, and finalization costs remain separate. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol`, reasoning effort `xhigh`, with tool use and code execution. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
133 lines
8.3 KiB
TypeScript
133 lines
8.3 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { mkdtempSync, mkdirSync, rmSync, readFileSync, writeFileSync, symlinkSync, statSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { afterEach, expect, it, vi } from "vitest";
|
|
import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
|
|
import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js";
|
|
|
|
const directories: string[] = [];
|
|
afterEach(() => { for (const dir of directories.splice(0)) rmSync(dir, { recursive: true, force: true }); });
|
|
const binding = { companyId: "company", agentId: "agent", workspaceId: "workspace", sessionId: "session",
|
|
runnerInstanceId: "runner", environmentLeaseId: "lease", remoteCwd: "/workspace", providerDigest: "provider" };
|
|
const runnerArtifact = () => ({ version: "0.0.0", digest: "sha256:" + "c".repeat(64) });
|
|
const manifest = (revision: string) => {
|
|
const payload = { runnerSourceRevision: revision.repeat(40), target: { platform: "linux", architecture: "x64" },
|
|
pins: { opencode: "1.18.34" }, artifacts: { nodeCommand: { path: "node", sha256: "sha256:" + "1".repeat(64) },
|
|
opencodeCommand: { path: "opencode", sha256: "sha256:" + "2".repeat(64) } } };
|
|
return { schema: "paperclip-runner/remote-provider-pack/v1", payload,
|
|
digest: "sha256:" + createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex") };
|
|
};
|
|
it("reattaches with the verified launch authority after the controller release changes", () => {
|
|
const stateDirectory = mkdtempSync(join(tmpdir(), "provider-pack-authority-")); directories.push(stateDirectory);
|
|
const original = manifest("a"), next = manifest("b");
|
|
const launched = resolveRemoteProviderPackAuthority({ stateDirectory, binding, reattach: false, runnerArtifact, current: () => original });
|
|
launched.verified();
|
|
const current = vi.fn(() => next);
|
|
const recovered = resolveRemoteProviderPackAuthority({ stateDirectory, binding, reattach: true, current, runnerArtifact });
|
|
expect(recovered.manifest).toEqual(original);
|
|
expect(recovered.runnerArtifact).toEqual(runnerArtifact());
|
|
expect(current).not.toHaveBeenCalled();
|
|
});
|
|
|
|
function fixture() {
|
|
const stateDirectory = mkdtempSync(join(tmpdir(), "provider-pack-authority-")); directories.push(stateDirectory);
|
|
const current = vi.fn(() => manifest("a"));
|
|
const input = { stateDirectory, binding, reattach: false, current, runnerArtifact };
|
|
return { input, path: join(stateDirectory, "provider-pack-authority.json") };
|
|
}
|
|
it("uses the new release for a fresh launch without changing the old receipt until verification", () => {
|
|
const { input, path } = fixture();
|
|
resolveRemoteProviderPackAuthority(input).verified();
|
|
input.current.mockReturnValue(manifest("b"));
|
|
const fresh = resolveRemoteProviderPackAuthority(input);
|
|
expect(fresh.manifest).toEqual(manifest("b"));
|
|
expect(JSON.parse(readFileSync(path, "utf8")).manifest).toEqual(manifest("a"));
|
|
fresh.verified();
|
|
expect(JSON.parse(readFileSync(path, "utf8")).manifest).toEqual(manifest("b"));
|
|
expect(statSync(path).mode & 0o777).toBe(0o600);
|
|
});
|
|
it("keeps historical sessions without a receipt at the current release integrity gate", () => {
|
|
const { input } = fixture();
|
|
const recovered = resolveRemoteProviderPackAuthority({ ...input, reattach: true });
|
|
expect(recovered.manifest).toEqual(manifest("a"));
|
|
expect(input.current).toHaveBeenCalledOnce();
|
|
});
|
|
it.each(Object.keys(binding) as (keyof typeof binding)[])("rejects a saved authority for a different %s", field => {
|
|
const { input } = fixture();
|
|
resolveRemoteProviderPackAuthority(input).verified();
|
|
expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true,
|
|
binding: { ...binding, [field]: "other" } })).toThrow("does not match this execution");
|
|
expect(input.current).toHaveBeenCalledOnce();
|
|
});
|
|
it.each(["digest", "schema", "symlink", "oversized", "invalid_json"])("fails closed on %s instead of trusting remote or current bytes", kind => {
|
|
const { input, path } = fixture();
|
|
resolveRemoteProviderPackAuthority(input).verified();
|
|
const saved = JSON.parse(readFileSync(path, "utf8"));
|
|
if (kind === "digest") saved.manifest.payload.runnerSourceRevision = "c".repeat(40);
|
|
if (kind === "schema") saved.schema = "unknown";
|
|
writeFileSync(path, JSON.stringify(saved));
|
|
if (kind === "invalid_json") writeFileSync(path, "{");
|
|
if (kind === "oversized") writeFileSync(path, " ".repeat(512 * 1024 + 1));
|
|
if (kind === "symlink") { const copy = path + ".copy"; writeFileSync(copy, JSON.stringify(saved)); rmSync(path); symlinkSync(copy, path); }
|
|
expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true })).toThrow("saved provider pack authority");
|
|
expect(input.current).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("uses current authority for a replacement lease instead of restoring the prior pack", () => {
|
|
const { input, path } = fixture();
|
|
resolveRemoteProviderPackAuthority(input).verified();
|
|
const currentRunner = { version: "0.0.1", digest: "sha256:" + "d".repeat(64) };
|
|
const replacement = resolveRemoteProviderPackAuthority({ ...input, binding: { ...binding, environmentLeaseId: "replacement" },
|
|
current: () => manifest("b"), runnerArtifact: () => currentRunner });
|
|
expect(replacement.manifest).toEqual(manifest("b"));
|
|
expect(replacement.runnerArtifact).toEqual(currentRunner);
|
|
replacement.verified();
|
|
expect(JSON.parse(readFileSync(path, "utf8")).binding.environmentLeaseId).toBe("replacement");
|
|
});
|
|
it("retains the authenticated daemon identity across controller daemon replacement", () => {
|
|
const { input } = fixture();
|
|
resolveRemoteProviderPackAuthority(input).verified();
|
|
const changedRunner = vi.fn(() => ({ version: "new", digest: "sha256:" + "d".repeat(64) }));
|
|
const recovered = resolveRemoteProviderPackAuthority({ ...input, reattach: true, runnerArtifact: changedRunner });
|
|
expect(recovered.runnerArtifact).toEqual(runnerArtifact());
|
|
expect(changedRunner).not.toHaveBeenCalled();
|
|
});
|
|
it.each([{}, { version: "0.0.0", digest: "not-a-sha256" }])("rejects an invalid saved daemon identity: %j", runnerArtifact => {
|
|
const { input, path } = fixture();
|
|
resolveRemoteProviderPackAuthority(input).verified();
|
|
const saved = JSON.parse(readFileSync(path, "utf8"));
|
|
writeFileSync(path, JSON.stringify({ ...saved, runnerArtifact }));
|
|
expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true })).toThrow("saved provider pack authority");
|
|
});
|
|
|
|
it("keeps admitted session ownership with the caller if receipt storage or its warning fails", async () => {
|
|
const { input, path } = fixture();
|
|
mkdirSync(path);
|
|
const authority = resolveRemoteProviderPackAuthority(input);
|
|
const warning = vi.fn(() => { throw new Error("log unavailable"); });
|
|
await expect(recordAdmittedRemoteProviderPack(authority, warning)).resolves.toBe(false);
|
|
expect(warning).toHaveBeenCalledWith(expect.stringContaining("could not save remote release authority"));
|
|
expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true })).toThrow("saved provider pack authority");
|
|
});
|
|
it("retains the original projectless lease across warm run placeholders but rejects a different real workspace", () => {
|
|
const { input } = fixture();
|
|
const first = { ...binding, workspaceId: providerPackWorkspaceId("run-1", "run-1", "original-lease") };
|
|
resolveRemoteProviderPackAuthority({ ...input, binding: first }).verified();
|
|
const next = { ...binding, workspaceId: providerPackWorkspaceId("run-2", "run-2", "original-lease") };
|
|
expect(resolveRemoteProviderPackAuthority({ ...input, binding: next, reattach: true }).manifest).toEqual(manifest("a"));
|
|
const changed = { ...binding, workspaceId: providerPackWorkspaceId("project-workspace", "run-2", "original-lease") };
|
|
expect(() => resolveRemoteProviderPackAuthority({ ...input, binding: changed, reattach: true })).toThrow("does not match this execution");
|
|
});
|
|
|
|
it("persists the compatible installation selected at launch, then pins it on reattachment", () => {
|
|
const { input } = fixture();
|
|
const authority = resolveRemoteProviderPackAuthority(input);
|
|
const installed = manifest("b");
|
|
authority.select(installed);
|
|
authority.verified();
|
|
const attached = resolveRemoteProviderPackAuthority({ ...input, reattach: true });
|
|
expect(attached.manifest).toEqual(installed);
|
|
expect(() => attached.select(manifest("c"))).toThrow("cannot replace an attached pack authority");
|
|
});
|