Files
PaperClipAI/server/src/services/native-runtime/retained-provider-pack-source.test.ts
T
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00

479 lines
31 KiB
TypeScript

import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { createServer } from "node:http";
import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { pathToFileURL } from "node:url";
import { gzipSync, zstdCompressSync } from "node:zlib";
import tar from "tar-stream";
import { afterEach, expect, it, vi } from "vitest";
import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js";
import { uploadVerifiedProviderPack } from "./provider-pack-upload.js";
import * as archiveOwner from "./provider-pack-archive.js";
import { resolveProviderPackUploadSource } from "./retained-provider-pack-source.js";
const archiveScript = pathToFileURL(path.resolve(import.meta.dirname, "../../../../scripts/create-provider-pack-archive.mjs")).href;
const { createProviderPackLayers } = await import(archiveScript);
const exec = promisify(execFile);
const hash = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
const roots: string[] = [];
const servers: ReturnType<typeof createServer>[] = [];
const actualFetch = globalThis.fetch;
afterEach(async () => {
vi.unstubAllGlobals();
vi.unstubAllEnvs();
vi.restoreAllMocks();
await Promise.all(servers.splice(0).map(server => new Promise<void>((done, reject) => {
server.close(error => error ? reject(error) : done());
server.closeAllConnections();
})));
await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true })));
});
const runnerInputs = "c".repeat(64);
type RunnerImage = "published" | "one-transport-layer" | "three-transport-layers" | "unpublished" | "token-refused" | "slim-unreachable"
| "other-inputs" | "other-revision" | "no-slim-annotation" | "slim-annotation-is-a-tag" | "slim-is-other-bytes" | "slim-not-a-prefix" | "slim-is-the-full-image"
| "nine-transport-layers" | "pack-file-in-transport";
async function fixture(options: { wrongCode?: boolean; unsafeOverlay?: boolean; changedFileLayer?: boolean; hardlink?: "safe" | "escape" | "unknown"; runnerImage?: RunnerImage } = {}) {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "retained-pack-test-"));
roots.push(root);
const temporary = path.join(root, "tmp");
await fs.mkdir(temporary);
vi.stubEnv("TMPDIR", temporary);
const makePack = async (name: string, tool: string, source: string) => {
const pack = path.join(root, name);
await fs.mkdir(path.join(pack, "bin"), { recursive: true });
const artifact = { path: "bin/tool", sha256: `sha256:${hash(tool)}` };
const payload = { pins: { opencode: "1.18.34" }, target: { platform: "linux", architecture: "x64" }, runnerSourceRevision: source,
...(options.runnerImage ? { runnerInputsDigest: runnerInputs } : {}),
contractDigest: `sha256:${hash(tool)}`, artifacts: { nodeCommand: artifact, opencodeCommand: artifact, opencodeProxy: artifact } };
const manifest = { schema: "paperclip-runner/remote-provider-pack/v1", payload, digest: `sha256:${hash(canonicalProviderPackJson(payload))}` };
await fs.writeFile(path.join(pack, "provider-pack.json"), JSON.stringify(manifest));
await fs.writeFile(path.join(pack, "bin/tool"), tool, { mode: 0o755 });
await fs.writeFile(path.join(pack, "bin/large"), Buffer.alloc(1024 * 1024, 42), { mode: 0o755 });
await createProviderPackLayers(pack);
return { pack, manifest };
};
const selected = await makePack("old", "retained provider code", "a".repeat(40));
const current = await makePack("current", "current incompatible code", "b".repeat(40));
if (options.changedFileLayer) {
const unsafe = tar.pack(), chunks: Buffer[] = [];
const collected = (async () => { for await (const chunk of unsafe) chunks.push(Buffer.from(chunk)); })();
const body = Buffer.alloc(1024 * 1024, 43);
await new Promise<void>((done, reject) => unsafe.entry({ name: "bin/large", type: "file", size: body.length, mode: 0o755 }, body, error => error ? reject(error) : done()));
unsafe.finalize(); await collected;
const bytes = zstdCompressSync(Buffer.concat(chunks));
await fs.writeFile(selected.pack + ".archives/0.tar.zst", bytes);
const receipt = JSON.parse(await fs.readFile(selected.pack + ".archive.json", "utf8"));
receipt.files[0].archive = { bytes: bytes.length, sha256: hash(bytes) };
await fs.writeFile(selected.pack + ".archive.json", JSON.stringify(receipt));
}
if (options.unsafeOverlay || options.hardlink) {
const unsafe = tar.pack(), chunks: Buffer[] = [];
const collected = (async () => { for await (const chunk of unsafe) chunks.push(Buffer.from(chunk)); })();
for (const [name, body] of [["provider-pack.json", JSON.stringify(selected.manifest)], ["bin/tool", "retained provider code"], ...(options.unsafeOverlay ? [["../outside", "must not be written"]] : [])]) {
await new Promise<void>((done, reject) => unsafe.entry({ name: name!, type: "file", size: Buffer.byteLength(body!) }, body!, error => error ? reject(error) : done()));
}
if (options.hardlink) await new Promise<void>((done, reject) => unsafe.entry({ name: "bin/tool-copy", type: "link", linkname: options.hardlink === "safe" ? "./bin/tool" : options.hardlink === "escape" ? "../outside" : "bin/missing", mode: 0o644 }, error => error ? reject(error) : done()));
unsafe.finalize(); await collected;
const bytes = zstdCompressSync(Buffer.concat(chunks));
await fs.writeFile(selected.pack + ".archives/overlay.tar.zst", bytes);
const receipt = JSON.parse(await fs.readFile(selected.pack + ".archive.json", "utf8"));
receipt.overlay = { bytes: bytes.length, sha256: hash(bytes) };
await fs.writeFile(selected.pack + ".archive.json", JSON.stringify(receipt));
}
const transportPack = options.wrongCode ? current.pack : selected.pack;
const transportNames = ["provider-pack.archive.json", ...await fs.readdir(transportPack + ".archives").then(names => names.map(name => `provider-pack.archives/${name}`))];
const transportLayer = async (prefix: string, names: string[], directories: string[] = []) => {
const archive = tar.pack();
const chunks: Buffer[] = [];
const collected = (async () => { for await (const chunk of archive) chunks.push(Buffer.from(chunk)); })();
for (const name of directories) await new Promise<void>((done, reject) => archive.entry({ name: `${name}/`, type: "directory" }, error => error ? reject(error) : done()));
for (const name of names) {
const bytes = name.startsWith("provider-pack/") ? Buffer.from("{}") : await fs.readFile(path.join(path.dirname(transportPack), name.replace("provider-pack", path.basename(transportPack))));
await new Promise<void>((done, reject) => archive.entry({ name: `${prefix}${name}`, type: "file", size: bytes.length }, bytes, error => error ? reject(error) : done()));
}
archive.finalize();
await collected;
return gzipSync(Buffer.concat(chunks));
};
const layer = await transportLayer("opt/paperclip-runner/", transportNames);
const config = Buffer.from(JSON.stringify({ architecture: "amd64", os: "linux", config: { Labels: {
"org.opencontainers.image.revision": selected.manifest.payload.runnerSourceRevision,
"org.opencontainers.image.source": "https://github.com/paperclipai/paperclip",
} }, history: [{ created_by: "COPY /provider-pack-transport/ /opt/paperclip-runner/ # buildkit" }] }));
const manifest = Buffer.from(JSON.stringify({ schemaVersion: 2, mediaType: "application/vnd.oci.image.manifest.v1+json", config: { digest: `sha256:${hash(config)}`, size: config.length }, layers: [{ digest: `sha256:${hash(layer)}`, size: layer.length, mediaType: "application/vnd.oci.image.layer.v1.tar+gzip" }] }));
const index = Buffer.from(JSON.stringify({ schemaVersion: 2, mediaType: "application/vnd.oci.image.index.v1+json", manifests: [{ digest: `sha256:${hash(manifest)}`, size: manifest.length, platform: { os: "linux", architecture: "amd64" } }] }));
const routes = new Map<string, Buffer>([
["/token", Buffer.from(JSON.stringify({ token: "public-code-fixture-token" }))],
[`/v2/paperclipai/paperclip/manifests/sha-${selected.manifest.payload.runnerSourceRevision}`, index],
[`/v2/paperclipai/paperclip/manifests/sha256:${hash(manifest)}`, manifest],
[`/v2/paperclipai/paperclip/blobs/sha256:${hash(config)}`, config],
[`/v2/paperclipai/paperclip/blobs/sha256:${hash(layer)}`, layer],
]);
if (options.runnerImage && options.runnerImage !== "unpublished") {
const kind = options.runnerImage, repository = "/v2/paperclipai/paperclip-runner";
const stable = transportNames.filter(name => /\/[0-9]+\.tar\.zst$/.test(name)), rest = transportNames.filter(name => !stable.includes(name));
const archives = ["provider-pack.archives"];
const transport = kind === "one-transport-layer" ? [await transportLayer("", transportNames, archives)]
: kind === "three-transport-layers" ? [await transportLayer("", stable, archives), await transportLayer("", ["provider-pack.archive.json"]), await transportLayer("", rest.slice(1), archives)]
: [await transportLayer("", stable, archives), await transportLayer("", [...rest, ...(kind === "pack-file-in-transport" ? ["provider-pack/provider-pack.json"] : [])], archives)];
const described = (bytes: Buffer | string) => ({ digest: `sha256:${hash(bytes)}`, size: bytes.length, mediaType: "application/vnd.oci.image.layer.v1.tar+gzip" });
const shared = [described("large pack files"), described("the rest of the pack")];
const added = kind === "nine-transport-layers" ? Array.from({ length: 9 }, (_value, index) => described(`transport ${index}`)) : transport.map(described);
const imageOf = (layers: unknown[]) => Buffer.from(JSON.stringify({ schemaVersion: 2, config: { digest: `sha256:${hash("runner config")}`, size: 13 }, layers }));
const indexOf = (image: Buffer, annotations?: Record<string, string>) => Buffer.from(JSON.stringify({ schemaVersion: 2,
manifests: [{ digest: `sha256:${hash(image)}`, size: image.length, platform: { os: "linux", architecture: "amd64" } }], ...(annotations ? { annotations } : {}) }));
const full = imageOf([...shared, ...added]);
const slim = imageOf(kind === "slim-not-a-prefix" ? [shared[1], shared[0]] : kind === "slim-is-the-full-image" ? [...shared, ...added] : shared);
const slimIndex = indexOf(slim), slimDigest = `sha256:${hash(slimIndex)}`;
const fullIndex = indexOf(full, {
"io.paperclip.runner.inputs-digest": kind === "other-inputs" ? "d".repeat(64) : runnerInputs,
"org.opencontainers.image.revision": kind === "other-revision" ? "e".repeat(40) : selected.manifest.payload.runnerSourceRevision,
...(kind === "no-slim-annotation" ? {} : { "io.paperclip.runner.slim-digest": kind === "slim-annotation-is-a-tag" ? `inputs-${runnerInputs}-slim` : slimDigest }),
});
routes.set(`${repository}/manifests/inputs-${runnerInputs}`, fullIndex);
routes.set(`${repository}/manifests/inputs-${runnerInputs}-slim`, slimIndex);
routes.set(`${repository}/manifests/sha256:${hash(full)}`, full);
routes.set(`${repository}/manifests/sha256:${hash(slim)}`, slim);
if (kind !== "slim-unreachable") routes.set(`${repository}/manifests/${slimDigest}`, kind === "slim-is-other-bytes" ? indexOf(slim, { moved: "true" }) : slimIndex);
for (const bytes of transport) routes.set(`${repository}/blobs/sha256:${hash(bytes)}`, bytes);
}
const requests: string[] = [];
const stalled = new Set<string>();
const observers = new Map<string, () => void>();
const server = createServer((request, response) => {
const url = new URL(request.url!, "http://fixture");
const route = url.pathname;
requests.push(route);
const refused = options.runnerImage === "token-refused" && route === "/token" && url.searchParams.get("scope") === "repository:paperclipai/paperclip-runner:pull";
const bytes = refused ? undefined : routes.get(route);
response.writeHead(bytes ? 200 : refused ? 403 : 404, { "content-length": bytes?.length ?? 0 });
response.flushHeaders();
observers.get(route)?.();
if (stalled.has(route)) return;
response.end(bytes);
});
servers.push(server);
await new Promise<void>(done => server.listen(0, "127.0.0.1", done));
const address = server.address();
if (!address || typeof address === "string") throw Error("fixture server unavailable");
vi.stubGlobal("fetch", (input: string | URL | Request, init?: RequestInit) => {
const url = new URL(typeof input === "string" || input instanceof URL ? input : input.url);
expect(url.hostname).toBe("ghcr.io");
return actualFetch(`http://127.0.0.1:${address.port}${url.pathname}${url.search}`, init);
});
const bin = path.join(root, "commands");
await fs.mkdir(bin);
const tarCommand = (await exec("sh", ["-c", "command -v gtar || command -v tar"])).stdout.trim();
await fs.symlink(tarCommand, path.join(bin, "tar"));
const execute = vi.fn(async (input: { command: string; args?: string[] }) => {
try {
const result = await exec(input.command, input.args ?? [], { env: { ...process.env, PATH: `${bin}:${process.env.PATH}` } });
return { exitCode: 0, timedOut: false, stdout: result.stdout, stderr: result.stderr };
} catch (error) { return { exitCode: 1, timedOut: false, stdout: "", stderr: String(error) }; }
});
const syncIn = vi.fn(async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => {
for (const operation of operations) for (const file of operation.files) {
await fs.mkdir(path.dirname(file.targetPath), { recursive: true });
await fs.copyFile(file.sourcePath, file.targetPath);
}
return { operations: [] };
});
return { root, selected, current, requests, routes, runner: { execute, syncIn }, targetPath: path.join(root, "target"),
temporary, layerRoute: `/v2/paperclipai/paperclip/blobs/sha256:${hash(layer)}`,
stall: (route: string) => { stalled.add(route); },
observed: (route: string) => new Promise<void>(done => { if (requests.includes(route)) done(); else observers.set(route, done); }),
};
}
function upload(input: Awaited<ReturnType<typeof fixture>>, extra: Partial<Parameters<typeof uploadVerifiedProviderPack>[0]> = {}) {
return uploadVerifiedProviderPack({ root: input.current.pack, sourceDigest: input.current.manifest.digest, expectedDigest: input.selected.manifest.digest,
manifest: input.selected.manifest, runner: input.runner, cwd: input.root, targetPath: input.targetPath,
reuseRoots: [], verifyDirectory: () => { throw Error("current_directory_fallback_forbidden"); },
stageDirectory: async () => { throw Error("current_directory_fallback_forbidden"); }, verifyUploaded: async () => {}, ...extra } as Parameters<typeof uploadVerifiedProviderPack>[0]);
}
it("installs the exact retained pack through the real uploader on an empty host instead of the current image archive", async () => {
const input = await fixture();
const verifyUploaded = vi.fn(async () => {
expect(await fs.readFile(path.join(input.targetPath, "bin/tool"), "utf8")).toBe("retained provider code");
expect(JSON.parse(await fs.readFile(path.join(input.targetPath, "provider-pack.json"), "utf8"))).toEqual(input.selected.manifest);
});
await upload(input, { verifyUploaded });
expect(verifyUploaded).toHaveBeenCalledOnce();
expect(input.requests.some(route => route.includes("/blobs/"))).toBe(true);
});
const fromRunnerImage = (route: string) => route.startsWith("/v2/paperclipai/paperclip-runner/");
const fromApplicationImage = (route: string) => route.startsWith("/v2/paperclipai/paperclip/");
async function importedSource(input: Awaited<ReturnType<typeof fixture>>) {
const source = await resolveProviderPackUploadSource({ root: input.current.pack, sourceDigest: input.current.manifest.digest, manifest: input.selected.manifest });
expect(source.archiveOnly).toBe(true);
expect(JSON.parse(await fs.readFile(path.join(source.root, "provider-pack.json"), "utf8"))).toEqual(input.selected.manifest);
expect(await fs.readFile(source.root + ".archive.json")).toEqual(await fs.readFile(input.selected.pack + ".archive.json"));
for (const name of await fs.readdir(input.selected.pack + ".archives")) {
expect(await fs.readFile(path.join(source.root + ".archives", name))).toEqual(await fs.readFile(path.join(input.selected.pack + ".archives", name)));
}
return source;
}
const transportBlobs = (input: Awaited<ReturnType<typeof fixture>>) => input.requests.filter(fromRunnerImage).filter(route => route.includes("/blobs/"));
it("reads a retained pack that names its Runner image from only the layers that image adds to its slim image, and never reads the slim tag", async () => {
const input = await fixture({ runnerImage: "published" });
const source = await importedSource(input);
expect(input.requests.filter(fromRunnerImage)).toHaveLength(6);
expect(transportBlobs(input)).toHaveLength(2);
expect(input.requests.filter(fromApplicationImage)).toEqual([]);
expect(input.requests.filter(route => route.endsWith("-slim"))).toEqual([]);
expect((await fs.readdir(path.dirname(source.root))).sort()).toEqual(["provider-pack", "provider-pack.archive.json", "provider-pack.archives"]);
const pointer = JSON.parse(await fs.readFile(path.join(path.dirname(path.dirname(source.root)), input.selected.manifest.digest.slice(7) + ".json"), "utf8"));
expect(pointer.runnerImage).toMatchObject({ inputsDigest: runnerInputs });
expect(pointer.runnerImage.transportLayerDigests).toHaveLength(2);
expect(pointer.transportLayerDigest).toBe(pointer.runnerImage.transportLayerDigests[1]);
});
it.each([["one-transport-layer", 1], ["three-transport-layers", 3]] as const)("takes every layer after the slim image's, however many (%s)", async (runnerImage, layers) => {
const input = await fixture({ runnerImage });
await importedSource(input);
expect(transportBlobs(input)).toHaveLength(layers);
expect(input.requests.filter(fromApplicationImage)).toEqual([]);
});
it.each(["unpublished", "token-refused", "slim-unreachable"] as const)("falls back to the application image when the Runner image cannot be read (%s)", async runnerImage => {
const input = await fixture({ runnerImage });
const source = await importedSource(input);
expect(transportBlobs(input)).toEqual([]);
expect(input.requests.filter(fromApplicationImage).some(route => route.includes("/blobs/"))).toBe(true);
expect((await fs.readdir(path.dirname(source.root))).sort()).toEqual(["provider-pack", "provider-pack.archive.json", "provider-pack.archives"]);
});
it.each(["other-inputs", "other-revision", "no-slim-annotation", "slim-annotation-is-a-tag", "slim-is-other-bytes", "slim-not-a-prefix", "slim-is-the-full-image",
"nine-transport-layers", "pack-file-in-transport"] as const)(
"refuses a Runner image that does not match the pack or the layer contract, without trying another source (%s)", async runnerImage => {
const input = await fixture({ runnerImage });
await expect(upload(input)).rejects.toThrow("runner_remote_provider_source_invalid");
expect(input.requests.filter(fromApplicationImage)).toEqual([]);
expect(input.requests.filter(route => route.endsWith("-slim"))).toEqual([]);
if (runnerImage !== "pack-file-in-transport") expect(transportBlobs(input)).toEqual([]);
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
});
it("keeps a matching current pack on its existing path without registry or cache reads", async () => {
const input = await fixture();
await upload(input, { root: input.selected.pack, sourceDigest: input.selected.manifest.digest });
expect(input.requests).toEqual([]);
await expect(fs.stat(path.join(input.temporary, `paperclip-provider-pack-sources-${process.getuid?.()}`))).rejects.toMatchObject({ code: "ENOENT" });
expect(await fs.readFile(path.join(input.targetPath, "bin/tool"), "utf8")).toBe("retained provider code");
});
it("refuses a different historical code pack even when OCI source labels and all transport hashes are valid", async () => {
const input = await fixture({ wrongCode: true });
await expect(upload(input)).rejects.toThrow("runner_remote_provider_archive_invalid");
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
expect(await fs.readFile(path.join(input.current.pack, "bin/tool"), "utf8")).toBe("current incompatible code");
});
it("does not mask a corrupt matching current archive by looking up a different code source", async () => {
const input = await fixture();
await fs.writeFile(input.selected.pack + ".archive.json", "{");
await expect(upload(input, { root: input.selected.pack, sourceDigest: input.selected.manifest.digest })).rejects.toThrow("runner_remote_provider_archive_invalid");
expect(input.requests).toEqual([]);
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
});
it("rejects a changed controller code identity before looking up a historical source", async () => {
const input = await fixture();
await expect(upload(input, { sourceDigest: input.selected.manifest.digest })).rejects.toThrow("runner_remote_provider_source_invalid");
expect(input.requests).toEqual([]);
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
});
it("refuses an unknown historical source without clearing or falling back to the current image", async () => {
const input = await fixture();
input.routes.delete(`/v2/paperclipai/paperclip/manifests/sha-${input.selected.manifest.payload.runnerSourceRevision}`);
await expect(upload(input)).rejects.toThrow("runner_remote_provider_source_unavailable: HTTP 404");
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
});
it("rejects escaping inner tar paths despite valid OCI and layer digests before any remote effects", async () => {
const input = await fixture({ unsafeOverlay: true });
await expect(upload(input)).rejects.toThrow("runner_remote_provider_source_invalid");
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
await expect(fs.stat(path.join(input.temporary, "outside"))).rejects.toMatchObject({ code: "ENOENT" });
});
it("checks the file layer's actual plaintext against its declared code digest before any remote effects", async () => {
const input = await fixture({ changedFileLayer: true });
await expect(upload(input)).rejects.toThrow("runner_remote_provider_source_invalid");
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
});
it("preserves historical code hardlinks only to earlier confined regular files", async () => {
const input = await fixture({ hardlink: "safe" });
await upload(input);
const original = await fs.stat(path.join(input.targetPath, "bin/tool")), copy = await fs.stat(path.join(input.targetPath, "bin/tool-copy"));
expect(copy.ino).toBe(original.ino);
expect(await fs.readFile(path.join(input.targetPath, "bin/tool-copy"), "utf8")).toBe("retained provider code");
});
it.each(["escape", "unknown"] as const)("refuses a historical code hardlink with an %s target before remote effects", async hardlink => {
const input = await fixture({ hardlink });
await expect(upload(input)).rejects.toThrow("runner_remote_provider_source_invalid");
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
});
it("cancels a stalled source body and removes only the unfinished private import", async () => {
const input = await fixture();
input.stall(input.layerRoute);
const controller = new AbortController();
const operation = upload(input, { signal: controller.signal });
const rejected = expect(operation).rejects.toThrow("owned caller stopped");
await input.observed(input.layerRoute);
controller.abort(new Error("owned caller stopped"));
await rejected;
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
const cache = path.join(input.temporary, `paperclip-provider-pack-sources-${process.getuid?.()}`);
await vi.waitFor(async () => expect(await fs.readdir(cache)).toEqual([]));
});
it("enforces the source deadline while the registry is stalled", async () => {
const input = await fixture();
input.stall(input.layerRoute);
const nativeTimeout = AbortSignal.timeout;
vi.spyOn(AbortSignal, "timeout").mockImplementation(milliseconds => {
expect(milliseconds).toBe(180_000);
return nativeTimeout(80);
});
await expect(upload(input)).rejects.toMatchObject({ name: "TimeoutError" });
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
});
it("reuses a verified derived cache without registry reads and reconstructs it after an empty-host reset", async () => {
const input = await fixture();
await upload(input);
const first = input.requests.length;
await upload(input, { targetPath: input.targetPath + "-cached" });
expect(input.requests).toHaveLength(first);
await fs.rm(path.join(input.temporary, `paperclip-provider-pack-sources-${process.getuid?.()}`), { recursive: true });
await upload(input, { targetPath: input.targetPath + "-empty" });
expect(input.requests.length).toBeGreaterThan(first);
expect(await fs.readFile(path.join(input.targetPath + "-empty", "bin/tool"), "utf8")).toBe("retained provider code");
});
it("shares warm-cache verification while a stopped caller leaves another waiter active", async () => {
const input = await fixture();
await upload(input);
const requests = input.requests.length;
const firstController = new AbortController(), secondController = new AbortController();
let enter!: () => void, release!: () => void, joined!: () => void, duplicate!: () => void;
const entered = new Promise<void>(done => { enter = done; });
const gate = new Promise<void>(done => { release = done; });
const secondJoined = new Promise<void>(done => { joined = done; });
const repeated = new Promise<void>(done => { duplicate = done; });
const originalRead = archiveOwner.readProviderPackArchive;
const verification = vi.spyOn(archiveOwner, "readProviderPackArchive").mockImplementation(async (...args) => {
enter();
if (verification.mock.calls.length > 1) duplicate();
await gate;
return originalRead(...args);
});
const originalAny = AbortSignal.any;
vi.spyOn(AbortSignal, "any").mockImplementation(signals => {
const combined = originalAny(signals);
if (signals.includes(secondController.signal)) {
const add = combined.addEventListener.bind(combined);
vi.spyOn(combined, "addEventListener").mockImplementation((type, listener, options) => {
const result = add(type, listener, options);
if (type === "abort") joined();
return result;
});
}
return combined;
});
const source = { root: input.current.pack, sourceDigest: input.current.manifest.digest, manifest: input.selected.manifest };
const first = resolveProviderPackUploadSource({ ...source, signal: firstController.signal });
const firstStopped = expect(first).rejects.toThrow("owned shared-cache waiter stopped");
await entered;
const second = resolveProviderPackUploadSource({ ...source, signal: secondController.signal });
try {
await Promise.race([secondJoined, repeated.then(() => { throw Error("warm-cache verification repeated"); })]);
expect(verification).toHaveBeenCalledOnce();
firstController.abort(new Error("owned shared-cache waiter stopped"));
await firstStopped;
expect(verification.mock.calls[0]?.[2]?.aborted).toBe(false);
release();
expect(await second).toMatchObject({ archiveOnly: true });
expect(input.requests).toHaveLength(requests);
} finally {
firstController.abort(new Error("owned shared-cache waiter stopped"));
secondController.abort(new Error("owned shared-cache fixture cleanup"));
release();
await Promise.allSettled([first, second, firstStopped]);
}
});
it("rejects a changed cached archive before any remote command or transfer", async () => {
const input = await fixture();
await upload(input);
const cache = path.join(input.temporary, `paperclip-provider-pack-sources-${process.getuid?.()}`);
const publication = JSON.parse(await fs.readFile(path.join(cache, input.selected.manifest.digest.slice(7) + ".json"), "utf8"));
await fs.appendFile(path.join(cache, publication.entry, "provider-pack.archives/overlay.tar.zst"), "changed");
input.runner.execute.mockClear();
input.runner.syncIn.mockClear();
await expect(upload(input, { targetPath: input.targetPath + "-new" })).rejects.toThrow("runner_remote_provider_archive_invalid");
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
expect(await fs.readFile(path.join(input.targetPath, "bin/tool"), "utf8")).toBe("retained provider code");
});
it("refuses an unsafe cache directory instead of following it", async () => {
const input = await fixture();
const outside = path.join(input.root, "outside");
await fs.mkdir(outside);
await fs.symlink(outside, path.join(input.temporary, `paperclip-provider-pack-sources-${process.getuid?.()}`));
await expect(upload(input)).rejects.toThrow("runner_remote_provider_source_invalid");
expect(await fs.readdir(outside)).toEqual([]);
expect(input.requests).toEqual([]);
expect(input.runner.execute).not.toHaveBeenCalled();
});
it("leaves a target intact when its archive transport cannot install the verified source", async () => {
const input = await fixture();
await fs.mkdir(input.targetPath);
await fs.writeFile(path.join(input.targetPath, "marker"), "untouched");
input.runner.execute.mockResolvedValue({ exitCode: 127, timedOut: false, stdout: "", stderr: "unavailable" });
await expect(upload(input)).rejects.toThrow("runner_remote_provider_archive_transport_unavailable");
expect(input.runner.execute).toHaveBeenCalledOnce();
expect(input.runner.execute.mock.calls[0]?.[0].args?.join(" ")).not.toContain("rm -rf");
expect(input.runner.syncIn).not.toHaveBeenCalled();
expect(await fs.readFile(path.join(input.targetPath, "marker"), "utf8")).toBe("untouched");
});
it("refuses an oversized historical layer specification before any target effects", async () => {
const input = await fixture();
await fs.mkdir(input.targetPath);
await fs.writeFile(path.join(input.targetPath, "marker"), "untouched");
const reuseRoots = Array.from({ length: 2000 }, (_, index) => `/qualified-image-${index}/` + "a".repeat(64));
await expect(upload(input, { reuseRoots })).rejects.toThrow("runner_remote_provider_archive_transport_unavailable");
expect(input.runner.execute).not.toHaveBeenCalled();
expect(input.runner.syncIn).not.toHaveBeenCalled();
expect(await fs.readFile(path.join(input.targetPath, "marker"), "utf8")).toBe("untouched");
});