mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud deploys images that contain the native Rust Runner. > - The image already builds that Runner before copying ordinary app source. > - A Rust source change still invalidates its entire compiled dependency layer. > - Compiled dependencies can survive source changes when their recipe is unchanged. > - This PR adds a separate locked dependency build before compiling the real workspace. ## Linked Issues or Issue Description Refs #13195. A search of related Docker and Cargo cache PRs found no duplicate dependency-recipe change. **What existing behavior does this improve?** Docker image build time after Rust source or embedded protocol changes. **Current behavior** The `runner-build` stage compiles dependencies and workspace code in one layer. In Cloud readiness run 34698143548, that stage took about 3m48s when its cache was unavailable. **Proposed behavior** Generate a recipe with pinned cargo-chef 0.1.73. Build locked release dependencies in `runner-deps`, then copy and compile real Rust source and embedded protocol inputs in `runner-build`. Source edits can reuse the dependency layer from the existing registry cache. **Reason and benefit** Reduce dependency recompilation during source changes and merge bursts. Expected savings are roughly 2–4 minutes when the old native layer would miss but dependency layers are available. Full cold builds also pay for the recipe tool installation. Ordinary app-only cache hits gain little from this change. **Breaking changes** None to the shipped application or image tags. The recipe tool and compiled dependencies remain in build stages. ## What Changed - Install a pinned recipe generator with its locked dependencies and the existing package-owned compiler. - Add recipe planning and compiled dependency stages. Use the same release profile, package, binary, and lockfile enforcement as the real native build. - Remove generated source stubs before copying actual source. Preserve protocol inputs, timestamp normalization, binary staging, and application checks. - Add Docker cache wiring regressions and update the Docker cache documentation. - Run a two-build probe in Docker Runner check. It requires dependency reuse, changed real binary metadata after a source edit, and a changed recipe after a dependency declaration edit. It uses a disposable tracked-source context and exports only small metadata files. ## Verification - Passed all five Docker build-stamp and dependency-cache tests with `pnpm exec vitest run server/src/__tests__/docker-build-stamp.test.ts`. - Passed the local ARM64 `docker buildx build --target runner-build --progress plain`. Local Docker then hit storage errors during a runtime probe; cache invalidation verification continues on GitHub-hosted Linux. - Passed `bash -n scripts/check-docker-runner-cache.sh`, `actionlint`, and `git diff --check`. - Passed a [Linux AMD64 cache probe](https://github.com/paperclipai/paperclip/actions/runs/34711042199) against the PR source: dependencies compiled in 3m49s for the baseline and were `CACHED` after a source edit; real source compilation took about 37 seconds. Binary metadata changed and dependency declaration changes altered the recipe. The permanent probe is also running in latest-head Docker Runner check. - Passed latest-head [Docker Runner check](https://github.com/paperclipai/paperclip/actions/runs/34711145160), including the permanent source/dependency invalidation probe. - Passed full [PR verification](https://github.com/paperclipai/paperclip/actions/runs/34711145352/attempts/2): typecheck, all grouped tests, native verification, build, release dry run, and browser checks. One unrelated signoff-policy browser test failed waiting for a heartbeat run on attempt 1; only that failed shard and dependent checks were retried, and passed. - Latest-head Greptile is 5/5 with no unresolved findings. Full local tests/build were limited by local disk exhaustion; Linux CI completed those checks. ## Risks - The two-build CI probe has a 20-minute job limit to cover the cold build and source rebuild. It adds no AWS routing. - A fully cold build must install cargo-chef and populate the dependency layer. Both become reusable registry layers; no Actions cache is added. - The recipe and final build must keep the same compiler, build profile, package, binary, and directory layout. A source-change rebuild probe checks real cache reuse and binary invalidation. - Dependency or compiler changes still require rebuilding dependencies. Existing image verification and full-SHA publication gates remain unchanged. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
108 lines
5.7 KiB
TypeScript
108 lines
5.7 KiB
TypeScript
import { readFileSync } from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import { describe, expect, it } from "vitest";
|
|
|
|
/**
|
|
* Drift guard for the Docker build-stamp wiring.
|
|
*
|
|
* The server build runs scripts/write-build-stamp.mjs, which stamps the built
|
|
* commit into dist/build-info.json. The build context has no .git, so the
|
|
* script reads PAPERCLIP_BUILD_COMMIT instead. Docker exposes an ARG to the
|
|
* next RUN as an environment variable, but an ARG goes out of scope at the end
|
|
* of its stage. So the build stage must declare `ARG PAPERCLIP_BUILD_COMMIT`
|
|
* before the server build; the production ARG alone stamps nothing, because
|
|
* the server build already ran in the earlier stage.
|
|
*
|
|
* This guard fails if a refactor drops the build-stage ARG, moves it after the
|
|
* server build, or removes the build-arg the docker workflow passes.
|
|
*/
|
|
|
|
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
|
|
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
|
|
const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker.yml"), "utf8");
|
|
const cloudWorkflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker-cloud.yml"), "utf8");
|
|
|
|
/**
|
|
* Return the text of the Dockerfile stage that starts at the named target.
|
|
* A stage runs from its `FROM ... AS <name>` line to the next `FROM` line.
|
|
*/
|
|
function stageBody(source: string, stageName: string): string {
|
|
const froms = [...source.matchAll(/^FROM .*$/gm)];
|
|
const startIdx = froms.findIndex((m) => new RegExp(`\\bAS ${stageName}\\b`).test(m[0]));
|
|
expect(startIdx, `Dockerfile must declare a '${stageName}' stage`).toBeGreaterThanOrEqual(0);
|
|
const start = froms[startIdx].index ?? 0;
|
|
const end = froms[startIdx + 1]?.index ?? source.length;
|
|
return source.slice(start, end);
|
|
}
|
|
|
|
it("keeps per-build runtime metadata out of the weekly CLI-install cache", () => {
|
|
const production = stageBody(dockerfile, "production");
|
|
const tools = production.search(/^RUN echo "cli-tools-epoch:/m);
|
|
const entrypoint = production.search(/^RUN chmod \+x \/usr\/local\/bin\/docker-entrypoint\.sh/m);
|
|
const runtime = production.search(/^ENV NODE_ENV=production/m);
|
|
const epoch = production.search(/^ARG CLI_TOOLS_CACHE_EPOCH\b/m);
|
|
expect(tools).toBeGreaterThanOrEqual(0);
|
|
expect(entrypoint).toBeGreaterThan(tools);
|
|
expect(epoch).toBeGreaterThanOrEqual(0);
|
|
expect(epoch).toBeLessThan(tools);
|
|
for (const name of ["PAPERCLIP_BUILD_VERSION", "PAPERCLIP_BUILD_COMMIT"]) {
|
|
const declarations = [...production.matchAll(new RegExp(`^ARG ${name}\\b`, "gm"))];
|
|
expect(declarations).toHaveLength(1);
|
|
expect(declarations[0].index).toBeGreaterThan(entrypoint);
|
|
expect(declarations[0].index).toBeLessThan(runtime);
|
|
expect(production.slice(runtime)).toContain(`${name}=\${${name}}`);
|
|
}
|
|
});
|
|
|
|
describe("docker build-stamp wiring", () => {
|
|
it("declares PAPERCLIP_BUILD_COMMIT in the build stage before the server build", () => {
|
|
const build = stageBody(dockerfile, "build");
|
|
const argIdx = build.search(/^ARG PAPERCLIP_BUILD_COMMIT\b/m);
|
|
const serverBuildIdx = build.search(/^RUN pnpm --filter @paperclipai\/server build\b/m);
|
|
expect(argIdx, "build stage must declare ARG PAPERCLIP_BUILD_COMMIT").toBeGreaterThanOrEqual(0);
|
|
expect(serverBuildIdx, "build stage must run the server build").toBeGreaterThanOrEqual(0);
|
|
expect(
|
|
argIdx,
|
|
"ARG PAPERCLIP_BUILD_COMMIT must precede the server build so the stamp script reads it",
|
|
).toBeLessThan(serverBuildIdx);
|
|
});
|
|
|
|
it("passes PAPERCLIP_BUILD_COMMIT as a build-arg for both image targets", () => {
|
|
const argLines = [...`${workflow}\n${cloudWorkflow}`.matchAll(/^\s*PAPERCLIP_BUILD_COMMIT=.*$/gm)];
|
|
expect(
|
|
argLines.length,
|
|
"the docker workflow must pass PAPERCLIP_BUILD_COMMIT for the production and cloud builds",
|
|
).toBeGreaterThanOrEqual(2);
|
|
});
|
|
});
|
|
|
|
|
|
describe("Docker Rust dependency cache", () => {
|
|
it("caches the locked dependency recipe separately from source and per-build metadata", () => {
|
|
const chef = stageBody(dockerfile, "rust-chef");
|
|
const planner = stageBody(dockerfile, "runner-plan");
|
|
const dependencies = stageBody(dockerfile, "runner-deps");
|
|
expect(chef).toContain("FROM rust-toolchain AS rust-chef");
|
|
expect(chef).toMatch(/cargo install cargo-chef --version \d+\.\d+\.\d+ --locked/);
|
|
expect(planner).toContain("COPY packages/paperclip-runner/runner ./runner");
|
|
expect(planner).toContain("cargo chef prepare --recipe-path /tmp/runner-recipe.json");
|
|
expect(dependencies).toContain("FROM rust-chef AS runner-deps");
|
|
expect(dependencies).toContain("COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json");
|
|
expect(dependencies).toContain("cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd");
|
|
expect(dependencies).not.toMatch(/COPY .*\.\/runner|COPY .*\.\/protocol|COPY \. \.|PAPERCLIP_BUILD_COMMIT/);
|
|
});
|
|
|
|
it("rebuilds real workspace code and embedded protocol inputs after cooking dependencies", () => {
|
|
const native = stageBody(dockerfile, "runner-build");
|
|
expect(native).toContain("FROM runner-deps AS runner-build");
|
|
for (const source of ["runner", "protocol"]) {
|
|
expect(native.indexOf(`COPY packages/paperclip-runner/${source} ./${source}`))
|
|
.toBeLessThan(native.indexOf("cargo build --release"));
|
|
expect(native).toContain(`COPY packages/paperclip-runner/${source} ./${source}`);
|
|
}
|
|
expect(native).toContain("cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd");
|
|
expect(stageBody(dockerfile, "build")).toContain("FROM runner-build AS build");
|
|
});
|
|
});
|