mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
perf(docker): keep dependencies and provider binaries in layers that survive a commit (#21167)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each merge to master builds the production Docker image for amd64 and arm64 and pushes it to GHCR > - Each build pushes three layers again, about 3.5 GB compressed of a 4.7 GB image > - A file-by-file comparison of two consecutive master images shows that about 95% of those bytes do not change between commits > - The unchanged bytes are in the same layers as small files that change on every commit, so the registry sees new layers > - This pull request moves the unchanged bytes into their own layers > - The benefit is a smaller push for each master build and a smaller pull for each deploy ## Linked Issues or Issue Description **What existing behavior does this improve?** The production image build. It pushes about 3.5 GB of layers for each architecture on each master commit. Each deploy pulls the same bytes again. **Subsystem affected** Docker image build (`Dockerfile`, `production` and `runner-provider-pack` stages). **Current behavior** I compared the amd64 images of two consecutive master commits (`2174f2d` and `dfc510b`, 15 changed server source files). I hashed each file in the three layers that changed. | Layer | Compressed | Content that is identical between the two builds | | --- | --- | --- | | `COPY --from=build /app /app` | 1,404 MB | 97.5%. The root `node_modules` is 2.86 GB of 4.13 GB and has 0 changed files. | | `COPY /provider-pack` | 1,008 MB | 100% of the 112 files of 1 MiB or more (2.54 GB). Only 32 small files change (manifest, `.modules.yaml`, `.bin` shims). | | `COPY /provider-pack-transport/` | 1,059 MB | The archives differ only in tar timestamps. This pull request does not change this layer. | **Proposed behavior** - Copy the root `node_modules` from the `deps` stage as its own layer. Copy the remainder of `/app` from the `build` stage with `--exclude=node_modules`. - In the `runner-provider-pack` stage, move pack files of 1 MiB or more to `/provider-pack-large` after the transport archive is made. Set their mtimes to a fixed value. - Copy `/provider-pack-large` with `COPY --link`, then copy the small remainder of the pack to the same directory. **Reason and benefit** The dependency layer changes only when the lockfile changes. The large provider-pack layer changes only when a provider version changes. The expected push for a commit that changes source only goes from about 3.5 GB to about 1.5 GB for each architecture. The master build runs about 140 times a day on AWS runners, and this traffic is about 570 GB a day of billed data transfer. Deploy pulls become smaller by the same amount. **Breaking changes** None. The file tree of the image is the same. Provider-pack files of 1 MiB or more now have an mtime of 1970-01-01. ## What Changed - `Dockerfile`: the `production` stage copies `/app/node_modules` from `deps` first, then `/app` from `build` without the root `node_modules`. - `Dockerfile`: the `runner-provider-pack` stage splits large pack files into `/provider-pack-large` with fixed mtimes. The `production` stage copies that tree with `--link` before the remainder of the pack. - `server/src/__tests__/docker-image-layer-reuse.test.ts`: a drift guard for the copy order, the exclude, the `--link`, and the position of the split after the archive step. ## Verification - `cd server && npx vitest run src/__tests__/docker-image-layer-reuse.test.ts src/__tests__/docker-build-stamp.test.ts`: 9 tests pass. The 2 new tests fail against the `Dockerfile` on master. - `node scripts/check-docker-deps-stage.mjs`: PASS. - `pnpm verify`: exit 2 (the `Dockerfile` is a global CI input). Required CI supplies the broad coverage. - A small Dockerfile with the same syntax version (`docker/dockerfile:1.20`) and the same four `COPY` instructions, built two times with a changed stamp file: - `--exclude=node_modules` leaves out only the root `node_modules`. A nested `node_modules` stays. - The dependency layer has the same digest in the two builds. - The large-file layer has the same digest in the two builds only with `--link` and fixed mtimes. Without one of them the digest changes. - Hard links between large files stay hard links. The merged directory contains each file one time. - Evidence that the `build` stage writes nothing to the root `node_modules`: in the published image, all 107,413 files there have an mtime in the minute of the `deps` install. The app build ran 15 hours later. **Not verified:** a full build of the production image with this `Dockerfile`. The local Docker disk was full, and no pre-merge workflow builds this image. The first full build is the master build after merge. The check after merge is to compare the layer digests of two consecutive master images. ## Risks - Medium until the first master build. If the image build fails, master image builds stop until a revert. No broken image can be published, because the build fails before the push. - If a future build step writes to the root `node_modules` in the `build` stage, the image will not contain that change. The comment in the `Dockerfile` states this rule. - A file that crosses the 1 MiB boundary moves between the two pack layers. This only changes which layer is pushed. - The first build after merge pushes all the new layers one time. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with tool use (shell, AWS CLI, Docker, registry API). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
eaef46b60b
commit
a37f3056b6
2 files changed
+75
-1
No files matched your search
+19
-1
@@ -179,6 +179,16 @@ RUN mkdir -p /provider-pack /provider-pack-transport \
|
||||
echo "Skipping remote provider pack: supply a full PAPERCLIP_BUILD_COMMIT to enable remote OpenCode/ACPX execution"; \
|
||||
fi \
|
||||
&& chmod -R a+rX /provider-pack
|
||||
# The pack is ~2.5 GB of provider binaries that change only when a provider
|
||||
# version does, next to small files that change on every commit (the manifest
|
||||
# carries the source revision). Move the large files to their own tree, after
|
||||
# the transport archive has read the whole pack, so production can copy them
|
||||
# as a layer that stays identical between commits. Normalized mtimes keep that
|
||||
# layer's bytes identical when this stage reruns.
|
||||
RUN mkdir -p /provider-pack-large \
|
||||
&& cd /provider-pack \
|
||||
&& find . -type f -size +1048575c -exec sh -c 'mkdir -p "/provider-pack-large/$(dirname "$1")" && mv "$1" "/provider-pack-large/$1"' _ {} \; \
|
||||
&& find /provider-pack-large -exec touch -h -d @0 {} +
|
||||
|
||||
FROM base AS production
|
||||
ARG USER_UID=1000
|
||||
@@ -203,8 +213,16 @@ RUN echo "cli-tools-epoch: ${CLI_TOOLS_CACHE_EPOCH}" \
|
||||
COPY scripts/docker-entrypoint.sh /usr/local/bin/
|
||||
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
|
||||
|
||||
COPY --chown=node:node --from=build /app /app
|
||||
# Installed dependencies are ~2.9 GB of /app and change only with the lockfile.
|
||||
# Copied from the deps stage as their own layer, a commit that leaves the
|
||||
# lockfile alone does not push or pull them again. The build stage writes
|
||||
# nothing under the root node_modules, so the rest of /app is copied without it.
|
||||
COPY --chown=node:node --from=deps /app/node_modules /app/node_modules
|
||||
COPY --chown=node:node --exclude=node_modules --from=build /app /app
|
||||
|
||||
# Large provider binaries first, linked so the layer does not depend on the
|
||||
# per-commit /app layer above; then the small per-commit remainder of the pack.
|
||||
COPY --link --chown=0:0 --from=runner-provider-pack /provider-pack-large/ /opt/paperclip-runner/provider-pack/
|
||||
COPY --chown=0:0 --from=runner-provider-pack /provider-pack /opt/paperclip-runner/provider-pack
|
||||
COPY --from=runner-provider-pack /provider-pack-transport/ /opt/paperclip-runner/
|
||||
# Managed deployments can remap node's UID at startup. This immutable pack
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
/**
|
||||
* Drift guard for the layers of the production image that must stay identical
|
||||
* between commits.
|
||||
*
|
||||
* Installed dependencies (~2.9 GB) and the large provider-pack binaries
|
||||
* (~2.5 GB) change only when the lockfile or a provider version changes. When
|
||||
* they shared a layer with files that change on every commit, each master
|
||||
* build pushed about 3.5 GB of unchanged bytes and each deploy pulled them.
|
||||
*/
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
|
||||
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
|
||||
|
||||
function stageBody(stageName: string): string {
|
||||
const froms = [...dockerfile.matchAll(/^FROM .*$/gm)];
|
||||
const startIdx = froms.findIndex((m) => new RegExp(`\\bAS ${stageName}\\s*$`).test(m[0]));
|
||||
expect(startIdx, `Dockerfile must declare a '${stageName}' stage`).toBeGreaterThanOrEqual(0);
|
||||
return dockerfile.slice(froms[startIdx].index ?? 0, froms[startIdx + 1]?.index ?? dockerfile.length);
|
||||
}
|
||||
|
||||
it("copies installed dependencies as their own layer, before the per-commit app files", () => {
|
||||
const production = stageBody("production");
|
||||
const dependencies = production.search(/^COPY --chown=node:node --from=deps \/app\/node_modules \/app\/node_modules$/m);
|
||||
const app = production.search(/^COPY --chown=node:node --exclude=node_modules --from=build \/app \/app$/m);
|
||||
expect(dependencies, "dependencies must come from the deps stage").toBeGreaterThanOrEqual(0);
|
||||
expect(app, "the app copy must leave out the root node_modules").toBeGreaterThan(dependencies);
|
||||
// A copy of all of /app from the build stage would put the dependencies
|
||||
// back into the per-commit layer.
|
||||
expect(production).not.toMatch(/^COPY (?!.*--exclude=node_modules).*--from=build \/app \/app$/m);
|
||||
});
|
||||
|
||||
it("copies the large provider-pack files as a linked layer with fixed timestamps", () => {
|
||||
const pack = stageBody("runner-provider-pack");
|
||||
const archive = pack.indexOf("create-provider-pack-archive.mjs");
|
||||
const split = pack.indexOf("mv \"$1\" \"/provider-pack-large/$1\"");
|
||||
const timestamps = pack.indexOf("find /provider-pack-large -exec touch -h -d @0 {} +");
|
||||
// The transport archive reads the whole pack, so the split must follow it.
|
||||
expect(archive).toBeGreaterThanOrEqual(0);
|
||||
expect(split).toBeGreaterThan(archive);
|
||||
expect(timestamps).toBeGreaterThan(split);
|
||||
|
||||
const production = stageBody("production");
|
||||
const large = production.search(
|
||||
/^COPY --link --chown=0:0 --from=runner-provider-pack \/provider-pack-large\/ \/opt\/paperclip-runner\/provider-pack\/$/m,
|
||||
);
|
||||
const rest = production.search(
|
||||
/^COPY --chown=0:0 --from=runner-provider-pack \/provider-pack \/opt\/paperclip-runner\/provider-pack$/m,
|
||||
);
|
||||
expect(large, "large files need --link so the layer does not depend on the app layer").toBeGreaterThanOrEqual(0);
|
||||
expect(rest).toBeGreaterThan(large);
|
||||
});
|
||||
Reference in new issue
Block a user