perf(docker): keep dependencies and provider binaries in layers that survive a commit (#21167)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each merge to master builds the production Docker image for amd64
and arm64 and pushes it to GHCR
> - Each build pushes three layers again, about 3.5 GB compressed of a
4.7 GB image
> - A file-by-file comparison of two consecutive master images shows
that about 95% of those bytes do not change between commits
> - The unchanged bytes are in the same layers as small files that
change on every commit, so the registry sees new layers
> - This pull request moves the unchanged bytes into their own layers
> - The benefit is a smaller push for each master build and a smaller
pull for each deploy

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The production image build. It pushes about 3.5 GB of layers for each
architecture on each master commit. Each deploy pulls the same bytes
again.

**Subsystem affected**

Docker image build (`Dockerfile`, `production` and
`runner-provider-pack` stages).

**Current behavior**

I compared the amd64 images of two consecutive master commits (`2174f2d`
and `dfc510b`, 15 changed server source files). I hashed each file in
the three layers that changed.

| Layer | Compressed | Content that is identical between the two builds
|
| --- | --- | --- |
| `COPY --from=build /app /app` | 1,404 MB | 97.5%. The root
`node_modules` is 2.86 GB of 4.13 GB and has 0 changed files. |
| `COPY /provider-pack` | 1,008 MB | 100% of the 112 files of 1 MiB or
more (2.54 GB). Only 32 small files change (manifest, `.modules.yaml`,
`.bin` shims). |
| `COPY /provider-pack-transport/` | 1,059 MB | The archives differ only
in tar timestamps. This pull request does not change this layer. |

**Proposed behavior**

- Copy the root `node_modules` from the `deps` stage as its own layer.
Copy the remainder of `/app` from the `build` stage with
`--exclude=node_modules`.
- In the `runner-provider-pack` stage, move pack files of 1 MiB or more
to `/provider-pack-large` after the transport archive is made. Set their
mtimes to a fixed value.
- Copy `/provider-pack-large` with `COPY --link`, then copy the small
remainder of the pack to the same directory.

**Reason and benefit**

The dependency layer changes only when the lockfile changes. The large
provider-pack layer changes only when a provider version changes. The
expected push for a commit that changes source only goes from about 3.5
GB to about 1.5 GB for each architecture. The master build runs about
140 times a day on AWS runners, and this traffic is about 570 GB a day
of billed data transfer. Deploy pulls become smaller by the same amount.

**Breaking changes**

None. The file tree of the image is the same. Provider-pack files of 1
MiB or more now have an mtime of 1970-01-01.

## What Changed

- `Dockerfile`: the `production` stage copies `/app/node_modules` from
`deps` first, then `/app` from `build` without the root `node_modules`.
- `Dockerfile`: the `runner-provider-pack` stage splits large pack files
into `/provider-pack-large` with fixed mtimes. The `production` stage
copies that tree with `--link` before the remainder of the pack.
- `server/src/__tests__/docker-image-layer-reuse.test.ts`: a drift guard
for the copy order, the exclude, the `--link`, and the position of the
split after the archive step.

## Verification

- `cd server && npx vitest run
src/__tests__/docker-image-layer-reuse.test.ts
src/__tests__/docker-build-stamp.test.ts`: 9 tests pass. The 2 new tests
fail against the `Dockerfile` on master.
- `node scripts/check-docker-deps-stage.mjs`: PASS.
- `pnpm verify`: exit 2 (the `Dockerfile` is a global CI input).
Required CI supplies the broad coverage.
- A small Dockerfile with the same syntax version
(`docker/dockerfile:1.20`) and the same four `COPY` instructions, built
two times with a changed stamp file:
- `--exclude=node_modules` leaves out only the root `node_modules`. A
nested `node_modules` stays.
  - The dependency layer has the same digest in the two builds.
- The large-file layer has the same digest in the two builds only with
`--link` and fixed mtimes. Without one of them the digest changes.
- Hard links between large files stay hard links. The merged directory
contains each file one time.
- Evidence that the `build` stage writes nothing to the root
`node_modules`: in the published image, all 107,413 files there have an
mtime in the minute of the `deps` install. The app build ran 15 hours
later.

**Not verified:** a full build of the production image with this
`Dockerfile`. The local Docker disk was full, and no pre-merge workflow
builds this image. The first full build is the master build after merge.
The check after merge is to compare the layer digests of two consecutive
master images.

## Risks

- Medium until the first master build. If the image build fails, master
image builds stop until a revert. No broken image can be published,
because the build fails before the push.
- If a future build step writes to the root `node_modules` in the
`build` stage, the image will not contain that change. The comment in
the `Dockerfile` states this rule.
- A file that crosses the 1 MiB boundary moves between the two pack
layers. This only changes which layer is pushed.
- The first build after merge pushes all the new layers one time.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, AWS CLI, Docker, registry API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-12 13:48:38 -06:00
1 parent eaef46b60b
commit a37f3056b6
2 files changed
+75 -1

No files matched your search

+19 -1
View File
@@ -179,6 +179,16 @@ RUN mkdir -p /provider-pack /provider-pack-transport \
echo "Skipping remote provider pack: supply a full PAPERCLIP_BUILD_COMMIT to enable remote OpenCode/ACPX execution"; \
fi \
&& chmod -R a+rX /provider-pack
# The pack is ~2.5 GB of provider binaries that change only when a provider
# version does, next to small files that change on every commit (the manifest
# carries the source revision). Move the large files to their own tree, after
# the transport archive has read the whole pack, so production can copy them
# as a layer that stays identical between commits. Normalized mtimes keep that
# layer's bytes identical when this stage reruns.
RUN mkdir -p /provider-pack-large \
&& cd /provider-pack \
&& find . -type f -size +1048575c -exec sh -c 'mkdir -p "/provider-pack-large/$(dirname "$1")" && mv "$1" "/provider-pack-large/$1"' _ {} \; \
&& find /provider-pack-large -exec touch -h -d @0 {} +
FROM base AS production
ARG USER_UID=1000
@@ -203,8 +213,16 @@ RUN echo "cli-tools-epoch: ${CLI_TOOLS_CACHE_EPOCH}" \
COPY scripts/docker-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
COPY --chown=node:node --from=build /app /app
# Installed dependencies are ~2.9 GB of /app and change only with the lockfile.
# Copied from the deps stage as their own layer, a commit that leaves the
# lockfile alone does not push or pull them again. The build stage writes
# nothing under the root node_modules, so the rest of /app is copied without it.
COPY --chown=node:node --from=deps /app/node_modules /app/node_modules
COPY --chown=node:node --exclude=node_modules --from=build /app /app
# Large provider binaries first, linked so the layer does not depend on the
# per-commit /app layer above; then the small per-commit remainder of the pack.
COPY --link --chown=0:0 --from=runner-provider-pack /provider-pack-large/ /opt/paperclip-runner/provider-pack/
COPY --chown=0:0 --from=runner-provider-pack /provider-pack /opt/paperclip-runner/provider-pack
COPY --from=runner-provider-pack /provider-pack-transport/ /opt/paperclip-runner/
# Managed deployments can remap node's UID at startup. This immutable pack
@@ -0,0 +1,56 @@
import { readFileSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { expect, it } from "vitest";
/**
* Drift guard for the layers of the production image that must stay identical
* between commits.
*
* Installed dependencies (~2.9 GB) and the large provider-pack binaries
* (~2.5 GB) change only when the lockfile or a provider version changes. When
* they shared a layer with files that change on every commit, each master
* build pushed about 3.5 GB of unchanged bytes and each deploy pulled them.
*/
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
function stageBody(stageName: string): string {
const froms = [...dockerfile.matchAll(/^FROM .*$/gm)];
const startIdx = froms.findIndex((m) => new RegExp(`\\bAS ${stageName}\\s*$`).test(m[0]));
expect(startIdx, `Dockerfile must declare a '${stageName}' stage`).toBeGreaterThanOrEqual(0);
return dockerfile.slice(froms[startIdx].index ?? 0, froms[startIdx + 1]?.index ?? dockerfile.length);
}
it("copies installed dependencies as their own layer, before the per-commit app files", () => {
const production = stageBody("production");
const dependencies = production.search(/^COPY --chown=node:node --from=deps \/app\/node_modules \/app\/node_modules$/m);
const app = production.search(/^COPY --chown=node:node --exclude=node_modules --from=build \/app \/app$/m);
expect(dependencies, "dependencies must come from the deps stage").toBeGreaterThanOrEqual(0);
expect(app, "the app copy must leave out the root node_modules").toBeGreaterThan(dependencies);
// A copy of all of /app from the build stage would put the dependencies
// back into the per-commit layer.
expect(production).not.toMatch(/^COPY (?!.*--exclude=node_modules).*--from=build \/app \/app$/m);
});
it("copies the large provider-pack files as a linked layer with fixed timestamps", () => {
const pack = stageBody("runner-provider-pack");
const archive = pack.indexOf("create-provider-pack-archive.mjs");
const split = pack.indexOf("mv \"$1\" \"/provider-pack-large/$1\"");
const timestamps = pack.indexOf("find /provider-pack-large -exec touch -h -d @0 {} +");
// The transport archive reads the whole pack, so the split must follow it.
expect(archive).toBeGreaterThanOrEqual(0);
expect(split).toBeGreaterThan(archive);
expect(timestamps).toBeGreaterThan(split);
const production = stageBody("production");
const large = production.search(
/^COPY --link --chown=0:0 --from=runner-provider-pack \/provider-pack-large\/ \/opt\/paperclip-runner\/provider-pack\/$/m,
);
const rest = production.search(
/^COPY --chown=0:0 --from=runner-provider-pack \/provider-pack \/opt\/paperclip-runner\/provider-pack$/m,
);
expect(large, "large files need --link so the layer does not depend on the app layer").toBeGreaterThanOrEqual(0);
expect(rest).toBeGreaterThan(large);
});