mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after this image is published and attested. > - A master commit takes 29 min (p50) and 56 min (p90) to reach a published image. Every minute here delays a staging fix. > - The run logs show where the time goes. The exact-source gate (about 14 min) is the critical path today. The build legs finish about 3 min before it. After the gate, `merge-and-push` spends about 95 s to pull the 4.9 GB amd64 image for one probe. > - This pull request moves that probe into the build legs, which run before the gate. It also removes a duplicate 187 MB layer from the amd64 image. > - The benefit is about 95–100 s less from gate to signed image on every master commit, with the same proof. arm64 is now also probed. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time from a commit landing on master to the signed standard image (`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation). **Subsystem affected** `.github/workflows/docker.yml`, `Dockerfile`, `scripts/standard-image-contract.mjs`. **Current behavior** Measured on run 43133035209 (sha42fd790e, total 999 s): - Push 01:13:47. The slowest exact-source job, `General tests (server (7/10))`, ends 01:27:08. `Cloud source verified v1` waits 20 s for a runner and ends 01:27:44. The watcher sees it at its next 30 s poll, 01:27:58. - `build-and-push (amd64)` ends 01:24:52 and arm64 ends 01:21:59. Both have slack before the gate. Run 43134250912 shows the same shape: the gate opened 01:44:37 and amd64 ended 01:41:53. - `merge-and-push` starts 01:28:03 and pushes the manifest at 01:28:34. Then "Verify PID 1 reaps orphaned processes" pulls the full amd64 image (01:28:34 to 01:30:09, 95 s). "Verify the immutable standard image" runs the same probe again on the same bytes (5 s). The attestation ends 01:30:22. Last 60 master pushes (2026-11-11 20:17Z to 2026-11-12 03:21Z): 43 cancelled or superseded, 14 succeeded, 2 failed, 1 in progress. Push to first published image that contains the commit: p50 29.4 min, p90 55.5 min, max 85.6 min. A successful run takes 22.5 min (p50) from push. The median gap between pushes is 4.5 min. Where the amd64 "Build and push by digest" step spends its 582 s: | Time | Phase | What happens | | --- | --- | --- | | 0–8 s | Load | Dockerfile frontend, cache manifest import | | 8–54 s | Pull cached layers (46 s) | `deps`, `rust-chef`, `runner-deps`, `runner-build` and `base` are **all cache hits**. The `build` stage starts `FROM runner-build` and copies `deps`, so BuildKit downloads and extracts them: a 932 MB node_modules layer (13 s download, 24.5 s extract) and about 650 MB of Rust `target/` | | 54–108 s | Per-commit build (54 s) | `COPY . .` 10 s, mtime normalization 11 s, UI 5 s, plugin SDK 3 s, server tsc 18.5 s, runtime bundle 5.6 s | | 108–206 s | `runner-provider-pack` (98 s) | Rebuilt every commit: it is `FROM build` and stamps the commit. pnpm install about 25 s, pack 9 s, **Kimi smoke 37 s (amd64 only)**, archive 28 s | | parallel | `production` | Tool layer is a cache hit but pulls 1.11 GB (11 s). `COPY --from=build /app` 24 s | | 206–294 s | Pack copy and chmod | `COPY` pack 7.5 s. `chmod -R a+rX` plus UID check **71 s** (arm64: 20.5 s) | | 294–420 s | Image export (126 s) | 115 s compressing new layers: `/app` 1.40 GB, pack 1.01 GB, pack archives 1.06 GB, chmod copy 0.19 GB (compressed). Push 10 s | | 420–582 s | Cache export (172 s) | `mode=max`: 152 s "preparing" (compressing the per-commit layers of `build` and `runner-provider-pack`, which no later commit can reuse), 20 s upload | What rebuilds every commit, and why: only the `build` stage and everything after it. `PAPERCLIP_BUILD_COMMIT` is already declared after the dependency layers, `CLI_TOOLS_CACHE_EPOCH` is weekly, and dependency install already copies manifests first. The commit ARG is used by the UI build, the server stamp, and the provider pack, so those must rebuild. A pnpm `--mount=type=cache` gives nothing here: the runners are ephemeral, and cache mounts are not exported to the registry cache. Why amd64 is slower than arm64: both legs run on runs-on AWS 32-core instances (m6a.8xlarge and m6g.8xlarge), not on `ubuntu-latest`. The difference is content, not CPU. amd64 has a larger provider pack with the Kimi smoke test (37 s), a 187 MB chmod copy layer (arm64: 0 MB), a 4.9 GB image (arm64: 2.9 GB), and a 172 s cache export (arm64: 100 s). **Proposed behavior** Run the PID 1 probe in each build leg, on its own architecture, against the digest that leg pushed. Before signing, require that every Linux image in the standard index is one of those probed digests. Set the provider pack modes in the stage that writes the pack. **Reason and benefit** The probe pull leaves the post-gate path. The leg has about 3 min of slack before the gate. The chmod change removes about 50 s and 187 MB from the amd64 leg, which offsets most of the probe pull that the leg now does. **Breaking changes** None. Tags, digests, attestation subject, and signer identity are unchanged. **Additional context** The concurrency proposal and the build-in-PR plan are below, in "Risks". They are not implemented. ## What Changed - `docker.yml` `build-and-push`: new step "Verify PID 1 reaps orphaned processes". It runs `scripts/assert-orphan-reaping.sh` in `ghcr.io/...@<pushed digest>` with `--platform` set to the leg platform. It runs before "Export digest", so `merge-and-push` only receives probed digests. - `docker.yml` `merge-and-push`: removed the two `docker run` probes (tag and digest). "Resolve standard image digest" now runs `standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests /tmp/digests`. - `standard-image-contract.mjs`: `--verified-digests <dir>` reads each probed leg digest from the registry, checks that its bytes hash to that digest, and collects its Linux image digests. It fails unless every Linux image in the standard index is in that set. Registry content is addressed by digest, so this proves that the signed index names exactly the probed bytes, without a second pull. - `Dockerfile`: `chmod -R a+rX /provider-pack` now runs in the same `RUN` that writes the pack. `COPY` keeps the modes. The production stage keeps the readability check as UID 65534. - Tests: new contract tests (membership, bare manifest legs, digest mismatch, empty set, lookup by digest). New workflow test: the probe runs between the build and "Export digest", and the publisher does not `docker run`. Updated the provider-pack Dockerfile test. - `doc/standard-image-contract.md`: describes the per-leg probe and the membership check. Expected savings, per master commit: | Change | Saving | Confidence | | --- | --- | --- | | Probe moved out of `merge-and-push` | 95–100 s from gate to attestation (measured: 95 s pull + 5 s second probe) | High | | Pack chmod in its own stage | About 50 s and 187 MB in the amd64 leg (71 s vs 20.5 s on arm64, where the chmod layer is empty) | Medium-high on size, medium on seconds | | Net amd64 leg | About +10 to +45 s (probe pull estimated at 60–95 s, minus about 50 s). Still before the gate | Medium | ## Verification - `node --test scripts/standard-image-contract.test.mjs`: 10 pass (2 new). - `node --test` on `.github/scripts/tests/{cloud-readiness,docker-canary-promotion,post-merge-runner-routing,lockfile-refresh-workflows}.test.mjs`, `scripts/__tests__/release-verify-workflow.test.mjs`, `scripts/reuse-canary-image.test.mjs`, `scripts/standard-image-contract.test.mjs`, `scripts/cloud-source-verification.test.mjs`: 484 pass. - `vitest run server/src/__tests__/docker-build-stamp.test.ts server/src/__tests__/container-init-reaping.test.ts`: 22 pass. Red check: with the `origin/master` `Dockerfile` and `docker.yml`, the 2 new or changed assertions fail. - Real registry, read only: `node scripts/standard-image-contract.mjs --resolve42fd790e80--verified-digests <dir>` with the two leg digests from run 43133035209 returns the published index digest. With only the arm64 leg digest, it fails: "linux/amd64 image sha256:0f0642e6… did not pass the build-leg orphan-reaping probe". - `actionlint .github/workflows/docker.yml`: the same 6 shellcheck findings as `origin/master`, none new. - Shell syntax check (`sh -n`) of the changed pack `RUN`. Not exercised: the workflow itself (it runs only on master pushes, and I did not dispatch it). No local `docker build`: the Docker daemon is not running on this machine, and a full build does not fit a few minutes. The leg probe pull time is an estimate from the publisher runner. ## Risks - The leg now pulls its own image (4.9 GB amd64). AWS legs have about 175 GB free. Hosted tag builds run "Free runner disk" first, and the old publisher pulled the same image on a hosted runner for tag builds. - If the registry returns a leg index in an unexpected form, the resolve step fails closed and no master image is signed. The real-registry check above used today's exact shapes (OCI index with one platform image and one attestation manifest per leg). - Tag builds (`v*`, `nightly/v*`, `beta/v*`) keep the probe, now per leg. They have no standard-image signing step, and the manifest list is made only from probed digests. ### Deliberately not changed - **The exact-source gate.** It is a deliberate "never publish an unverified image" rule (`watch-release-source.mjs --require-proof` and `--check-proof`, `doc/cloud-build-readiness.md`). I kept it. The owner levers are: the slowest `cloud-readiness` shard (`server (7/10)`, 12 min, sets the gate), the 20 s runner wait for `Cloud source verified v1`, and the watcher's 30 s poll (15 s average delay). I did not shorten the poll because the org API rate limit was exhausted today. - **Cache export (172 s amd64, 100 s arm64).** `mode=max` compresses about 2 GB of per-commit pack layers every build. A fix: a separate cache-only build of a small aggregate stage (`deps`, `runner-build`, the production tool layer) with `mode=max`, and the main build with `cache-from` only. Not done here: the legs are not on the master critical path today, and I cannot measure it without a master run. It is worth doing for tag builds or if the gate gets faster. - **Cached layer pull (46 s).** The `build` stage starts `FROM runner-build`, so it downloads about 650 MB of Rust `target/` that it deletes at the end. The server build's `prepare:runner-vendor` uses that target, so this needs a separate change. - **Provider pack per commit (98 s amd64).** It rebuilds because it is `FROM build` and stamps the commit. Its 1.06 GB transport archives also duplicate the 1.01 GB pack inside the image. Keying the pack on its own inputs is a product-contract change. ### Concurrency proposal (not implemented) Today one run holds `docker-${{ github.ref }}` for 17–23 min. Most of that is waiting for the gate, not building. A newer commit waits for that run before its build starts (run 43134250912 waited 6.4 min). Proposal: 1. Remove the workflow-level group. Add job-level groups: `build-and-push` uses `docker-build-${{ github.ref }}-${{ matrix.arch }}` with `cancel-in-progress: false`. At most one build per architecture runs and the newest commit replaces the pending one. So not every commit is built. 2. `merge-and-push` uses `docker-publish-${{ github.ref }}` with `cancel-in-progress: false`. It skips publication when a newer master commit already has a signed image, so the image order stays monotonic. 3. A superseded run must not keep its source watcher alive, or the API load grows with each push. Fold the early watcher into the gate job, or end it when its build job is cancelled. Effect: the newest commit starts building at push, so its image publishes at about gate time plus 45 s, instead of after the previous run ends. That removes the 0–17 min queue wait, which is most of the gap between the p50 of 29 min and the gate time of about 14 min. ### Plan: build in PR CI and promote on merge (not in this PR) What must change: 1. Turn on the GitHub merge queue for `master`. The queue tests a `gh-readonly-queue/...` commit and then fast-forwards `master` to that exact commit, so the SHA stamped in the image (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack `runnerSourceRevision`, the UI bundle) is the final master SHA. 2. On `merge_group`: build both architectures with **no secrets and no registry credentials**, because the code under test is not yet on master. Write each image as an OCI layout artifact. A second job that runs no PR code pushes those artifacts by digest to a staging repository, keyed by the merge-group SHA. 3. Run `cloud-readiness` on `merge_group` for the same SHA, so exact-source proof exists before merge. 4. On the master push, `docker.yml` looks up the staged digests for `github.sha`, checks that the tree and SHA match, retags them as `sha-<full>`, and signs with the same identity (`docker.yml@refs/heads/master`). If no staged image exists, it falls back to the current build. Guarantees to keep: never publish or sign an image without exact-source proof for that SHA. The signer identity and the attested digest stay the same. No secrets in any job that runs non-CODEOWNER code. The PID 1 probe runs on both architectures before signing. Superseded queue entries do not publish. Expected saving: from master push to signed image, about 1–2 min (lookup, retag, attest) instead of 29 min p50 and 56 min p90 today. The work moves before the merge, so the merge itself waits for the queue (about the current gate time of 14 min). A staging fix then reaches Cloud about 25 min sooner at the median. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used via Claude Code with tool use (shell, file edits, read-only GitHub API and registry reads). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
351 lines
20 KiB
Docker
351 lines
20 KiB
Docker
# syntax=docker/dockerfile:1.20
|
|
FROM node:24-trixie-slim AS base
|
|
ARG USER_UID=1000
|
|
ARG USER_GID=1000
|
|
RUN apt-get update \
|
|
&& apt-get install -y --no-install-recommends ca-certificates gosu curl gh git wget ripgrep python3 tini \
|
|
&& rm -rf /var/lib/apt/lists/* \
|
|
&& corepack enable
|
|
|
|
# Modify the existing node user/group to have the specified UID/GID to match host user
|
|
RUN usermod -u $USER_UID --non-unique node \
|
|
&& groupmod -g $USER_GID --non-unique node \
|
|
&& usermod -g $USER_GID -d /paperclip node
|
|
|
|
FROM base AS deps
|
|
WORKDIR /app
|
|
COPY package.json pnpm-workspace.yaml pnpm-lock.yaml .npmrc ./
|
|
COPY cli/package.json cli/
|
|
COPY server/package.json server/
|
|
COPY ui/package.json ui/
|
|
COPY packages/shared/package.json packages/shared/
|
|
COPY packages/db/package.json packages/db/
|
|
COPY packages/adapter-utils/package.json packages/adapter-utils/
|
|
COPY packages/google-sheets-mcp-server/package.json packages/google-sheets-mcp-server/
|
|
COPY packages/kv-demo-mcp-server/package.json packages/kv-demo-mcp-server/
|
|
COPY packages/mcp-server/package.json packages/mcp-server/
|
|
COPY packages/paperclip-eval-kernel/package.json packages/paperclip-eval-kernel/
|
|
COPY packages/paperclip-runner/package.json packages/paperclip-runner/
|
|
COPY packages/skills-catalog/package.json packages/skills-catalog/
|
|
COPY packages/tailscale-https-broker/package.json packages/tailscale-https-broker/
|
|
COPY packages/teams-catalog/package.json packages/teams-catalog/
|
|
COPY packages/adapters/claude-local/package.json packages/adapters/claude-local/
|
|
COPY packages/adapters/codex-local/package.json packages/adapters/codex-local/
|
|
COPY packages/adapters/cursor-cloud/package.json packages/adapters/cursor-cloud/
|
|
COPY packages/adapters/cursor-local/package.json packages/adapters/cursor-local/
|
|
COPY packages/adapters/gemini-local/package.json packages/adapters/gemini-local/
|
|
COPY packages/adapters/grok-local/package.json packages/adapters/grok-local/
|
|
COPY packages/adapters/kimi-local/package.json packages/adapters/kimi-local/
|
|
COPY packages/adapters/hermes/package.json packages/adapters/hermes/
|
|
COPY packages/adapters/hermes-gateway/package.json packages/adapters/hermes-gateway/
|
|
COPY packages/adapters/openclaw-gateway/package.json packages/adapters/openclaw-gateway/
|
|
COPY packages/adapters/opencode-local/package.json packages/adapters/opencode-local/
|
|
COPY packages/adapters/pi-local/package.json packages/adapters/pi-local/
|
|
COPY packages/plugins/sdk/package.json packages/plugins/sdk/
|
|
COPY --parents packages/plugins/sandbox-providers/./*/package.json packages/plugins/sandbox-providers/
|
|
COPY packages/plugins/paperclip-plugin-fake-sandbox/package.json packages/plugins/paperclip-plugin-fake-sandbox/
|
|
COPY packages/plugins/plugin-llm-wiki/package.json packages/plugins/plugin-llm-wiki/
|
|
COPY packages/plugins/plugin-workspace-diff/package.json packages/plugins/plugin-workspace-diff/
|
|
COPY patches/ patches/
|
|
COPY scripts/link-plugin-dev-sdk.mjs scripts/
|
|
COPY packages/paperclip-runner/scripts/install-copilot-assets.mjs packages/paperclip-runner/scripts/materialize-copilot-binary.mjs packages/paperclip-runner/scripts/copilot-inner-distribution.mjs packages/paperclip-runner/scripts/
|
|
COPY packages/paperclip-runner/src/drivers/acpx/copilot-installation.ts packages/paperclip-runner/src/drivers/acpx/
|
|
COPY server/scripts/install-copilot-assets.mjs server/scripts/
|
|
COPY server/src/services/copilot-connection-probe.ts server/src/services/
|
|
|
|
# CI owns repository lock updates. Resolve manifest changes in the image's
|
|
# private build tree, then install from that exact generated lock.
|
|
RUN pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile && pnpm install --frozen-lockfile
|
|
|
|
FROM base AS rust-toolchain
|
|
WORKDIR /app
|
|
# Debian's packaged rust lags the ecosystem (trixie ships 1.85) and the
|
|
# runner's dependency tree now requires a newer rustc. Install rustup from a
|
|
# version-pinned, checksum-verified installer and let the runner's own
|
|
# rust-toolchain.toml choose the compiler — one pin, owned by the runner
|
|
# package, shared by CI and image builds alike.
|
|
#
|
|
# The C toolchain is explicit: apt's cargo used to pull gcc in as a
|
|
# dependency, and rustup does not — without it every build script dies on
|
|
# "linker `cc` not found".
|
|
RUN apt-get update \
|
|
&& apt-get install -y --no-install-recommends gcc libc6-dev pkg-config \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
ENV RUSTUP_HOME=/usr/local/rustup \
|
|
CARGO_HOME=/usr/local/cargo \
|
|
PATH=/usr/local/cargo/bin:$PATH
|
|
ARG RUSTUP_VERSION=1.29.0
|
|
ARG RUSTUP_SHA256_AMD64=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10
|
|
ARG RUSTUP_SHA256_ARM64=9732d6c5e2a098d3521fca8145d826ae0aaa067ef2385ead08e6feac88fa5792
|
|
RUN set -eux; \
|
|
arch="$(dpkg --print-architecture)"; \
|
|
case "$arch" in \
|
|
amd64) rustTarget="x86_64-unknown-linux-gnu"; sha256="$RUSTUP_SHA256_AMD64" ;; \
|
|
arm64) rustTarget="aarch64-unknown-linux-gnu"; sha256="$RUSTUP_SHA256_ARM64" ;; \
|
|
*) echo "unsupported architecture: $arch" >&2; exit 1 ;; \
|
|
esac; \
|
|
curl -fsSLo /tmp/rustup-init "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/${rustTarget}/rustup-init"; \
|
|
echo "${sha256} /tmp/rustup-init" | sha256sum -c -; \
|
|
chmod +x /tmp/rustup-init; \
|
|
/tmp/rustup-init -y --no-modify-path --profile minimal --default-toolchain none; \
|
|
rm /tmp/rustup-init
|
|
# Install the package-owned compiler before any application source enters the
|
|
# stage. rustup-init above installs rustup itself, not the selected compiler.
|
|
COPY packages/paperclip-runner/rust-toolchain.toml /tmp/runner-toolchain/rust-toolchain.toml
|
|
RUN cd /tmp/runner-toolchain && rustup show
|
|
|
|
# Pin the recipe generator and its dependency lockfile. It is a build-only tool
|
|
# and uses the same package-owned compiler as both native build stages.
|
|
FROM rust-toolchain AS rust-chef
|
|
RUN cd /tmp/runner-toolchain && cargo install cargo-chef --version 0.1.73 --locked
|
|
|
|
FROM rust-chef AS runner-plan
|
|
WORKDIR /app/packages/paperclip-runner
|
|
COPY packages/paperclip-runner/rust-toolchain.toml ./
|
|
COPY packages/paperclip-runner/runner ./runner
|
|
RUN cd runner && cargo chef prepare --recipe-path /tmp/runner-recipe.json
|
|
|
|
FROM rust-chef AS runner-deps
|
|
WORKDIR /app/packages/paperclip-runner/runner
|
|
COPY packages/paperclip-runner/rust-toolchain.toml ../
|
|
# The recipe changes only when dependency manifests, the lockfile, or target
|
|
# metadata change. Source edits can reuse this compiled dependency layer.
|
|
COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json
|
|
RUN cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd --recipe-path /tmp/runner-recipe.json \
|
|
&& find . -mindepth 1 -maxdepth 1 ! -name target -exec rm -rf {} +
|
|
|
|
FROM runner-deps AS runner-build
|
|
WORKDIR /app/packages/paperclip-runner
|
|
# Rust embeds protocol schemas and fixtures with include_str!. Keep those
|
|
# alongside the complete Cargo workspace so every compile-time input keys
|
|
# this layer. Ordinary server/UI edits can then reuse the native build.
|
|
COPY packages/paperclip-runner/rust-toolchain.toml ./
|
|
COPY packages/paperclip-runner/runner ./runner
|
|
COPY packages/paperclip-runner/protocol ./protocol
|
|
COPY packages/paperclip-runner/acpx-profiles.json packages/paperclip-runner/antigravity-distributions.json ./
|
|
# Cargo fingerprints source mtimes. Normalize them here and after the full
|
|
# source copy below so a fresh checkout cannot invalidate unchanged inputs.
|
|
RUN find runner protocol -type f -exec touch -d @0 {} + \
|
|
&& touch -d @0 rust-toolchain.toml acpx-profiles.json antigravity-distributions.json \
|
|
&& cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd
|
|
|
|
FROM runner-build AS build
|
|
WORKDIR /app
|
|
COPY --from=deps /app /app
|
|
COPY . .
|
|
RUN find packages/paperclip-runner/runner packages/paperclip-runner/protocol -type f -exec touch -d @0 {} + \
|
|
&& touch -d @0 packages/paperclip-runner/rust-toolchain.toml packages/paperclip-runner/acpx-profiles.json packages/paperclip-runner/antigravity-distributions.json
|
|
# Both the browser bundle and server stamp need the source commit. Declare it
|
|
# after the stable dependency layers, before either application build.
|
|
ARG PAPERCLIP_BUILD_COMMIT=""
|
|
RUN pnpm --filter @paperclipai/ui build
|
|
RUN pnpm --filter @paperclipai/plugin-sdk build
|
|
# The server build runs scripts/write-build-stamp.mjs, which stamps the built
|
|
# commit into dist/build-info.json. The build context has no .git, so the
|
|
# script reads PAPERCLIP_BUILD_COMMIT instead. Docker exposes an ARG to the
|
|
# next RUN as an environment variable. The production stage below declares the
|
|
# same ARG again for the runtime fallback; an ARG goes out of scope at the
|
|
# end of its stage. Empty for local `docker build`, which then writes no stamp.
|
|
# NODE_OPTIONS covers the Node build steps. tsc is a native binary and ignores it.
|
|
ENV NODE_OPTIONS=--max-old-space-size=4096
|
|
# The image needs no declarations, and CI type-checks this source separately.
|
|
# Emitting without either cuts the tsc step from more than 10 GiB to about 2.5 GiB.
|
|
RUN PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build
|
|
RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1)
|
|
RUN pnpm --filter @paperclipai/server build:runtime \
|
|
&& test -f server/dist-runtime/index.mjs || (echo "ERROR: server runtime bundle missing" && exit 1)
|
|
RUN rm -rf packages/paperclip-runner/runner/target
|
|
|
|
# Remote OpenCode and ACPX runs require a controller-owned provider pack to
|
|
# verify the sandbox installation or stage matching assets. Ship it in the
|
|
# standard image so downstream Cloud compositions inherit the same artifacts.
|
|
# Grok's native executable stays an external sandbox prerequisite; this pack
|
|
# contains only its launcher.
|
|
FROM build AS runner-provider-pack
|
|
# Unstamped local builds remain usable, but cannot qualify a remote pack.
|
|
# Never invent a source revision to make an unqualified pack look verified.
|
|
# Managed deployments can remap node's UID at startup, so the pack must stay
|
|
# readable by any UID. Grant that here, in the same layer that writes the
|
|
# files: COPY keeps these modes. A `chmod -R` after the COPY into production
|
|
# rewrote every changed file into a second copy of the pack (187 MB compressed
|
|
# and ~50 s of build time on amd64), and the production stage still checks
|
|
# readability as an unrelated UID below.
|
|
RUN mkdir -p /provider-pack /provider-pack-transport \
|
|
&& if [ -n "${PAPERCLIP_BUILD_COMMIT}" ]; then \
|
|
PAPERCLIP_RUNNER_SOURCE_REVISION="${PAPERCLIP_BUILD_COMMIT}" node packages/paperclip-runner/scripts/build-provider-pack.mjs /provider-pack \
|
|
&& node scripts/create-provider-pack-archive.mjs /provider-pack \
|
|
&& mv /provider-pack.archives /provider-pack.archive.json /provider-pack-transport/; \
|
|
else \
|
|
echo "Skipping remote provider pack: supply a full PAPERCLIP_BUILD_COMMIT to enable remote OpenCode/ACPX execution"; \
|
|
fi \
|
|
&& chmod -R a+rX /provider-pack
|
|
|
|
FROM base AS production
|
|
ARG USER_UID=1000
|
|
ARG USER_GID=1000
|
|
# Refreshes the tool layer below when it changes (CI stamps an ISO week, so
|
|
# the @latest CLI tools advance weekly). Without it the cached layer would
|
|
# freeze the tools until an unrelated cache bust.
|
|
ARG CLI_TOOLS_CACHE_EPOCH=""
|
|
WORKDIR /app
|
|
# Tool and OS layer BEFORE the app copy: it references nothing from /app, and
|
|
# the app copy changes on every commit — ordered the other way around, this
|
|
# (the single most expensive layer: four CLI toolchains + apt, per arch) can
|
|
# never hit the layer cache and rebuilds on every build.
|
|
RUN echo "cli-tools-epoch: ${CLI_TOOLS_CACHE_EPOCH}" \
|
|
&& npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest @moonshot-ai/kimi-code@latest \
|
|
&& apt-get update \
|
|
&& apt-get install -y --no-install-recommends openssh-client jq \
|
|
&& rm -rf /var/lib/apt/lists/* \
|
|
&& mkdir -p /paperclip \
|
|
&& chown node:node /paperclip
|
|
|
|
COPY scripts/docker-entrypoint.sh /usr/local/bin/
|
|
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
|
|
|
|
COPY --chown=node:node --from=build /app /app
|
|
|
|
COPY --chown=0:0 --from=runner-provider-pack /provider-pack /opt/paperclip-runner/provider-pack
|
|
COPY --from=runner-provider-pack /provider-pack-transport/ /opt/paperclip-runner/
|
|
# Managed deployments can remap node's UID at startup. This immutable pack
|
|
# contains public code and integrity metadata, never credentials; it must remain
|
|
# readable afterward. The runner-provider-pack stage grants a+rX before this
|
|
# COPY, which preserves modes.
|
|
# Keep it root-owned and verify access as an unrelated unprivileged UID.
|
|
RUN if [ -f /opt/paperclip-runner/provider-pack/provider-pack.json ]; then \
|
|
gosu 65534:65534 node -e 'const fs = require("node:fs"); const path = require("node:path"); const root = "/opt/paperclip-runner/provider-pack"; const manifest = JSON.parse(fs.readFileSync(path.join(root, "provider-pack.json"), "utf8")); for (const artifact of Object.values(manifest.payload.artifacts)) fs.readFileSync(path.join(root, artifact.path)); fs.accessSync(path.join(root, manifest.payload.artifacts.nodeCommand.path), fs.constants.X_OK);'; \
|
|
fi
|
|
ENV PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH=/opt/paperclip-runner/provider-pack
|
|
|
|
# Declare per-build metadata after the stable RUN layers. Docker includes
|
|
# in-scope ARG values in a RUN's environment even when its command does not
|
|
# mention them; declaring these earlier invalidates the weekly tool cache.
|
|
# The build stage still receives the commit before writing dist/build-info.json.
|
|
# Empty for local builds, preserving the server's normal version fallbacks.
|
|
ARG PAPERCLIP_BUILD_VERSION=""
|
|
ARG PAPERCLIP_BUILD_COMMIT=""
|
|
ENV NODE_ENV=production \
|
|
HOME=/paperclip \
|
|
HOST=0.0.0.0 \
|
|
PORT=3100 \
|
|
SERVE_UI=true \
|
|
PAPERCLIP_HOME=/paperclip \
|
|
PAPERCLIP_INSTANCE_ID=default \
|
|
PAPERCLIP_BUILD_VERSION=${PAPERCLIP_BUILD_VERSION} \
|
|
PAPERCLIP_BUILD_COMMIT=${PAPERCLIP_BUILD_COMMIT} \
|
|
USER_UID=${USER_UID} \
|
|
USER_GID=${USER_GID} \
|
|
PAPERCLIP_CONFIG=/paperclip/instances/default/config.json \
|
|
PAPERCLIP_DEPLOYMENT_MODE=authenticated \
|
|
PAPERCLIP_DEPLOYMENT_EXPOSURE=private \
|
|
OPENCODE_ALLOW_ALL_MODELS=true \
|
|
GEMINI_SANDBOX=false
|
|
|
|
EXPOSE 3100
|
|
|
|
# tini, not node, is PID 1. The entrypoint ends in `exec`, so without an init
|
|
# node inherits PID 1 and never wait()s the orphans the kernel re-parents onto
|
|
# it -- agent runs spawn git/claude/esbuild/sh descendants that outlive their
|
|
# leader, so they pile up as permanent zombies (~79/h measured) until the
|
|
# cgroup pid limit is exhausted and *every* fork() in the container fails.
|
|
# tini reaps adopted orphans and forwards signals, so the exec chain below and
|
|
# graceful shutdown are unchanged. Mirrors docker/agent-runtime/Dockerfile.base.
|
|
ENTRYPOINT ["/usr/bin/tini", "-s", "--", "docker-entrypoint.sh"]
|
|
CMD ["node", "server/dist-runtime/index.mjs"]
|
|
|
|
# Cloud image variant (build with `--target cloud`): the production image
|
|
# plus built bundled sandbox-provider plugins. Managed instances receive a
|
|
# `plugins.autoInstall` key list through PAPERCLIP_MANAGED_CONFIG and
|
|
# install those plugins from the bundled catalog at boot
|
|
# (server/src/services/bundled-plugins.ts), which requires each plugin's
|
|
# dist/ to exist in the image — the default image ships only their source,
|
|
# so auto-install logs "bundle not present" and skips. The plugins are
|
|
# built in this separate target so the default (self-hosted) image stays
|
|
# lean; CI pins the default build to `--target production`. Both targets
|
|
# inherit the build-owned remote provider pack from the standard image.
|
|
#
|
|
# The sandbox providers are intentionally excluded from the pnpm workspace
|
|
# (see pnpm-workspace.yaml), so each installs standalone exactly as its
|
|
# README prescribes. Installing in a `build`-based stage (not `production`)
|
|
# keeps devDependencies available for tsc: `production` sets
|
|
# NODE_ENV=production, which would make pnpm skip them.
|
|
#
|
|
# CLOUD_BUNDLED_PLUGINS is the space-separated list of sandbox-provider
|
|
# directory names to build into the variant. Only what managed deployments
|
|
# actually auto-install belongs here — every entry adds its node_modules
|
|
# to the image. Growing the list is a one-line workflow change.
|
|
FROM build AS cloud-plugins
|
|
ARG CLOUD_BUNDLED_PLUGINS="daytona"
|
|
RUN set -eu; \
|
|
for name in $CLOUD_BUNDLED_PLUGINS; do \
|
|
dir="packages/plugins/sandbox-providers/$name"; \
|
|
test -d "$dir" || { echo "ERROR: unknown sandbox provider '$name'" >&2; exit 1; }; \
|
|
pnpm -C "$dir" install --ignore-workspace --no-lockfile; \
|
|
pnpm -C "$dir" build; \
|
|
test -f "$dir/dist/manifest.js" || { echo "ERROR: $dir is missing dist/manifest.js after build" >&2; exit 1; }; \
|
|
done
|
|
|
|
# The hosted image variant ships selected optional peer packages
|
|
# pre-installed. A managed tenant then needs no separate install step.
|
|
# The self-hosted image stays on the opt-in contract: it never runs this
|
|
# stage, so a package like `@sentry/node` stays a true optional peer
|
|
# dependency. A self-hosted operator installs it by hand (see
|
|
# doc/observability.md).
|
|
#
|
|
# CLOUD_BUNDLED_SERVER_DEPS names the optional peer packages to install.
|
|
# The value is a space-separated list, the same shape as
|
|
# CLOUD_BUNDLED_PLUGINS above. The stage reads each package's version
|
|
# from the `peerDependencies` block of `server/package.json` at build
|
|
# time, so the version has one committed home.
|
|
#
|
|
# The stage fails the build in three cases:
|
|
# - the argument is empty
|
|
# - server/package.json declares no version for a named package
|
|
# - the named package is not an optional peer
|
|
#
|
|
# This check keeps the argument limited to packages the server already
|
|
# treats as optional.
|
|
#
|
|
# The install happens in its own isolated directory, not inside
|
|
# `server`'s own workspace install. The self-hosted target above never
|
|
# gains these packages this way. The directory sits under `/app`, not
|
|
# `server/`, and `--ignore-workspace` below excludes it from the pnpm
|
|
# workspace. From that directory, pnpm still finds the `packageManager`
|
|
# pin in the repo's own `package.json` by walking up — the same pnpm
|
|
# version the rest of the build uses.
|
|
#
|
|
# The install writes no lock file (`--no-lockfile`, the same flag the
|
|
# `cloud-plugins` stage above uses). Two builds of the same commit can
|
|
# therefore install different transitive versions of a named package.
|
|
# Three facts make this an accepted trade-off:
|
|
# - the `cloud-plugins` stage above already has the same property, with
|
|
# the same flag
|
|
# - the direct version of each named package comes from one exact,
|
|
# single-sourced place: the `peerDependencies` block of
|
|
# `server/package.json`
|
|
# - an automated check asserts the installed direct version after every
|
|
# build, so a transitive drift that breaks the package still fails the
|
|
# build
|
|
FROM build AS cloud-server-deps
|
|
WORKDIR /app/.cloud-server-deps
|
|
ARG CLOUD_BUNDLED_SERVER_DEPS="@sentry/node"
|
|
RUN set -eu; \
|
|
test -n "$CLOUD_BUNDLED_SERVER_DEPS" || { echo "ERROR: CLOUD_BUNDLED_SERVER_DEPS is empty; name at least one optional peer package to install" >&2; exit 1; }; \
|
|
echo '{"name":"paperclip-cloud-server-deps","private":true}' > package.json; \
|
|
specifiers=""; \
|
|
for name in $CLOUD_BUNDLED_SERVER_DEPS; do \
|
|
version="$(node -e "const pkg=require('/app/server/package.json'); const name=process.argv[1]; const version=(pkg.peerDependencies||{})[name]; if(!version){console.error('ERROR: server/package.json declares no peerDependencies version for '+JSON.stringify(name));process.exit(1);} const meta=(pkg.peerDependenciesMeta||{})[name]; if(!meta||meta.optional!==true){console.error('ERROR: '+JSON.stringify(name)+' is not declared as an optional peer dependency in server/package.json; CLOUD_BUNDLED_SERVER_DEPS may name only optional peer packages');process.exit(1);} process.stdout.write(version);" "$name")"; \
|
|
test -n "$version" || { echo "ERROR: could not resolve a version for '$name'" >&2; exit 1; }; \
|
|
specifiers="$specifiers ${name}@${version}"; \
|
|
done; \
|
|
test -n "$specifiers" || { echo "ERROR: CLOUD_BUNDLED_SERVER_DEPS names no package" >&2; exit 1; }; \
|
|
pnpm add --ignore-workspace --no-lockfile $specifiers
|
|
|
|
FROM production AS cloud
|
|
COPY --chown=node:node --from=cloud-plugins /app/packages/plugins/sandbox-providers /app/packages/plugins/sandbox-providers
|
|
# Land the isolated install inside the server's own `node_modules`, the
|
|
# directory Node's module resolution walks up to from `/app/server` for
|
|
# both a CommonJS `require.resolve` and an ECMAScript `import` — an entry
|
|
# on `NODE_PATH` would satisfy only the first and silently fail the second.
|
|
COPY --chown=node:node --from=cloud-server-deps /app/.cloud-server-deps/node_modules /app/server/node_modules
|