mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner qualification uses a Daytona runner image and a nightly Runner Full-Stack E2E workflow > - The nightly workflow fails at "Validate catalog and select cells", and a local image build fails at `sha256sum -c` > - Operators cannot build a current Daytona image for restart tests until both failures are understood > - This pull request removes a stale Dockerfile lock digest default, fixes the local recipes, and makes the nightly failure name its real cause > - The benefit is that a local build fails clearly when the digest is missing, and an expired overhead allowance is obvious from the log ## Linked Issues or Issue Description **What happened?** 1. The nightly `Runner Full-Stack E2E` runs 41997383733 and 42190298486 fail in "Validate catalog and select cells". The log shows `digest-mismatch: error`, but that line is only the input list of `actions/download-artifact`. The real error is `Operating overhead allowance needs review before paid admission`. The repository variable `RUNNER_E2E_OPERATING_OVERHEAD_JSON` has `expiresAt: 2026-11-03T19:49:53Z`. `scheduledOperatingOverhead` rejects an allowance that expires within 4 hours. Manual dispatch (run 42046147744) does not use the scheduled overhead path, so it passed this job. 2. `docker/daytona-runner/Dockerfile` set `ARG PAPERCLIP_RUNNER_LOCK_SHA256=342a01e4…`. That value went stale when `pnpm-lock.yaml` changed (#19443). The local recipes in `tests/runner-e2e/README.md` and `docker/daytona-runner/README.md` did not pass the argument, so local builds failed with a confusing checksum mismatch. **Expected behavior** A local build computes and passes the lockfile digest. A missing digest fails by name. An expired overhead allowance tells the operator which allowance expired and which variable to renew. **Steps to reproduce** 1. Run `gh run view 42190298486 --log` and read the "Validate catalog and select cells" job. 2. Follow the old "## Daytona image" recipe in `tests/runner-e2e/README.md` on current master. ## What Changed - `docker/daytona-runner/Dockerfile`: remove the fixed `PAPERCLIP_RUNNER_LOCK_SHA256` default. Reject a missing or non-SHA-256 value by name before `sha256sum -c` and before installation. CI already passes the real value. - `tests/runner-e2e/README.md` and `docker/daytona-runner/README.md`: compute `shasum -a 256 pnpm-lock.yaml` and pass `PAPERCLIP_RUNNER_LOCK_SHA256` in the local build recipes. Replace the text about a "fixed Dockerfile default". - `tests/runner-e2e/daytona-image.test.ts`: new test. It requires a default-free `ARG`, the shape check before the checksum, and the argument in every documented local recipe. - `tests/runner-e2e/operating-overhead.ts`: the scheduled admission error now names the allowance reference, its validity window, the 4-hour margin, and `RUNNER_E2E_OPERATING_OVERHEAD_JSON`. The check itself is unchanged. ## Verification - `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/operating-overhead.test.ts tests/runner-e2e/daytona-image.test.ts tests/runner-e2e/scheduled-operations.test.ts tests/runner-e2e/cost-ledger.test.ts`: 55 passed. - `pnpm test:e2e:runner:typecheck`: passed. - `pnpm verify`: exit 2 (README is a global CI input). Required CI supplies broad coverage. - `tests/runner-e2e/workflow-security.test.ts` has 5 local failures on files that this PR does not touch (`pr-trusted.yml`, `chat-catalog-testops-case.yml`). They are not caused by this change. - I did not run a Docker build or a paid campaign. ## Risks - Low risk. Any external script that builds this Dockerfile without `PAPERCLIP_RUNNER_LOCK_SHA256` now fails with a clear message. Before, it failed with a checksum mismatch whenever the lockfile changed. Both CI build paths already pass the argument. - The Dockerfile is an image content-ID input, so the next image gets a new content tag. Current master needs a new image anyway. - The nightly workflow stays red until an operator reviews and renews `RUNNER_E2E_OPERATING_OVERHEAD_JSON`. This PR does not renew the allowance, because that is a reviewed spend decision. - Scheduled runs set `needs_daytona=false`, so the nightly run does not publish a Daytona image even when the catalog passes. A current image comes from a manual dispatch that selects Daytona cells. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
197 lines
11 KiB
Markdown
197 lines
11 KiB
Markdown
# Paperclip Daytona runner image
|
||
|
||
Ordinary provider packs now include pinned Cursor assets for Linux x64; no
|
||
`--candidate-providers=cursor` flag is needed. Cursor admission is enabled after
|
||
the [production readiness gates](../../doc/plans/2026-10-03-cursor-production-readiness.md)
|
||
passed. The public CLI installation command is `paperclipai runtime setup cursor`;
|
||
it runs explicitly on the execution host and is never an npm installation hook.
|
||
|
||
This image is the Paperclip Cloud fleet sandbox image plus a source-built
|
||
`paperclip-runnerd` and immutable provider pack. The pack contains Node 24.11,
|
||
OpenCode 1.18.34, the compiled OpenCode proxy, ACPX 0.13.1 sidecar, qualified ACP
|
||
agents, and the production lockfile. Its manifest digests each executable bridge
|
||
and records the runner source revision as provenance. Its interface declaration
|
||
allows compatible app releases to reuse it without uploading or installing a pack.
|
||
|
||
The fleet pins are intentionally copied from
|
||
[`paperclip-cloud/fleet-sandbox-image/Dockerfile`](https://github.com/paperclipai/paperclip-cloud/blob/master/fleet-sandbox-image/Dockerfile).
|
||
Update both definitions together until the fleet base is published as a stable
|
||
image that this Dockerfile can extend directly.
|
||
|
||
## Harness versions
|
||
|
||
The October 2026 refreshes pin Codex 0.160.0, Claude Agent SDK
|
||
0.3.286 (Claude Code 2.1.286), and OpenCode 1.18.34 in the shared provider
|
||
pack. Claude Code 2.1.280 is the minimum for
|
||
[Opus 5.5](https://code.claude.com/docs/en/model-config) and 2.1.284 is the
|
||
minimum for Sonnet 5.5; the pinned runtime also supports Fable 5.1. Codex
|
||
0.160.0 ships bundled metadata for the current
|
||
[GPT-6.1 Sol, GPT-6 Sol, and GPT-6 Luna model IDs](https://learn.chatgpt.com/docs/models).
|
||
The sandbox layer pins Grok CLI 1.0.70, Gemini CLI 0.65.0, Kimi Code 2.3.3,
|
||
Cursor CLI 2026.10.22-27dcf5f, and GitHub CLI 2.103.3 (October 2026
|
||
refresh). The native Cursor provider uses its separately verified distribution,
|
||
pinned to 2026.09.26-dd393fe; it does not use the sandbox layer’s global Cursor
|
||
executable. Grok CLI 1.0.70 supports the current
|
||
[Grok 4.7](https://docs.x.ai/developers/grok-4-7) model family. Hermes stays
|
||
at 0.19.0, the newest release on PyPI.
|
||
|
||
Keep the patched ACP bridge versions separate from their CLI runtime pins.
|
||
Their executable digests do not change when only the runtime dependency
|
||
changes. Refresh the runtime executable digests from integrity-verified npm
|
||
release archives for every supported platform, and keep the native runner,
|
||
provider manifest, and remote controller version checks aligned.
|
||
|
||
## Build and verify
|
||
|
||
Run `pnpm --filter @paperclipai/paperclip-runner test:opencode:qualification`
|
||
after installing dependencies to exercise the actual pinned OpenCode executable.
|
||
It checks health/version, session creation and retrieval, SSE messages, an async
|
||
prompt, and session deletion against a loopback mock provider. It uses an
|
||
isolated home, starts no paid model request, and retires its process group.
|
||
Set `PAPERCLIP_TEST_OPENCODE_BINARY` to the materialized Linux executable when
|
||
qualifying an assembled provider pack.
|
||
|
||
|
||
The fleet image is currently amd64-only because the pinned Cursor and GitHub CLI
|
||
checksums cover amd64.
|
||
|
||
```bash
|
||
content_id="$(pnpm --silent test:e2e:runner:image-id)"
|
||
docker buildx build \
|
||
--platform linux/amd64 \
|
||
--build-arg PAPERCLIP_RUNNER_CONTENT_ID="${content_id}" \
|
||
--build-arg PAPERCLIP_RUNNER_SOURCE_REVISION="$(git rev-parse HEAD)" \
|
||
--build-arg PAPERCLIP_RUNNER_LOCK_SHA256="$(shasum -a 256 pnpm-lock.yaml | cut -d ' ' -f 1)" \
|
||
--tag "paperclip-daytona-runner:e2e-content-${content_id}" \
|
||
--load \
|
||
--file docker/daytona-runner/Dockerfile \
|
||
.
|
||
|
||
docker run --rm --platform linux/amd64 \
|
||
--entrypoint paperclip-runnerd \
|
||
"paperclip-daytona-runner:e2e-content-${content_id}" \
|
||
--build-metadata
|
||
```
|
||
|
||
The metadata must advertise `dial_ws_loopback`, `dial_wss`, and `listen_ws`.
|
||
The explicit entrypoint is needed only for this local probe because Daytona's
|
||
base image uses its own long-running sandbox entrypoint.
|
||
|
||
`test:e2e:runner:image-id` hashes the audited Docker build dependency closure,
|
||
target platform, the immutable Dockerfile syntax-frontend digest, and every
|
||
immutable `FROM` reference. It fails before the paid workflow can build when
|
||
the frontend or a base is not pinned to a sha256 digest. When updating the
|
||
syntax version, resolve and review its registry digest and update both values in
|
||
the first Dockerfile line. Git commits that do not change those inputs reuse the
|
||
same content tag.
|
||
|
||
`PAPERCLIP_RUNNER_SOURCE_REVISION` remains the full Git SHA that built the first
|
||
published copy and is retained as provenance rather than cache identity.
|
||
|
||
## Use in Paperclip
|
||
|
||
Publish the image to a registry Daytona can pull, or use the environment
|
||
editor's **Configure image** flow to produce a Daytona snapshot. Set the
|
||
environment image to that immutable tag or snapshot. Paperclip probes the
|
||
sandbox user's `PATH` for `paperclip-runnerd` and `codex` and checks
|
||
`/opt/paperclip-runner/provider-pack` for OpenCode and ACPX. It uses the pack
|
||
when its selected provider interface and platform are compatible. App source
|
||
revisions and unrelated provider pins do not force an update. Startup reads the
|
||
small manifest in the existing runner metadata query. It does not scan the pack,
|
||
run a version command, or transfer files on this path. Otherwise it stages the pack configured by
|
||
`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH`. Remote OpenCode and ACPX never
|
||
fall back to host-local processes.
|
||
|
||
`src/live/provider-pack-contract.json` in the Runner package declares the
|
||
OpenCode and ACPX interface revisions. Change a revision only for an incompatible
|
||
runner/pack boundary change, and qualify the new pack before publishing it.
|
||
Bumping a revision also retires every older image pack.
|
||
The pack build also records a `contractDigest`, computed by
|
||
`scripts/contracts/contract-fingerprints.mjs` from the JSON contracts under
|
||
`protocol/` and `generated/`, from Rust serde shapes, payload keys, match
|
||
literals and env keys, and from the Runner's ACPX patch, rather than from
|
||
source bytes. An image pack is reused
|
||
only when its digest equals the controller's bundled pack; comments and
|
||
refactors leave it unchanged, so a mismatch means a contract changed and the
|
||
controller uploads its bundled pack. Packs without a digest are not reused.
|
||
Define a new runner/pack contract in a schema under `protocol/` and generate the
|
||
TypeScript from it, as the sidecar's `session.open` parameter allowlist is
|
||
generated from `acpx-sidecar.schema.json`; TypeScript-only definitions are not
|
||
part of the digest.
|
||
Dependency versions and Git SHAs remain build provenance. ACPX retains its
|
||
selected agent's qualified profile checks.
|
||
|
||
The image build makes the pack root-owned and non-writable to the execution user. Fast admission
|
||
requires protected ancestors and manifest, with no writable path alias. Task-owned
|
||
packs and older packs without an interface declaration keep full content
|
||
verification. An attached session retains the exact pack and daemon identity
|
||
admitted at launch. Publish one new image with the interface declaration to enable
|
||
reuse across later compatible app releases.
|
||
|
||
Do not promote `paperclip-runner-e2e-20260826-v2` for OpenCode or ACPX. Build a
|
||
new immutable image or snapshot from a clean committed revision and pass that
|
||
full Git SHA as `PAPERCLIP_RUNNER_SOURCE_REVISION`.
|
||
|
||
Do not bake provider credentials, Paperclip bootstrap tickets, or Daytona
|
||
preview tokens into this image. They remain per-run secret material.
|
||
|
||
The provider-pack build pins the official Linux x64 Node 24.21.0 image by
|
||
manifest digest. Its bundled Undici is 7.29.1, which fixes
|
||
[GHSA-3wwx-pv8p-q78v](https://github.com/advisories/GHSA-3wwx-pv8p-q78v).
|
||
Pi separately verifies its private Node executable and nested npm dependency;
|
||
changing the outer interpreter does not replace either provider-owned pin.
|
||
|
||
Provider CLI updates are manifest-only changes: repository CI owns the root
|
||
lockfile. Resolve the complete workspace manifest graph in the build context
|
||
before invoking Docker, matching CI when a source commit precedes the lockfile
|
||
bot. The trusted workflow supplies this resolved lockfile as an immutable artifact.
|
||
The complete resolved lockfile must match `PAPERCLIP_RUNNER_LOCK_SHA256` before
|
||
package installation or lifecycle execution. Review and refresh that digest
|
||
with source dependency changes; registry-time resolution drift fails closed.
|
||
The Product E2E workflow resolves one lockfile before the image build. It
|
||
verifies the downloaded artifact, then passes that artifact's SHA-256 as the
|
||
`PAPERCLIP_RUNNER_LOCK_SHA256` build argument. The Dockerfile checks the resolved
|
||
lock against this value before installation. The Dockerfile has no default
|
||
digest: standalone builds pass the SHA-256 of the lockfile in their build
|
||
context, and a missing or malformed value fails before installation instead of
|
||
surfacing as a checksum mismatch. When a source change needs a fresh lockfile,
|
||
resolve it from the clean tracked lockfile using the exact
|
||
`pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile` command,
|
||
and verify a second resolution preserves the digest. A lockfile left by a
|
||
filtered or incremental install can retain stale importer patch identities.
|
||
Refresh exact runtime versions and qualification digests together; do not
|
||
download dependencies when a task starts.
|
||
|
||
## Candidate ACP qualification assets
|
||
|
||
Provider branches can build their pinned assets with
|
||
`node packages/paperclip-runner/scripts/build-provider-pack.mjs /absolute/pack --candidate-providers=cursor`
|
||
(or `copilot` or `pi`). The source revision must include the named provider's
|
||
builder. Assets are installed at build time under `provider-assets/<provider>/<platform>-<architecture>`.
|
||
The pack manifest binds each complete asset tree. Runtime admission separately
|
||
checks the provider's source-owned closure pins and copies a verified launch snapshot.
|
||
A pack with candidate assets does not qualify or enable that provider.
|
||
|
||
For an isolated Linux x64 Daytona qualification image, pass
|
||
`--build-arg PAPERCLIP_RUNNER_CANDIDATE_PROVIDERS=cursor` with the normal build arguments.
|
||
Compute its content ID with the same selection:
|
||
`pnpm --silent test:e2e:runner:image-id --candidate-providers=cursor`.
|
||
Candidate assets and the default pack have distinct image identities. Never reuse
|
||
the default image's content ID for a candidate build.
|
||
Use each provider branch's recorded version and explicit model. Keep credentials
|
||
out of images. Paid qualification requires bound provider and Daytona credentials,
|
||
inspectable spend, and the shared $100 ceiling recorded in the capability report.
|
||
|
||
The pack builder tests its copied Node interpreter after relocation. Use a
|
||
standalone Node distribution if the host interpreter depends on a package manager's
|
||
private shared libraries. Pi additionally pins its complete portable interpreter
|
||
and npm dependency graph.
|
||
Refresh exact runtime versions and qualification digests together; never download
|
||
dependencies when a task starts. Grok's additive native ACP profile keeps its
|
||
qualified 1.0.13 executable at the verified sandbox prerequisite path. It does not replace the
|
||
legacy adapter's `grok` command on PATH.
|
||
|
||
Native Grok is an image prerequisite at `/opt/paperclip/providers/grok/1.0.13/grok`.
|
||
Its checksum-verified provisioning is separate from the provider pack, which ships
|
||
only the built-in launcher. Public npm installation never downloads this binary.
|