Commit Graph
75 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
Nicky LeachandClaude Fable 5.2 999709b0a6 fix(docker): stop a stale-lockfile build early and keep beta and stable releases buildable (#21333)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The application image now takes its Runner from the
`runner-artifacts` stages, which install with a frozen lockfile
> - A contributor cannot commit `pnpm-lock.yaml`. After a change to a
manifest of the Runner, the lockfile on `master` is behind until the
refresh merges
> - In that time a plain `docker build` fails late, inside a long step,
with a message that is easy to miss. And a beta or a stable release can
be cut from such a commit, because the release workflows refresh the
lockfile for their own build
> - This pull request adds an early check with a clear instruction to
the image build, and the same check to the beta and the stable release
> - The benefit is a failure in seconds that says what to run, and beta
and stable tags that always build from source

## Linked Issues or Issue Description

This is the third part of a chain. The first part is #21331. The second
part takes the Runner of the application image from the
`runner-artifacts` stage. Refs #21204 and #21306 (the Runner image and
its lockfile rule). No issue exists for this change.

**What happened?**

A tree with a lockfile behind the manifests of the Runner fails `docker
build` after the daemon compile (169 s in the test). The error is
`ERR_PNPM_OUTDATED_LOCKFILE` in the middle of the largest `RUN` of the
`Dockerfile`.

**Expected behavior**

The build fails within seconds of the check running. The message names
the files and the command to run. The application's dependency install
starts from the check, so it cannot start before the check passes.
Stages that do not depend on it (the Runner's toolchain download and
daemon compile, and the tool layer of `production`) can already be
running, and the failed check cancels them. A beta or a stable release
from such a commit does not start.

How often this happens: of 72 nightly tags since 2026-08-24, 9 are on a
commit whose lockfile is behind the manifests of the Runner (2026-09-01,
09-18 through 09-23, 09-28 and 10-26). Of 12 beta tags, none is. A beta
is the promotion of a nightly, so a beta can be cut from such a commit.

**Steps to reproduce**

1. Change a dependency range in
`packages/paperclip-runner/package.json`. Do not refresh
`pnpm-lock.yaml`.
2. Run `docker build .`.

**Paperclip version or commit**

`master` with the previous pull request of this chain.

**Deployment mode**

Not applicable. This is the image build and the release tooling.

## What Changed

- `Dockerfile`: a `runner-lockfile` stage copies only the manifests the
Runner build installs from, the lockfile and the patches. It runs the
frozen, filtered, offline lockfile check, with the arguments of
`runnerLockState`. `deps` starts from it. For a stale lockfile it prints
the files and `pnpm install --resolution-only --ignore-scripts
--no-frozen-lockfile`. For any other failure it prints the output of
pnpm and says so.
- `scripts/check-release-runner-lockfile.mjs`: exports the Runner inputs
of a commit with `git archive` and gives them to `runnerLockState`
(`scripts/runner-image-publish.mjs`). Exit 0 for current, 1 for behind,
2 for an error.
- `scripts/release.sh` and `scripts/release-lib.sh`:
`require_runner_lockfile_current` runs for the `beta` and the `stable`
channel, after `--print-version` exits and before any build. It runs in
a dry run too. It does not run for `canary` or `nightly`.
- Both beta paths end in the `publish_beta` job: the promotion of a
nightly, and a `candidate/beta-*` branch. That job checks out the commit
that `select_beta` chose and runs `release.sh beta` there. So the check
reads the committed files of the commit the beta is built from. A beta
is refused before its soak starts. The stable that promotes it is
checked again.
- Tests: `server/src/__tests__/runner-lockfile-guards.test.ts` (18
tests). Three of them pin the checkout of `publish_beta`,
`preview_stable` and `publish_stable` to the selected commit. Two tests
plus one stub in `scripts/__tests__/release-dry-run-notes.test.mjs`
cover the beta, the candidate beta and the stable path.
- Docs: `doc/DOCKER.md`, `doc/RELEASING.md`,
`doc/runner-image-contract.md`.

## Verification

Builds: `linux/arm64`, local. They ran before the rebase onto `master`,
except the last one. `master` changed neither the `Dockerfile` nor a
manifest in between.

- A tree with `ajv` changed from `^8.20.0` to `^8.19.0` in the manifest
of the Runner: `docker buildx build --target build` fails after 6 s at
`runner-lockfile`, with the message.
- The same tree after `pnpm install --resolution-only --ignore-scripts
--no-frozen-lockfile` on the host, which is what the image workflows
run: `--target build` builds. The check passes, the Runner stages
install, and the vendor step logs `reusing the current Runner build` in
both of its runs.
- After the rebase: `build` and `runner-provider-pack` built from this
branch without a build commit. The check passed.
- The release check with the real pnpm: `behind` for the first tree,
`current` for the second, exit 0 for this checkout.
- `vitest run server/src/__tests__/runner-lockfile-guards.test.ts` with
the other changed server tests, on macOS and in `node:24.22.3-bookworm`
(249 pass in 11 files on Linux). The test file typechecks.
- `node --test scripts/__tests__/release-dry-run-notes.test.mjs`: 14
pass. `scripts/release-lib.test.mjs`: 15 pass on Linux; on macOS 4 of
its tests fail on `master` too. `release-verify-workflow.test.mjs` and
`acpx-patch-packaging.test.mjs`, which read the release scripts: pass.
- `docker buildx build --check`: no warning.
- The Docker builds above ran before the second part of the chain got
its expectation step and before the beta guard. The expectation step was
built on the second part alone. This branch as it is now was not built
again; its tests ran.
- Not run: the release workflow, any `linux/amd64` build, the full test
suite. Only CI and a release dry run can prove the release path.

## Risks

- **A beta can be refused.** When the newest nightly is on a commit
whose lockfile is behind, `release.sh beta` stops, in the dry run and in
the real run. The operator must promote a nightly whose commit includes
the lockfile refresh, or cut a candidate branch that includes it. `node
scripts/check-release-runner-lockfile.mjs --commit <ref>` gives the
answer before the dispatch.
- **A stable can be refused** only when its beta was cut before this
change, or with release tooling that does not have the check. Promotions
run the release tooling of the source commit.
- The check runs in `release.sh`. In the beta workflow it runs in
`publish_beta`, after the approval of the `npm-beta` environment and
after the candidate verification. In the stable workflow it runs after
verification, not in the `preflight_stable` job. The selection jobs have
no pnpm.
- `release.sh stable` now needs `git`, `tar` and `pnpm` on `PATH` before
its first build step. The workflow has them.
- `scripts/__tests__/release-dry-run-notes.test.mjs` is not in a CI
command today. The Vitest file carries the enforced tests.
- The image build has one more small stage. It reads only manifests, so
a source change does not run it again.
- The digest of the Runner image changes again, because its identity
covers the whole `Dockerfile`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:17:15 -08:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:52:02 -08:00
Nicky LeachandClaude Fable 5.2 5d3fa712aa feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Today the application image build compiles the Runner. Other images
that need the Runner build it again, or pull the whole application image
to copy it out
> - Each of those builds can produce a slightly different Runner, and
each one costs time
> - The Runner should be built one time for each change to its inputs,
and every image should copy the same bytes
> - This pull request adds the build target `runner-artifacts`, a
contract for its layout, and a workflow that builds and checks it. It
publishes nothing
> - Nothing consumes the target in this pull request. The application
image build does not change
> - The benefit is one checked Runner build that later changes can
publish and copy by digest

## Linked Issues or Issue Description

Refs #21096 (the merge-queue image build) and #21167 (provider pack
layers). Those are earlier work on the same build. No issue exists for
this change.

**Problem or motivation**

The Runner is compiled inside the application image build. The daemon
comes from the `runner-build` stage. The provider pack is built on top
of the full server and UI build. Downstream sandbox and VM images need
the same Runner. They rebuild it from source, or they pull the
application image and copy paths out of it. There is no contract for
those paths and no single identity for a Runner build.

**Proposed solution**

Build the Runner alone, from its own inputs, as a plain file tree. Give
each build an identity: a digest of its inputs. Document the layout as a
contract, and check every built tree against it. Keep every check that
proves the Runner is good in this one build.

**Alternatives considered**

- Keep the current builds and only document the paths in the application
image. This does not remove the duplicate builds, and consumers still
need the full application image.
- Build the Runner in `docker.yml`. That workflow has no pull-request
trigger, so a change to the Runner build would have no check before it
merges.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication (this pull request)
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

This pull request had one larger form before. Publication, the image
layers and the second variant are now parts 3 and 4.

## What Changed

- `Dockerfile`: eight new stages that end in the target
`runner-artifacts` (`FROM scratch`). They install with a filtered,
frozen lockfile from a context that holds only the Runner's inputs. The
base is a Node image on Debian bookworm, pinned by digest. No existing
stage changed. `cloud` is still the default target.
- The tree holds: the stripped daemon, the Runner package's complete
`dist` with the build stamp that its own build records, the provider
pack with its transport archives, the Grok provisioning inputs, the
transport contract source and a bundled transport module, the lockfile,
and `runner-image.json`.
- `scripts/runner-image-identity.mjs`: computes the image identity. It
hashes every file under the paths that the build copies from the build
context (path, executable bit, content; symbolic links by target), the
whole `Dockerfile`, and the two platforms. The build runs it on the
files that the build context supplies after `.dockerignore`.
- `scripts/runner-image-layout.mjs`: the layout contract. `write`
assembles the tree in the build. `check` verifies a built tree: every
documented path, the build stamp, every pack artifact and tree digest in
`provider-pack.json`, and every transport archive. It also extracts the
transport archives the way a sandbox does and compares the result with
the pack. No checked path may resolve outside the tree.
- Publication form. The build argument
`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it
does the pack manifest carry `payload.runnerInputsDigest` (part 1), and
`runner-image.json` records `publication: true`. Another image build
that runs the stage inline builds the same Runner, but its pack names no
image. `check` requires the field in a publication build and refuses it
in any other. `--expect-inputs-digest` passes only for a publication
build.
- `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd
--build-metadata` on each base image that a consumer uses. Each base is
pinned by digest.
- `.github/workflows/runner-image.yml`: two jobs, on pull requests that
touch an input and on manual dispatch. `identity` resolves the lockfile
and computes the digest in the build context. `build` builds each
architecture on a native runner, checks the layout, and runs the smoke
matrix. The workflow has `contents: read` only. It holds no credential,
reads no registry, and publishes nothing. It builds the publication
form, so the check can bind the tree to the digest that the identity job
resolved. The tree stays on the runner.
- `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`:
the tree digest function moves here from `build-provider-pack.mjs`
without a change in behavior, so the layout check can use it. A test
pins its value.
- `.github/workflows/pr-merge-review.yml`: lists the new workflow, as
`doc/PR-MERGE-REVIEW-WEBHOOK.md` requires.
- `doc/runner-image-contract.md` (127 lines): the contract for
consumers. `doc/DOCKER.md` links to it.

Changes from the earlier form of this pull request:

- The identity hashes the whole `Dockerfile`. The script no longer
parses stages, and it no longer refuses recipes.
`scripts/runner-image-identity.mjs` went from 397 lines to 166. A short
test keeps the three rules that make the hash enough: the Runner stages
start only from pinned images, only `runner-artifacts-context` reads the
build context, and it copies only the paths that the identity hashes.
- The contract document went from 384 lines to 127.
- The workflow tests keep only the security properties: job permissions,
no credential and no registry access, triggers, actions pinned by
commit, and no expression inside a shell script. They no longer pin step
names, order, or wording.

## Verification

Verified on `linux/arm64` only, on a local Docker host:

- `docker buildx build --target runner-artifacts --platform linux/arm64
--build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg
PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .`
builds in 166 seconds without a cache. The image is one layer of
850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files.
- `node scripts/runner-image-layout.mjs check <tree>
--expect-inputs-digest <digest> --expect-source-revision <sha>
--expect-architecture arm64` passes on the exported tree.
- `docker buildx build --file docker/runner-image/Dockerfile.smoke
--build-context runner-artifacts=<tree> docker/runner-image` passes on
all five base images.
- The digest from `--target runner-artifacts-identity` equals the digest
in the built `runner-image.json`:
`d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`.
- The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication
argument builds. Its pack has `runnerSourceRevision` and no
`runnerInputsDigest`. `check` passes on it, and `check
--expect-inputs-digest` fails with "this is not a publication build".
- An edit to `ui/package.json` and to `server/src` leaves the digest
unchanged. A comment added to an application stage of the `Dockerfile`
changes it.
- The two points above ran one commit state earlier. The only difference
to this commit is one type annotation in a comment of
`scripts/runner-image-layout.mjs`.
- A named build context replaces the `runner-artifacts` stage, and no
Runner stage runs. A build with the application build's view of the
context accepts the tree's `dist` and build stamp as current
(`assessRunnerBuild`).
- The daemon has the same bytes in the five cold builds of this stack
that were compared.
- Tests: `cd server && node node_modules/vitest/vitest.mjs run` on
`runner-image-identity.test.ts`, `runner-image-layout.test.ts` and
`runner-image-workflow.test.ts` (132 tests), with
`remote-provider-pack-verification.test.ts` and the existing Dockerfile
tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node
--test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the
image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33
pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and
`docker buildx build --check` report nothing.

Never run:

- The `linux/amd64` build. It is the leg that downloads the Cursor and
Copilot assets for the pack.
- The workflow. No job has run on GitHub.
- The transport archive round trip with GNU tar outside the build. The
build runs it with GNU tar. The local host ran it with bsdtar.
- `actionlint` and `hadolint`. They are not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk to the application image. The change adds stages and files.
No existing target reaches the new stages.
- The first run of the workflow is the first test of the amd64 build.
This pull request triggers it.
- The `Dockerfile` is one file for all images. Any edit to it, also to
an application stage, changes the Runner image identity. That is the
price of not parsing it.
- The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag,
not a digest. A new frontend release under that tag does not change the
identity.
- The image does not run the drift checks of the Runner's `build`
script. They need files outside the Runner's inputs, and repository CI
runs them. A test fails when the `build` script gains a step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 17:30:51 -08:00
DottaandClaude Fable 5.2 a37f3056b6 perf(docker): keep dependencies and provider binaries in layers that survive a commit (#21167)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each merge to master builds the production Docker image for amd64
and arm64 and pushes it to GHCR
> - Each build pushes three layers again, about 3.5 GB compressed of a
4.7 GB image
> - A file-by-file comparison of two consecutive master images shows
that about 95% of those bytes do not change between commits
> - The unchanged bytes are in the same layers as small files that
change on every commit, so the registry sees new layers
> - This pull request moves the unchanged bytes into their own layers
> - The benefit is a smaller push for each master build and a smaller
pull for each deploy

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The production image build. It pushes about 3.5 GB of layers for each
architecture on each master commit. Each deploy pulls the same bytes
again.

**Subsystem affected**

Docker image build (`Dockerfile`, `production` and
`runner-provider-pack` stages).

**Current behavior**

I compared the amd64 images of two consecutive master commits (`2174f2d`
and `dfc510b`, 15 changed server source files). I hashed each file in
the three layers that changed.

| Layer | Compressed | Content that is identical between the two builds
|
| --- | --- | --- |
| `COPY --from=build /app /app` | 1,404 MB | 97.5%. The root
`node_modules` is 2.86 GB of 4.13 GB and has 0 changed files. |
| `COPY /provider-pack` | 1,008 MB | 100% of the 112 files of 1 MiB or
more (2.54 GB). Only 32 small files change (manifest, `.modules.yaml`,
`.bin` shims). |
| `COPY /provider-pack-transport/` | 1,059 MB | The archives differ only
in tar timestamps. This pull request does not change this layer. |

**Proposed behavior**

- Copy the root `node_modules` from the `deps` stage as its own layer.
Copy the remainder of `/app` from the `build` stage with
`--exclude=node_modules`.
- In the `runner-provider-pack` stage, move pack files of 1 MiB or more
to `/provider-pack-large` after the transport archive is made. Set their
mtimes to a fixed value.
- Copy `/provider-pack-large` with `COPY --link`, then copy the small
remainder of the pack to the same directory.

**Reason and benefit**

The dependency layer changes only when the lockfile changes. The large
provider-pack layer changes only when a provider version changes. The
expected push for a commit that changes source only goes from about 3.5
GB to about 1.5 GB for each architecture. The master build runs about
140 times a day on AWS runners, and this traffic is about 570 GB a day
of billed data transfer. Deploy pulls become smaller by the same amount.

**Breaking changes**

None. The file tree of the image is the same. Provider-pack files of 1
MiB or more now have an mtime of 1970-01-01.

## What Changed

- `Dockerfile`: the `production` stage copies `/app/node_modules` from
`deps` first, then `/app` from `build` without the root `node_modules`.
- `Dockerfile`: the `runner-provider-pack` stage splits large pack files
into `/provider-pack-large` with fixed mtimes. The `production` stage
copies that tree with `--link` before the remainder of the pack.
- `server/src/__tests__/docker-image-layer-reuse.test.ts`: a drift guard
for the copy order, the exclude, the `--link`, and the position of the
split after the archive step.

## Verification

- `cd server && npx vitest run
src/__tests__/docker-image-layer-reuse.test.ts
src/__tests__/docker-build-stamp.test.ts`: 9 tests pass. The 2 new tests
fail against the `Dockerfile` on master.
- `node scripts/check-docker-deps-stage.mjs`: PASS.
- `pnpm verify`: exit 2 (the `Dockerfile` is a global CI input).
Required CI supplies the broad coverage.
- A small Dockerfile with the same syntax version
(`docker/dockerfile:1.20`) and the same four `COPY` instructions, built
two times with a changed stamp file:
- `--exclude=node_modules` leaves out only the root `node_modules`. A
nested `node_modules` stays.
  - The dependency layer has the same digest in the two builds.
- The large-file layer has the same digest in the two builds only with
`--link` and fixed mtimes. Without one of them the digest changes.
- Hard links between large files stay hard links. The merged directory
contains each file one time.
- Evidence that the `build` stage writes nothing to the root
`node_modules`: in the published image, all 107,413 files there have an
mtime in the minute of the `deps` install. The app build ran 15 hours
later.

**Not verified:** a full build of the production image with this
`Dockerfile`. The local Docker disk was full, and no pre-merge workflow
builds this image. The first full build is the master build after merge.
The check after merge is to compare the layer digests of two consecutive
master images.

## Risks

- Medium until the first master build. If the image build fails, master
image builds stop until a revert. No broken image can be published,
because the build fails before the push.
- If a future build step writes to the root `node_modules` in the
`build` stage, the image will not contain that change. The comment in
the `Dockerfile` states this rule.
- A file that crosses the 1 MiB boundary moves between the two pack
layers. This only changes which layer is pushed.
- The first build after merge pushes all the new layers one time.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, AWS CLI, Docker, registry API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:38 -06:00
DottaandClaude Opus 5.5 f8e1fb0278 perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds and signs the standard Core image for every
master push. Paperclip Cloud deploys a Core commit only after this image
is published and attested.
> - A master commit takes 29 min (p50) and 56 min (p90) to reach a
published image. Every minute here delays a staging fix.
> - The run logs show where the time goes. The exact-source gate (about
14 min) is the critical path today. The build legs finish about 3 min
before it. After the gate, `merge-and-push` spends about 95 s to pull
the 4.9 GB amd64 image for one probe.
> - This pull request moves that probe into the build legs, which run
before the gate. It also removes a duplicate 187 MB layer from the amd64
image.
> - The benefit is about 95–100 s less from gate to signed image on
every master commit, with the same proof. arm64 is now also probed.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The time from a commit landing on master to the signed standard image
(`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation).

**Subsystem affected**
`.github/workflows/docker.yml`, `Dockerfile`,
`scripts/standard-image-contract.mjs`.

**Current behavior**
Measured on run 43133035209 (sha 42fd790e, total 999 s):

- Push 01:13:47. The slowest exact-source job, `General tests (server
(7/10))`, ends 01:27:08. `Cloud source verified v1` waits 20 s for a
runner and ends 01:27:44. The watcher sees it at its next 30 s poll,
01:27:58.
- `build-and-push (amd64)` ends 01:24:52 and arm64 ends 01:21:59. Both
have slack before the gate. Run 43134250912 shows the same shape: the
gate opened 01:44:37 and amd64 ended 01:41:53.
- `merge-and-push` starts 01:28:03 and pushes the manifest at 01:28:34.
Then "Verify PID 1 reaps orphaned processes" pulls the full amd64 image
(01:28:34 to 01:30:09, 95 s). "Verify the immutable standard image" runs
the same probe again on the same bytes (5 s). The attestation ends
01:30:22.

Last 60 master pushes (2026-11-11 20:17Z to 2026-11-12 03:21Z): 43
cancelled or superseded, 14 succeeded, 2 failed, 1 in progress. Push to
first published image that contains the commit: p50 29.4 min, p90 55.5
min, max 85.6 min. A successful run takes 22.5 min (p50) from push. The
median gap between pushes is 4.5 min.

Where the amd64 "Build and push by digest" step spends its 582 s:

| Time | Phase | What happens |
| --- | --- | --- |
| 0–8 s | Load | Dockerfile frontend, cache manifest import |
| 8–54 s | Pull cached layers (46 s) | `deps`, `rust-chef`,
`runner-deps`, `runner-build` and `base` are **all cache hits**. The
`build` stage starts `FROM runner-build` and copies `deps`, so BuildKit
downloads and extracts them: a 932 MB node_modules layer (13 s download,
24.5 s extract) and about 650 MB of Rust `target/` |
| 54–108 s | Per-commit build (54 s) | `COPY . .` 10 s, mtime
normalization 11 s, UI 5 s, plugin SDK 3 s, server tsc 18.5 s, runtime
bundle 5.6 s |
| 108–206 s | `runner-provider-pack` (98 s) | Rebuilt every commit: it
is `FROM build` and stamps the commit. pnpm install about 25 s, pack 9
s, **Kimi smoke 37 s (amd64 only)**, archive 28 s |
| parallel | `production` | Tool layer is a cache hit but pulls 1.11 GB
(11 s). `COPY --from=build /app` 24 s |
| 206–294 s | Pack copy and chmod | `COPY` pack 7.5 s. `chmod -R a+rX`
plus UID check **71 s** (arm64: 20.5 s) |
| 294–420 s | Image export (126 s) | 115 s compressing new layers:
`/app` 1.40 GB, pack 1.01 GB, pack archives 1.06 GB, chmod copy 0.19 GB
(compressed). Push 10 s |
| 420–582 s | Cache export (172 s) | `mode=max`: 152 s "preparing"
(compressing the per-commit layers of `build` and
`runner-provider-pack`, which no later commit can reuse), 20 s upload |

What rebuilds every commit, and why: only the `build` stage and
everything after it. `PAPERCLIP_BUILD_COMMIT` is already declared after
the dependency layers, `CLI_TOOLS_CACHE_EPOCH` is weekly, and dependency
install already copies manifests first. The commit ARG is used by the UI
build, the server stamp, and the provider pack, so those must rebuild. A
pnpm `--mount=type=cache` gives nothing here: the runners are ephemeral,
and cache mounts are not exported to the registry cache.

Why amd64 is slower than arm64: both legs run on runs-on AWS 32-core
instances (m6a.8xlarge and m6g.8xlarge), not on `ubuntu-latest`. The
difference is content, not CPU. amd64 has a larger provider pack with
the Kimi smoke test (37 s), a 187 MB chmod copy layer (arm64: 0 MB), a
4.9 GB image (arm64: 2.9 GB), and a 172 s cache export (arm64: 100 s).

**Proposed behavior**
Run the PID 1 probe in each build leg, on its own architecture, against
the digest that leg pushed. Before signing, require that every Linux
image in the standard index is one of those probed digests. Set the
provider pack modes in the stage that writes the pack.

**Reason and benefit**
The probe pull leaves the post-gate path. The leg has about 3 min of
slack before the gate. The chmod change removes about 50 s and 187 MB
from the amd64 leg, which offsets most of the probe pull that the leg
now does.

**Breaking changes**
None. Tags, digests, attestation subject, and signer identity are
unchanged.

**Additional context**
The concurrency proposal and the build-in-PR plan are below, in "Risks".
They are not implemented.

## What Changed

- `docker.yml` `build-and-push`: new step "Verify PID 1 reaps orphaned
processes". It runs `scripts/assert-orphan-reaping.sh` in
`ghcr.io/...@<pushed digest>` with `--platform` set to the leg platform.
It runs before "Export digest", so `merge-and-push` only receives probed
digests.
- `docker.yml` `merge-and-push`: removed the two `docker run` probes
(tag and digest). "Resolve standard image digest" now runs
`standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests
/tmp/digests`.
- `standard-image-contract.mjs`: `--verified-digests <dir>` reads each
probed leg digest from the registry, checks that its bytes hash to that
digest, and collects its Linux image digests. It fails unless every
Linux image in the standard index is in that set. Registry content is
addressed by digest, so this proves that the signed index names exactly
the probed bytes, without a second pull.
- `Dockerfile`: `chmod -R a+rX /provider-pack` now runs in the same
`RUN` that writes the pack. `COPY` keeps the modes. The production stage
keeps the readability check as UID 65534.
- Tests: new contract tests (membership, bare manifest legs, digest
mismatch, empty set, lookup by digest). New workflow test: the probe
runs between the build and "Export digest", and the publisher does not
`docker run`. Updated the provider-pack Dockerfile test.
- `doc/standard-image-contract.md`: describes the per-leg probe and the
membership check.

Expected savings, per master commit:

| Change | Saving | Confidence |
| --- | --- | --- |
| Probe moved out of `merge-and-push` | 95–100 s from gate to
attestation (measured: 95 s pull + 5 s second probe) | High |
| Pack chmod in its own stage | About 50 s and 187 MB in the amd64 leg
(71 s vs 20.5 s on arm64, where the chmod layer is empty) | Medium-high
on size, medium on seconds |
| Net amd64 leg | About +10 to +45 s (probe pull estimated at 60–95 s,
minus about 50 s). Still before the gate | Medium |

## Verification

- `node --test scripts/standard-image-contract.test.mjs`: 10 pass (2
new).
- `node --test` on
`.github/scripts/tests/{cloud-readiness,docker-canary-promotion,post-merge-runner-routing,lockfile-refresh-workflows}.test.mjs`,
`scripts/__tests__/release-verify-workflow.test.mjs`,
`scripts/reuse-canary-image.test.mjs`,
`scripts/standard-image-contract.test.mjs`,
`scripts/cloud-source-verification.test.mjs`: 484 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass. Red
check: with the `origin/master` `Dockerfile` and `docker.yml`, the 2 new
or changed assertions fail.
- Real registry, read only: `node scripts/standard-image-contract.mjs
--resolve 42fd790e80 --verified-digests
<dir>` with the two leg digests from run 43133035209 returns the
published index digest. With only the arm64 leg digest, it fails:
"linux/amd64 image sha256:0f0642e6… did not pass the build-leg
orphan-reaping probe".
- `actionlint .github/workflows/docker.yml`: the same 6 shellcheck
findings as `origin/master`, none new.
- Shell syntax check (`sh -n`) of the changed pack `RUN`.

Not exercised: the workflow itself (it runs only on master pushes, and I
did not dispatch it). No local `docker build`: the Docker daemon is not
running on this machine, and a full build does not fit a few minutes.
The leg probe pull time is an estimate from the publisher runner.

## Risks

- The leg now pulls its own image (4.9 GB amd64). AWS legs have about
175 GB free. Hosted tag builds run "Free runner disk" first, and the old
publisher pulled the same image on a hosted runner for tag builds.
- If the registry returns a leg index in an unexpected form, the resolve
step fails closed and no master image is signed. The real-registry check
above used today's exact shapes (OCI index with one platform image and
one attestation manifest per leg).
- Tag builds (`v*`, `nightly/v*`, `beta/v*`) keep the probe, now per
leg. They have no standard-image signing step, and the manifest list is
made only from probed digests.

### Deliberately not changed

- **The exact-source gate.** It is a deliberate "never publish an
unverified image" rule (`watch-release-source.mjs --require-proof` and
`--check-proof`, `doc/cloud-build-readiness.md`). I kept it. The owner
levers are: the slowest `cloud-readiness` shard (`server (7/10)`, 12
min, sets the gate), the 20 s runner wait for `Cloud source verified
v1`, and the watcher's 30 s poll (15 s average delay). I did not shorten
the poll because the org API rate limit was exhausted today.
- **Cache export (172 s amd64, 100 s arm64).** `mode=max` compresses
about 2 GB of per-commit pack layers every build. A fix: a separate
cache-only build of a small aggregate stage (`deps`, `runner-build`, the
production tool layer) with `mode=max`, and the main build with
`cache-from` only. Not done here: the legs are not on the master
critical path today, and I cannot measure it without a master run. It is
worth doing for tag builds or if the gate gets faster.
- **Cached layer pull (46 s).** The `build` stage starts `FROM
runner-build`, so it downloads about 650 MB of Rust `target/` that it
deletes at the end. The server build's `prepare:runner-vendor` uses that
target, so this needs a separate change.
- **Provider pack per commit (98 s amd64).** It rebuilds because it is
`FROM build` and stamps the commit. Its 1.06 GB transport archives also
duplicate the 1.01 GB pack inside the image. Keying the pack on its own
inputs is a product-contract change.

### Concurrency proposal (not implemented)

Today one run holds `docker-${{ github.ref }}` for 17–23 min. Most of
that is waiting for the gate, not building. A newer commit waits for
that run before its build starts (run 43134250912 waited 6.4 min).
Proposal:

1. Remove the workflow-level group. Add job-level groups:
`build-and-push` uses `docker-build-${{ github.ref }}-${{ matrix.arch
}}` with `cancel-in-progress: false`. At most one build per architecture
runs and the newest commit replaces the pending one. So not every commit
is built.
2. `merge-and-push` uses `docker-publish-${{ github.ref }}` with
`cancel-in-progress: false`. It skips publication when a newer master
commit already has a signed image, so the image order stays monotonic.
3. A superseded run must not keep its source watcher alive, or the API
load grows with each push. Fold the early watcher into the gate job, or
end it when its build job is cancelled.

Effect: the newest commit starts building at push, so its image
publishes at about gate time plus 45 s, instead of after the previous
run ends. That removes the 0–17 min queue wait, which is most of the gap
between the p50 of 29 min and the gate time of about 14 min.

### Plan: build in PR CI and promote on merge (not in this PR)

What must change:
1. Turn on the GitHub merge queue for `master`. The queue tests a
`gh-readonly-queue/...` commit and then fast-forwards `master` to that
exact commit, so the SHA stamped in the image (`PAPERCLIP_BUILD_COMMIT`,
`build-info.json`, the provider pack `runnerSourceRevision`, the UI
bundle) is the final master SHA.
2. On `merge_group`: build both architectures with **no secrets and no
registry credentials**, because the code under test is not yet on
master. Write each image as an OCI layout artifact. A second job that
runs no PR code pushes those artifacts by digest to a staging
repository, keyed by the merge-group SHA.
3. Run `cloud-readiness` on `merge_group` for the same SHA, so
exact-source proof exists before merge.
4. On the master push, `docker.yml` looks up the staged digests for
`github.sha`, checks that the tree and SHA match, retags them as
`sha-<full>`, and signs with the same identity
(`docker.yml@refs/heads/master`). If no staged image exists, it falls
back to the current build.

Guarantees to keep: never publish or sign an image without exact-source
proof for that SHA. The signer identity and the attested digest stay the
same. No secrets in any job that runs non-CODEOWNER code. The PID 1
probe runs on both architectures before signing. Superseded queue
entries do not publish.

Expected saving: from master push to signed image, about 1–2 min
(lookup, retag, attest) instead of 29 min p50 and 56 min p90 today. The
work moves before the merge, so the merge itself waits for the queue
(about the current gate time of 14 min). A staging fix then reaches
Cloud about 25 min sooner at the median.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used via Claude Code
with tool use (shell, file edits, read-only GitHub API and registry
reads). Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:29:11 -06:00
Nicky LeachandClaude 6ef37398c8 fix(build): fit typecheck and image build in 4 GiB (#18737)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors, agents, and image builds run `tsc` to check and
compile the repository.
> - The repository uses TypeScript 7, which is a native binary. Node
heap flags do not limit it.
> - The server is one large program, so one `tsc` process needs about 10
GiB.
> - The operating system stops `tsc` under a 4 GiB memory limit.
> - This pull request checks the server in layers, removes the server
from the CLI program, and removes declarations and checks from the image
emit.
> - The benefit is that the full typecheck and the image `tsc` step now
complete in 4 GiB.

## Linked Issues or Issue Description

I found no duplicate pull request or issue. A search for `tsc`, memory,
typecheck, and OOM found only runtime memory work, which is a different
subject.

**What happened?**

`tsc` used too much memory, and the kernel stopped it with exit code 137
in resource-constrained environments. Measured peaks before this change:

| Command | Peak |
| --- | --- |
| Server `tsc --noEmit` | About 10 GiB |
| Server `tsc` emit, as in the Docker build | About 10.7 GiB |
| CLI `tsc --noEmit` | About 7 GiB |
| `pnpm -r typecheck`, whole process tree | About 10.2 GiB |

**Expected behavior**

The typecheck and the image `tsc` step complete under a 4 GiB memory
limit.

**Steps to reproduce**

1. Start a Linux container with `--memory=4g --memory-swap=4g --cpus=4`.
2. Install the dependencies on the base commit.
3. Run `pnpm -r typecheck`. The kernel stops the process with exit code
137.
4. Run `pnpm --filter @paperclipai/server build`. The `tsc` step needs
more than 4 GiB.

**Paperclip version or commit**

Base commit `09162fe893`, with TypeScript 7.0.2.

**Deployment mode**

Not applicable. This issue applies to development, CI, and image builds.
It does not change the server at runtime.

## What Changed

- Add `server/scripts/typecheck-layers.mjs`. It reads the import graph
on each run and runs one `tsc` process per layer. The layers are the
files below the largest import cycle, the cycle, the files above the
cycle, and the tests. Each layer reads the earlier layers as
declarations.
- Add `scripts/run-tsc.mjs`. It adds `--singleThreaded` when
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1` is set. The server layers and the UI
typecheck use it.
- Add `scripts/typecheck-low-memory.mjs`. It checks one package at a
time in this mode. It has no `package.json` entry, as
`.greptile/rules.md` requires.
- Change the server `typecheck` script to build only the Runner
TypeScript output. It does not build the Runner release binary now.
`pnpm build` and `typecheck:build-gaps` still run the full Runner build
and its generated-file checks.
- Change the `tsc` step of the server `build` script to
`scripts/run-tsc.mjs`. The `Dockerfile` sets `PAPERCLIP_TSC_EMIT_ONLY=1`
for that build, so the image emits JavaScript without checks or
declarations. Without this variable, `build` still does a full check for
npm packages and CI.
- Use a non-literal specifier for the `@paperclipai/server` import in
`cli/src/commands/run.ts`. The import result already had no useful type,
and the literal specifier added the full server to the CLI program.
- Add a layered server typecheck step to the `pr-trusted` typecheck job,
because `typecheck:build-gaps` does not run this script.
- Update `AGENTS.md` and `doc/DEVELOPING.md`.

## Verification

Results on Linux use a clean copy of this branch, a new `pnpm install`,
Node 24, Rust 1.97.1, 4 CPUs, and a 4 GiB memory limit with no swap.
Results before the merge of `master`:

| Command | Result | Peak non-reclaimable memory |
| --- | --- | --- |
| `node scripts/typecheck-low-memory.mjs` | Exit 0, 0 OOM kills | 3,492
MiB |
| `PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build` |
Exit 0 | 3,656 MiB, from the Rust release build |
| Server single-program `tsc --noEmit --singleThreaded` | Exit 137, 1
OOM kill | Reached the limit |

Results on macOS on the merged head, with peak memory for the full
process tree:

| Command | Before | After |
| --- | --- | --- |
| `pnpm -r typecheck` | 10.2 GiB, exit 0 | 5.47 GiB, exit 0 |
| `node scripts/typecheck-low-memory.mjs` | Not available | 3.81 GiB,
exit 0 |
| CLI typecheck | 7.06 GiB | 1.68 GiB, before the merge |

Other checks:

- Compiler tests: new tests run the real compiler on a small fixture.
They prove that the layers emit declarations and pass, that a renamed
export fails the layer that imports it, and that a changed type fails
the test layer.
- Declaration imports: a new compiler test proves that types from
`.d.mts` and JSON imports stay correct in later layers. On the real
server, a wrong use of `AgentFileManifest` in the cycle layer failed
with `TS2353`.
- Coverage: a test on the real server source asserts that each source
file is in one layer and that no import goes to a later layer. The test
layer contains the remaining files. `src/__tests__` stays out of the
root list, as in `server/tsconfig.json`.
- Error detection: I added one type error to one file in each of the
four layers. The typecheck reported all four errors and exited with 1.
- Cross-layer errors: I renamed an export in the lower layer. The cycle
and upper layers reported the error and the typecheck exited with 1. I
reverted these temporary edits.
- Emit: full `tsc` and `--noCheck --declaration false --declarationMap
false` gave byte-identical JavaScript and source maps for 2,249 files on
the base commit.
- Runtime: `scripts/server-boot-smoke.sh node
server/dist-runtime/index.mjs` passed with a bundle from the emit-only
build.
- CLI: the esbuild bundle keeps the dynamic import, and `node --check`
passed.
- Tests on the merged head: `typecheck-layers`, `prepare-runner-vendor`,
`cli-invocation-safety`, the CLI `runtime` tests, and all 503
`.github/scripts/tests` tests passed. Before the merge,
`release-verify-workflow` and `runner-release-crate-cache` passed.
- Full `pnpm --filter @paperclipai/server build` passed before the
merge.

Failures during this work:

- The first low-memory run failed in the UI with `TS5023: Unknown
compiler option '-b'`, because the flag came before `-b`. I corrected
the argument order and added a test that runs build mode with the flag.
- The first rename check did not rename the export, because BSD `sed`
does not support `\b`. Its exit 0 was not evidence. I ran the check
again with `perl`, and the result above is from that run.
- The first version of the real-compiler fixture had a `TS2774` error in
its own source. The test found this error, and I corrected the fixture.
- CI failed `server-package-build-script.test.ts` on `1abc5e5a43`,
because that test reads the steps of the server `build` script, and I
had moved them to `build:prepare` and `build:assets`. My first search
for readers of this script did not include `server/src`. I removed those
scripts and `build:image`, restored one `build` script, and kept the
test unchanged.
- `GOMEMLIMIT` did not help enough, and I removed it from the plan. On
these steps, `3GiB` saved at most about 300 MiB. With default checkers,
the cycle layer was stopped with and without it, and the lower layer
went from 7 seconds to 151 seconds.

Not run or not proven:

- I did not run the full `pnpm test:run`, a root `pnpm build`, or a full
`docker build`. I expect CI to run the test and build gates.
- I did not measure the Docker build steps other than the server build.
- I did not run the 4 GiB checks on x86-64.
- I did not measure the merged head in the 4 GiB container. The macOS
result on the merged head is 3.81 GiB, and the merge added one source
file to the lower layer.

## Risks

- The margin is small. In the 4 GiB container, the cycle layer peaks
about 600 MiB below the limit, and the Rust release build in the image
build peaks about 440 MiB below it. Growth of the largest import cycle
uses this margin first.
- `node scripts/typecheck-low-memory.mjs` applies only to `pnpm -r`
packages. These commands still check the server as one program and need
more than 4 GiB: `pnpm build`, `test:e2e:runner:typecheck`,
`test:lifecycle-baseline:typecheck`, and the `tests/speko-native`
project.
- A local flow that used the Runner binary from a previous typecheck
must now run `pnpm --filter @paperclipai/paperclip-runner build`.
- The image does not include server declarations now. I found no
consumer in this repository. A downstream image that reads
`server/dist/*.d.ts` must change.
- An image build does not find type errors now. A typecheck in CI must
find them before the image build.
- The import scan uses a regular expression. A false edge can only make
a layer larger. A missed edge causes a `TS6307` failure, so it cannot
skip a file.
- The new CI step adds about one minute to the typecheck job.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with
reasoning, tool use, and code execution. The runtime does not show the
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-03 08:53:35 -08:00
DottaandPaperclip 54cc522e86 feat(runner): add gated native Antigravity ACP execution (#18188)
## Thinking Path

> - Paperclip manages agents and their work.
> - Native Runner must bind each provider turn to the selected task,
account, model, and instructions.
> - Antigravity needs native ACP admission and physical process
ownership across continuation.
> - A saved conversation must not reuse an old process owner or silently
switch its effective model.
> - This pull request adds the native candidate behind exact operator
admission.
> - The default release gate remains closed while qualification is
pending.

## Linked Issues or Issue Description

Builds on merged #18187. Product account setup and the candidate
producer are in dependent #18189. The repeatable browser journey is in
#18190. Refs: #12503, #14938; those proposals use legacy CLI adapters.

**Agent or provider**

Google Antigravity ACP 1.4.3.

**Why this adapter is useful**

Users need to set up an Antigravity agent and run real work through
their managed Google subscription.

**How the agent is invoked**

Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact
operator admission is required while qualification remains pending.

## What Changed

- Add native Antigravity ACP model and instruction admission,
selected-account homes, and fenced refresh ownership.
- Carry exact candidate authority from the control plane through Runner,
Rust, the sidecar, and each current run. Clamp daemon lifetime to the
absolute permission expiry, including replacement owners.
- Preserve a rotated OAuth refresh token before later identity
verification can fail, including failed or cancelled startup admission.
Make private lease-close retries safe.
- Bound native probe retirement and retry the exact original owner; do
not delete credential homes before physical proof.
- Preserve Gemini API-key recovery when shared Google subscription
capability is added.
- Resume the same tagged native conversation with fresh per-turn process
ownership and physical native retirement.
- Include native manifests in isolated Rust image stages, release cache
keys, and dev rebuild detection. The workflow changes only extend
existing cache input lists.
- Keep current master provider behavior and append Grok 13, Pi 35, and
Copilot 84 input identities. Preserve prior identities, including saved
Copilot 82 and 83 runs.

## Verification

- Runner TypeScript main checks, profile generators, Rust formatting,
and offline workspace checks pass. The native backend integration target
passes 28 tests.
- The complete transport and cleanup group passes 387 tests without
unhandled rejections. These checks use local scripted children and
actual process and port ownership.
- Saved-profile decoding, native-execution parsing, installation, and
selected-credential controls pass 208 focused tests. All 109
pre-existing profile identity files remain byte-for-byte unchanged.
Copilot, Grok, and shared TypeScript/Rust generation checks pass.
- The final recovery fixture passes all seven cases. After the master
rebase, the combined recovery and owned-Node-runtime group passes all 10
cases. Each expiry case sets one 45-second absolute grant before
startup. It includes the deliberate six-second cold delay, four existing
eight-second command budgets, and active-turn headroom. The test proves
an active turn before expiry and then checks the daemon, owned process
group, child processes, and all three native port fences. The 60-second
test deadline does not renew provider authority.
- The consolidated model-discovery and pending-release group passes 22
tests. Google credential compatibility and auth recovery pass 25 tests,
including existing Gemini API-key recovery. The full product settings
test group remains intact in #18189.
- Earlier failures remain in their original CI runs and local records.
They exposed the Copilot 82 decoder gap, an expired five-second fixture
bootstrap ticket, and a fixture grant consumed before activation on
loaded CI. Each repair has a separate result. Production ticket limits
and admission behavior are unchanged by the final test commits.
- An unrelated decision-expiry check failed in [the earlier CI
run](https://github.com/paperclipai/paperclip/actions/runs/40941986698/job/123688668997).
Its owning code is unchanged from the base. The unchanged suite passes
20 tests on isolated PostgreSQL. A controlled database rollback
reproduces the earlier wake: clean startup schedules the 120-second
decision expiry, while rollback reconciliation leaves a 30-second
continuation-coalescing wake. The decision stays open, with no wake
delivery or error. The fresh final-head server shard passes; the failed
result is preserved.
- Final native commit `ec6bc7e4d77f5ae0cc4946dbba6c662e95a87f5b` passed
`ci / verify` and `ci / e2e`. Fresh Greptile and Paperclip reviews both
score 5/5, with no open findings. Antigravity retains the recorded
native `.5` closure and exact low-model profile. This PR does not claim
global or cross-platform qualification.

## Risks

- The new transport carries private managed credentials. Exact
ownership, account generation, model, instructions, and native
retirement are fail-closed.
- Only local macOS ARM64 has a verified complete Antigravity runtime.
Default Antigravity task execution remains disabled without current
trusted operator admission. The Google server sign-in lane and
production candidate producer ship in #18189; this native slice does not
expose the subscription catalog choice.
- Shared launch inputs change provider identities. New identities are
append-only; existing recorded identities remain available.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, and
review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 16:36:12 -05:00
DottaandPaperclip 9ae4c25e7e perf(runner): reuse compatible protected provider packs (#17749)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner starts provider runtimes in remote execution
environments.
> - Sandbox images already contain qualified provider packs.
> - Startup required the image pack to match the entire current app
release manifest.
> - An unrelated app commit could therefore cause pack verification,
transfer, and installation again.
> - This pull request selects protected packs by their provider
interface and platform.
> - The benefit is no extra pack probe, directory scan, version command,
link, or upload on the compatible image path.

## Linked Issues or Issue Description

**What happened?**

Starting a remote task can reject a compatible image pack because its
source revision or unrelated provider differs from the controller. Pack
replacement added about 16 seconds in the investigated startup. Startup
also repeated full content checks.

**Expected behavior**

Use an already qualified, protected pack when its selected provider
interface and target platform are compatible. Keep the selected artifact
identity for process launch and recovery.

**Steps to reproduce**

1. Install a qualified provider pack in a sandbox image.
2. Start a controller built from another app commit.
3. Start a remote OpenCode task. The old complete-manifest comparison
rejects the image pack even when the provider interface still works.

Related repairs: #16493 reduced transfer cost. #17512 retained exact
artifact authority during reattachment. This change preserves that
recovery rule and changes fresh-launch pack selection.

## What Changed

- Add explicit OpenCode and ACPX interface revisions to produced pack
manifests. App Git SHAs and dependency pins remain provenance.
- Read bounded protected-installation metadata in the existing final
runner metadata query. Accept only root-owned paths that the execution
user cannot write, including ancestors and manifest. Reject writable
aliases.
- Select and prepare the pack in server-owned environment setup before
constructing the existing transport. Bind provider launch paths and
digests from the selected installation. Save that identity for exact
reattachment.
- Keep full content verification for uploads, writable installations,
and legacy packs. Reuse successful verification within one startup. Keep
older attached runners compatible with the optional metadata request.
- Seal image packs and document the one-time image refresh that enables
reuse across later compatible releases. Keep selected ACPX agent
qualification checks.

## Verification

- Red/green regression: the old code rejects a protected pack from
another app revision with `manifest mismatch`; the fixed path selects it
without executing verification commands.
- Native startup regression: select the installed pack's own Node digest
with one existing runner metadata request and no pack verification or
version commands.
- Targeted pack, artifact, recovery, execution-driver, and Rust metadata
tests pass. Incompatible interfaces/platforms, malformed identities,
writable roots, path aliases, and exact reattachment are covered.
- Final revision `91dc1676ff67bdbe5866c8586daac10240530cca`: workspace
typecheck and build pass. All 53 current-head checks are successful,
with no pending or failed checks. Greptile is 5/5 with no open findings.
- The broad local `pnpm test:run` attempt reported a workspace-runtime
conflict test failure; its isolated rerun passed. The broad run was
stopped after that failure. Full CI now passes, including the server
test shards. This is not a claim that the full local run passed.
- Linux filesystem evidence: the production metadata functions read a
real root-owned preinstalled pack as a non-root user in 16 microseconds
median and 20 microseconds p95 across 100 warm reads. This measures
metadata admission, not model or hosted response latency.
- All 910 affected server tests and 57 Copilot attestation tests pass
locally on the final implementation. Full CI covers the unchanged Runner
transport. No provider qualification hash was refreshed.
- Built the actual Linux daemon at
`416e7ec7425d0d3565b4a5e1808002d650a60414` and ran its metadata command
as UID 65534. The protected installation returned its real manifest with
`immutable: true`; writable alias and missing-pack cases returned
`immutable: false`. This is metadata/permission evidence, not provider
execution.
- Hosted rollout and a hosted end-to-end latency measurement have not
run.

## Risks

- Existing images need one rebuild to publish the compatibility
declaration. Packs without it keep the legacy content-verification path.
- Fast admission relies on image publication producing a qualified
protected tree. Root execution and task-writable installations retain
full verification.
- Bump an interface revision when required runner/pack behavior changes
incompatibly. Deploy a new image to roll out compatible runtime fixes or
upgrades.
- This removes provider-pack preparation delay. Other runner startup,
context, model, and finalization costs remain separate.

## Model Used

OpenAI Codex, configured model `gpt-6.1-sol`, reasoning effort `xhigh`,
with tool use and code execution. The context-window size is not exposed
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 14:43:12 -05:00
Devin FoleyandPaperclip 6b804bfeba Fix Docker dependency installation for Copilot lifecycle hooks (#16983)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The standard Docker image contains the server and Runner.
> - Its dependency stage installs packages before copying application
source.
> - New Copilot postinstall hooks need files absent from that stage.
> - Both image architectures stop before compilation.
> - This change supplies the lifecycle inputs and tests both
entrypoints.

## Linked Issues or Issue Description

**What happened?**

Master Docker builds fail with a missing
`packages/paperclip-runner/scripts/install-copilot-assets.mjs`. The
server postinstall also needs its bootstrap script and source marker.
See the failed public [Docker
build](https://github.com/paperclipai/paperclip/actions/runs/39450988012).

**Expected behavior**

Dependency installation recognizes the existing source workspace. The
later build compiles and verifies provider assets through the existing
path.

**Steps to reproduce**

1. Build the standard Docker image from master.
2. Reach `pnpm install` in the `deps` stage.
3. Observe the missing lifecycle-script error before compilation.

**Paperclip version or commit**

Reproduced in the Docker build for
`24758b9da129a5e424fb7ba6b423d748ce23f3bc`. This PR starts at
`33cc567f75ae8a3fecf4442d20772e62944f1552`.

**Deployment mode**

Docker, Linux amd64 and arm64. No database is involved.

Searched open PRs and issues for Docker, Copilot, installer and the
exact missing script. No matching fix was found.

## What Changed

- Copy the Runner installer and its two helper modules into the
dependency layer.
- Copy the server installer and both existing source-workspace markers.
- Execute both real postinstall entrypoints in a temporary fixture
populated from the relevant Docker COPY inputs.
- Run the fixture in the existing trusted PR dependency-stage check.

## Verification

- `node --test scripts/docker-deps-postinstall.test.mjs` passed.
- The same test fails with the original Dockerfile because the installer
is missing. The fixed Dockerfile was restored and the test passed again.
- `node scripts/check-docker-deps-stage.mjs` passed.
- The fixture passed through the existing timing wrapper in default and
`PAPERCLIP_CI_PROFILE=true` modes. The latter wrote test timing records.
- `pnpm install --frozen-lockfile` passed.
- All three `node --test
packages/paperclip-runner/scripts/install-copilot-assets.test.mjs` cases
passed. These test fresh Runner/server assets, source bootstrap and
published-package failure.
- Full workspace `pnpm -r typecheck` and `pnpm build` passed. The full
local test command is running. All 55 checks passed on exact head
`8ab57339cb9a0dcefa3ae9c2c0f92dc4f6210a23`, with one neutral security
scan and four conditional skips. This includes full server, browser,
Runner, typecheck, build, package-smoke and dependency-stage checks.
Apex reviewed the same head at 5/5 with its prior timing finding
resolved. No review thread remains unresolved.
- The local Docker daemon does not respond to an info probe. A complete
local image build has not been run. The post-merge canonical image build
must pass before any deployment uses this change.

## Risks

The dependency cache now includes these small lifecycle inputs. Source
edits to the two marker files invalidate that cache. The existing
source-workspace detection and compiled asset verification remain in
place. This does not add an installer bypass, change provider pins or
alter runtime behavior. No schema change is included.

## Model Used

OpenAI GPT-6 through Codex, with source inspection, code execution and
GitHub tools. The exact served model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run focused tests locally and recorded the results above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 09:46:48 -07:00
DottaandPaperclip 98ca6ff7c5 perf: reuse verified provider files during sandbox startup (#16493)
Build provider-pack transport layers once, reuse verified image files in private directories, and upload only missing layers. Preserve archive integrity checks and directory fallback on unsupported hosts.

Verified with 44 focused tests, all 48 main CI jobs, cloud Docker Runner checks, and Greptile 5/5 on the merged head.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-19 08:45:09 -05:00
Devin FoleyandPaperclip e30550ef39 refactor: remove code comments added by recent PRs (#16681)
CONTRIBUTING.md says PRs add no code comments, but #16485, #16586, #16621 and #16672 added some. This removes the 215 comment-only lines git blame on master attributes to those merge commits, across 12 code files, plus one blank line the removal left doubled. Rewritten (not added) comments in scripts/assert-cloud-image-sentry.mjs and comments in .github workflows are left alone. No code changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-17 18:55:39 -07:00
Devin FoleyandPaperclip 27ab902d3d perf(docker): start the server from a precompiled runtime bundle (#16672)
The production image started the server through the tsx loader, which resolved about 3,100 modules and transpiled about 1,200 workspace TypeScript files on every boot (7.6-20 s before listen on a live instance). A build:runtime step now bundles server/dist/index.js with esbuild, compiling in the workspace packages, and the image starts node server/dist-runtime/index.mjs. Server dependencies stay bare so runtime resolution (including the cloud image's optional peers) is unchanged; each module keeps its original import.meta location; dynamic imports stay split; the entry presents the original dist/index.js as argv[1]. The avatar worker loads through tsx's API when the main thread has no loader, tini runs as a subreaper (-s), and PR CI boots the bundle with scripts/server-boot-smoke.sh.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-17 16:22:35 -07:00
DottaandPaperclip 57e977be72 feat: integrate Pi 1.0 into the experimental Runner (#14921)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The experimental Runner owns provider processes and durable
sessions.
> - Pi needs working task execution and human controls.
> - The five-PR stack must preserve changes already on master.
> - Each layer now carries the complete integrated source for a safe
sequential fallback.
> - This PR belongs to native GitHub stack #15602, ending at #14956.

## Linked Issues or Issue Description

Refs #14436, #14631, #14743 and #14956.

Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are
#14921, #14922, #14923, #14924 and #14956. The user authorized the
complete merge after checks pass. Existing `pi_local` execution is
unchanged. Accounting and wider provider/platform qualification remain
deferred.

## What Changed

- Recover missing final replies after workspace finalization changes
owners, using accepted-turn evidence without rerunning work or granting
external-chat publication.
- Preserve the admitted Pi instruction root across warm runs, while
retaining changed-root rejection.
- Give Pi a bounded 15-second default shutdown grace so stop, drain
acknowledgement and durable suspension can complete. Explicit deadlines
and other providers retain their existing behavior.
- Integrate the Pi 1.0 runtime and master contracts.
- Use Pi profile 22. Preserve explicit caller-selected models and exact
native thinking levels. Keep Pi's wrapper, helper, extension and
question/control behavior unchanged from the qualified profile-19
runtime.
- Preserve master's Dot lifecycle and consent fields, configured task
environment, status guards and current Codex/Claude dependency versions.
Cursor stays qualified. Copilot stays pending; profile 17 binds the
changed shared protocol validation sources.
- Exclude general AWS IAM credentials from Pi static/custom provider
bindings and selected task projections; preserve the provider-scoped
Bedrock bearer key. Profile 21 is retained as historical provenance.
Rust and cloud install probes use the current declaration.
- Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12
payload. Pin the patch and complete runtime closures. Include the patch
in normal installed setup tooling. Keep the upstream Pi shrinkwrap as
provenance and permit only this exact security correction.
- Include current attestation files in the Docker build context. Keep
the repository lockfile unchanged from master. CI and private image
builds resolve manifest changes before their frozen installation.

## Verification

- Full local `pnpm -r typecheck` passes, including Runner Rust, server
and UI. Focused integration checks pass: 194 Runner
admission/environment tests, 63 profile/credential tests with one
expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice
tests.
- Full local `pnpm build` passes on the final source.
- Fresh final-source checks pass: all 698 Rust workspace tests (32
binaries), 156 credential/profile/controller tests with one expected
skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests.
- The profile-21 Pi materializer passes on the native host with the
official pinned Node 24.21.0 and its npm. It verifies all 150 locked
packages, the patched dependency and the exact closure. Setup/package
bundle tests and UI token gates pass.
- The old hashes were reproduced for all three supported targets before
calculating the patched graph. New closure hashes are darwin-arm64
`282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`,
darwin-x64
`64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and
linux-x64
`713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`.
Independent native platform execution is separate from graph identity
reproduction.
- Historical cloud qualification remains unchanged: all seven core cases
pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`,
harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image
`sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`,
and accepted Sonnet 4.6/low fixture. All 215 canonical files and all
seven cleanup checks pass independent verification. These are profile-19
results and are not relabeled as fresh profile-22 runs.
- Current Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.
[The readiness
plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md)
preserves campaign and failed-attempt provenance.
- Merge only after every PR's current-head CI and fresh review pass.
Linux CI covers the full suites, build and browser tests. The local
embedded Postgres API-authority suite cannot start on this macOS/Node 26
host, so Linux CI must confirm that suite.

### Fresh profile-22 core qualification — 2026-10-08

All seven accepted core cases pass canonically on Pi profile 22, with
`openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low
thinking. This model is a fixture; production accepts the caller's
explicit Pi provider/model.

Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`.
Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`.
Immutable cloud image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`.
Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.

[Hosted Linux image and clean-install
verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023)
passes, including all 20 source-bound archives, normal CLI/Pi setup,
companion import and the production pack reader. This exact installation
source includes the latest master integration and the corrected Pi warm
instruction-root fence. Full local typecheck/build and current-head
hosted CI verify the final stack. All 13 focused real-root regressions
pass. The full local executor suite passed 662 tests; 15 database tests
could not start the Mac embedded PostgreSQL service. Hosted Linux CI
passes the full required verification and E2E checks. These fresh
results keep their own source identity; profile-19 results remain
historical.

| Core path | Canonical campaign | Retained archive SHA-256 |
| --- | --- | --- |
| File edit, validation, download and Done |
`pi-core22-replyfix-0-1791511228` | 23 files;
`a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` |
| Pending question and controller restart |
`pi-core22-replyfix-1-1791511376` | 33 files;
`6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` |
| Three-turn session/process/workspace continuity |
`pi-core22-replyfix-2-1791511587` | 23 files;
`7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` |
| Four typed questions and browser reconnects |
`pi-core22-replyfix-3-1791511881` | 42 files;
`9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` |
| Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22
files;
`a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` |
| Same-turn steering and permission denial |
`pi-core22-replyfix-5-1791512261` | 39 files;
`c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` |
| Stop during pending permission | `pi-core22-replyfix-6-1791512390` |
33 files;
`7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` |

All 215 canonical files (28714587 bytes) are independently
hash-verified. All seven cleanup grades pass, with no owned runtime
process or temporary root after each case. Automatic retries are zero.
The owned cloud host stopped normally after retention. The prior
profile-22 warm attempt remains failed and separately retained: archive
SHA-256
`1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its
original canonical classification is preserved. Diagnosis reproduced a
product bug comparing an agent-files root against an unset
checkpoint-only field. The fix stores the admitted physical root
separately from the adopted per-run collection capability. The real-root
regression fails before the fix and passes afterward, including
rejection of a changed physical root. Fixture, grader, model and all
seven accepted case IDs are unchanged; this fresh campaign tests
final-reply publication after file registration first. The intermediate
restart attempt also remains failed and retained: archive SHA-256
`5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its
original canonical classification is preserved. Pi resumed, wrote the
verified answer and completed its task; exact runner suspension was
proven, but idle stop consumed about 5.2s and left under 3s for the
drain acknowledgement. The Pi-only default shutdown grace is now 15s,
preserving a full 5s drain round trip and a finite suspension reserve.
Explicit caller deadlines, other provider defaults, literal drain
receipts and exact suspension identity checks remain unchanged. The
timing regression fails before this correction and passes afterward; all
18 focused settlement tests and Runner typecheck pass. The final-source
file attempt is also preserved as failed (`candidate_failure`), archive
SHA-256
`db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`.
Native edit, validation, exact downloadable artifact and Done/succeeded
all passed, and the exact final reply was durably recorded. A workspace
recovery owner completed before the live heartbeat reached presentation,
leaving that reply absent from task chat. Recovery now materializes only
a completed final reply from the accepted turn of an ordinary internal
Done task, preserving issue/run/contract binding, suppression,
external-chat authorization and same-run deduplication. The database
regression covers the generated file-preparation receipt, suppression,
unapproved external continuation and replay. Server typecheck and all 49
response-selection tests pass; hosted Linux verifies the database
regression because embedded PostgreSQL cannot start on this Mac.

The delayed-final-answer database regression passes on [the final
root-source Linux server
shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152),
alongside 1,108 passing tests. The first root Runner shard had one
unchanged durable-resume test exceed its 5-second timeout; the identical
top-source shard and the isolated exact test passed. One rerun of that
failed job and its required aggregate passed without source or test
changes. The original failed job log and the single-rerun receipt remain
retained.

### October 9 merge verification

Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All
current-head checks pass, including `ci / verify` and `ci / e2e`;
exact-head Greptile review is 5/5 with no unresolved threads. Current
master conflicts are resolved. The user authorized the maintainer
override of the code-owner review gate after these checks. The seven
retained live core cases remain bound to source
`3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image.

## Risks

- The security correction changes the dependency closure and profile
identity. Old sessions must reopen on the new profile. Exact identities
and credential bindings fail closed.
- The runner remains experimental and requires explicit selection.
Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E
model is a fixture.
- Accounting and the broad platform/provider matrix remain deferred.
This merge does not publish a release or deploy a service.

## Model Used

OpenAI GPT-6 through Codex assisted with reasoning, repository
inspection, editing and tool use. The exact serving ID and context
window are not exposed in this session. Final live qualification uses Pi
1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed
low thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 08:56:41 -05:00
DottaandPaperclip 6395cae072 fix(runner): ship provider pack in the standard Docker image (#14854)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote OpenCode and ACPX runs need a provider pack from the
application build.
> - Cloud now builds its application image from the standard production
image.
> - The provider pack was added only to the legacy cloud image target.
> - The standard image therefore cannot supply the pack to downstream
Cloud images.
> - This pull request adds the pack to the production image and lets the
cloud target inherit it.
> - Remote runs can then use the pack that matches the application
source commit.

## Linked Issues or Issue Description

Refs #13827. Refs #14024.

The standard production image does not include the remote provider pack.
Downstream Cloud images inherit that omission. Remote OpenCode and ACPX
runs fail with `runner_remote_provider_artifact_incompatible` and ask
for `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH`.

## What Changed

- Build and copy the provider pack into the standard production image.
- Set the pack path and check that an unprivileged user can read its
artifacts and execute Node.
- Let the legacy cloud target inherit the pack from production.
- Add regression checks for production packaging and cloud inheritance.
- Document stamped image behavior and the default pack path.

## Verification

- The 12 focused Docker stamp and provider-pack reuse tests pass on
commit `4a11f8d52aead55f85527c8e82c6d7f2644ce0da`.
- The new packaging regression failed against the old Dockerfile and
passed with the fix.
- On the current commit, `pnpm build` and `pnpm -r typecheck` pass. All
seven standard-image contract tests also pass.
- The current-head CI build, typecheck, test, browser, and native Runner
checks passed. The local full suite hit one chat-channel assertion
failure; that exact test passed in isolation. The remaining local run
was stopped after CI completed to avoid duplicating its full suite. An
earlier run on the pre-rebase base had a heartbeat comment batching
timeout; the external chat wait integration suite passed all 142 tests
in isolation.
- [The stamped preview image build
passed](https://github.com/paperclipai/paperclip/actions/runs/36885002850/job/110446106393),
including the production-stage provider pack build, copy, and
unprivileged artifact readability/executable check. Publication,
compatibility validation, and deployment of this exact commit to a
staging QA instance passed.
- Reproduced the exact missing-pack error on an existing staging image
with Paperclip Runner, ACPX, and Claude in a remote Daytona computer.
The legacy Claude adapter succeeds with the same account and computer.
After deploying this commit, the same native task succeeded: it computed
`5050` with a real remote shell command, wrote a proof file, read it
back in a separate call, uploaded the file as a deliverable, and
completed the task. The uploaded file contents and Done state persisted
after a page reload. The run trace confirms Paperclip Runner, ACPX, and
Claude. The first run took 2m 59s, including approximately 97s of remote
artifact preparation. A second native run read the unchanged file from
the prior run and completed successfully. Its startup took about 120s;
this verifies repeated execution and file persistence, not fast
provider-pack reuse.

## Risks

- Stamped standard images now include the provider pack and its build
cost. A pack build failure now fails the production image build.
- Unstamped local builds still skip pack generation. Setting the path
alone does not create a pack.
- No database, provider authentication, or runner verification rules
change.

## Model Used

OpenAI Codex, GPT-6. The exact serving model identifier and context
window are not exposed in this session. Capabilities used: repository
inspection, code editing, shell verification, and browser testing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-01 11:07:50 -05:00
DottaandPaperclip 18e8c121d9 fix(runner): include Grok support in public installs with sandbox prerequisites (#14024)
## Thinking Path

> - Paperclip manages agents through a shared native runner.
> - Built-in harness support should ship with Paperclip's public
distribution.
> - Grok already speaks ACP; it does not require a new public bridge
package.
> - Sandbox provisioning owns the native executable and its pinned
version.
> - The runner must verify that prerequisite without downloading it
during npm installation.
> - This change separates built-in launcher identity from external
runtime identity.
> - Clean npm installation and live staging checks verify the
distribution boundary.

## Linked Issues or Issue Description

Refs #13882, #13973, #13977, #13979.

This follow-up now targets master after #13882 was squash-merged. It
replaces the private `@paperclipai/grok-acp` workspace package with
runner-owned assets. Current master is included so the branch also
contains the merged scheduler, complete-event capture, and durable
cleanup fixes.

## What Changed

- Ship Grok launcher and qualification metadata inside the runner's
compiled output and the public server's vendored runner tree.
- Remove the separate Grok npm package and all package-manager install
hooks for this runtime.
- Require the checksum-verified Grok Build 1.0.13 binary at
`/opt/paperclip/providers/grok/1.0.13/grok` in the selected execution
environment. Provision it explicitly in the Daytona image and CI setup.
- Keep native binaries outside the provider pack. Bind the built-in
launcher into the pack manifest.
- Preserve executable leases, descriptor-backed startup, credential
fences, permissions, and exact ACP model admission.
- Use `builtin:grok-acp` and `native:grok` as profile identities.
Historical package-profile sessions fail closed on resume rather than
being silently reinterpreted.
- Resolve built-in assets from the authenticated sidecar location,
including public server npm layouts. Keep the controller path out of
provider environments.
- Add clean npm tarball installation verification to the existing
trusted canary CI job and the admitted manual EC2 verification path. It
stages a unified release version and runs npm lifecycle scripts, then
verifies missing-prerequisite rejection and admission after separate
provisioning without credentials or inference.
- Include the controller-owned provider pack in stamped Cloud images.
Unstamped local images omit the pack and remain usable; remote ACPX
requires full source provenance.
- Correct CLI approval-page metadata for an already authenticated Cloud
board user; approval authorization remains unchanged.
- Honor explicit native-runner enablement in the Cloud agent picker and
direct setup page, keeping the flag disabled by default.
- Allow selecting the execution environment before connecting
credentials. Include Grok in the existing authenticated hello-probe
flow, targeting its pinned native prerequisite for runner setup.
- Recover an existing subscription sign-in conflict through an explicit
cancel-and-retry action, serialized after cancellation succeeds.
- Preserve the selected ACPX harness before normalizing config fields,
so new Grok agents use the Grok default model.
- Keep the credential-free Cloud provider pack root-owned and readable
after runtime UID remapping; verify manifest and referenced asset access
under an unrelated unprivileged UID during image builds.
- Archive prior failover backups alongside explicitly replaced harness
state, preserving evidence while preventing stale backups from blocking
a fresh replacement.
- Update Daytona image content inputs and contract tests for the
built-in assets and explicit provisioner.
- Document and regression-test the shared `approve-all` default for Grok
setup, saved configuration, and native execution. Explicitly saved
restrictions remain unchanged.

## Verification

Current merge-repair head `df09eb3e1a619430ad8419a0ee9aedd486689b05`
incorporates master `f1a394bd30cb56fb9e479f98b9f50176fe921858` after the
base PR was squash-merged. All 12 conflicts came from incoming files
identical to the tested pre-squash base. The final tree exactly matches
a three-way merge using that original base, preserving built-in Grok
distribution and removal of the obsolete private package. All 252
focused runner/UI tests, six npm-isolation tests, and token gates pass.
Fresh exact-head Greptile review is 5/5 with no outstanding findings;
security scans and EC2 native compilation pass. All current-head CI is
green: 56 successful checks/statuses and four intentional skips ([run
36468768035](https://github.com/paperclipai/paperclip/actions/runs/36468768035)).
The repository owner explicitly authorized bypassing code-owner approval
after all checks passed; no CI checks or repository protection settings
are bypassed or changed. The only remaining PR was removed from the
completed stack metadata to permit native auto-merge.

Earlier integration head `78cb306ecc41b5c96577c26c1d89153b0ef865a1`
includes master `3447609d2247e75e55d91493dda91a608364f672` (2026-09-28).
Two master advances during verification overlapped the eval catalog; the
final merge preserves Grok qualification, completion updates, and
bounded API-response reading in all 348 cells. All 77 focused
catalog/eval/workflow tests pass. Both native stack layers (#14397) are
mergeable, and both exact-head Greptile reviews are 5/5 with successful
security scans and no unresolved review threads. All current-head CI is
green: 56 successful checks/statuses and four intentional skips ([CI
attempts](https://github.com/paperclipai/paperclip/actions/runs/36447124691)).
The initial attempt lost two EC2 runners to shutdown signals and stalled
a third shard during dependency preparation; all three passed the
same-commit failed-job-only retry. Trunk code-owner requirements remain
enforced. The review summary’s non-blocking saved-asset offset
classification note concerns code already merged in #14301; those
runtime files are identical to master and outside this stack’s diff.
Historical live evidence below retains its original source revisions.
[Final public npm
verification](https://github.com/paperclipai/paperclip/actions/runs/36445542764)
passed on `76ea70cd4d13786a042af9df82f0fd7a8c85ae30`: 17 public
packages, an executed offline lifecycle sentinel, unchanged consumer
lock, built-in launcher, missing-prerequisite rejection, and verified
separately provisioned binary/command lease. Provisioning and cleanup
require no host privilege elevation; only the positive probe mounts the
temporary native binary read-only. The verifier is unchanged by the
final master merge. All six isolation tests and an offline npm smoke
test pass. The prior head had 56 green CI checks and a 5/5 review after
two unchanged tests timed out and passed a failed-job-only retry ([CI
attempts](https://github.com/paperclipai/paperclip/actions/runs/36444597313)).
All 56 recovery-display/lineage tests pass; re-review cleared the
already-covered missed-retry concern. Earlier EC2 failures remain
retained: [npm lockfile
rejection](https://github.com/paperclipai/paperclip/actions/runs/36436311203),
[missing compiler in the slim
image](https://github.com/paperclipai/paperclip/actions/runs/36440210984),
and the aggregate 15-minute test timeouts in those broad runs. Both
broad attempts passed typecheck, token gates, Product E2E type/unit
checks and build. The focused EC2 lane preserves the existing
trusted-actor and immutable-source gates.


Earlier documentation/test checkpoint
`ff244c4fd78a7ede5a3e00efe09f475f133ef33e` leaves runtime behavior
unchanged. 154 focused tests pass across configuration building, native
provider resolution, permission policy, credentials, UI configuration,
and new-agent setup (including both Grok auth modes); token gates pass.
All fresh CI is green for this head: 56 successful checks/statuses and
two intentional skips ([run
36367065119](https://github.com/paperclipai/paperclip/actions/runs/36367065119)).
Greptile is 5/5 with no new findings. Grok already inherits the shared
`approve-all` default, so unattended setup requires no manual permission
change.

Runtime head `bb5a9307991f1ac567b781970ef11b39d518e19b` fixes a final
staging continuation failure before provider startup: explicit
replacement archived the old harness but left its failover backups
active, which caused `runner_harness_state_mismatch`. The regression
fails before the fix and passes after it; all eight adjacent
recovery-safety cases also pass. Old backups remain inspectable inside
the continuity archive. All fresh CI is green at this head ([run
36360839248](https://github.com/paperclipai/paperclip/actions/runs/36360839248)),
with a 5/5 review. One unrelated Cursor test timed out in the initial
server shard; the same-commit failed-job rerun passed, and both attempts
are retained. Staging deployment is confirmed healthy on this revision.
The controller image is
`ghcr.io/paperclipai/paperclip@sha256:6ad91c487910ccd2596ff7aed0a3a3ea5233d12b51b83cd6e1402237749b9673`.
The final browser-created staging task passed on this exact revision
with API authentication: context read → structured human question →
controller restart → answer submission → same native provider session
resumed → document saved → task Done. The two turns took approximately
119s and 77s. The actual write receipt was applied, and the saved
document has exactly one revision containing the selected answer and
requested marker. Usage and cost were not reported. [Controller image
build](https://github.com/paperclipai/paperclip/actions/runs/36360889243).

- Previous integration head `a44f7dbb6b6f77cd9ed893756ca453307f281e5f`:
all CI green (53 successful checks/statuses, two intentional skips),
including repository typecheck/build/tests, native Runner tests, browser
shards, and canary installation checks. [CI run
36358672529](https://github.com/paperclipai/paperclip/actions/runs/36358672529).
Greptile is 5/5 with no unresolved findings.
- Focused checks cover Grok credentials, executable admission, launcher
assets, provider-pack paths/permissions, workflow contracts, setup
defaults, CLI authorization, and subscription conflict recovery. All 39
protocol definitions validate. Final integration checks pass 124
catalog/evidence/cache tests and nine project-form tests; token gates
pass. Some local dependency checks could not load the stale installed
dependency tree; the corresponding fresh EC2 checks pass.
- Clean public npm installation passed on EC2 at
`8b172ebcf8e02e30662d830c00f3961e3bd459ec` ([run
36164964900](https://github.com/paperclipai/paperclip/actions/runs/36164964900)):
17 unified-version packages, lifecycle scripts enabled, built-in
launcher present, no separate Grok package or npm-downloaded binary,
missing prerequisite rejected, separately provisioned native executable
and command lease verified. No credentials or inference were used.
Subsequent changes preserve this npm asset layout.
- The immutable Daytona prerequisite image is
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:98957d5be0ac774d086b6402b5849e8e6356fec70fb8c09fca6eb4ed6de918e0`,
built from `5a2db471f3ddabe77f9f80e76ed27f996cb97fba`. The previous
Cloud controller image was
`ghcr.io/paperclipai/paperclip@sha256:fd914e1ab1e45f741e8e078ff452d16f082d7ac05f9b4b3506d3a3c64150d204`,
built from `a44f7dbb6b6f77cd9ed893756ca453307f281e5f`; it is superseded
by the latest image above. Its EC2 build verified provider-pack access
under an unrelated unprivileged UID.
- Browser staging at `40f898bc4cba73c1dff4e6344a3983ba0fb247ef` passed
full Grok onboarding with the correct `grok-4.7` model, saved credential
delivery, and pinned Daytona execution. A browser-created task read
context and asked the structured human question. After a controller
restart, answering the persisted question resumed the same native
provider session, saved the requested document, and completed the task.
Actual tool outcomes and durable state agree: one question and one
document revision. The two successful turns took 42.7s and 63.1s; usage
and cost were not reported.
- Restricted policy returned the expected `approval_required` outcome.
Functional staging tests explicitly selected `approve-all`; controller
authorization and governed approvals remain enforced. Temporary board
CLI access was revoked and verified rejected (HTTP 401), and the
disposable onboarding agent was paused.

Failures remain retained: the pre-fix continuation failure (its task
remains blocked; the passing final task is fresh), the original Cloud
provider-pack permission failure, the expected restricted-policy denial,
the superseded npm staging failure, and an earlier monolithic CI
infrastructure timeout. Browser CI exposed a project alias/form race;
the final stack uses master's stronger draft-preservation fix and all
browser shards pass. Historical full subscription/API protocol and
Product rosters retain their original source revisions and do not
qualify this packaging revision. No local Docker or Rust build was used.

## Risks

The branch includes master’s draft-preservation fix for project URL
aliases. It keeps the same project’s edit form mounted and clears prior
data when the project or company changes.

Custom sandboxes and local execution hosts must provision the pinned
binary before Grok starts. Missing, changed, unsupported-platform, and
symlinked executables fail admission. The new builtin profile cannot
resume sessions created with the former private-package profile.
Existing Claude/Codex npm bridge profiles retain their package pins.
Grok restricted modes preserve the selected policy but cannot
automatically admit Paperclip calls: ACP permission metadata does not
independently bind tool authority, so those calls stop with
`approval_required`. New Grok configurations default to `approve-all`,
including API configurations that omit the mode. Existing explicitly
restricted configurations remain restricted; controller authorization
and governed approvals remain enforced.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact
serving model identifier and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 14:54:44 -05:00
DottaandPaperclip f1a394bd30 feat(runner): add Grok Build through native ACP (#13882)
## Thinking Path

> - Paperclip manages AI agents and governs their work.
> - Its native runner uses structured provider protocols for sessions
and tools.
> - Grok Build supports ACP over stdio, but the runner did not expose
it.
> - Native execution requires company-scoped credentials, verified
identities, and permission gates.
> - This change adds Grok through ACPX for local and Daytona execution.
> - Subscription login and explicit API-key execution have separate
credential paths.
> - Qualification grades real tool outcomes, durable state, and browser
workflows.

## Linked Issues or Issue Description

Refs #13845, #13847, #13850, #13878, #13901, #13973, #13977, #13979.

Add **Grok Build** to `paperclip_runner` with `provider: "acpx"`,
`acpxAgent: "grok"`, and model `grok-4.7`. Existing legacy Grok agents
keep their adapter. Merge the three companion fixes (#13973, #13977,
#13979) before treating the integrated Product qualification as deployed
behavior.

## What Changed

- Synchronize shared, TypeScript, Rust, server, validation, and UI
provider contracts.
- Run Grok native ACP stdio through ACPX and the authenticated Paperclip
MCP bridge. Verify the pinned executable and exact ACP model identity.
- Prefer company subscription login. Support an explicit company-secret
API key without automatic paid fallback. Fence refresh and copyback to
the same account and remove private runtime credentials after
containment.
- Preserve selected permissions, cancellation, durable session identity,
resume, and restart recovery. Keep unsupported steering and goals
unavailable. Preserve missing usage and cost as unknown.
- Package checksum-verified Grok Build 1.0.13 for Daytona with an
immutable, signed image built on EC2.
- Add deterministic admission, protocol, permissions, identity,
credential, failure, and cleanup checks. Add the maintained 39-case
protocol roster and separate subscription/API Product profiles.
- Fix live-test findings in reasoning events, reloads, idle-owner
retirement, credential-home cleanup, expired-login model discovery,
launcher pinning, and rerun evidence selection.
- Align control-plane state readers with the transport's 64 MiB bound
while retaining identity, ownership, lifecycle, and size rejection
checks.
- Stabilize two asynchronous CI assertions while retaining actual
outcome and filesystem-evidence checks.

## Verification

Current integration head `f114948376056fe0b6b34c1496ae8667b59daa63`
includes master `3447609d2247e75e55d91493dda91a608364f672` (2026-09-28).
Two master advances during verification overlapped the eval catalog; the
final merge preserves Grok qualification, completion updates, and
bounded API-response reading in all 348 cells. All 77 focused
catalog/eval/workflow tests pass. Both native stack layers (#14397) are
mergeable, and both exact-head Greptile reviews are 5/5 with successful
security scans and no unresolved review threads. All current-head CI is
green: 56 successful checks/statuses and four intentional skips ([CI
run](https://github.com/paperclipai/paperclip/actions/runs/36447097232)).
Trunk code-owner requirements remain enforced. The review summary’s
non-blocking saved-asset offset classification note concerns code
already merged in #14301; those runtime files are identical to master
and outside this stack’s diff. Historical live evidence below retains
its original source revisions.


Earlier integration checkpoint:
`24fc9b94ca0afb21ccdc8d26dbb2e4b258ad72cb`. Refreshed against master
`0f14d2612`, preserving Grok qualification alongside the new accounting
and lifecycle suites. All 124 focused catalog, evidence, and
service-worker checks pass. The current base workflow includes the
explicitly selected public-install verification lane; follow-up #14024
supplies its verifier script. CI at that earlier checkpoint was green
(56 successful checks/statuses, four intentional skips), and the review
is 5/5 with no unresolved findings. Prior feature CI at
`fd73f0a9b1ecdf4094685054028df71739ddc3e1` passed ([run
36148259902](https://github.com/paperclipai/paperclip/actions/runs/36148259902));
that is historical evidence, not a current-head result.

Paid Product measurements use frozen integrated source
`2d939a92b21dcaf5c77c88b54d96784d2ddd0699`, which combines the feature
with #13973, #13977, and #13979. That source passed all 52 CI checks and
clean 5/5 review. Later master syncs incorporate upstream changes. Their
checks remain separate from these pinned live measurements.

| Check | Result and source-pinned report |
| --- | --- |
| Subscription protocol roster | [39/39 first attempts; 206
assertions](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-36046839612-1/index.html),
runtime `bc6833f7`, evals `92bb4b8c` |
| API protocol roster | [39/39 first attempts; 206
assertions](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-35926577007-1/index.html),
runtime `4a1061c8`, evals `3213dbec` |
| Subscription full Product matrix | [16/16 first attempts; 144
assertions; cleanup
passed](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36096908572-1/index.html),
source `2d939a92` |
| Subscription core repetitions | 18/18: tool use, planning approval,
and Stop/resume each passed three times in local and Daytona profiles.
The full matrix contains repetition one; [repeat
two](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36104551060-1/index.html)
and [repeat
three](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36108100404-1/index.html)
each passed 6/6. Total: 28 unique subscription attempts at `2d939a92`. |
| API smoke and question continuation | [4/4 first attempts; cleanup
passed](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36147315401-1/index.html),
both environments at `2d939a92` |
| Historical API Product coverage | [16/16 full
matrix](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35875144860-1/index.html)
and 18/18 core repetitions at `4a1061c8`; retained as measurements of
that revision |
| Native Daytona proof | Three subscription and three API
MCP/permissions/resume runs passed at `bc6833f7`. Three expired-login
admission and fenced refresh checks passed without inference. All test
sandboxes were removed. |
| Inspectable artifacts and UI | Current-source screenshots verify
planning approval, direct Ask completion, question continuation after
controller restart, and two downloadable project revisions. The project
downloads pass 12 and 18 tests; all 40 independent artifact oracle
checks pass. |
| Provider-free checks | 116 eval-validator tests, 39 Grok definitions,
and 359 enabled/external campaign cells pass. Continuation regressions
above 2 MiB and 16 MiB failed before their fixes; 32 focused
recovery/ownership/size checks pass. |

The 32 unique current-source Product attempts have no failures, retries,
or skipped cells, and all cleanup checks pass. Whole-workflow timing,
model identity, image and provider-pack provenance, attempts, and
accounting coverage are retained in the canonical reports. The report
publisher's conservative `complete=false` flag is preserved; independent
audits verify the exact selected source catalog and immutable result
rows.

Pins: Grok Build `1.0.13 (5e9a58528b76)`, ACPX `0.13.1`, ACP model
`grok-4.7`. Linux binary SHA-256:
`edf79521581bb5e6b95abef848491a6a742e860da3e237ebe86a280d30dce4c1`.
Launcher SHA-256:
`f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55`.
Image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:76b24edfd850219e949418b19e4ceba690e84d51d199ade426e484953329b5e9`.
Image build source is `4196a4cd`, recorded separately from application
source `2d939a92`; each campaign verifies the image signature and
provider pack.

Original failed campaigns remain available: [continuation
bound](https://github.com/paperclipai/paperclip/actions/runs/36057718059),
[scheduler/event
capture](https://github.com/paperclipai/paperclip/actions/runs/36071063537),
and [startup cleanup plus EC2
interruption](https://github.com/paperclipai/paperclip/actions/runs/36080870743).
They retain their original grades. No Docker or Rust builds ran on the
developer laptop for these follow-ups.

## Risks

Merge packaging follow-up #14024 with this base before public release.
The follow-up replaces the private Grok bridge package with a built-in
launcher and makes the native binary an explicit sandbox prerequisite.

Three separate, reviewed fixes are part of the tested integrated
behavior: #13973 serializes task-run admission; #13977 captures complete
event evidence; #13979 durably reconciles failed Daytona creation. Each
has green CI and clean 5/5 review. Failed-create recovery has 277 plugin
tests, 92 SDK tests, host-runtime recovery tests, and a real Daytona
lost-deletion-receipt proof. The live proof uses a private file for
journal persistence; database durability is covered by host tests.
Worker death before delivery of a failure envelope remains outside that
recovery mechanism.

Subscription fixtures stage an authorized company login; interactive
browser sign-in is not qualified. Local Product profiles ran on EC2
Linux. The temporary subscription credential was removed from the
protected GitHub environment after all subscription audits, with absence
verified. Runtime homes and refresh copyback remain ownership-fenced.

Protocol results remain pinned to their original revisions; they are not
relabeled as tests of the latest feature commit. New binary/model
versions require qualification. Missing token usage and model cost
remain unknown; runtime estimates do not establish a full bill.
Automatic paid Grok scheduling remains disabled pending separate
reviewed enablement. The 64 MiB bound can increase memory use for
verbose sessions, and larger files still fail closed. No automatic
legacy-agent migration occurs.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact
serving model identifier and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 13:54:36 -05:00
Devin FoleyandPaperclip 600e552d7b fix: attribute Sentry errors to the loaded source release (#13719)
Attribute optional server and browser Sentry events to their source build.
Use validated build commits for Docker and source/npm artifacts, preserve
explicit server release overrides, and keep cached browser bundles tied
to the commit they loaded.

Verify 127 focused tests, server/UI typechecks, Docker and source build
stamps, all 53 CI checks, and Greptile 5/5 with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-20 08:08:33 -07:00
Devin FoleyandPaperclip 7435b2ee9c ci: cache compiled Docker Rust dependencies separately from source (#13329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys images that contain the native Rust Runner.
> - The image already builds that Runner before copying ordinary app
source.
> - A Rust source change still invalidates its entire compiled
dependency layer.
> - Compiled dependencies can survive source changes when their recipe
is unchanged.
> - This PR adds a separate locked dependency build before compiling the
real workspace.

## Linked Issues or Issue Description

Refs #13195. A search of related Docker and Cargo cache PRs found no
duplicate dependency-recipe change.

**What existing behavior does this improve?**

Docker image build time after Rust source or embedded protocol changes.

**Current behavior**

The `runner-build` stage compiles dependencies and workspace code in one
layer. In Cloud readiness run 34698143548, that stage took about 3m48s
when its cache was unavailable.

**Proposed behavior**

Generate a recipe with pinned cargo-chef 0.1.73. Build locked release
dependencies in `runner-deps`, then copy and compile real Rust source
and embedded protocol inputs in `runner-build`. Source edits can reuse
the dependency layer from the existing registry cache.

**Reason and benefit**

Reduce dependency recompilation during source changes and merge bursts.
Expected savings are roughly 2–4 minutes when the old native layer would
miss but dependency layers are available. Full cold builds also pay for
the recipe tool installation. Ordinary app-only cache hits gain little
from this change.

**Breaking changes**

None to the shipped application or image tags. The recipe tool and
compiled dependencies remain in build stages.

## What Changed

- Install a pinned recipe generator with its locked dependencies and the
existing package-owned compiler.
- Add recipe planning and compiled dependency stages. Use the same
release profile, package, binary, and lockfile enforcement as the real
native build.
- Remove generated source stubs before copying actual source. Preserve
protocol inputs, timestamp normalization, binary staging, and
application checks.
- Add Docker cache wiring regressions and update the Docker cache
documentation.
- Run a two-build probe in Docker Runner check. It requires dependency
reuse, changed real binary metadata after a source edit, and a changed
recipe after a dependency declaration edit. It uses a disposable
tracked-source context and exports only small metadata files.

## Verification

- Passed all five Docker build-stamp and dependency-cache tests with
`pnpm exec vitest run server/src/__tests__/docker-build-stamp.test.ts`.
- Passed the local ARM64 `docker buildx build --target runner-build
--progress plain`. Local Docker then hit storage errors during a runtime
probe; cache invalidation verification continues on GitHub-hosted Linux.
- Passed `bash -n scripts/check-docker-runner-cache.sh`, `actionlint`,
and `git diff --check`.
- Passed a [Linux AMD64 cache
probe](https://github.com/paperclipai/paperclip/actions/runs/34711042199)
against the PR source: dependencies compiled in 3m49s for the baseline
and were `CACHED` after a source edit; real source compilation took
about 37 seconds. Binary metadata changed and dependency declaration
changes altered the recipe. The permanent probe is also running in
latest-head Docker Runner check.
- Passed latest-head [Docker Runner
check](https://github.com/paperclipai/paperclip/actions/runs/34711145160),
including the permanent source/dependency invalidation probe.
- Passed full [PR
verification](https://github.com/paperclipai/paperclip/actions/runs/34711145352/attempts/2):
typecheck, all grouped tests, native verification, build, release dry
run, and browser checks. One unrelated signoff-policy browser test
failed waiting for a heartbeat run on attempt 1; only that failed shard
and dependent checks were retried, and passed.
- Latest-head Greptile is 5/5 with no unresolved findings. Full local
tests/build were limited by local disk exhaustion; Linux CI completed
those checks.

## Risks

- The two-build CI probe has a 20-minute job limit to cover the cold
build and source rebuild. It adds no AWS routing.
- A fully cold build must install cargo-chef and populate the dependency
layer. Both become reusable registry layers; no Actions cache is added.
- The recipe and final build must keep the same compiler, build profile,
package, binary, and directory layout. A source-change rebuild probe
checks real cache reuse and binary invalidation.
- Dependency or compiler changes still require rebuilding dependencies.
Existing image verification and full-SHA publication gates remain
unchanged.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 11:57:32 -07:00
Devin FoleyandPaperclip 59d74b68b2 ci: cache the native Runner in a separate Docker stage (#13195)
Compile the native Runner from its complete Cargo and protocol inputs in a separate cached Docker stage. Preserve Cargo validation and generated-contract checks during the normal application build, normalize input timestamps across checkouts, and compile the isolated target in PR CI.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-10 20:36:55 -07:00
Devin FoleyandPaperclip 3fd556b8f6 ci: preserve weekly Docker tool cache across commits (#13190)
Keep stable Docker tool installation layers independent of application build version and commit metadata. Preserve the existing weekly tool refresh and runtime build stamp.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-10 20:10:31 -07:00
Devin FoleyandPaperclip 314ff24b7a refactor(docker): declare the build stage's C toolchain explicitly (#12673)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Docker images compile the Rust runner (paperclip-runnerd) during
the build stage
> - The previous change replaced apt's cargo/rustc with a pinned rustup
install
> - apt's cargo package pulled gcc in as a dependency; rustup does not
install a C toolchain
> - Every master docker build now fails with "linker `cc` not found" in
the build scripts
> - This pull request installs gcc, libc6-dev, and pkg-config explicitly
in the build stage
> - The benefit is that the build stage declares its own C toolchain
instead of inheriting one by accident

## Linked Issues or Issue Description

**What happened?**

Master `docker.yml` builds fail in `pnpm --filter @paperclipai/server
build`: cargo build scripts (`libc`, `quote`, `proc-macro2`) die with ``
error: linker `cc` not found ``, and the job exits with code 101.

**Expected behavior**

Master docker builds compile the runner and publish images.

**Steps to reproduce**

1. Run the `docker.yml` workflow on current `master`.
2. Observe the `build-and-push` job fail with the linker error above.

**Paperclip version or commit**

`master` after commit `317394456` (the rustup change); example failing
run: docker.yml on `86ebdf842`.

## What Changed

- The build stage installs `gcc`, `libc6-dev`, and `pkg-config`
explicitly, with a comment recording why: the old apt cargo brought gcc
in as a dependency and the pinned rustup install does not.

## Verification

- Replicated the build stage's exact package sequence in
`node:24-trixie-slim` — base-stage packages only (which include no
compiler), then this new line, then the pinned, checksum-verified rustup
install: the runner compiles (`Finished release`) with `rustc 1.97.1`
and `cc 14.2.0`.
- Note: `docker.yml` triggers only on master pushes, so a PR run cannot
exercise the image build itself; the container replication above is the
pre-merge check. No test files: build-infrastructure fix (`fix:` on the
Dockerfile only).

## Risks

- Low risk. Three packages added to the build stage only; the production
stage is unchanged.

## Model Used

- Claude Fable 5 (`claude-fable-5`), Anthropic — Claude Code harness,
extended thinking + tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-01 10:38:53 -07:00
Dotta 5458940a6e feat(runner): add offline evaluation tooling (#12653)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner needs repeatable evaluation contracts.
> - Evaluation code must stay separate from provider launch and
production orchestration.
> - Offline fixtures need stable compatibility, scoring, traceability,
and report rules.
> - Published Runner consumers need only the supported evaluation
contract surface.
> - This pull request adds offline evaluation tooling and a
workspace-private matrix kernel.
> - The benefit is deterministic evaluation without credentials or paid
provider calls.

## Linked Issues or Issue Description

Refs #11297

This pull request extracts the offline evaluation unit from the earlier
aggregate Runner work.

## What Changed

- Add a workspace-private, provider-neutral evaluation matrix kernel.
- Add the public `@paperclipai/paperclip-runner/evals` compatibility and
native execution contracts.
- Add fail-closed runnerd artifact and protocol compatibility checks.
- Add deterministic workflow catalogs, scoring, traceability, and report
generation.
- Add sanitized Codex, OpenCode, and ACPX fixtures.
- Add package-boundary and clean-consumer checks.
- Add the eval package manifest to the Docker dependency stage.
- Add the generated protocol fixture digest without changing the
lockfile.

## Verification

GitHub Actions must run:

- Runner TypeScript and Rust type checks.
- Runner unit and protocol tests.
- Evaluation kernel tests.
- Workflow traceability checks.
- Clean-consumer and package-boundary checks.
- Repository test, type-check, build, policy, and security gates.

No local test command was run. The repository owner requested
GitHub-only verification.

## Risks

This is a large greenfield review surface with 51 files. The code does
not launch a live provider or load credentials. Package and protocol
drift fail closed. The workspace lockfile remains under the existing
CI-owned process.

## Model Used

OpenAI Codex with the GPT-5 agent model. The work used high reasoning,
repository inspection, tool use, and parallel code review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-01 05:19:47 -05:00
Devin FoleyandPaperclip 3173944561 refactor(docker): install rust via a verified rustup, pinned by rust-toolchain.toml (#12605)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Docker images ship the server together with the Rust runner
binary (paperclip-runnerd)
> - The Dockerfile installed rust from Debian trixie's apt archive,
which provides rustc 1.85
> - The runner's dependency tree and its `rust-toolchain.toml` now
require rustc 1.97.1, so every master docker build fails while hosted CI
(with a newer preinstalled rust) stays green
> - No image has been published since the dependency refresh, so
deployments cannot receive current builds
> - This pull request installs rustup from a version-pinned,
checksum-verified installer and defers the compiler choice to the
runner's own `rust-toolchain.toml`
> - The benefit is one toolchain pin, owned by the runner package,
shared by CI and image builds, installed without executing an unverified
remote script

## Linked Issues or Issue Description

**What happened?**

Every `docker.yml` build on `master` fails since the runner dependency
refresh. The failing step is `pnpm --filter @paperclipai/server build`,
which runs `cargo build` for `paperclip-runnerd`. Cargo reports: `rustc
1.85.0 is not supported by the following packages: icu_collections@2.3.0
requires rustc 1.88` (and sibling icu crates). The build exits with code
101 and no image is published.

**Expected behavior**

Master docker builds compile the runner and publish images.

**Steps to reproduce**

1. Run the `docker.yml` workflow on current `master`.
2. Observe the `build-and-push` and `build-and-push-cloud` jobs fail in
the server build step with the rustc version error.

**Paperclip version or commit**

`master` (first failing build ~2026-08-31 12:21 UTC; last successful
image build `fd7cb77d8`).

## What Changed

- The docker build stage downloads a pinned `rustup-init` (1.29.0) for
the build architecture, verifies it against an embedded sha256, and
installs with `--default-toolchain none`.
- The compiler version comes from
`packages/paperclip-runner/rust-toolchain.toml` (1.97.1) — one pin, no
drift between the Docker layer and the runner package.
- A comment records why apt rust is not used: Debian's archive lags the
ecosystem.

## Verification

- In the exact base image (`node:24-trixie-slim`): the checksum check
passes, rustup installs with no default toolchain, and `cargo build
--release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --bin paperclip-runnerd` completes with `rustc
1.97.1` selected from the toml.
- This PR's own docker build exercises the same path end to end for both
architectures.
- No test files: this is a build-infrastructure refactor with no runtime
code change (hence the `refactor:` prefix); the docker build itself is
the executable check.

## Risks

- Low risk. The change is scoped to the docker build stage; the
production stage is unchanged. The rustup installer version and
checksums are pinned; bumping rust later means editing only
`rust-toolchain.toml`.

## Model Used

- Claude Fable 5 (`claude-fable-5`), Anthropic — Claude Code harness,
extended thinking + tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-31 14:16:17 -07:00
Nicky LeachandPaperclip 7895f7f2b0 Install the declared Sentry server package into the hosted image (#12330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip supports opt-in Sentry error monitoring for server and
browser errors.
> - The hosted image must include the server package when an operator
sets SENTRY_DSN.
> - The server package is an optional peer in the source tree, so the
image did not include it.
> - This pull request installs the declared server package in the hosted
image and checks the result.
> - The benefit is a hosted tenant can send server errors without a
manual package install.

## Linked Issues or Issue Description

No public issue exists for this change.

**What happened?**

The hosted image did not include the declared @sentry/node server
package. A hosted tenant could set SENTRY_DSN, but the server could not
load the package from the image.

**Expected behavior**

The hosted image must include the exact @sentry/node version from
server/package.json. The self-hosted image must remain without this
optional package.

**Steps to reproduce**

1. Build or pull the hosted image.
2. Resolve @sentry/node from the server package path.
3. Compare its version with server/package.json.
4. Confirm that the tsx loader path still resolves.

**Paperclip version or commit**

Commit b6ff556a33ebdbe764b7f495951cd59009776608.

**Deployment mode**

Docker hosted image.

## What Changed

- Add a cloud-server-deps Docker stage that installs the declared
@sentry/node version in isolation.
- Copy the isolated package into the cloud image without changing the
production image.
- Add a probe that checks the tsx loader and the resolved Sentry
version.
- Run the probe after the hosted image push in the Docker workflow.
- Add server tests and update the observability documentation.

## Verification

- Run `pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/cloud-image-sentry.test.ts`.
- Confirm that the changed test passes in CI.
- Confirm that all pull request checks pass.
- Note that the Docker workflow does not run for pull requests. It runs
after a push to master, for configured tags, or after manual dispatch.

## Risks

- Low risk. The production image body stays unchanged.
- The cloud image adds the declared Sentry package and a small
dependency tree.
- The workflow probe fails if the image loses the tsx loader or resolves
a different Sentry version.

## Model Used

OpenAI GPT-5; exact model version supplied by the execution service;
tool use and code execution; context window not specified.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-27 19:10:49 -07:00
Dotta 397de98193 feat(runner): add flagged Codex execution adapter (#12188)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner now has protocol, provider, tool, package,
persistence, and hidden server boundaries.
> - The server still cannot select that path for a real agent heartbeat.
> - A new runtime must not change any existing direct adapter.
> - An experimental runtime must fail closed when its rollout flag is
off.
> - This pull request adds one guarded Codex vertical slice through
runnerd.
> - The benefit is a production-built runner path that users cannot
start by default.

## Linked Issues or Issue Description

Refs #11962

Refs #12111

Refs #12169

Refs #12176

**Subsystem affected**

Cross-cutting. The change affects the runner package, server
orchestration, shared settings, and adapter configuration UI.

**Problem or motivation**

The hidden PRP coordinator cannot execute a real heartbeat. The
application also needs an explicit rollout boundary before it can expose
the experimental runner. Existing direct adapters must keep their
current execution and finalization behavior.

**Proposed solution**

Add `paperclip_runner` as a Codex-only adapter behind the default-off
`enableNativeRunner` instance flag. Select the native runtime only for
that adapter. Persist the run binding before runnerd starts. Wait for
the durable PRP result and terminal event. Resume the real Codex
provider thread on later heartbeats. Keep persisted native runs readable
and recoverable after the flag changes.

**Alternatives considered**

The server could route `codex_local` through runnerd. That option would
change an existing adapter and weaken rollback safety. The server could
expose all providers now. That option would add unreviewed provider
behavior. The build could depend on a prebuilt runner binary. That
option would make source builds architecture-dependent and difficult to
verify.

**Roadmap alignment**

This work supports the shipped enforced-outcomes, governed-tool, and
self-healing-run milestones. It does not add a new roadmap surface. It
is the guarded execution step after the merged hidden runner boundaries.

**Additional context**

This is the next replacement for the closed large runner pull request.
Task-thread presentation remains a separate follow-up so this change can
preserve the current direct-adapter UI.

## What Changed

- Add `paperclip_runner` as an explicit Codex-only adapter.
- Add the default-off `enableNativeRunner` instance flag.
- Reject fresh create, hire, import, switch, and execution requests
while the flag is off.
- Allow edits to persisted runner agents while the flag is off.
- Recover an already persisted native run even after the flag is
disabled.
- Keep every built-in direct adapter on its existing runtime path.
- Persist an immutable native run binding and revisioned completion
contract before runnerd starts.
- Execute server to PRP to runnerd to Codex to server through the hidden
coordinator.
- Validate the durable result against the terminal event and exact
completion criteria before finalization.
- Preserve the Codex provider thread ID and use `thread/resume` on the
next heartbeat.
- Strip unsupported Codex configuration fields from the experimental
adapter.
- Build a target-native release runner binary from source and vendor it
into the server distribution.
- Install Rust only in the Docker build stage. Do not add a workflow or
lockfile change.
- Stop the runner process group on completion, cancellation, and forced
shutdown.

## Verification

- Run `pnpm --filter @paperclipai/paperclip-runner check:all`. All 69
TypeScript tests and 58 Rust tests pass. Protocol, conformance, replay,
formatting, and generated-file checks pass.
- Run the 12 focused adapter, settings, runtime-selection, coordinator,
direct-isolation, and real Codex integration test files. All 186 tests
pass.
- The real integration test uses PostgreSQL, HTTP, WebSocket, runnerd,
and a fake Codex app server. It proves one `thread/start` followed by
one `thread/resume`.
- Run `pnpm -r typecheck`.
- Run `pnpm build`.
- Run `pnpm check:token-gates`.
- Build the Docker `build` target from a clean context. Confirm that the
server distribution contains an executable `paperclip-runnerd` built
with Debian Rust 1.85.
- Start the server through the source-mode tsx entry point with the
package `dist` directory absent. Confirm the vendor shim resolves source
exports and the server boots.
- Run `pnpm test:run` twice. On this macOS host, 405 files pass and 1
file skips. Eight untouched workspace and loopback tests fail because
macOS resolves `/tmp` and `/var` through `/private` and because
PID-derived test ports exceed 65535. Linux CI must pass the full suite.
- Confirm that the diff contains 52 files. Confirm that it contains no
`.github` or `pnpm-lock.yaml` change.

## Risks

- The feature flag is off by default. A fresh native start fails with a
stable error while the flag is off.
- A persisted native run remains recoverable after the flag changes.
This prevents rollout changes from corrupting recorded work.
- Only local Codex execution is accepted. Other providers and remote
work modes fail closed.
- Existing direct adapters do not start runnerd, create native rows, use
native status arbitration, or enter native finalization.
- The runner receives its one-use bootstrap ticket through the child
environment. The server does not put the ticket in command arguments or
logs.
- The server validates the company, task, agent, run, runner, session,
completion contract, result, and terminal binding before it accepts
completion.
- The build compiles a target-native Rust binary. Cross-platform release
packaging remains a later concern. Source builds and Docker builds
compile for their current target.
- Docker needs enough build memory for the existing server TypeScript
compile. The Docker build stage sets a 4 GB V8 heap limit.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with GPT-5. The exact deployment ID and context-window
size are not exposed. The model used agentic reasoning, repository
tools, code execution, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and applicable tests pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-25 16:03:41 -05:00
Zannis Kalampoukisandzannis 5db8ce3c44 fix(docker): make tini PID 1 in the server image so adopted orphans are reaped (#12137)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute inside the server container, and they spawn many
short-lived descendants: git, the adapter CLI, esbuild, sh
> - The server image sets `ENTRYPOINT ["docker-entrypoint.sh"]`, and
that entrypoint ends in `exec`, so node becomes PID 1
> - Node reaps only the children it spawned itself. It installs no
`SIGCHLD`/`waitpid` handler for orphans that the kernel re-parents onto
PID 1, so those orphans stay as zombies forever
> - Zombies accumulate monotonically. When the cgroup pid limit is
reached, every `fork()` in the container fails and the instance is dead
> - This pull request installs `tini` and makes it PID 1 in front of the
existing entrypoint, adds a behavioural test that proves reaping, and
adds a `pids_limit` backstop to both compose files
> - The benefit is that a long-running container no longer degrades into
total fork failure, and a future regression is caught by CI instead of
by an outage

Depends-on: none — this change is self-contained in the image build and
its tests, and it touches no other in-flight branch

## Linked Issues or Issue Description

No public GitHub issue exists for this defect. It was found on a live
long-running instance. Description follows the bug report template.

**What happened?**

The server container ran for 22 hours and reached 2039 of 2048 pids in
its cgroup. Of 1760 processes, 1731 were zombies, and all 1731 had PID 1
as their parent. PID 1 was `node --import
./server/node_modules/tsx/dist/loader.mjs server/dist/index.js`. Zombies
accrued at about 79 per hour and were never reaped. The oldest zombie
was 20.8 hours old against a container uptime of 22.0 hours, so nothing
had been reaped since boot. Once the pid limit was reached, `git` and
`gh` failed with `pthread_create failed: Resource temporarily
unavailable`.

**Expected behavior**

PID 1 reaps orphaned processes that the kernel re-parents onto it. The
pid count of a long-running container stays flat instead of growing
without bound.

**Steps to reproduce**

1. Start the server image without `docker run --init` and without `init:
true`.
2. Run agent work that spawns descendants which outlive their immediate
parent.
3. Read `/sys/fs/cgroup/pids.current` and count processes in `Z` state
over several hours.
4. The zombie count grows monotonically and every zombie has PPID 1.

**Relevant logs or output**

```
cgroup pids.current / pids.max : 2039 / 2048
total processes                : 1760
  zombies                      : 1731  (98.4%)
  parent of every zombie       : PID 1  (1731/1731)
PID 1 cmdline                  : node --import .../tsx/dist/loader.mjs server/dist/index.js
container uptime               : 22.0 h
oldest zombie                  : 20.8 h    median: 14.4 h
zombie names                   : git 717, claude 280, MainThread 167, sleep 141,
                                 esbuild 138, postgres 76, sh 65, sccache 50
```

**Additional context**

The fix pattern is already in this repository.
`docker/agent-runtime/Dockerfile.base` installs `tini` and sets
`ENTRYPOINT ["/usr/bin/tini", "--"]`. It was never applied to the server
image.

## What Changed

- `Dockerfile`: install `tini` in the `base` stage and set `ENTRYPOINT
["/usr/bin/tini", "--", "docker-entrypoint.sh"]`. The entrypoint stays
in the exec chain, so UID/GID remapping, `gosu`, and graceful shutdown
are unchanged.
- `scripts/assert-orphan-reaping.sh` (new): a behavioural probe. It
spawns a leader that forks a grandchild, exits the leader, and asserts
that the orphaned grandchild leaves `Z` state instead of persisting. It
fails closed if the grandchild is not re-parented onto PID 1, so a pass
cannot mean the check ran too early.
- `.github/workflows/docker.yml`: run that probe against the pushed
image after the publish step. The publish step is multi-arch with `push:
true`, so nothing is loaded into the runner daemon and the pushed tag is
the only thing to test. The cloud variant is `FROM production` and
inherits the same `ENTRYPOINT`.
- `scripts/docker-build-test.sh`: run the same probe against a local
build.
- `docker/docker-compose.yml` and
`docker/docker-compose.quickstart.yml`: add `pids_limit: 2048` as a
backstop, so a future leak dies visibly at its own ceiling instead of
starving the host of pids.
- `server/src/__tests__/container-init-reaping.test.ts` (new): 13
assertions that guard the configuration the probe depends on.

No per-orchestrator init lever was added. The image owning PID 1 covers
compose, plain `docker run`, the quadlet units, and the ECS task
definition in one place. Adding `init: true` in compose or
`initProcessEnabled` on the ECS task would nest a second init around
`tini`, and `tini` then warns on every boot that it is not PID 1. The
new test asserts the absence of both levers across all three manifests,
so the decision survives the next edit.

## Verification

| Check | Result |
|---|---|
| `scripts/assert-orphan-reaping.sh` against a real init | Grandchild
re-parented to PPID 1, then reaped. Exit 0. |
| Same probe forced against a genuine zombie | Reports `Z` and fails.
The failure branch is not vacuous. |
| Config guard against the pre-fix files | Exactly the 3 relevant
assertions turn red. |
| Config guard with `tini` removed from `apt-get` but the comments kept
| Red. It checks the install, not a mention of the name. |
| `cd server && npx vitest run
src/__tests__/container-init-reaping.test.ts` | 13 passed |
| `npx tsc --noEmit -p server` | Clean |
| `node scripts/check-docker-deps-stage.mjs` | PASS |
| `node --test scripts/release-verify-workflow.test.mjs` | 8 passed |

Not verified locally: no container runtime is available in the authoring
environment, so the probe has not run against a build of this image. The
new `docker.yml` step runs it against the pushed image on this PR.

## Risks

Low risk, but it is an image and entrypoint change, so it affects
deployments.

- `tini` adds one small package to the `base` stage.
`docker/agent-runtime/Dockerfile.base` already installs it from the same
Debian archive.
- Signal handling changes shape: `tini` receives `SIGTERM` and forwards
it to the entrypoint, which `exec`s node. `tini` forwards signals to its
direct child by default, and the exec chain keeps node as that child, so
graceful shutdown is preserved. A reviewer should confirm this on a real
stop.
- `pids_limit: 2048` is new for compose users. A deployment that
legitimately needs more than 2048 processes would now hit the ceiling.
The measured steady state on a busy instance was under 400.
- If a deployment already passes `--init` or `init: true`, `tini` runs
under another init and prints a warning that it is not PID 1. Reaping
still works because the outer init handles it. The compose files in this
repository do not set `init: true`.

## Model Used

Claude Opus 5 (`claude-opus-5`), extended thinking, with tool use and
code execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local issues or links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: zannis <1011451+zannis@users.noreply.github.com>
2026-08-25 09:52:39 -07:00
DottaandPaperclip fdbc69172d feat(runner): add PRP v1 schemas and fixtures (#12087)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner needs a language-neutral contract between the
server and the runner process.
> - A shared contract must exist before TypeScript, Rust, transport, or
provider implementations can depend on it.
> - Required protocol versions must fail closed, while safe optional
fields must remain compatible.
> - The contract also needs deterministic fixtures and a drift gate for
later cross-language work.
> - This pull request adds that contract without adding runtime
behavior.
> - The benefit is a small, reviewable source of truth for the next
implementation pull requests.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting. This pull request adds a private package contract for
later server, TypeScript, and Rust work.

**Problem or motivation**

Paperclip Runner does not have a small language-neutral protocol
boundary on `master`. A runtime implementation without this boundary can
drift between languages, accept unsupported required versions, or
silently change canonical fixtures.

**Proposed solution**

Add PRP v1 JSON Schemas, accepted and rejected fixtures, a Codex
structured-question fixture, and a generated SHA-256 manifest. Run
compatibility and manifest checks during the package build. Keep the
package private and export nothing in this pull request.

**Alternatives considered**

The combined runner branch contains schemas together with providers,
SDKs, labs, and server behavior. That change is too large for normal
review. Generating TypeScript validators in this pull request would also
cross into the next review unit.

**Roadmap alignment**

This contract supports the governed tool and control-plane direction in
`ROADMAP.md`. It does not enable a new production adapter or endpoint.

**Additional context**

Refs #12084 and #11962. This pull request was reviewed as a stack on
#12084, then rebased and retargeted to `master` after #12084 merged. The
current delta is 38 files.

## What Changed

- Added 20 PRP v1 JSON Schemas with stable identifiers and resolved
references, including explicit cross-language conformance input and
output schemas.
- Added canonical replay, cross-language, and Codex question fixtures.
- Added accepted cases for additive optional fields and a rejected case
for an unsupported required protocol version.
- Added a deterministic manifest with SHA-256 digests for every schema
and fixture.
- Added package-local schema-instance, schema-reference, compatibility,
question-ID, conformance-pair, and drift checks.
- Added a private workspace package with no public exports and no
production runtime behavior.
- Added the package manifest to the Docker dependency-stage inventory
required for every workspace package. This does not copy or build runner
runtime code into the production image.
- Kept the provider descriptor and question fixture Codex-only. No
deferred provider package or dependency is present.

## Verification

- `pnpm install --frozen-lockfile` passed with Node 24.19.0 and pnpm
9.15.4. No lockfile change is committed.
- `pnpm --filter @paperclipai/paperclip-runner check:protocol` passed
with 8 tests.
- The committed AJV 2020-12 gate accepted every canonical v1 replay,
question, and cross-language conformance fixture. It rejected the
required v2 fixture, a replay fixture with a missing required command
ID, and conformance output with a missing session ID.
- `pnpm -r typecheck` passed.
- `pnpm build` passed and ran the protocol manifest drift check.
- `pnpm check:token-gates` passed.
- `node ./scripts/check-docker-deps-stage.mjs` passed.
- `git diff --check` passed.
- The delta against its declared base is 38 files.
- `pnpm test:run` completed with 4,687 passing tests, 19 skipped tests,
and 29 failures across 9 unchanged server files. The failures reproduce
macOS path aliases, local listener probes, workspace-runtime
assumptions, and one connection-retry timeout. No changed-file test
failed. Linux CI must pass before this pull request is ready.
- `pnpm check:tokens` reports existing personal-name references outside
this pull request. A scoped scan of `packages/paperclip-runner` found no
secret-like values, internal references, or deferred-provider names.
- PR #12084 was squash-merged, and this branch was rebased onto that
merge and retargeted to `master`. The first master-base policy run
correctly caught the missing Docker dependency-stage manifest copy;
commit `4fa1ea7c` fixes that gate, and the complete Linux matrix is
green.
- Serialized server shard 1 initially hit an unchanged heartbeat
test-harness timeout and a later assertion in the same file. Its
isolated rerun passed in 3m57s. All other shards passed on their first
attempt.
- Greptile reviewed the final commit at 5/5 with no blocking failure.
Both earlier actionable validation threads are resolved, and no review
thread remains open.

## Risks

Low production risk. The package is private and has no exports, server
adapter, endpoint, or process. AJV is a package-only development
dependency that the server workspace already uses. The main risk is
contract churn before the TypeScript and Rust consumers land. The
generated manifest and compatibility fixtures make that churn explicit.

I checked `ROADMAP.md`. This change defines a contract for planned
control-plane work and does not add overlapping product behavior.

## Model Used

OpenAI Codex with GPT-5 was used. The exact serving model ID and context
size were not exposed. The model used high reasoning, repository tools,
GitHub tools, and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-24 09:59:03 -05:00
Nicky LeachandPaperclip 38d8f37172 fix(build): enforce Node 24 across Paperclip (#11792)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip runs across the CLI, server, adapters, plugins, CI, and
container images.
> - These surfaces declared different Node.js versions from 20 through
24.
> - A newer `@types/node` major can expose APIs that the supported
runtime does not provide.
> - Node.js 20 is no longer a suitable project baseline, and Node.js 24
is the current LTS line.
> - This pull request sets Node.js 24.11.0 as one repository-wide
baseline, adds a drift check, and gives users actionable startup
guidance when their runtime is too old.
> - The benefit is one clear runtime contract for development, release,
installation, and published packages.

## Linked Issues or Issue Description

Refs #2734

Refs #11727

Refs #739

## What Changed

- Require Node.js 24.11.0 or newer in all 42 package manifests and
runtime checks.
- Use Node.js 24 in GitHub Actions, Docker images, smoke images, sandbox
setup, portable installs, and esbuild targets.
- Align every direct `@types/node` declaration on `^24.0.0`.
- Prevent Dependabot from opening major `@types/node` upgrades without a
matching runtime decision.
- Add `.nvmrc` and a CI policy check for Node version drift.
- Update ACP version gates, tests, and user documentation for the new
minimum.
- Print a non-blocking warning on CLI and server startup when Node is
unsupported, with remediation through a version manager or the
documented downloaded `install.sh` workflow.
- Deduplicate that warning when `paperclipai run` boots the CLI and
server in the same process.

## Verification

- `node scripts/check-node-version-policy.mjs`
- `node --check scripts/check-node-version-policy.mjs`
- `node --check cli/esbuild.config.mjs`
- `node --check scripts/generate-npm-package-json.mjs`
- `bash -n scripts/install.sh scripts/test-install-sh-docker.sh
scripts/e2e-install-lifecycle.sh`
- Parsed all 42 package manifests and confirmed `engines.node` is
`>=24.11.0`.
- `git diff --check`
- `vitest run
packages/adapter-utils/src/sandbox-install-command.test.ts` passed with
3 tests.
- `vitest run cli/src/node-version.test.ts` passed with 4 tests.
- Directly exercised the shared warning helper for unsupported-version
messaging and same-process deduplication.
- The focused exe.dev suite could not resolve the locally unbuilt plugin
SDK from this isolated worktree. A full offline workspace install was
also blocked because the package-manager signature verifier requires
registry access. The full suite was not run locally; draft CI performs a
clean install and evaluates the wider impact.

## Risks

- This is a breaking runtime change for users, plugins, and deployments
that still use Node.js 20 or 22.
- Published workspace packages will now produce an engine warning or
failure in strict package managers on older Node.js releases.
- Node.js 24 can reveal dependency, native module, Playwright, or agent
CLI compatibility issues in CI.
- The bootstrap installer now installs Node.js 24 when the current
runtime is older than 24.11.0.
- The portable sandbox fallback is pinned to Node.js 24.11.0 and depends
on that upstream tarball remaining available.
- Unsupported runtimes continue booting after a warning, so a later
incompatibility can still fail at its point of use.
- The CLI and server share the warning policy through the published
`@paperclipai/shared` package; packaging checks must keep that subpath
export available.
- This PR does not commit `pnpm-lock.yaml` because repository policy
assigns lockfile generation to CI.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex based on GPT-5. The exact deployment ID and context
window are not exposed in this session. Reasoning, repository tools,
shell execution, and GitHub tools were enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-21 10:17:52 -07:00
233c12f029 feat: add kimi-local adapter for Kimi Code CLI (CLI + ACP engines) (#9967)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Local agent adapters (`claude_local`, `gemini_local`, `grok_local`,
…) are the integration surface that lets Paperclip run coding CLIs on
the host machine
> - The Kimi Code CLI (`kimi`, Moonshot AI) has a documented
non-interactive mode, `kimi -p --output-format stream-json` with session
resume via `kimi -r`, but Paperclip has no built-in adapter for it
> - So Kimi users (especially Kimi membership / OAuth subscribers)
cannot onboard their CLI to Paperclip agent teams
> - This pull request adds a complete built-in `kimi_local` adapter
(both execution engines, session management, instructions + skills
delivery, thinking-effort control, environment test, UI and CLI modules,
docs) following the established `gemini_local`/`grok_local` package
pattern
> - Kimi Code ships an ACP server (`kimi acp`), so the adapter runs on
Paperclip's shared acpx engine by default (streaming transcript with
live tool status, like `claude_local`/`gemini_local`) and falls back to
a headless CLI lane (`kimi -p --output-format stream-json`) when ACP
prerequisites are unavailable
> - The benefit is that Kimi Code becomes a first-class Paperclip agent
lane: selectable in the UI, resumable across heartbeats, with the same
operating context (instruction bundle, skills, effort) and streaming
transcript the other local adapters get

## Linked Issues or Issue Description

- Supersedes #9880 (same branch; expanded from the CLI-only lane into a
complete adapter with the default ACP engine lane, control-plane skill
install, and live transcript wiring)
- Refs #9879 (adapter request for Kimi Code CLI, filed with this PR)
- Refs #163 (original Kimi support request)

Duplicate/related prior PRs, per the dedup search (both appear stale: no
updates or maintainer review since May 2026, and both target an older
Kimi CLI interface; calling them out for reviewer context per
CONTRIBUTING.md):

- Refs #6276 (`feat: add kimi-local adapter`): targets an older
array-based content format (`{type: think}`/`{type: text}` blocks), not
the current documented stream-json schema
- Refs #5202 (`feat(adapter): add Kimi CLI local adapter with Wire
protocol support`): builds on a `--wire` JSON-RPC interface that current
Kimi Code CLI (0.27.0) no longer documents; the current documented
headless interface is `-p --output-format stream-json`

This PR is a fresh implementation against current master and the
currently documented/verified Kimi CLI behavior (see Verification).
Happy to fold in anything useful from the earlier attempts if a reviewer
prefers.

## What Changed

- **New adapter package** `packages/adapters/kimi-local`
(`@paperclipai/adapter-kimi-local`), modeled on
`gemini-local`/`grok-local`:
- `src/server/execute.ts`: spawns `kimi -p <prompt> --output-format
stream-json` (argv array, no shell), `-m <model>` only when configured,
`-r <sessionId>` when the stored session cwd matches the run cwd,
automatic fresh-session retry on unrecoverable-session errors,
headless-safe env (`CI=1`, `NO_COLOR=1`, `KIMI_CODE_NO_AUTO_UPDATE=1`,
`TERM=dumb`; user-configured values win), full remote (ssh/sandbox)
execution lane with runtime install via `@moonshot-ai/kimi-code`
- **Instruction bundle delivery**: the prompt path directive now names
the sibling instruction files (`./HEARTBEAT.md`, `./SOUL.md`,
`./TOOLS.md`) alongside the prepended entry file, and local runs pass
`--add-dir <instructions-dir>` so Kimi can actually open them (matching
`claude_local`). Without this, only the entry file reached Kimi and
agents improvised the operating workflow that `HEARTBEAT.md` documents
- **Thinking effort**: a configured `effort` is forwarded as the
`KIMI_MODEL_THINKING_EFFORT` operational override (Kimi has no
per-invocation effort flag). It is only sent for models that advertise
`support_efforts` (currently `kimi-code/k3`) to avoid provider
rejections, and `medium` maps to `high` since Kimi has no medium tier
(`low`/`high`/`max` pass through)
- **Skills delivery**: desired Paperclip skills are delivered via Kimi's
`--skills-dir` flag from a dedicated per-run directory (a local
snapshot, or the synced snapshot on remote targets), so skills load
reliably and in isolation. Paperclip never overwrites the shared
`$KIMI_CODE_HOME/skills` home, so skills installed by the operator or
other agents are left intact. `--skills-dir` is only passed when at
least one skill is desired, so unconfigured agents keep Kimi's default
skill discovery
- **Live run status**: the adapter now forwards each streamed
stream-json line to `onEvent` (assistant `content` as an assistant
snippet, `tool_calls` as tool-name events), which drives the
issue-thread activity indicator (`currentToolName` /
`lastAssistantSnippet` / `lastEventAt`). Previously the adapter only
wrote the raw run log, so the issue thread showed a stale "no output for
N s" line with no tool or reasoning context while Kimi worked. Tool
results are omitted so the last meaningful "Using X" / snippet is not
overwritten by a generic label
- `src/server/parse.ts`: parses the verified Kimi stream-json event
shapes (`assistant` text, `assistant.tool_calls` with JSON-string
arguments, `tool` results, trailing `meta.session.resume_hint` for
session-id capture) plus failure classifiers (`kimi_auth_required`,
transient network, unrecoverable session). A signaled exit (null exit
code, not a timeout) is now reported as a failure rather than coalesced
to success, and the error message names the terminating signal
- `src/server/skills.ts`: lists/syncs Paperclip skills for the adapter's
skill-management surface
- `src/server/test.ts`: environment test covering CLI resolution + `kimi
--version`, cwd check, auth detection (OAuth credential dirs, keyed
`[providers.*]` in config.toml, or the `KIMI_MODEL_NAME` +
`KIMI_MODEL_API_KEY` env pair), and a live hello probe
- `src/ui/` (stdout-line parser for transcripts, config builder) and
`src/cli/` (stream event formatter) modules
- Root metadata: three managed model aliases
(`kimi-code/kimi-for-coding`, `kimi-code/kimi-for-coding-highspeed`,
`kimi-code/k3`), effort-capable-model metadata (`EFFORT_CAPABLE_MODELS`,
effort mapping helpers), `agentConfigurationDoc`
- Tests: 101 tests across parse, execute (args building, resume gating,
retry, auth error code, timeout, signaled-exit failure, effort
forwarding/gating/mapping, `--add-dir` instructions directive,
`--skills-dir` gating, `onEvent` runtime-event forwarding), ACP engine
(engine resolution, acpx config build, node-version gate), ACP
transcript delegation, environment test, UI parse/build-config
- **ACP engine lane (default)** (`src/server/acp.ts` + shared
`adapter-utils/acpx-engine`): Kimi Code ships an ACP server (`kimi
acp`), so `kimi_local` now runs on Paperclip's shared acpx engine by
default, matching `claude_local`/`codex_local`/`gemini_local`. The
issue-thread transcript streams live (assistant text deltas, tool calls
with a `pending`->`completed` status lifecycle) instead of the CLI
lane's bursty complete-message output. Registered `kimi_local -> "kimi"`
in `ACPX_ADAPTER_AGENT_IDS` and resolved the built-in agent command to
`kimi acp`; `execute.ts` dispatches to the ACP executor first with an
automatic CLI fallback when ACP prerequisites fail (`engine=acp`
requires ACP, `engine=cli` pins the headless lane); `index.ts` falls
back to the shared acpx session codec; the UI/CLI delegate `acpx.*`
events to the shared acpx transcript parser and event formatter. The
headless CLI lane (above) remains as the fallback
- **Registration** (one entry each, mirroring existing adapters): server
adapter registry + `BUILTIN_ADAPTER_TYPES`, `AGENT_ADAPTER_TYPES`
(shared), UI adapter registry + display registry (`Kimi Code`, Moon
icon) + capabilities defaults, CLI adapter registry, `Dockerfile`
(package copy + `npm install --global @moonshot-ai/kimi-code@latest`),
`vitest.config.ts` workspace, `scripts/release-package-manifest.json`
- **Behavioral sets** mirroring `gemini_local` (Kimi resumes sessions
the same way): `GIT_SENSITIVE_LOCAL_ADAPTER_TYPES`,
`SESSIONED_LOCAL_ADAPTERS` (heartbeat + recovery),
`REMOTE_MANAGED_ADAPTERS`, ssh/sandbox execution-target allow-lists,
`ADAPTER_DEFAULT_RULES_BY_TYPE` (`timeoutSec: 0`, `graceSec: 15`), and
`LEGACY_SESSIONED_ADAPTER_TYPES` + `ADAPTER_SESSION_MANAGEMENT` in
adapter-utils
- **UI touch-points**: New Agent default-model branch, AgentConfigForm
command map (`kimi_local: "kimi"`) + model defaults + a Kimi-specific
thinking-effort option list (`Low`/`High`/`Max`, reflecting Kimi's tiers
rather than borrowing Claude's), OnboardingWizard (command map, model
default, `kimi login` / `KIMI_MODEL_NAME + KIMI_MODEL_API_KEY` auth
hints, manual-debug command line), InviteLanding enabled adapters
- **Control-plane skill install** (`cli/src/commands/client/agent.ts`):
`paperclipai agent local-cli` seeded the Paperclip control-plane skills
into `~/.codex/skills` and `~/.claude/skills` so Codex/Claude agents
auto-discover the API reference every run. Kimi had no equivalent
target, so `kimi_local` agents began each session without the
control-plane skill and rediscovered routes (e.g. the company-scoped
`POST /api/companies/{companyId}/issues`) by trial and error. Added
`~/.kimi-code/skills` (honoring `KIMI_CODE_HOME`) as a third install
target for parity. Independent of the per-run `--skills-dir` delivery,
which only applies to explicitly configured skills.
- **Docs**: `docs/adapters/kimi-local.md` (prerequisites, auth options,
config fields including `effort`, session resume, instruction bundle,
skills delivery, control-plane skill install) + a row in
`docs/adapters/overview.md`

Out of scope (deliberately): model profiles, built-in agent
`allowedAdapterTypes` additions.

## Verification\n\nCurrent-master rebase verification (OpenAI Codex,
2026-08-03): 13 focused files / 231 tests pass; adapter-utils, server,
UI, CLI, and Kimi adapter typechecks pass; full repository build and UI
token gates pass. The branch is conflict-free against master at head
`1249df117c5e12e5771b9a570a6340866450619e`.\n\nAutomated (all from repo
root, pnpm 9.15.4, Node 22):

- `vitest run packages/adapters/kimi-local`: 89/89 pass (includes
coverage for the instruction `--add-dir` directive, effort
forwarding/gating/mapping, `--skills-dir` gating, the signaled-exit
failure path, and `onEvent` runtime-event forwarding with cross-chunk
line buffering)
- `vitest run server/src/__tests__/adapter-registry.test.ts
server/src/__tests__/adapter-routes.test.ts
server/src/services/heartbeat-stop-metadata.test.ts
ui/src/adapters/adapter-display-registry.test.ts`: 37/37 pass
- `vitest run cli/src/__tests__/skills.test.ts`: 13/13 pass (the
control-plane skill install target follows the existing Codex/Claude
install path, whose symlink logic is unchanged)
- `vitest run packages/shared`: 307/307 pass; `vitest run
packages/adapter-utils`: pass except one pre-existing, unrelated failure
(`mcp-isolation.integration.test.ts` requires Claude CLI ≥ 2.1.207; host
has 2.1.185, fails identically on unmodified master)
- `pnpm --filter @paperclipai/adapter-kimi-local typecheck|build`, plus
typecheck of `server`, `ui`, `cli`, `adapter-utils`: all clean
- `pnpm install --frozen-lockfile`: passes (the PR diff itself contains
no lockfile changes, per repo policy; verified against a locally
regenerated lockfile)
- `node scripts/check-no-git-push.mjs` and `node
scripts/check-forbidden-tokens.mjs`: pass
- CI note: the `policy` job's release-bootstrap step is expected to stay
red until a maintainer bootstraps the first npm publish of
`@paperclipai/adapter-kimi-local`; see the CI Note for Maintainers
comment. All other contributor-actionable checks are green.

Manual end-to-end (real Kimi CLI 0.27.0, OAuth login, dev server on an
isolated instance):

1. Server `GET /api/adapters` lists `kimi_local` as builtin with correct
capability flags; models endpoint returns the three Kimi models
2. `POST .../adapters/kimi_local/test-environment`: all checks pass,
including a live `kimi -p` hello probe
3. Created a `kimi_local` agent and invoked two heartbeats: run 1
spawned `kimi -p ... --output-format stream-json`, Kimi used its `Read`
tool, produced the expected answer, and the session id was captured from
the `session.resume_hint` meta event; run 2 resumed the **same** Kimi
session (`sessionIdBefore == sessionIdAfter`) via `-r`
4. UI: adapter appears in the New Agent dropdown; selecting it shows the
Kimi command placeholder, the three models, and the Kimi config fields;
the run transcript renders Kimi tool calls via the adapter's stdout
parser

The instruction-bundle, thinking-effort, and `--skills-dir` changes
landed after the manual run above. They are covered by the unit tests
listed under Automated, and the Kimi CLI flags they rely on
(`--add-dir`, `--skills-dir`, `KIMI_MODEL_THINKING_EFFORT`) were
confirmed against the installed Kimi Code CLI 0.27.0 (`kimi --help`,
config-file thinking-effort docs).

Screenshots (assets branch on the fork, not part of the diff):

![Kimi Code in the Add a new agent runtime
picker](https://raw.githubusercontent.com/hawikk/paperclip/assets/kimi-local-pr-screenshots/shots/00-kimicode.png)

![Adapter dropdown with Kimi
Code](https://raw.githubusercontent.com/hawikk/paperclip/assets/kimi-local-pr-screenshots/shots/01-adapter-dropdown-kimi.png)

![Kimi adapter selected: command, model, config
fields](https://raw.githubusercontent.com/hawikk/paperclip/assets/kimi-local-pr-screenshots/shots/02-kimi-adapter-selected.png)

![Kimi models in the model
dropdown](https://raw.githubusercontent.com/hawikk/paperclip/assets/kimi-local-pr-screenshots/shots/03-kimi-model-dropdown.png)

![Successful resumed heartbeat run (kimi_local invocation + parsed
transcript)](https://raw.githubusercontent.com/hawikk/paperclip/assets/kimi-local-pr-screenshots/shots/04-successful-resumed-run.png)

![Agents list showing the Kimi Code
label](https://raw.githubusercontent.com/hawikk/paperclip/assets/kimi-local-pr-screenshots/shots/05-agents-list.png)

## Risks

- Low risk to existing behavior: the change is additive, one new
workspace package plus single-entry registrations alongside existing
adapters; no existing adapter code paths are modified.
- The adapter invokes the locally installed `kimi` CLI; like other local
adapters, run behavior depends on the host's Kimi version. The parser is
written against the documented/verified 0.27.0 stream-json schema and
degrades gracefully (malformed lines are skipped, failures surface as
run errors).
- `--skills-dir` overrides Kimi's auto-discovery of user and project
skills for the run. This is intentional (paperclip-managed agents get a
reproducible, isolated skill set), and it is only passed when at least
one Paperclip skill is desired, so unconfigured agents keep default
discovery.
- Thinking effort is only forwarded to models that advertise
`support_efforts` (currently `kimi-code/k3`); `EFFORT_CAPABLE_MODELS`
must be extended when more Kimi models gain support, otherwise a
configured effort is silently ignored for them.
- `Dockerfile` now installs `@moonshot-ai/kimi-code@latest` globally
alongside the other agent CLIs, so image size increases slightly.
- Maintainer action needed for the npm bootstrap gate: the `policy`
job's release-bootstrap step fails until the first npm publish of
`@paperclipai/adapter-kimi-local` (the gate from #5146 that every new
adapter package has passed through). Enrollment with `publishFromCi:
true` is required by the manifest validator (dropping the entry,
`false`, or `private` are all rejected), so this is intentionally left
to a maintainer. Remaining CI lanes are expected to run once it is done.

## Model Used\n\n- **Current-master rebase, conflict adaptation, and
registry-parity coverage:** OpenAI, **GPT-5 Codex** (Codex agent; exact
serving model ID and context-window size were not exposed to the
runtime), with repository, shell, Git, and GitHub tooling. It preserved
Hawik’s commit authorship, reconciled ACPX and environment-capability
changes, added current registry tests, and ran the verification
above.\n- **Adapter implementation and initial review:** Moonshot AI,
**Kimi K3 Coding** (latest), via **Kimi Code CLI v0.27.0**
(`kimi-code/k3` alias, 1M-token context window, thinking mode, agentic
tool use). The CLI agent explored the repo, wrote the adapter
implementation (delegated to a coder sub-agent of the same model), ran
tests, and drafted the first version of this PR body. A second
model-driven review pass (read-only, same model) audited the diff for
security/correctness before submission; its findings (shell-quoting
hardening, auth-detection false positive, session-compaction
registration, test gaps) were fixed and are included.
- **Harness-context fixes and review responses:** Anthropic, **Claude
Opus 4.8** (`claude-opus-4-8`) via Claude Code. Diagnosed from run logs
that Kimi received only the entry instructions file (not the
`HEARTBEAT.md`/`SOUL.md`/`TOOLS.md` bundle) and that `effort` was never
wired, then implemented the instruction `--add-dir` delivery,
`KIMI_MODEL_THINKING_EFFORT` forwarding, and `--skills-dir` skill
delivery, added the accompanying tests and docs, and addressed the
automated review comments (preserving external skills on remote sync,
treating a signaled exit as a failure). Also extended the `paperclipai
agent local-cli` installer to seed the control-plane skills into
`~/.kimi-code/skills` for Codex/Claude parity, wired `onEvent` runtime
events so the issue-thread activity indicator reflects Kimi's tool and
reasoning output live, and built the ACP engine lane (`kimi acp` via the
shared acpx engine, default) so the transcript streams with live tool
status like the other ACP adapters. The Kimi CLI flags, subcommand, and
env var relied on here were verified against the installed Kimi Code CLI
0.27.0.
- All CLI behaviors claimed here (`-p`, `--output-format stream-json`,
`-r` resume, event shapes, `--add-dir`, `--skills-dir`,
`KIMI_MODEL_THINKING_EFFORT`) were verified empirically against the
installed Kimi CLI, not assumed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green *(only the release-bootstrap step
remains red, pending the maintainer npm publish described in Risks)*
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
*(will address all Greptile comments as they arrive)*
- [x] I will address all Greptile and reviewer comments before
requesting merge





---

## Maintainer Addendum (2026-08-20)

The shared acpx-engine and issue-chat changes (run-summary segmentation,
placeholder tool-event coalescing,
`ISSUE_CHAT_TRANSCRIPT_MAX_VISIBLE_ENTRIES` 30 → 400, live-reasoning UI)
have been **extracted to #11761** so the cross-adapter behavior changes
review and revert independently — both commits there preserve @hawikk's
authorship. This PR is now the kimi-specific adapter only (60 files,
+3,793/−8, essentially pure addition); the only shared-engine touch left
is the `kimi acp` command resolution. `publishFromCi` is `true` — the
package name is bootstrapped on npm.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Dotta <bippadotta@protonmail.com>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-08-20 12:06:33 -07:00
Nicky LeachandPaperclip 5a1ce7aed8 fix(server): stamp built commit into service.version (#11748)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server emits OpenTelemetry spans so operators can trace agent
work
> - Each span needs a service version that identifies the code that
produced it
> - The current service version comes from a static environment value
and can become stale after a rebuild
> - This pull request records the built commit and resolves the service
version from the build stamp, runtime Git, the environment, or an
unknown fallback
> - The benefit is trace data that identifies the correct built commit
during development and deployment

## Linked Issues or Issue Description

**What happened?**

The server used a static `OTEL_SERVICE_VERSION` value for every
OpenTelemetry span. Rebuilds could produce traces with an old commit
value.

**Expected behavior**

The server should report the built commit when a build stamp exists. It
should use runtime Git, the environment value, or `unknown` as fallback.

**Steps to reproduce**

1. Set `OTEL_SERVICE_VERSION` to an old commit value.
2. Build the server at a different commit.
3. Start the server and inspect the OpenTelemetry service version.
4. Confirm that the built commit takes precedence over the old
environment value.

## What Changed

- Add a build script that writes the short Git commit to
`dist/build-info.json`.
- Resolve `service.version` from the build stamp, runtime Git, the
environment, or `unknown`.
- Log the resolved service version once during server startup.
- Add tests for the resolution order and safe behavior without Git.
- Document the resolution order in `doc/observability.md`.

## Verification

- `pnpm --filter @paperclipai/server build`
- `npx vitest run server/src/__tests__/service-version.test.ts`
- `pnpm --filter @paperclipai/server typecheck`
- Confirm that the build stamp contains the short commit.
- Confirm that the stamp wins over the environment value.
- Confirm that a build without Git exits successfully without a stamp.

## Risks

The server now prefers the built commit over `OTEL_SERVICE_VERSION`. A
build without Git uses the existing environment value or `unknown`. The
change needs no schema migration and has a single-commit rollback path.

## Model Used

OpenAI Codex, GPT-5, tool use and code execution. The runtime does not
expose the context window size or reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-19 21:20:40 -07:00
Dotta f4802b1bbc feat(runtime-exposure): least-privilege Tailscale HTTPS broker, shared contract, and persisted exposure state (#11524)
<!-- Simplified Technical English (ASD-STE100). -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip starts and supervises managed runtime services for a
project's execution workspaces, so an agent's branch can be previewed
while it works
> - Those services only listen on plain loopback HTTP. A person on
another device, or on a phone, cannot open the preview
> - A Tailscale HTTPS mapping solves this, but `tailscale serve` needs
host privileges that the Paperclip server process must not hold
> - This pull request adds the foundation only: a separate
least-privilege host broker, the shared exposure contract, and the
database columns that hold exposure state
> - Nothing calls the broker yet, so there is no behavior change. The
benefit is that the privileged surface is small, reviewable, and
isolated before any lifecycle code depends on it

## Linked Issues or Issue Description

No public GitHub issue exists. The change follows the feature request
template.

**Subsystem affected**

Managed workspace runtime services, the shared type and validator
package, and the database schema.

**Problem or motivation**

A managed runtime service binds to loopback only. There is no supported
way to reach that preview from another device. Adding HTTPS directly to
the server would mean the server process runs `tailscale serve`, which
needs privileges far wider than the task requires. A compromised or
buggy server could then map any port to the tailnet.

**Proposed solution**

Split the privileged work into a separate broker process with a narrow
protocol, and define one shared contract that the server, the UI, the
runtime, and the broker all read. Land this foundation first, with no
caller, so the privileged code can be reviewed on its own.

**Alternatives considered**

- Call `tailscale serve` from the server process. This was rejected
because it gives the server unrestricted mapping authority.
- Use `sudo` for single `tailscale` commands. This was rejected because
the argument list is the only guard, and it is easy to widen by
accident.
- Use a generic reverse proxy. This was rejected because it does not
remove the need for a privileged Tailscale mapping step.

**Roadmap alignment**

This supports the existing managed workspace runtime capability. It adds
no new product surface on its own.

**Additional context**

The broker is the security boundary of the feature, so it is
deliberately the first slice. Three later pull requests build on it: the
server exposure lifecycle, the runtime lease and recovery integration,
and the leased-port mediator.

## What Changed

- Add the `@paperclipai/tailscale-https-broker` workspace package. The
broker listens on a unix socket, authorizes each peer with
`SO_PEERCRED`, and answers a small request protocol.
- Restrict what the broker will map. It accepts only same-number
HTTPS-to-loopback pairs inside the Paperclip port range, refuses
protected ports, and confirms that the loopback port belongs to a
Paperclip-owned listener.
- Parse every request with a strict JSON reader that rejects duplicate
keys, prototype keys, and unknown fields.
- Write an append-only audit record for each broker decision.
- Add the shared exposure contract in `@paperclipai/shared`: the
`RuntimeExposureConfig`, `RuntimeExposureState`, and
`RuntimeExposureStatus` types, their zod validators, the app and HMR
port rules, and the loopback-bind helpers.
- Persist exposure state on `workspace_runtime_services` with the new
`exposure` column, plus the server-private `exposure_handle` and
`backend_url` columns that are never serialized to API clients.
- Add the `execution_workspace_runtime_leases` table that the later
lease slice uses.
- Extend the runtime read-model test fixture for the three new columns.

## Verification

Focused checks, all run on this branch:

- `pnpm --filter @paperclipai/tailscale-https-broker test` — 12 files,
82 tests pass. This covers peer credentials, port policy, protected
ports, the serve config writer, the strict JSON reader, argv parsing,
and the socket server.
- `pnpm --filter @paperclipai/tailscale-https-broker typecheck` — clean.
- `npx vitest run --root packages/shared src/runtime-exposure
src/validators/runtime-exposure.test.ts` — 3 files, 40 tests pass.
- `pnpm --filter @paperclipai/db typecheck` — runs `check:migrations`
first. Migration numbering and migration safety both pass.
- `pnpm --filter @paperclipai/shared typecheck` — clean.
- `pnpm --filter @paperclipai/ui typecheck` — clean.
- `npx vitest run --root server
src/services/workspace-runtime-read-model.test.ts` — 3 tests pass.
- `npx tsc --noEmit -p server/tsconfig.json` — 139 errors, which is
exactly the count on `master` before this branch. All 139 come from the
unbuilt `@paperclipai/plugin-sdk` package.

To confirm the exposure state is inert, start a managed runtime service
as usual. The new columns stay null and the service behaves as it does
today.

## Risks

- Migration risk is low. Both migrations only add a table and three
nullable columns. No column is backfilled and no existing column
changes. The migration safety check passes.
- Behavior risk is low. No code path calls the broker in this pull
request, and the shared exposure fields are optional.
- The broker is privileged, so it is the real risk surface. It is
mitigated by peer-credential authorization, a fixed port range, a
protected-port deny list, same-number pair enforcement,
listener-ownership checks, strict JSON parsing, and an audit trail.
Reviewers should read
`packages/tailscale-https-broker/src/authorization.ts` and
`src/port-policy.ts` closely.
- The broker requires a `tailscale` version floor, which its README
records. An older host CLI makes the broker refuse to start rather than
map incorrectly.
- `pnpm-lock.yaml` changes because a new workspace package is added. The
diff is the new importer block, plus one duplicate `tinyexec` entry that
pnpm removed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR.

## Model Used

Claude Opus 5 (`claude-opus-5`), 1M context window, extended thinking,
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-17 05:54:12 -04:00
Devin Foley ea0dd3917e build: make image layer caching actually hit (#10571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Published container images are the deployable unit for self-hosted
and managed instances, so merge-to-image latency bounds every deploy
iteration
> - The docker workflow configures BuildKit caching, but builds still
ran ~12+ minutes essentially cold
> - Two causes: the most expensive layer (four CLI toolchains + apt) is
ordered after the always-changing app copy so it can never cache, and
the type=gha cache's 10GB repo cap means the two multi-arch mode=max
jobs evict each other
> - This pull request reorders the tool layer above the app copy (with a
weekly epoch so @latest tools keep advancing) and switches both jobs to
registry-backed cache in ghcr
> - The benefit is that warm builds shrink to roughly the app build +
push, targeting the sub-5-minute range together with the amd64-only
cloud variant

## Linked Issues or Issue Description

No existing public issue — inline description following the feature
request template:

**Subsystem affected**

CI / release publishing (docker workflow, Dockerfile)

**Problem or motivation**

Despite `cache-from/cache-to` being configured, image builds run
effectively cold: (1) the production stage installs four CLI toolchains
+ apt packages *after* `COPY --from=build /app /app`, and since the app
copy changes every commit, that most-expensive layer rebuilds every
build, per arch; (2) the `type=gha` BuildKit cache is capped at 10GB per
repository, and two multi-arch `mode=max` jobs overflow and evict each
other's entries.

**Proposed solution**

Order the tool/OS layer before the app copy (it references nothing from
`/app`), refresh it weekly via a `CLI_TOOLS_CACHE_EPOCH` build arg so
the `@latest` tools don't freeze in the cache, and move both jobs to
registry-backed BuildKit cache (`:buildcache` / `:buildcache-cloud` refs
in ghcr, no size cap, separate refs so the parallel jobs don't clobber
each other).

**Alternatives considered**

Pinning CLI tool versions instead of the weekly epoch — more
deterministic, but adds a version-bump chore; the weekly epoch preserves
current freshness semantics with bounded staleness. Keeping type=gha
with `mode=min` — smaller cache but loses intermediate-stage reuse,
which is where most of the win is.

**Roadmap alignment**

Not on ROADMAP.md; CI/publishing speed improvement only.

## What Changed

- `Dockerfile`: the production stage's tool/OS `RUN` (npm --global CLIs,
apt, `/paperclip` setup) moves above `COPY --from=build /app /app`; new
`CLI_TOOLS_CACHE_EPOCH` arg consumed by that layer. The `cloud` stage is
unaffected — it only layers plugin dists on top of the finished
production stage.
- `.github/workflows/docker.yml`: both jobs stamp the ISO week into
`CLI_TOOLS_CACHE_EPOCH`, and both switch `cache-from/cache-to` from
`type=gha` to `type=registry` with per-job refs.
- Includes the one-line amd64-only cloud-variant commit from #10570 so
the two PRs can't conflict; if #10570 merges first, this PR rebases down
to a single commit automatically.

## Verification

- Image content is unchanged by layer reordering: the moved `RUN`
references nothing from `/app`, and Docker layer ordering only affects
caching, not the final filesystem (tool installs and app copy touch
disjoint paths).
- The cache ref is written only by this workflow — `docker.yml` runs on
master/tag pushes, never on PRs — so the workflow's existing "no shared
caches into build inputs" supply-chain stance is unchanged (BuildKit
layer cache was already accepted via type=gha; the registry backend has
the same writer trust).
- Runtime proof lands with the first two master builds after merge: the
first warms the cache, the second should show the tool layer and
deps/build stages as CACHED in the build log, with wall clock dropping
accordingly. I'll be watching those as part of managed-deploy work.

## Risks

- Low. Worst case the registry cache misses (cold-build behavior, same
as today). The weekly epoch means CLI tools update at most a week late
inside images; a release built mid-week ships the tools from that week's
first build. Cache refs add two small artifacts to ghcr.

## Model Used

Claude Fable 5 (`claude-fable-5`, extended thinking, via Claude Code
with tool use and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no test-affecting changes)
- [x] I have added or updated tests where applicable (n/a — build config
and layer ordering only)
- [x] I have updated relevant documentation to reflect my changes
(in-file comments document both mechanisms)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-07-31 14:47:25 -07:00
Devin Foley 521271ebb7 build: bake the build commit into published images (#10566)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Instances commonly run from published container images, and
operators need to observe which build a container actually serves
> - `/api/health` now reports the running build commit, and server-info
already falls back to `PAPERCLIP_BUILD_COMMIT` when git is unavailable
> - But published images carry no `.git` and never received
`PAPERCLIP_BUILD_COMMIT`, so containers report `commit: null` — verified
live against a current image
> - That leaves the new deployment-verification field inert exactly
where it matters most: containerized deploys
> - This pull request bakes the exact build commit into both image
variants at build time, mirroring how `PAPERCLIP_BUILD_VERSION` is
already stamped
> - The benefit is that containers report their true commit on
`/api/health`, so deploy tooling can verify a rollout actually shipped

## Linked Issues or Issue Description

Companion to #10563 (which exposed the `commit` field on `/api/health`).
Inline description following the bug report template:

**What happened?**

A container from a published image responds to `GET /api/health` with
`"commit": null`. The image has no `.git` directory and the
`PAPERCLIP_BUILD_COMMIT` fallback that `server-info` supports is never
provided at build time, so git metadata resolves as unavailable.

**Expected behavior**

A container reports the commit it was built from, the same way it
already reports its build version via the baked
`PAPERCLIP_BUILD_VERSION`.

**Steps to reproduce**

Run any published image (e.g.
`ghcr.io/paperclipai/paperclip:sha-c4f6264-cloud`) and `curl
/api/health` — `commit` is `null` even though the build commit is known
at image-build time.

**Paperclip version or commit**

`sha-c4f6264-cloud` (first image containing #10563).

## What Changed

- `Dockerfile`: new `PAPERCLIP_BUILD_COMMIT` build arg, exported as an
ENV in the production stage (the `cloud` stage inherits it), directly
parallel to `PAPERCLIP_BUILD_VERSION`. Empty for local `docker build`,
which keeps the normal fallbacks.
- `.github/workflows/docker.yml`: both build jobs pass
`PAPERCLIP_BUILD_COMMIT=${{ github.sha }}`.

## Verification

- Reviewed the plumbing end-to-end: `build-commit.ts` reads
`PAPERCLIP_BUILD_COMMIT` (validated as a full SHA), `server-info.ts`
`readGitInfo` falls back to it when the git CLI fails, producing
`available: true, fullSha` — which `/api/health` surfaces as `commit`.
- Verified live that a current published image reports `commit: null`;
this change repairs that on the next build. Post-merge, the first master
image should report its commit — I'll be verifying that as part of
managed-deploy validation.
- No test changes: the fallback path is already covered by existing
server-info tests; this PR only supplies the env at image build.

## Risks

- Low. Two build-time stamps; no runtime code changes. A wrong SHA would
only mislabel the build (same failure mode `PAPERCLIP_BUILD_VERSION`
already carries), and `${{ github.sha }}` is the exact commit the
workflow builds.

## Model Used

Claude Fable 5 (`claude-fable-5`, extended thinking, via Claude Code
with tool use and code execution); diagnosis included live probes of a
running container's `/api/health`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no test-affecting changes;
server suites unaffected)
- [x] I have added or updated tests where applicable (n/a — build-time
stamps only)
- [x] I have updated relevant documentation to reflect my changes
(Dockerfile comments document the arg)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-07-31 13:01:52 -07:00
Devin FoleyandPaperclip d1b9448b57 fix(server): stamp the real build version into images instead of the package.json placeholder (#10257)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work; it ships as a Docker image that self-hosters and managed
deployments run.
> - The server resolves its own version at runtime in
`server/src/version.ts` (`resolveServerVersion()`), which feeds
analytics and the server debug panel.
> - That resolver derives the real version from `git describe`, and
falls back to `server/package.json`'s `version` when git isn't
available.
> - But `server/package.json`'s version is a static placeholder — CI
only stamps the real CalVer at publish, so in source it is never the
real version (currently `0.3.1`).
> - A Docker image has no `.git` (it's dockerignored), so `git describe`
can't run inside it. Every image therefore falls back to the placeholder
and reports `0.3.1` in analytics and the debug panel, regardless of
which commit it was built from.
> - This PR computes the real version once on the CI build runner (where
`.git` and tags exist), bakes it into the image, and has
`resolveServerVersion()` prefer that stamp when `git describe` is
unavailable.
> - The benefit: self-hosted and cloud images report their true version
instead of a misleading placeholder, with no change to dev checkouts,
`git describe`-based resolution, or local `docker build`.

## Linked Issues or Issue Description

No public issue exists — describing the bug inline (per the bug report
template).

**What happened?**
Docker images built from `master` (and release tags) report the server
version as the `0.3.1` placeholder in analytics and the server debug
panel, instead of the real version of the commit the image was built
from.

**Expected behavior**
An image reports the real version of its build commit (e.g.
`2026.722.0+51.git.<sha>`), so operators can tell which build is
running.

**Steps to reproduce**
1. Build the server Docker image from any `master` commit (the `Docker`
workflow, `production` target).
2. Run the image and open the server debug panel (or inspect the version
reported to analytics).
3. Observe the version is `0.3.1` rather than the commit's real version.

**Root cause**
`resolveServerVersion()` derives the real version from `git describe`,
but the image has no `.git` (dockerignored), so it falls back to
`server/package.json`'s `version` — a static placeholder CI only
replaces with the real CalVer at publish time. Nothing bakes the real
version into the image.

**Paperclip version or commit:** reproduces on `master` (`4c55f0d8`) and
any published image.
**Deployment mode:** self-hosted and managed (both the `production` and
`-cloud` images).
**Installation method:** Docker image (`ghcr.io/paperclipai/paperclip`).

**Related PRs (dedup search):** #9103 (merged — added the `git
describe`-based source-install resolution this builds on) and #9637
(closed). Neither bakes a version into the image; this PR closes that
gap. No duplicate found.

## What Changed

- **`.github/workflows/docker.yml`** — checkout with full history + tags
(`fetch-depth: 0`), and a new `Compute build version` step that runs
`git describe --tags --match 'v*' --long --dirty` on the pristine runner
checkout. The result is passed as a `PAPERCLIP_BUILD_VERSION` build-arg
to both the `production` and `-cloud` image builds.
- **`Dockerfile`** — the `production` stage takes an `ARG
PAPERCLIP_BUILD_VERSION` (default empty) and bakes it into the runtime
`ENV`; the `cloud` stage inherits it via `FROM production`.
- **`server/src/build-version.ts`** (new) — `readBuildVersion()` /
`parseBuildVersion()`, mirroring `build-commit.ts`: reads
`PAPERCLIP_BUILD_VERSION` (or a `.paperclip-build-version` file) as a
single-token stamp.
- **`server/src/version.ts`** — `resolveServerVersion()` prefers the
baked build version when `git describe` is unavailable, parsing it with
the same rules as a live checkout (`parseGitDescribeVersion`), and
falling through to the existing `build-commit` stamp and package version
when unset. A live checkout's `git describe` still wins over any stamp.
- Tests for the new behavior and the precedence.

## Verification

- `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && tsc
--noEmit` in `server/` — clean.
- `vitest run server/src/__tests__/version.test.ts
server/src/__tests__/build-version.test.ts` — **23 tests pass**,
covering: stamped version used when git describe fails, stamp parsed to
real CalVer, stamp preferred over the build-commit fallback, on-tag
stamp collapses to the release version, a pre-resolved stamp used
verbatim, and a live git describe still winning over a stamp.
- `git describe --tags --match 'v*' --long` for this commit →
`v2026.722.0-51-g<sha>`, which `resolveServerVersion()` reports as
`2026.722.0+51.git.<sha>` — no longer `0.3.1`.
- Not run locally: the full multi-arch image build (CI-only). The
workflow change is verified by inspection; the version is computed on
the pristine checkout before any lockfile refresh, so it carries no
spurious `-dirty`.

## Risks

Low. Additive and image-only:
- No runtime behavior changes for dev checkouts (git describe still
primary and wins over any stamp) or for local `docker build` (empty arg
→ server keeps its existing fallbacks).
- Not a breaking change; no schema or API surface. The stamp is
informational (version reporting only).
- `fetch-depth: 0` makes the release-image checkout fetch full
history/tags — a modest cost on a workflow that already runs at release
cadence with a 60-minute budget.
- Rollback: revert the commit; images simply return to reporting the
placeholder.

## Model Used

Claude Opus 4.8 (`claude-opus-4-8`, 1M-context variant), extended
thinking, with tool use / code execution — agentic edits, `tsc` +
`vitest` runs, and a `git describe` resolution check.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work (bugfix, not core feature work)
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#9103, #9637 — related, not duplicates)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (`fix/build-version-stamp`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user-facing docs affected; behavior is documented inline in `version.ts`
/ `build-version.ts` and the workflow/Dockerfile)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-25 10:06:28 -07:00
Devin Foley 965a827ee7 feat(docker): publish a cloud image variant with built bundled plugins (#10157)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Managed (cloud-hosted) deployments configure instances through
`PAPERCLIP_MANAGED_CONFIG`, including a `plugins.autoInstall` key list
that the boot-time installer resolves against the bundled plugin catalog
> - The installer requires each bundled plugin's `dist/manifest.js`
(`server/src/services/bundled-plugins.ts`), but the published image only
ships the sandbox providers' *source* — they are intentionally excluded
from the pnpm workspace, and the Dockerfile never builds them
> - Every managed auto-install therefore logs `bundled plugin bundle not
present; skipping auto-install` and no sandbox provider can be
provisioned through managed config
> - Baking built plugins into the single published image would fix it
but makes every self-hosted pull carry the providers' `node_modules` for
a managed-only mechanism
> - This pull request adds a `cloud` Dockerfile target extending
`production` with built bundled plugins — parameterized by build arg and
currently just `daytona` — published alongside the default image with a
`-cloud` tag suffix
> - The benefit is working plugin auto-provisioning for managed
deployments while the self-hosted image stays byte-identical and the
cloud variant only carries what is actually deployed

## Linked Issues or Issue Description

Fixes #10158 (filed for this problem; no prior issue existed — searched
for duplicate/related PRs and issues around bundled plugins, docker
image variants, and auto-install). Summary: **What happened:** on a
managed instance with `plugins.autoInstall: ["daytona"]` delivered via
`PAPERCLIP_MANAGED_CONFIG`, boot logs `bundled plugin bundle not
present; skipping auto-install` with `pluginPath:
/app/packages/plugins/sandbox-providers/daytona`, and the plugin is
never installed. **Expected:** the advertised bundled-catalog keys are
installable from the published image. **Why:** the image ships plugin
source without `dist/` — nothing in the Dockerfile builds the
workspace-excluded sandbox providers.

## What Changed

- `Dockerfile`: new `cloud-plugins` stage (based on `build`, so
devDependencies are available for `tsc`) that installs and builds each
provider named in the `CLOUD_BUNDLED_PLUGINS` build arg standalone
(`pnpm install --ignore-workspace --no-lockfile && pnpm build`, exactly
as the providers' READMEs prescribe), asserting `dist/manifest.js`
exists per plugin and failing loudly on unknown names; new `cloud` stage
= `production` + the built plugin tree. The arg defaults to `daytona` —
the only provider managed deployments auto-install today; every entry
adds its `node_modules` to the image, so the list grows only with actual
need (a one-line workflow change).
- `.github/workflows/docker.yml`: the existing build step is pinned to
`target: production` (without this, the new trailing stage would
silently become the default build target — this pin is what keeps the
self-hosted image identical); new metadata + build-push steps publish
the `cloud` target (with `CLOUD_BUNDLED_PLUGINS=daytona`) under the same
tag set with a `-cloud` suffix (`sha-<short>-cloud`, `latest-cloud`,
`<version>-cloud`), same schema labels, reusing the GHA layer cache

## Verification

- All seven sandbox providers build standalone from a clean checkout
with the exact commands the new stage runs, each producing
`dist/manifest.js` — so the current `daytona` default works and future
list additions are known-good
- The stage's shell loop was dry-run against the checkout (directory
existence + per-plugin assertion logic)
- Workflow YAML lints clean
- **Not run:** a full multi-arch `docker build` (no local docker
daemon). The `cloud` stage is additive and the default target is pinned,
so the risk is contained to the new build step; the first master build
after merge proves it end-to-end

## Risks

- Self-hosted behavior: unchanged. The default image build is pinned to
the `production` target, which produces the same layers as before this
change; the `cloud` stages run only for the new build step.
- The plugin installs in the `cloud-plugins` stage use `--no-lockfile`
(the providers are workspace-excluded and lockfile-less by design), so
plugin dependency resolution is not pinned at image-build time. This
mirrors the existing Plugins-page install path, which resolves from npm
at install time.
- CI cost: one additional build-push per master push. It reuses the
layer cache from the production build, so the marginal work is the
single plugin's build layers.
- An unknown name in `CLOUD_BUNDLED_PLUGINS`, or a provider that stops
producing `dist/manifest.js`, fails the cloud build loudly rather than
publishing a broken variant.

## Model Used

Claude (Anthropic), model ID `claude-fable-5[1m]` via Claude Code CLI —
extended thinking and tool use (code edits, standalone plugin build
verification, workflow lint).

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] Self-hosted behavior unchanged (default build target pinned to
`production`)
- [x] One clear change: publish a cloud image variant with built bundled
plugins
2026-07-24 08:22:34 -07:00
DottaandPaperclip 7b35de65aa feat(mcp) [split 1/8]: add fixture demo servers (#9556)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Governed MCP access spans contracts, runtime enforcement, adapters,
UI surfaces, and operator verification
> - The parity reference PR #9534 is too large for effective automated
or human review
> - The feature therefore needs a linear stack whose individual diffs
stay below the 100-file review limit
> - This pull request is split 1/8 and focuses on fixture and demo MCP
servers
> - The benefit is a standalone, testable review boundary while
preserving byte-for-byte parity at the top of the stack

## Linked Issues or Issue Description

- Related parity reference: #9534
- Problem: Developers need deterministic local MCP fixtures and visible
demo servers without pulling in the governed production runtime.
- Proposed solution: Adds the Google Sheets and KV demo MCP packages,
fixture catalog/servers, smoke harness, guide, and the root
smoke/typecheck registration hunks.
- Alternatives considered: keeping #9534 as one 403-file review, or
rewriting the feature to manufacture seams; both were rejected in favor
of path extraction plus compile-driven boundary moves.
- Roadmap alignment: this advances the existing governed MCP/tool-access
work already represented by #9534; it does not introduce a separate
roadmap initiative.
- Stack position: base branch is `master`.
- Merge policy: merge bottom-up, in order, only after the complete
eight-PR stack has been reviewed and the top-of-stack parity gate
remains empty.
- Requested review: QA for fixture and smoke coverage; Greptile on every
PR.

## What Changed

- Adds the Google Sheets and KV demo MCP packages, fixture
catalog/servers, smoke harness, guide, and the root smoke/typecheck
registration hunks.
- Keeps this PR below 100 changed files and independently typecheckable.
- Preserves the final tree from #9534 when combined with the other seven
stack levels.

## Verification

- `pnpm typecheck`
- `pnpm --filter @paperclipai/google-sheets-mcp-server test` — 27 tests
passed
- `pnpm --filter @paperclipai/kv-demo-mcp-server test` — 12 tests passed

## Risks

- The new packages add dependencies that are intentionally not committed
to `pnpm-lock.yaml`, per repository policy.
- Stack risk: merging out of order can expose incomplete layers;
mitigate by following the documented bottom-up merge policy.
- Parity risk: later edits to an intermediate branch can drift from
#9534; mitigate by re-running the empty top-of-stack diff before merge.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, exact model ID `gpt-5.4`; runtime-managed context
window; medium reasoning with repository, shell, Git, GitHub CLI, and
code-execution tools enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] Internal references are omitted except the execution-plan link
explicitly required for this coordinated split stack
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge


## Stack Coordination

- Internal execution plan:
[PAP-13874](/PAP/issues/PAP-13874#document-plan)
- Parity reference: #9534
- Stack: #9556 → #9557 → #9558 → #9559 → #9560 → #9561 → #9562 → #9563
- Merge bottom-up only after full-stack review and an empty parity diff
at #9563.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-14 12:56:21 -05:00
Devin FoleyandPaperclip eedc7ddef2 Make ACP the default engine for local adapters (#9238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Adapter packages are the bridge between the control plane and local
agent harnesses such as Claude Code, Codex, and Gemini CLI.
> - ACP support was concentrated in a separate `acpx_local` adapter,
which made ACP feel like a separate agent choice instead of an execution
capability of the harness adapters.
> - Claude, Codex, and Gemini now have ACP-capable harnesses, so the
native adapter should own ACP selection, fallback, config, transcript
parsing, and environment diagnostics.
> - The standalone ACPX adapter still needs a compatibility path for
existing rows, but it should not be offered as an active adapter for new
agents.
> - This pull request moves the shared ACP runtime into
`@paperclipai/acpx-engine`, wires Claude/Codex/Gemini local adapters to
prefer ACP when prerequisites are available, and retires `acpx_local` to
a tombstone.
> - The benefit is one adapter per harness, richer ACP transcripts by
default where possible, and a migration path for existing Claude/Codex
ACPX agents.

## Linked Issues or Issue Description

Closes #5932 — the broken default `acpx_local` Claude path is replaced
by native `claude_local` ACP support, existing Claude/Codex ACPX rows
migrate to native adapters, and new agents no longer choose the
standalone ACPX adapter.

Refs #4893 — original merged ACPX local adapter runtime that this PR
replaces with native per-harness ACP engines.
Refs #6590 — prior ACPX-Claude seamlessness work folded into the new
native Claude ACP path.
Refs #197 — related open generic ACP/Kiro adapter work; this PR does not
close it because Kiro/custom generic ACP remains a separate adapter
decision.
Refs #7018 — related Kimi-specific `acpx_local` shell failure; this PR
retires the built-in standalone adapter but does not add a native Kimi
adapter.
Refs #8864 — related ACPX prompt/API guidance PR; this PR moves runtime
guidance into the shared/native ACP engine path instead of the old
standalone adapter.
Refs #8881 — related `acpx_local` POSIX shell failure from the old
`acpx` pin; this PR updates ACP dependencies but does not claim
custom/OMP ACP support as a first-class native adapter.
Refs #8964 — related open `acpx_local` stderr cleanup PR; this PR makes
the old runtime path obsolete for new agents but keeps it as a
non-closing reference.

Problem description:

- The standalone `acpx_local` adapter duplicates Claude/Codex agent
choices that already have first-class local adapters.
- ACP should be an execution engine capability of each harness adapter
when the underlying harness supports ACP.
- Existing `acpx_local` agents should either migrate to native harness
adapters or fail with an explicit retirement message instead of silently
falling back to the process adapter.

## What Changed

- Added `@paperclipai/acpx-engine` as the shared ACP execution,
session-codec, CLI formatter, and UI parser package.
- Wired `claude_local`, `codex_local`, and `gemini_local` to auto-select
ACP by default when prerequisites pass, with `engine=cli` opt-out and
`engine=acp` strict mode.
- Added ACP config schema/UI fields, environment checks, session-codec
preservation, transcript parsing, and adapter capability metadata for
the native adapters.
- Retired `acpx_local` to a server tombstone, removed its
UI/package/runtime image surface, and added a migration for existing
Claude/Codex ACPX agents.
- Updated package manifests, lockfile, release tooling, docs, Kubernetes
sandbox defaults, and tests.

## Verification

- `corepack pnpm --filter @paperclipai/acpx-engine typecheck`
- `corepack pnpm --filter @paperclipai/adapter-claude-local typecheck`
- `corepack pnpm --filter @paperclipai/adapter-codex-local typecheck`
- `corepack pnpm --filter @paperclipai/adapter-gemini-local typecheck`
- `corepack pnpm --filter @paperclipai/acpx-engine exec vitest run`
- `corepack pnpm --filter @paperclipai/adapter-claude-local exec vitest
run src/server/acp.test.ts src/server/execute.acp-fallback.test.ts
src/ui/build-config.test.ts`
- `corepack pnpm --filter @paperclipai/adapter-codex-local exec vitest
run src/server/acp.test.ts src/ui/build-config.test.ts`
- `corepack pnpm --filter @paperclipai/adapter-gemini-local exec vitest
run src/server/acp.test.ts src/ui/build-config.test.ts
src/ui/parse-stdout.test.ts`
- `corepack pnpm --filter @paperclipai/plugin-sdk ensure-build-deps &&
corepack pnpm --filter @paperclipai/server exec tsc --noEmit`
- `corepack pnpm --filter @paperclipai/server exec vitest run
src/__tests__/adapter-routes.test.ts
src/__tests__/adapter-session-codecs.test.ts
src/__tests__/adapter-models.test.ts`
- `corepack pnpm --filter @paperclipai/ui typecheck`
- `corepack pnpm --filter @paperclipai/ui exec vitest run
src/adapters/metadata.test.ts
src/adapters/adapter-display-registry.test.ts
src/components/AgentConfigForm.test.ts
src/components/AgentConfigForm.render.test.tsx
src/components/transcript/RunTranscriptView.test.tsx`
- `node --test scripts/bootstrap-npm-package.test.mjs
scripts/release-package-map.test.mjs
scripts/verify-release-registry-state.test.mjs`

Note: the server typecheck script calls `pnpm` internally; this dev
shell exposes pnpm through Corepack only, so I ran the two script steps
manually with `corepack pnpm`.

## Risks

- Migration changes existing `acpx_local` Claude/Codex agents to native
adapter types and clears old ACPX task sessions/runtime state.
- Custom ACP commands remain on the retired tombstone and will need a
separate future adapter/plugin path.
- ACP auto-selection depends on local Node and ACP server command
prerequisites; remote and unsupported environments fall back to CLI
unless `engine=acp` is explicit.
- `@paperclipai/acpx-engine` is a new public package and needs npm
trusted-publishing bootstrap before release automation can publish it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI GPT-5 via Codex coding agent. Exact hosted model build and
context-window size are not exposed in this runtime. Tool use included
shell execution, repository editing, GitHub CLI operations, and local
test/typecheck execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-08 19:05:03 -07:00
DottaandPaperclip fd2f82ac5b [codex] Add built-in Hermes adapters (#8543)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent adapters are the boundary between the control plane and the
runtimes that actually do work.
> - Hermes support needs to be available as first-class local and
gateway adapters while still preserving the adapter-manager override
path for external packages.
> - The adapter work touches runtime execution, UI adapter metadata,
onboarding prompts, scoped credentials, release packaging, and smoke
coverage, so the handoff needs concrete verification rather than only
unit tests.
> - This pull request adds built-in Hermes local and Hermes gateway
support, keeps external adapter overrides compatible, and
documents/tests the gateway flow end to end.
> - The benefit is that operators can hire Hermes-backed agents without
a manual plugin install, while self-hosted installs can still
override/shadow the built-ins through Adapter manager packages.

## Linked Issues or Issue Description

No public GitHub issue exists for this exact Hermes built-in adapter,
gateway onboarding, and release-source work.

Problem description:
- Hermes local and gateway adapters need a public, reviewable source
path in the monorepo so package artifacts and built-in adapter behavior
match the application source.
- Operators need built-in `hermes_local` and `hermes_gateway` adapter
choices without losing the ability to install external Hermes packages
as overrides.
- Gateway onboarding needs secure defaults for API server URLs, API
keys, and generated agent setup text.
- Hermes-originated task bridge credentials need narrower API-key scope
configuration.
- Related public PRs found during duplicate search include #3027, #2363,
#7544, #7950, #8095, and #8543.

## What Changed

- Added the unified Hermes adapter package with local and gateway
server/UI/CLI exports, config schemas, transcript parsing, model
detection, and package metadata.
- Registered `hermes_local` and `hermes_gateway` as built-in adapters
across shared constants, server registries, CLI packaging, and UI
adapter registries.
- Kept the external adapter override path compatible so installed Hermes
packages can shadow built-ins and restore the built-in parser when
disabled.
- Added Hermes gateway onboarding docs, board-operator docs, Docker
smoke assets, and shell smoke harnesses for join/e2e validation.
- Added scoped task-bridge API-key support, authorization checks,
issue-origin handling, and tests for Hermes-created Paperclip tasks.
- Hardened gateway transport and redaction behavior for API keys,
headers, session data, and smoke diagnostics.
- Updated release packaging/bootstrap checks for the Hermes packages
while leaving `pnpm-lock.yaml` out of the PR per repository policy.

## Verification

Targeted local verification recorded before PR handoff:
- `pnpm --filter @paperclipai/hermes-paperclip-adapter exec vitest run
src/gateway/server/execute.test.ts` — 14/14 passed.
- `pnpm test:hermes-gateway-smoke` — 6/6 passed.
- Hermes package typecheck/build checks passed.
- Focused server/UI adapter tests passed — 31/31.
- Release helper Node tests passed — 18/18.
- `git diff --check origin/master..HEAD` passed.

Fresh Docker E2E smoke evidence:
- Ran `pnpm smoke:hermes-gateway-e2e` on 2026-06-26 with a fresh state
directory and fresh Docker container against a live Paperclip dev
server.
- Hermes direct execution reached `completed`.
- Hermes stop/cancel path reached `cancelled`.
- Hermes gateway created a Paperclip task, Paperclip ran the Hermes
agent, and the task reached `done` with the expected marker response.
- Temporary board auth keys, token files, smoke state, and Docker
containers were cleaned up after the run.

PR checks on head `b5eae40ce`:
- GitHub Actions passed: `policy`, `review`, `Typecheck + Release
Registry`, all general test shards, all serialized server shards,
`Build`, `Canary Dry Run`, `e2e`, and aggregate `verify`.
- External checks passed: Snyk and Socket Project Report.
- External Socket Pull Request Alerts remained pending after the
first-party CI matrix completed.

## Risks

- Medium risk: this spans adapter registration, package publishing,
gateway execution, onboarding docs, API-key scoping, and UI adapter
metadata.
- Migration risk is low: the scope-config migration adds a nullable
column and does not rewrite existing keys.
- Gateway execution depends on operator-provided Hermes API
configuration; the smoke covers the Docker gateway path but real
deployments may differ by network/auth setup.
- Direct Greptile review on the latest expanded diff is file-count
limited, although the commitperclip review gate passed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex, GPT-5 coding agent, tool use enabled in a local repository
workspace. Context window size is not exposed in this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Commitperclip review gate is green; direct Greptile review is
file-count limited on the latest expanded diff
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-26 16:04:58 -05:00
br-creativeandClaude Opus 4.6 1b89a8e3b8 fix: remove VOLUME keyword for Railway compatibility (#2619)
Railway bans VOLUME in Dockerfiles — persistent storage is handled via
Railway volumes instead.

## Thinking Path

<!--
Required. Trace your reasoning from the top of the project down to this
  specific change. Start with what Paperclip is, then narrow through the
  subsystem, the problem, and why this PR exists. Use blockquote style.
  Aim for 5–8 steps. See CONTRIBUTING.md for full examples.
-->

> - Paperclip orchestrates AI agents for zero-human companies
> - [Which subsystem or capability is involved]
> - [What problem or gap exists]
> - [Why it needs to be addressed]
> - This pull request ...
> - The benefit is ...

## What Changed

<!-- Bullet list of concrete changes. One bullet per logical unit. -->

-

## Verification

<!--
  How can a reviewer confirm this works? Include test commands, manual
  steps, or both. For UI changes, include before/after screenshots.
-->

-

## Risks

<!--
  What could go wrong? Mention migration safety, breaking changes,
  behavioral shifts, or "Low risk" if genuinely minor.
-->

-

## Model Used

<!--
  Required. Specify which AI model was used to produce or assist with
  this change. Be as descriptive as possible — include:
    • Provider and model name (e.g., Claude, GPT, Gemini, Codex)
• Exact model ID or version (e.g., claude-opus-4-6,
gpt-4-turbo-2024-04-09)
    • Context window size if relevant (e.g., 1M context)
• Reasoning/thinking mode if applicable (e.g., extended thinking,
chain-of-thought)
• Any other relevant capability details (e.g., tool use, code execution)
  If no AI model was used, write "None — human-authored".
-->

-

## Checklist

- [ ] I have included a thinking path that traces from project context
to this change
- [ ] I have specified the model used (with version and capability
details)
- [ ] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [ ] I have updated relevant documentation to reflect my changes
- [ ] I have considered and documented any risks above
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-15 14:33:53 -05:00
Darren DavisonandClaude Opus 4.8 9802636be3 build(docker): bundle Gemini CLI in image for gemini_local adapter (#7693)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work, and it runs agents through pluggable adapters.
> - One of those adapters, `gemini_local`
(`packages/adapters/gemini-local/`), runs Google's Gemini CLI on the
same host as the server.
> - For local (in-container) execution, the adapter only probes `PATH`
for the binary (`packages/adapter-utils/src/execution-target.ts`); it
does **not** auto-install — only `sandbox` transport targets install on
demand via `SANDBOX_INSTALL_COMMAND`.
> - The production Docker image bakes in `claude`, `codex`, and
`opencode` so their `*_local` adapters work out of the box, but `gemini`
was never added — so `gemini_local` fails inside the container with a
missing-binary error.
> - This PR adds `@google/gemini-cli@latest` to the image's global
install so `gemini_local` works locally like the other bundled CLIs,
sets `GEMINI_SANDBOX=false` for safe in-container CLI use, and documents
all bundled CLIs.
> - The benefit is plug-and-play Gemini support in Docker with no
per-deployment CLI install step, plus accurate docs (including a
previously-undocumented `opencode` bundle) and a heads-up about Google's
imminent unrestricted-API-key block.

## Linked Issues or Issue Description

No existing issue. Following the **adapter/feature** template fields:

- **Capability:** Run the existing `gemini_local` adapter inside the
official Docker image without a manual CLI install.
- **Problem/motivation:** The image pre-installs `claude`, `codex`, and
`opencode`, but not `gemini`. Because local execution probes `PATH` and
never auto-installs (only sandbox targets do), `gemini_local` runs fail
in-container with a missing-binary error. Operators currently have to
maintain a forked image.
- **Proposed solution:** Add `@google/gemini-cli@latest` to the existing
global `npm install` line; set `GEMINI_SANDBOX=false` in `ENV`.
- **Alternatives considered:** Runtime install at container start
(slower, network-dependent, inconsistent with how the other three CLIs
are handled).

**Related / superseded PRs** (both make the same Dockerfile change and
can be closed if this merges):
- #5912 — `fix(docker): install gemini-cli and normalize persistent
volume permissions`. Same Dockerfile line, but also bundles unrelated
volume-permission and heartbeat-test changes, and a maintainer noted it
now has a merge conflict. I corrected Greptile's `GEMINI_SANDBOX`
reasoning there and it was re-scored 5/5 (see [resolved
thread](https://github.com/paperclipai/paperclip/pull/5912#discussion_r3367332785))
— this PR is the rebased, docs-complete, single-purpose version of that
change.
- #2731 — `Added @google/gemini-cli to the Dockerfile`. One-liner, now
stale and merge-conflicting (predates the `opencode-ai` addition).

> **Note for Greptile:** the `--sandbox=none` reasoning for the Gemini
CLI sandbox concern was already validated on #5912's `Dockerfile:59`
thread and accepted (re-scored 5/5). This PR additionally bakes
`GEMINI_SANDBOX=false` into `ENV` as defense-in-depth, so the concern is
covered both at the adapter layer and the image layer.

## What Changed

- **Dockerfile:** add `@google/gemini-cli@latest` to the production
global npm install (after `opencode-ai`).
- **Dockerfile:** add `GEMINI_SANDBOX=false` to the `ENV` block (matches
the existing `OPENCODE_ALLOW_ALL_MODELS=true` precedent) so a
manually-invoked `gemini` inside the container does not attempt a
Docker-in-Docker sandbox.
- **docs/deploy/docker.md:** rename the section to "Local Adapter CLIs
in Docker"; list all four bundled CLIs mapped to their adapter type keys
(fixes a pre-existing gap — `opencode` was bundled but undocumented);
add `GEMINI_API_KEY` to the example; document per-provider credentials,
the `GEMINI_SANDBOX=false` default, and Google's 2026-06-19
unrestricted-key block with the `gemini auth login` (OAuth) alternative.

## Verification

- `npm view @google/gemini-cli` confirms the package exists, provides
the `gemini` bin, and requires Node `>=20` (the base image is Node 22
LTS). ✅
- Adapter already disables the CLI sandbox per run:
`packages/adapters/gemini-local/src/server/execute.ts` pushes
`--sandbox=none` whenever `config.sandbox` is false (the default). ✅
- Confirmed neither file was modified on `upstream/master`, so this
rebases cleanly with no conflicts. ✅
- Full image build is exercised by CI. (I did not run the multi-stage
`docker build` locally; the change adds one package to an existing,
working `npm install` line.)
- Reviewer manual check: `docker build -t paperclip-local . && docker
run --rm paperclip-local gemini --version` should print the CLI version.

## Risks

- **Low risk.** Adds one npm package to an existing global install and
one inert env var; no application code paths change.
- Minor image-size increase from the additional CLI (consistent with the
three already bundled).
- `@latest` is unpinned — intentionally consistent with the sibling
`@anthropic-ai/claude-code@latest` / `@openai/codex@latest` on the same
line; pinning all of them is a separate decision out of scope here.

## Model Used

Claude Opus 4.8 (model ID `claude-opus-4-8`, 1M-context variant), via
Claude Code with tool use / agentic file editing and web research.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues OR (b) described the
issue in-PR following the relevant issue template
- [ ] I have run tests locally and they pass (no unit tests cover the
Dockerfile; package/bin/engine verified via `npm view`, full build runs
in CI)
- [ ] I have added or updated tests where applicable (N/A — Docker image
+ docs change)
- [ ] If this change affects the UI, I have included before/after
screenshots (N/A)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (pending CI run on this PR)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending review)
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 10:42:07 -07:00
fff3832a01 [codex] Add teams catalog extraction (#7550)
Fixes #7551

## Thinking Path

> - Paperclip is the control plane for AI-agent companies, and reusable
company/team setup is part of making those companies faster to launch.
> - The teams catalog work introduces app-shipped team templates that
can be browsed, previewed, and installed into a company.
> - Catalog installation crosses several contracts: bundled package
contents, shared API types, server import/install behavior, CLI
workflows, and the board UI.
> - Agents also need a safe path through catalog installs: scoped
company selection, explicit source policy, approval fallback for agent
creation, and preserved catalog provenance.
> - This pull request extracts the completed teams catalog branch into
one reviewable PR on top of `public-gh/master`.
> - The benefit is a reusable teams catalog foundation with server, CLI,
package, docs, and hidden UI surfaces kept in sync.

## What Changed

- Added the `@paperclipai/teams-catalog` package with bundled/optional
team definitions, generated manifest, validators, catalog builder tests,
and migration notes.
- Added shared teams catalog types/validators plus server routes and
services for listing, previewing, and installing catalog teams.
- Integrated catalog install with company portability, skill/source
policy checks, provenance metadata, origin hashes, target-manager
reparenting, and installed/out-of-date detection.
- Added CLI `teams` commands and agent-safe company selection behavior,
including `company current` and approval fallback for forbidden
agent-run installs.
- Added hidden Team Catalog UI/API/query surfaces, Storybook fixtures,
and targeted UI tests while keeping the UI route out of primary
navigation.
- Added docs for CLI/company/teams catalog behavior and removed
generated screenshot artifacts from the PR diff.

## Verification

- `pnpm exec vitest run cli/src/__tests__/company.test.ts
cli/src/__tests__/teams.test.ts
packages/teams-catalog/src/catalog-builder.test.ts
packages/teams-catalog/src/shipped-catalog.test.ts
server/src/__tests__/agent-permissions-service.test.ts
server/src/__tests__/company-portability.test.ts
server/src/__tests__/company-skills-service.test.ts
server/src/__tests__/teams-catalog-routes.test.ts
server/src/__tests__/teams-catalog-service.test.ts
server/src/__tests__/teams-catalog-install-no-overrides.test.ts
ui/src/lib/company-routes.test.ts ui/src/pages/TeamCard.test.tsx
ui/src/pages/TeamCatalog.test.tsx
ui/src/pages/useInstallTeamCatalogEntry.test.tsx`
- `pnpm --filter @paperclipai/shared typecheck && pnpm --filter
@paperclipai/teams-catalog typecheck && pnpm --filter paperclipai
typecheck && pnpm --filter @paperclipai/server typecheck && pnpm
--filter @paperclipai/ui typecheck`
- Confirmed branch is rebased onto `public-gh/master` (`78dc3625a`) and
`public-gh/master` is an ancestor of `HEAD`.
- Confirmed PR diff excludes `pnpm-lock.yaml`, `.github/workflows/*`,
generated screenshot images, and screenshot helper scripts.

## Risks

- Medium review surface: this crosses package generation, shared
contracts, server install behavior, CLI, docs, and hidden UI code.
- Catalog install behavior creates agents/projects/tasks/skills and must
keep company scoping, permissions, source policy, and provenance checks
strict.
- `pnpm-lock.yaml` is intentionally excluded per repo policy;
CI/default-branch automation owns lockfile refresh.
- The Team Catalog UI is included but hidden from primary navigation, so
future enablement should re-check visual QA before exposure.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
>
> ROADMAP checked: this aligns with reusable companies/templates and
plugin-adjacent onboarding work. This PR packages work already developed
on the Paperclip task branch for review.

## Model Used

- OpenAI Codex, GPT-5 series coding agent in this Paperclip session;
exact runtime context window was not exposed. Used shell, git, `gh`, and
local test/typecheck tooling.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots, or documented why screenshots are intentionally omitted
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 12:55:49 -05:00
DottaandPaperclip 9eac727cf1 [codex] Add skills CLI and catalog management (#6782)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies through
company-scoped control-plane workflows.
> - Agents need reusable, inspectable skills that can be installed,
reset, audited, exported, and assigned without bespoke local setup.
> - The existing skill truth model needed cleanup so bundled skills,
optional catalog skills, runtime skills, and adapter-provided skills
have clear provenance.
> - Operators also need a practical CLI and board UI for discovering and
managing company skills.
> - This pull request adds the skills CLI, packaged skills catalog,
company skills APIs, and catalog-aware board UI.
> - The benefit is a more reusable Paperclip company setup where skills
are portable, auditable, and easier for operators and agents to manage.

## What Changed

- Added `paperclipai skills` CLI commands and coverage for catalog
listing, installing, resetting, and inspecting company skills.
- Added a packaged `@paperclipai/skills-catalog` workspace with bundled
and optional skill content plus validation/build tests.
- Added shared company-skill types and validators used across CLI,
server, and UI contracts.
- Added server catalog APIs/services for company skill catalog
operations, reset semantics, audit behavior, and portability provenance.
- Updated adapter skill handling so runtime/catalog provenance remains
explicit across local adapters.
- Added board UI support for browsing and managing catalog-backed
company skills.
- Updated docs for the skills CLI/catalog flow and the company skills
Paperclip skill reference.
- Rebased the branch onto current `paperclipai/paperclip:master`; no
`pnpm-lock.yaml`, `.github/workflows`, or migration files are included
in the final PR diff.

## Verification

- Passed: `pnpm run preflight:workspace-links && pnpm exec vitest run
cli/src/__tests__/skills.test.ts
packages/skills-catalog/src/catalog-builder.test.ts
packages/skills-catalog/src/shipped-catalog.test.ts
packages/shared/src/validators/company-skill.test.ts
packages/adapter-utils/src/server-utils.test.ts
packages/plugins/create-paperclip-plugin/src/entrypoints.test.ts
server/src/__tests__/company-skills-catalog-service.test.ts
server/src/__tests__/company-skills-routes.test.ts
server/src/__tests__/company-portability.test.ts`.
- Passed: `pnpm exec vitest run
server/src/__tests__/workspace-runtime.test.ts -t "default
branch|origin/master|symbolic-ref"`.
- Attempted: full `server/src/__tests__/workspace-runtime.test.ts`. Four
provisioning tests failed while seeding an isolated worktree database
from the local Paperclip instance because the local plugin schema dump
contains a duplicate-column foreign key
(`plugin_content_machine_18a7bc327b.content_case_signals`). The
default-branch tests touched by the rebase conflict passed in the
focused run above.
- Checked final diff: no `pnpm-lock.yaml`, no `.github/workflows`, and
no migration-file changes relative to `master`.

## Risks

- Medium: this is a broad skills/catalog change touching CLI, server
APIs, shared contracts, adapter skill sync, and UI.
- Catalog validation and reset semantics need careful reviewer attention
because they affect reusable company setup and portability.
- No database migrations are included in this PR, so there is no
migration ordering/idempotency risk in the final diff.
- No lockfile is included by design; dependency resolution will be
handled by the repository lockfile workflow.

## Model Used

- OpenAI Codex coding agent based on GPT-5, running in Paperclip via the
`codex_local` adapter with shell, git, GitHub CLI, and code-editing tool
access. Exact hosted model build/context-window metadata is not exposed
in this runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run targeted tests locally and documented the local
workspace-runtime seed failure above
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, screenshots were intentionally
omitted per PAP-10124 instructions; UI behavior is covered by tests and
reviewer inspection
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-28 07:33:51 -10:00
Devin FoleyandPaperclip f343bae119 fix(ci): copy link-plugin-dev-sdk.mjs into Docker deps stage (#6338)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - Releases ship via a Docker image built in the `build-and-push` CI
workflow
> - A recent change added `plugin-workspace-diff` to the pnpm workspace;
its `postinstall` hook calls `scripts/link-plugin-dev-sdk.mjs`
> - The Dockerfile's `deps` stage runs `pnpm install` before the full
repo is copied, so the script was missing and `pnpm install` failed with
`Cannot find module`
> - Sandbox-provider plugins have the same hook but are excluded from
the pnpm workspace, so they were unaffected — this was specific to
`plugin-workspace-diff`
> - This pull request copies `scripts/link-plugin-dev-sdk.mjs` into the
`deps` stage alongside the package.json files
> - The benefit is restoring the `build-and-push` CI workflow with a
minimal one-line change

## What Changed

- Add `COPY scripts/link-plugin-dev-sdk.mjs scripts/` to the
Dockerfile's `deps` stage so the `plugin-workspace-diff` postinstall
hook succeeds during `pnpm install`.

## Verification

- Reproduces the original failure on `master` by running `docker build
--target deps .` — fails at `pnpm install` with `Cannot find module
'/app/scripts/link-plugin-dev-sdk.mjs'`.
- With this patch, `docker build --target deps .` completes successfully
through the `deps` stage.
- CI `build-and-push` job (previously failing on
https://github.com/paperclipai/paperclip/actions/runs/26055610103/job/76602841176)
should now pass.

## Risks

- Low risk. One-line addition that copies a single script earlier in the
Docker build. No runtime behavior changes, no app code changes, no
schema changes.

## Model Used

- Claude (Anthropic), model ID `claude-opus-4-7`, extended thinking
enabled, 200K context. Used via Claude Code CLI with tool use (Bash,
Read, Edit, Grep) running inside the Paperclip agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-18 20:55:34 -07:00
DottaandPaperclip 5071c4c776 [codex] Add workspace diff viewer plugin (#6071)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies.
> - Operators need to inspect what agents changed inside execution and
project workspaces.
> - The existing workspace detail views did not provide a first-party
rich diff surface for staged, unstaged, head, renamed, binary,
oversized, and untracked changes.
> - The plugin system is the intended extension point for optional rich
UI surfaces.
> - This pull request adds a workspace diff plugin plus host services
and shared contracts so Changes tabs can render workspace diffs through
plugin slots.
> - The diff-renderer dependency should stay owned by the plugin package
rather than the core UI app.
> - The dependency surface must stay aligned with repository PR policy,
including intentionally omitting `pnpm-lock.yaml` from the PR.
> - The benefit is a more reviewable workspace surface without
hard-coding the renderer into every page.

## What Changed

- Added `@paperclipai/plugin-workspace-diff`, including diff
normalization, plugin manifest/worker/UI entrypoints, and focused plugin
tests.
- Kept `@pierre/diffs` scoped to `@paperclipai/plugin-workspace-diff`;
removed the core UI lab diff-renderer surface and direct UI package
dependency.
- Added shared workspace diff types and validators, plus plugin SDK
surface for workspace diff host services.
- Added server workspace diff service support and route coverage for
execution/project workspace diff flows.
- Wired Execution Workspace and Project Workspace Changes tabs to load
the diff plugin, including loading/error fallback behavior.
- Added UI tests and fixtures for the Changes tabs and plugin bridge
behavior.
- Added the new plugin package manifest to the Docker deps stage so PR
policy can validate dependency coverage.
- Addressed review hardening around empty untracked patches, workspace
path exposure, project workspace read capability checks, and default
base refs.

## Verification

- `pnpm --filter @paperclipai/plugin-workspace-diff test`
- `pnpm exec vitest run
packages/shared/src/validators/workspace-diff.test.ts
server/src/__tests__/workspace-diff-service.test.ts
ui/src/pages/ProjectWorkspaceDetail.test.tsx
ui/src/pages/ExecutionWorkspaceDetail.test.tsx`
- `pnpm exec vitest run ui/src/plugins/bridge.test.ts
server/src/__tests__/workspace-runtime-routes-authz.test.ts`
- `pnpm --filter @paperclipai/shared typecheck`
- `pnpm --filter @paperclipai/plugin-workspace-diff typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm --filter @paperclipai/ui typecheck`
- `node ./scripts/check-docker-deps-stage.mjs`
- Browser screenshot captured from the local worktree dev server:
https://files.catbox.moe/ofdpsp.png
- Confirmed branch is rebased onto `public-gh/master`,
`.github/workflows/pr.yml` is not included in the PR diff,
`ui/package.json` is not included in the PR diff, and `pnpm-lock.yaml`
is not included in the PR diff.

## Risks

- Medium UI integration risk: the Changes tab depends on the plugin slot
and host diff service path.
- Medium dependency risk: this adds `@pierre/diffs` in the plugin
package, but `pnpm-lock.yaml` is intentionally omitted per packaging
instructions because repository automation manages lockfile updates.
- Current CI blocker: downstream frozen installs fail until the
repository policy path for new plugin package dependencies is chosen.
- Diff rendering edge cases are covered for common working-tree and head
diff states, but very large repositories may still expose performance
limits.
- No migrations are included.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5 class coding model, tool-enabled local execution
environment. Exact context window was not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-18 08:50:06 -05:00
Devin Foley ab8b471685 Add built-in grok_local adapter (#6087)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies, so
adapter quality directly affects what runtimes the control plane can
supervise.
> - Local CLI adapters are one of the core execution surfaces because
they turn real coding tools into Paperclip-managed employees with
heartbeats, transcripts, and reviewability.
> - Grok Build was installed on the Paperclip host, but Paperclip had no
built-in `grok_local` adapter, so the runtime could not be configured
through the normal server/UI/CLI adapter path.
> - That gap needed to be closed with the same built-in registry,
environment diagnostics, transcript parsing, and skill/instructions
behavior that the other local adapters already rely on.
> - After the initial adapter landed, a real follow-up run showed that
Grok streaming text was being rendered one fragment per line, which made
transcripts harder to read even though the runtime itself was working.
> - This pull request adds the built-in `grok_local` adapter end-to-end
and then fixes the transcript parser so streamed Grok output is
coalesced into readable assistant/thinking blocks.
> - The benefit is that Grok Build becomes a first-class Paperclip
runtime with a usable operator experience instead of a partially wired
runtime with noisy transcript output.

## What Changed

- Added a new built-in `@paperclipai/adapter-grok-local` package with
server, UI, and CLI entrypoints.
- Implemented Grok execution, session handling, environment diagnostics,
config building, skill syncing, and parser coverage inside the new
adapter package.
- Registered `grok_local` across the built-in adapter inventories and
capability/display metadata in server, UI, CLI, and shared constants.
- Added adapter route coverage for the new built-in type.
- Fixed Grok transcript readability by emitting streamed `text` and
`thought` fragments as deltas so the shared transcript builder coalesces
them into readable message blocks.
- Added regression tests for the Grok parser and transcript coalescing
behavior.

## Verification

- `pnpm vitest run
packages/adapters/grok-local/src/ui/parse-stdout.test.ts
ui/src/adapters/transcript.test.ts`
- `pnpm --filter @paperclipai/adapter-grok-local build`
- Manual runtime verification on the Paperclip host during
implementation and follow-up review:
  - confirmed the Grok CLI was installed and authenticated
- confirmed the worktree dev server could be restarted cleanly and
health-checked after the parser follow-up
- No screenshots attached. This change is primarily adapter plumbing
plus transcript formatting behavior; reviewers can verify via the
Grok-backed run surfaces directly.

## Risks

- This adds a new built-in adapter, so any missed registration surface
could create inconsistencies between server, UI, and CLI behavior.
- The adapter depends on Grok Build's current event/output shape; if
upstream Grok streaming JSON changes, transcript parsing or session
extraction may need follow-up updates.
- The transcript readability fix intentionally changes how Grok
fragments are grouped, so any downstream code that implicitly expected
one entry per fragment would behave differently.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex via Paperclip `codex_local` agent runtime.
- GPT-5-class coding model with tool use, shell execution, file editing,
and repo inspection enabled.
- Exact backend model ID/context window were not surfaced to the agent
in this Paperclip session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-05-16 09:51:09 -07:00
DottaandPaperclip 508355b8fc [codex] Add LLM Wiki plugin package to master (#5716)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies.
> - The plugin system is the extension surface for optional product
capabilities without baking every workflow into core.
> - The LLM Wiki plugin package was reviewed in stacked PR #5592, which
targeted `pap-9173-llm-wiki-rest`.
> - The stack base PR #5597 merged to `master` before #5592 was merged
into that branch, so the plugin package never reached `master`.
> - A direct PR from `pap-9173-llm-wiki-rest` back to `master` would be
noisy because that branch has diverged from current `master`.
> - This pull request reapplies the reviewed
`packages/plugins/plugin-llm-wiki/` package onto current `master` and
updates Docker deps-stage manifest coverage.
> - The branch intentionally no longer changes `pnpm-workspace.yaml`
after maintainer feedback; because the new package is now a root
workspace importer, the remaining integration question is how
maintainers want the root lockfile handled under the current PR policy.

## What Changed

- Added the LLM Wiki plugin package under
`packages/plugins/plugin-llm-wiki/` from the merged PR #5592 head.
- Preserved the post-review cleanup from #5592: generated
design/screenshot artifacts are not committed, and `src/ui/index.tsx` /
`src/wiki.ts` are small public entrypoints.
- Added the new plugin package manifest to the Docker deps stage so
policy can validate package manifest coverage.
- Removed the earlier `pnpm-workspace.yaml` exclusion per maintainer
request, so the plugin is included by the existing `packages/plugins/*`
workspace glob.

## Verification

Current head:
- PGlite migration harness: ran migrations 001-003, verified old
non-space distillation unique constraints were removed, inserted
duplicate cursor and work-item keys in a second space, then reran
migration 003 successfully
- `node ./scripts/check-docker-deps-stage.mjs`
- `git diff --check`

Known current-head install result after removing the workspace
exclusion:
- `pnpm install --frozen-lockfile` fails because `pnpm-lock.yaml` has no
importer for `packages/plugins/plugin-llm-wiki/package.json`.

Previously verified on the same plugin source before the
workspace-exclusion removal:
- `pnpm --filter @paperclipai/plugin-sdk build`
- `cd packages/plugins/plugin-llm-wiki && pnpm install --lockfile=false
&& pnpm test`

## Risks

- The branch now includes `packages/plugins/plugin-llm-wiki` in the root
workspace but does not update `pnpm-lock.yaml`. Root frozen install will
fail until maintainers choose a lockfile path that fits repo policy.
- Committing `pnpm-lock.yaml` directly on this PR conflicts with the
current PR policy check, while excluding the package from
`pnpm-workspace.yaml` was rejected in maintainer feedback.
- The package includes UI code already reviewed in #5592; generated
screenshot/design artifacts were intentionally removed per maintainer
request, so visual review should regenerate screenshots locally if
needed.
- The package depends on plugin host support from #5597, which is
already merged to `master`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI GPT-5 Codex via Codex CLI, tool use and local code execution
enabled; context window not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run the targeted checks listed above
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

Stack context: #5592 was merged into `pap-9173-llm-wiki-rest` after
#5597 had already merged that branch to `master`, so this follow-up PR
is needed to carry the plugin package itself into `master`.

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-11 20:45:41 -05:00
DottaandPaperclip 21404e8a34 [codex] Fix Docker build without LLM wiki plugin package (#5714)
## Thinking Path

> - Paperclip is the control plane for autonomous AI companies, and its
Docker image needs to build from the checked-in core repository.
> - The Docker `deps` stage copies workspace package manifests before
running `pnpm install --frozen-lockfile` so dependency installation can
be cached.
> - Current `master` copied
`packages/plugins/plugin-llm-wiki/package.json`, but that plugin package
has not been merged into core yet.
> - Docker fails before install with a missing build-context path, so
the release image cannot build from the current repository state.
> - This pull request removes the premature plugin manifest copy while
leaving the plugin SDK and existing sandbox plugin package copies
intact.
> - The benefit is that the Docker build no longer depends on an
unmerged plugin package.

## What Changed

- Removed the `packages/plugins/plugin-llm-wiki/package.json` copy from
the Dockerfile `deps` stage.

## Verification

- `git diff --check`
- Static Dockerfile source validation: parsed non-stage `COPY` sources
and confirmed every source exists in the build context.
- Attempted `docker build --target deps --progress=plain -t
paperclip-pap-9235-deps-check .`, but Docker is unavailable in this
execution environment: `Cannot connect to the Docker daemon at
unix:///Users/dotta/.docker/run/docker.sock`.

## Risks

- Low risk. The removed path points to a package that is absent from the
repository, so retaining it is what breaks the build. The plugin can add
its manifest copy back when the package itself lands.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex using GPT-5, tool-enabled coding agent in a local
repository workspace. Exact context-window metadata is not exposed in
this runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-11 10:19:08 -05:00
Devin FoleyandPaperclip 534aee66ae Add cursor_cloud adapter for Cursor SDK + Cloud Agents API v1 (#5664)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - There are many adapter types, one per agent-runtime product (Claude,
Codex, OpenCode, Cursor local CLI, etc.)
> - Cursor shipped a public TypeScript SDK on 2026-04-29 that exposes
Cursor's full hosted-agent platform (cloud VMs, harness, MCP, skills,
hooks)
> - Paperclip had no first-class adapter for this — agents that wanted
to use Cursor's managed cloud runtime had to fall back to the local CLI
adapter, which loses the cloud session, streaming, and durable run model
> - This PR adds a new `cursor_cloud` adapter built directly on
`@cursor/sdk`, with Paperclip's heartbeat mapped to Cursor's
durable-agent + per-run model
> - The benefit is that any Paperclip agent can now drive a Cursor cloud
agent across heartbeats with native session reuse, streaming, and
cancellation, while Paperclip remains the source of truth for issue/task
state

## What Changed

- New built-in adapter package `packages/adapters/cursor-cloud` (15
files, ~1.7k LOC) backed by `@cursor/sdk` ^1.0.12
- `src/server/execute.ts` — SDK-first lifecycle: `Agent.create` /
`Agent.resume` / `Agent.getRun` / `agent.send` / `run.stream` /
`run.wait`, with session reuse keyed on the (runtime env type, env name,
repo set) tuple
- `src/server/session.ts` — codec for `cursorAgentId` + `latestRunId` +
repo metadata, persisted in `runtime.sessionParams`
- `src/server/test.ts` — environment probe via `Cursor.me()` and
optional model validation via `Cursor.models.list()`
- `src/ui/parse-stdout.ts` + `src/cli/format-event.ts` — normalize
Cursor SDK message types (`status`, `thinking`, `assistant`, `user`,
`tool_call`, `tool_result`, `result`) into Paperclip transcript events
for the UI and CLI
- Registrations: `packages/shared/src/constants.ts`,
`packages/adapter-utils/src/session-compaction.ts`,
`server/src/adapters/{registry,builtin-adapter-types}.ts`,
`ui/src/adapters/{registry,adapter-display-registry}.ts` +
`ui/src/adapters/cursor-cloud/index.ts`, `cli/src/adapters/registry.ts`,
plus workspace deps in `cli`/`server`/`ui` `package.json`
- `ui/src/components/AgentConfigForm.tsx` — hide local-Cursor
`mode`/thinking-effort field for `cursor_cloud` (different config
surface)
- 11 vitest tests covering execute paths (fresh create, matching-resume,
active-run reattach, non-finished result), session codec round-trip,
transcript parsing, and config building

## Verification

Reviewer steps:

```bash
pnpm install
pnpm --filter @paperclipai/adapter-cursor-cloud typecheck   # → clean
pnpm vitest run packages/adapters/cursor-cloud              # → 11/11 passing
```

End-to-end check against a real Cursor cloud agent (requires
`CURSOR_API_KEY` and Cursor GitHub-app install on the target repo):

1. Create a `cursor_cloud` agent in Paperclip with `repoUrl` set to the
test repo, `repoStartingRef: main`, and `env.CURSOR_API_KEY` set
2. Trigger a heartbeat → adapter calls `Agent.create({ cloud: { env: {
type: "cloud" }, repos: [...] } })`, streams events, terminates on
`finished`
3. Trigger a second heartbeat → adapter calls `Agent.resume` or
`agent.send` follow-up depending on prior-run state, reusing
`cursorAgentId`
4. The Paperclip UI/CLI transcript reflects Cursor `status` / `thinking`
/ `assistant` events as they stream
5. Cancellation from Paperclip maps to `run.cancel()` or Cloud API v1
`cancelRun` for cross-heartbeat cancellation

A direct-SDK smoke run against a real repo (devinfoley/my_test_project @
main) confirmed: `Cursor.me()` ok → `Agent.create` → `agent.send` →
`run.stream()` (30 events) → terminal status `finished` in ~11s.

## Risks

- **New adapter, additive only.** No existing adapter or registry is
replaced; current `cursor` local-CLI adapter is untouched. Default
behavior of any existing agent is unchanged.
- **External dependency on `@cursor/sdk`.** Cursor's SDK is v1.0.x and
may evolve. Mocked unit tests cover the public surface used here; if the
SDK breaks compatibility we update the adapter independently.
- **Cost/budget.** `cursor_cloud` runs on Cursor's billed cloud VMs;
operators must understand they are spending money outside Paperclip's
budget controls when they enable this adapter. Same shape as other
API-billed adapters.
- **No webhook support in V1.** The SDK already provides
stream/wait/cancel/reattach, so V1 does not require a public callback
URL. If a future use case needs out-of-band wakes, we add a Cloud API v1
webhook bridge as a separate change. This is called out in the issue
plan document.
- **Lockfile.** Per repo policy, `pnpm-lock.yaml` is intentionally not
in this PR — CI's lockfile workflow will update it on merge given the
manifest changes.

## Model Used

- Provider: Anthropic Claude (via Claude Code / Paperclip `claude_local`
adapter)
- Model: `claude-opus-4-7` (Claude Opus 4.7), knowledge cutoff January
2026
- Mode: standard tool-use with extended reasoning
- Context: ~200k token window
- Capabilities used: code generation, multi-file edits, shell/test
execution, GitHub PR workflow

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass (11/11 in
`packages/adapters/cursor-cloud`)
- [x] I have added or updated tests where applicable (4 new test files,
11 cases)
- [ ] If this change affects the UI, I have included before/after
screenshots (the only UI change is hiding the local-Cursor mode field on
the `cursor_cloud` adapter — happy to attach a screenshot if the
reviewer wants one)
- [x] I have updated relevant documentation to reflect my changes (issue
plan document supersedes the pre-SDK design; tracked in PAPA-203)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-10 17:21:04 -07:00