Commit Graph
2 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip 28ed2f1db0 fix(runner): support native OpenCode on macOS SSH hosts (#22534)
## Thinking Path
> - Paperclip coordinates agents across local, SSH, and sandbox
environments.
> - Native Runner must launch artifacts for the execution host.
> - A Linux controller currently selects Linux OpenCode assets for a
macOS SSH host.
> - Remote launch-file validation also accepts only ELF executables.
> - The standard Core image must carry a verified Mac runtime for this
host pair.
> - This pull request adds that runtime and selects it for Apple Silicon
SSH hosts.
> - Self-hosted and hosted controllers can use the same generic
implementation.

## Linked Issues or Issue Description
**What happened?**
A native OpenCode run on an Apple Silicon SSH host receives Linux
controller artifacts. Its private launch-file verifier also rejects
Mach-O files. A working legacy CLI agent does not exercise this path.

**Expected behavior**
The controller must select and verify the target's native daemon, Node,
OpenCode executable, and proxy. Existing Linux targets and legacy agents
must keep their current behavior.

**Steps to reproduce**
1. Run a standard Linux Core image.
2. Select an SSH environment on an Apple Silicon Mac.
3. Start a native OpenCode agent with a compatible model connection.
4. Observe the incompatible runtime selection before useful agent work.

**Paperclip version or commit**
Base: `324ec1f537a9ba1e84239faa81bcef326a5dd1b7`.

**Deployment mode**
Docker or another Linux controller, with macOS ARM64 execution over SSH.

Related work: #21662 covers local OpenCode confinement. #22236 covers
SSH process cleanup. This change does not depend on either PR. No
duplicate macOS OpenCode runtime PR was found.

## What Changed
- Select a verified `darwin-arm64` runtime for OpenCode SSH targets.
Preserve the Linux image default and explicit operator overrides.
- Verify Mach-O launch files against the admitted architecture, file
digests, and remote account. Keep existing Linux receipt and recovery
behavior.
- Build the Mac runtime on an Apple Silicon CI host with standalone Node
24.21.0. Keep OpenCode pinned to 1.18.34.
- Package the sealed OpenCode runtime artifacts and required package
metadata. Omit other harness distributions and their optional
qualification inventories.
- Verify archive provenance and safe extraction on Linux. Include the
runtime in standard and merge-queue Core images. Avoid storing the
transport archive in an image layer.
- Reuse artifacts only from successful protected master image builds
with identical runtime inputs. Test cache trust boundaries and register
the script tests in CI.
- Document runtime delivery, manual image inputs, and the initial Apple
Silicon scope. No Cloud source, schema, or agent configuration changes.

## Verification
- `pnpm -r typecheck` — passed.
- `pnpm build` — passed.
- Focused OpenCode artifact selection and private staging tests — 21
passed.
- Focused native executor tests for OpenCode, SSH, and provider packs —
60 passed.
- Real Rust daemon transport tests with the OpenCode protocol fixture —
4 passed.
- Installed OpenCode 1.18.34 server qualification with a local provider
fixture — 3 passed.
- Runtime packaging, cache trust, workflow, image-contract, and
merge-queue tests — 26 passed.
- The CI release-workflow test group, including canary reuse and
provenance — 218 passed.
- Docker workflow and build-stamp guards — 29 passed after adding the
Mac digest expectation, updating the excluded-source COPY expectation,
and removing the redundant production commit ARG.
- The source SDK registry, public-type, and evidence checks — 59 passed
after updating only the root package metadata hash. Lock, compiler,
parser, and model pins did not change.
- Fresh-host provider-pack upload harness — 7 passed. The new regression
first failed with `package_version:acpx`, then passed after the required
metadata was preserved.
- Node version policy — passed against a clean tracked-file snapshot.
Generated local provider assets make the direct workspace scan fail;
they are absent from the CI checkout.
- Built, packaged, installed, and verified the actual Mac runtime. Both
the controller verifier and the remote startup verifier accept its pins,
target, sealed files, manifest, and package metadata. The pre-change
private launch verifier rejects the real Mach-O files; the candidate
accepts those same files. The dedicated archive is about 101 MB.
- Live native smoke: real Mac daemon, private staged Mach-O files,
OpenCode 1.18.34, and a local GLM-5.3 backend. The model read two files,
edited the implementation, ran three tests, completed the semantic
finish call, and summarized the three passes. An independent test run
passed. The test file stayed unchanged. This used the existing `allow`
mode and a scratch account-home fixture on a Mac. It does not prove
hosted SSH dispatch.
- The full CI workflow passed on
`ae74808631f1933edec89aa1e80db7f2b756aca3`: server test shards, Runner
Rust and native server tests, type checking, builds, browser suites,
image checks, the native Mac producer, and Linux runtime installation.
Greptile is 5/5 with no unresolved threads.
- Local `pnpm test:run` used a disposable PostgreSQL 18 fixture after
embedded Postgres failed to start because its macOS package lacks
dynamic libraries. All six chat shards passed (1,971 tests). The
redundant unsharded local run was interrupted during the general-server
group after all current-head CI test jobs passed. The later local
workspace and serialized groups did not run. The owned PostgreSQL
fixture is stopped. This is not a local full-suite pass.

Retained review failures: the first PR revision omitted package metadata
and did not ignore downloaded image inputs. The fresh-host upload
regression reproduced the metadata failure before its fix. Generated
inputs are now ignored. CI also exposed a missing Node policy exception
for the pinned Mac producer; the exception is limited to that producer
job. A later policy attempt exposed an old exact-dependency assertion in
canary reuse tests. The assertion now covers the added runtime and
evaluates the publication gate for success, failure, cancellation, and
skip outcomes. The Runner support suite then found that the new test
registration changed a pinned root package input; the standard source
registry generator updated that one hash, and all 59 affected tests
passed. The server matrix then found four outdated image guard
expectations and a duplicate production commit ARG. The focused image
guards now pass after updating those expectations and keeping one commit
ARG before Mac verification.

Retained failures: the first producer attempt used Homebrew Node, which
failed the relocation check. The successful producer used standalone
Node 24.21.0. Live smoke attempts also encountered a stopped tunnel, an
upstream model failure, and the existing Rust restriction on OpenCode
permission requests in `ask` mode. The successful tool trace used the
existing `allow` mode. Its initial reporter looked for tool names in
generic item events; verification of the retained canonical tool events
and an independent rerun confirmed the result. The upstream model
failure is not explained by a later pass.

Not run locally: a full Docker image build, because the existing Docker
daemon does not respond. The PR adds native producer qualification and
Linux installation verification to CI. Hosted deployment, remote
service-account dispatch, and hosted continuation remain acceptance
steps after deployment.

## Risks
- A cold Mac runtime build adds CI time when Runner or producer inputs
change. Reuse is bounded and accepts only a successful protected master
producer. The artifact digest and runtime input identity remain
verified.
- Standard images gain a dedicated Mac runtime payload. Manual image
builds can omit it; Mac OpenCode SSH runs then fail with a named
missing-runtime error. Linux behavior remains available.
- This release covers Apple Silicon SSH distribution. It does not
qualify Intel Mac distribution or expand other harness support.
- The OpenCode permission bridge keeps its existing behavior. This PR
does not add `ask` approval support.
- No saved run/session schema or provider pin changes. Existing Linux
receipts retain their interpretation. A surviving admitted Runner
bypasses new artifact discovery. Legacy agents are not converted.
Existing account-change and old-receipt restoration tests remain in
coverage.
- No migration, cluster change, new sandbox image requirement, or Cloud
control-plane change. A staging application rebuild and deployment are
separate from this PR.

## Model Used
OpenAI GPT-6, through Codex, with repository tools, code execution,
tests, and SSH inspection. The session does not expose a more specific
model ID or context-window size.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-22 00:07:34 -08:00
DottaandPaperclip 374a918d67 fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must keep admitted provider state and useful
workspace files.
> - OpenCode binds its saved session to exact executable paths and
hashes.
> - A replacement image can remove those files even when session data is
saved.
> - Retaining a manifest alone cannot restore the missing launch files.
> - This pull request saves the qualified files before provider work and
restores the same owned paths.
> - The benefit is cold continuity without relaxing the Runner guard or
replaying a prior turn.

## Linked Issues or Issue Description

Refs #19558. That fix retains the selected pack for ordinary
continuation. This change adds durable file retention for a cold
allocation. Related #19487 adjusts the admission budget; it does not
retain launch files. A search of open and closed OpenCode cold/durable
artifact PRs found no duplicate.

**What happened?**

A saved OpenCode session could outlive the image that supplied its Node
executable, proxy, and native executable. A new image supplied different
bytes. The Runner then correctly refused the changed launch profile. The
saved harness did not contain the old files.

**Expected behavior**

Core must save qualified launch files before the first turn. It must
restore those exact files under the same owned account and absolute
paths. Unknown ownership, missing files, and changed hashes must still
stop recovery.

**Steps to reproduce**

Start an admitted remote OpenCode session. Save its workspace, encrypted
checkpoint, and harness data. Destroy the original allocation and
controller files. Restore into a new allocation with an image whose
qualified proxy has different bytes. Resume the same session. The new
opt-in local qualification command automates this path with actual
OpenCode and a local synthetic model.

## What Changed

- Store company-bound, encrypted launch objects and canonical run/backup
references. Generate migration 0438 through the normal database
workflow.
- Stage qualified Node, proxy, and native OpenCode files under a private
content-addressed OS-account home. Record the account UID and exact
launch paths. Preserve the Rust profile guard.
- Gate first provider work on durable capture under the current native
claim. Bound Stop/deadline waits while the actual late transfer keeps
its worker and scratch ownership until settlement. Restore cold caches
with verified staging and atomic rename. Keep live reattachment and
historical image-rooted sessions on their existing paths.
- Reuse bounded canonical expiration owners. Atomically fence old
checkpoint authority with a strict company/run-scoped retirement marker
before external deletion. Keep its indexed cleanup anchor across failed
deletes and restart. Preserve earlier resume handles and reject later
header or source-consumer writes. Add no idle scan or network call
inside a transaction.
- Add storage, race, executor, filesystem, and actual Rust resume
regressions. Add the opt-in local Linux qualification command and
operating limits in `doc/opencode-cold-recovery.md`.

## Verification

- RED before repair: real PostgreSQL/AES cold restore lacked the launch
files; normal executor preparation chose the new image pack. Separate
regressions exposed lost upload receipts, missing cache repair,
last-owner retirement, an account-UID mismatch, and a resume-handle race
during external deletion. Each expected failure was captured before its
repair.
- Local qualification before the final master integration: 1,169
executor tests, 233 storage/archive/admission/expiration tests, all
seven server typecheck layers, migration safety, and the actual Rust
cold-resume test passed.
- The maintained Linux command passed with two cached immutable
Paperclip images whose proxy hashes differ. It removed the original
account/controller trees, restored the production published cwd under a
new lease, resumed the same actual OpenCode session, kept a useful file
and prior model context, and made one new synthetic model request. No
external model call or network was available in either container.
- The object-store fixture uses real encrypted StorageService streams
with an independent owned local directory. It is not hosted S3
acceptance or a complete task HTTP/actor workflow. Historical tasks
still require normal effects reconciliation and recovery after
deployment.
- Review regressions reproduced partial-cache publication,
resume-versus-deletion, fresh-writer and source-consumer resurrection,
and unbounded Stop/deadline waits before each repair. They now pass
through real filesystem, encrypted storage, PostgreSQL, and SDK
boundaries. File validation is split from database ownership (384/831
source lines).
- Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the
maintained actual OpenCode cold qualification passed with `dirty:false`.
It preserved the provider session and recorded launch paths, restored
useful workspace content and prior model context, and made exactly one
resumed synthetic model request. The 314,560,000-byte launch capture
took 23.118 seconds; the later harness was 833,536 bytes. Subsequent
repairs change backup-reference retirement and governed continuity-cache
classification; capture, restore, launch, and published-workspace inputs
are unchanged, so that runtime receipt remains scoped to those paths.
The continuity-break consumer is verified separately.
- Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68
affected real PostgreSQL/AES storage tests and 15/15 focused executor
authority/admission cases pass. All seven server typecheck layers pass
(source layers and unchanged test shards, with the final 590-file test
layer repeated after fixture edits). All review threads have
evidence-backed resolutions. Its subsequent CI caught the separate
lifecycle-classification gap described below. The refreshed Paperclip
review reported paused work, not a completed new review.
- Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused
root-entry/executor cases and all three source declaration layers pass.
[Greptile reviewed this exact head at
5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874).
Final-head [CI
verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339)
passes: 60 successful checks, five conditional skips, and the explicitly
neutral paused Paperclip review. No check is pending or failing.
- CI also caught the launch cache missing from the session-root
lifecycle contract. Both the writer scan and a real governed-replacement
consumer failed locally before repair: old launch bytes remained in the
active root. The cache now archives with the prior session state; the
consumer verifies exact bytes are preserved in the archive and absent
from the active root.
- The latest review found that backup retirement could delete an
independent run pin while a selected session retained the checkpoint. A
real PostgreSQL/AES regression failed before the repair. Backup-only
ownership now uses a NULL-safe exact backup predicate; the run pin
belongs to fenced checkpoint expiration. The regression verifies
checkpoint decryption and launch-file restoration after pending-backup
cleanup, followed by deletion only after removing the final resume
handle.
- An executor first-session case checks new-session durable capture
through the real capture helper and receipt propagation into the actual
harness manifest at the backup boundary. Real storage cases verify the
run-profile receipt and canonical backup/run pins.
- Sequential closure reuse and simultaneous first admission are
different scopes. A deterministic two-uploader test measures both
independent uploads, one verified ready winner, loser-only deletion, and
usable restoration for both runs. Each capture is bounded to 512 MiB and
five minutes; up to three pending receipts per run are retained. There
is no claim of global single-flight capture or a company-wide egress
bound.
- Earlier optional warm-ownership verification encountered missing local
binaries. After exact-source setup, all nine ownership assertions
passed; two other suites encountered local PostgreSQL setup/teardown
contention, so those interrupted attempts are not passes. Required
current-head CI supplies the broad final coverage. No deployment or
hosted historical recovery is claimed.

## Risks

- This adds two tables and a forward migration. The file object is
bounded to 512 MiB. Capture adds one initial transfer; later harness
archives exclude those large files. Company-scoped ready objects are
shared and retained until the last canonical owner retires. Concurrent
initial admissions can duplicate bounded transfers; each keeps
independent ownership and cancellation rather than waiting on another
controller. A continuity break retains the controller cache with its
prior state until normal session-root cleanup, adding up to one bounded
launch closure per retired state.
- Cold continuity requires the same qualified Linux platform, supported
runtime dependencies, owned account UID/home, and published absolute
cwd. A mismatch fails closed. No alias or Rust guard change is included.
- Existing image-rooted profiles have no retained file object. This
change cannot backfill missing historical binaries. A fresh governed
session must reconcile prior effects first.
- Live reattachment does not read artifact objects. Storage or
provenance failures before a new turn hold admission. They do not grant
new execution, model, tool, or replay authority.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and local browser tools. The exact served model ID and
context window are not exposed in this session. No claim is made for
unavailable model metadata.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 17:30:00 -06:00