Files
PaperClipAI/doc/opencode-cold-recovery.md
T
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00

226 lines
14 KiB
Markdown

# OpenCode launch artifacts and cold recovery
OpenCode records the absolute paths and hashes of its Node executable, proxy,
and native executable in its durable provider state. An image upgrade can change
these files. A saved session must keep its admitted files. The Runner still
rejects a different launch profile.
For a new managed remote OpenCode session with durable object storage, Core puts
these three qualified files in the runtime account's private, content-addressed
launch directory. Core reads the home and UID from the runtime OS account
database. It does not infer ownership from `HOME`, a workspace path, or a provider
response. The launch receipt records this UID, the absolute directory, the pack
manifest, and the encrypted artifact identity.
Core saves the files before it releases the first provider turn. Artifact capture
uses the current native coordinator claim and the recorded Core admission. A
brief admission lock waits for at most one second and checks the current owner
after it acquires the lock. A stale, revoked, or unknown owner cannot publish.
The artifact operation does not grant execution or tool authority.
The archive is limited to 512 MiB of launch files. Core streams files and checks
hashes and file identity before and after capture. Pack-internal package links
can resolve to regular source files. Links outside the selected pack are refused.
The published launch directory contains regular files and private, owned
parents. Node and OpenCode must be qualified native ELF executables. A script
wrapper or a changed account fails closed. The selected runtime must still
supply the qualified native executables' platform and system dependencies.
The artifact object uses authenticated encryption and a company-bound key
wrapper. A company can reuse a ready object with the same file descriptor.
Canonical run and harness-backup references keep the object alive. Large launch
files are excluded from each later harness archive. A pending upload records its
exact encrypted object reference before upload, so an interrupted reply can be
retired without guessing an object key.
A ready company closure is reused. Before it copies anything, admission looks
for a ready, verified company object whose descriptor has the same file paths
and SHA-256 digests as the selected manifest. The database matches the target,
paths and digests, so the lookup reads only matching objects, not every
version the company keeps. When one exists, admission pins it
for the run and records the run receipt in one transaction. No file leaves the
sandbox, and nothing is encrypted, uploaded or downloaded. A durable artifact
therefore still exists before the first turn. Concurrent first admissions on
one server with the same company and manifest share one in-flight capture. The
other admissions wait for it, then reuse its ready object under their own
ownership fence. If the shared capture fails, each waiting admission captures on
its own. Admissions on different servers may still upload the same closure
independently.
Each capture has the 512 MiB limit; a run can retain at most three pending
uploads. There is no single deadline for the whole capture. Each network step
has its own bound instead: each export command has 120 seconds, and the sandbox
transfer keeps its provider's command timeouts. The upload and the download that
verifies it each have the larger of 120 seconds and the time for the object at
256 KiB per second. When an upload or download step exceeds its bound, the
failure is `native_provider_artifacts_transfer_timeout` and names the step. The
step is then retried from the local copy, up to three attempts in all, without
copying the files out of the sandbox again. Stop interrupts the capture and the
wait for a shared capture. Publication selects one verified ready object under
the closure lock. A loser pins that winner before deleting only its own object.
This bounds each admitted run, not total simultaneous company egress. A burst can
therefore transfer duplicate bytes; distributed upload election is not included.
On cold recovery, Core restores the canonical Runner checkpoint, workspace
snapshot, and native harness data through their normal paths. It also verifies
and decrypts the recorded launch files. The runtime must have the same owned
account UID and absolute home. Core stages the exact files at the recorded
content-addressed paths. It does not replace files in the current image.
Published workspaces retain their recorded directory basename across lease
changes. A replacement allocation with the same provider base therefore keeps
the native session's absolute cwd. The workspace service can restore files to a
different base, but that alone does not authorize changing a native session's
cwd. This change adds no directory aliases and makes no change to the Rust
profile guard.
An explicitly governed continuity break archives the controller launch cache
with the prior session state, receipt, and backups. The cache is not a new
execution authority and must not remain active for the fresh provider session.
Live Runner reattachment does not download or replace the launch files.
Historical sessions that recorded image-rooted paths keep those paths. A governed fresh-session
transition can use a new pack after existing effects and receipts are reconciled.
It must not replay a previous turn or treat a missing result as proof that no
effect occurred.
## Historical provider-pack code sources
An older session can retain its exact pack manifest while the current server
image contains a different pack. When neither the staged nor the preinstalled
pack verifies, the normal uploader resolves the selected historical code from
`ghcr.io/paperclipai/paperclip`. It uses the manifest's recorded source revision
and Linux architecture. The full source-SHA image tag is a lookup hint; a tag
name alone does not prove the code identity.
The resolver verifies the OCI index, selected platform manifest, image config,
source revision, and dedicated provider-pack transport layer. It then verifies
the selected canonical pack manifest, compressed layer hashes, tar paths and
links, and each file layer's plaintext bytes. The final installed pack still
passes the normal verification against the original selected manifest. The
OpenCode version alone cannot admit a different proxy or contract. This source
selection also serves other providers that use the full provider-pack uploader;
it does not change their compatibility or qualification rules.
The code source is downloaded and verified before the uploader can clear or
transfer anything to the target. A historical source requires the existing
qualified archive transport. An unavailable transport leaves the target intact.
Missing images, changed bytes, invalid source identity, and unsupported packs
fail closed. The resolver never resets provider state, substitutes the current
pack, changes the stored launch paths, or starts an old application image.
The temporary cache contains public code and source provenance only. It uses
private, process-account-owned directories and exact manifest keys. An empty
host can reconstruct the cache from the registry. A cached source is checked
again before upload. Imports stream data with a four-GiB code limit and a
three-minute deadline, allow two active imports or cache verifications, and bound queued distinct
sources to 32. Concurrent callers share verification of the same cached pack.
Import waits and downloads respond to cancellation; one cancelled
caller does not cancel a shared import still needed by another caller. Inactive
instances perform no registry reads or cache scans. Each import requires disk
headroom before writing; completed cached code remains until temporary storage
is removed.
The normal per-turn startup owns the source wait, including OpenCode starts
without a warm execution context. Stop, execution-lease loss, and startup
completion cancel that wait. An existing execution context can cancel it sooner.
The source deadline applies before Runner admission. It does not increase the
separate cold-admission or provider-turn time limits.
The resolver uses only an anonymous token scoped to reading this public code
repository. It does not use database, sandbox, model, or object-store credentials.
Its cache and registry dependency are separate from the encrypted S3 backup of
provider conversation state and workspace files.
The Docker publication workflow publishes full source-SHA tags. There is no
image-deletion workflow in this repository, but this is not a permanent registry
retention guarantee. GitHub package administrators can
[delete package versions](https://docs.github.com/en/packages/learn-github-packages/deleting-and-restoring-a-package).
Historical continuation therefore depends on retaining the corresponding code
image in GHCR, or retaining a verified source cache. Deleting both keeps the
session blocked with its original state intact; an S3 provider-state backup alone
does not reconstruct legacy image-owned executable code.
Artifact cleanup uses the existing bounded checkpoint and backup expiration
owners. It requires closed work, settled native ownership, and exact stopped
remote execution receipts. Backup retirement releases only that backup reference.
The separate run reference lasts until fenced checkpoint expiration, including
when an interrupted backup upload is retired while a resume handle still exists.
A remaining backup or run reference keeps the object. A failed external delete keeps its receipt and reference for a later normal
expiration attempt. There is no new idle scan or timer.
Checkpoint expiration checks current resume handles under its existing locks.
It then fences old generations and atomically replaces the expired header with
a versioned, company/run-scoped retirement marker. An earlier handle keeps the
original checkpoint and files. After retirement, readers and header writers
refuse the old checkpoint. Source consumers also check retirement in their
existing ownership query, so a cached old snapshot cannot bypass the barrier.
The marker stays in the existing header index until
external artifact deletion succeeds, including after a server restart. No
transaction waits for a network delete. Families with more than 100 related
runs keep every pin and checkpoint. Missing object storage keeps the marker and
pins for a later normal cleanup attempt.
The capture wait responds to Stop, startup completion, and the execution
deadline. Native bulk transfer does not offer a cancellation signal. Core keeps
that actual transfer registered as pending startup work after it stops waiting.
The late operation can write only its private local scratch files. It cannot
publish after the context closes. It removes its scratch files and releases its
worker responsibility only after actual settlement. Stop does not imply that
this transport has finished.
## Local qualification
Run the focused server suites with the repository's supported pnpm version:
```sh
pnpm --filter @paperclipai/server exec vitest run --config vitest.config.ts \
src/services/native-runtime/native-harness-backup-storage.integration.test.ts \
src/services/native-runtime/native-harness-backup-archive.test.ts \
src/services/native-runtime/remote-opencode-artifacts.test.ts \
src/services/native-runtime/native-provider-pack-admission.test.ts \
src/services/native-runtime/native-provider-artifact-capture.test.ts \
src/services/native-runtime/native-runner-checkpoint-retirement.test.ts \
src/services/native-runtime/native-session-executor.test.ts \
src/__tests__/native-harness-backup-expiration.test.ts \
src/__tests__/native-harness-backup-expiration-config.test.ts
```
The opt-in Linux qualification requires two locally cached Paperclip images with
full immutable image IDs and different OpenCode proxy hashes. Both images must
have the same Linux architecture and the standard qualified provider pack. The
new image must include the packaged Runner binary at
`/app/server/dist/vendor/paperclip-runner/bin/paperclip-runnerd`.
Build the current checkout's Runner TypeScript SDK first. Use the local Docker
context that has the cached images:
```sh
pnpm --filter @paperclipai/paperclip-runner build:typescript
pnpm --filter @paperclipai/server exec tsx scripts/qualify-opencode-cold-recovery.mts \
--context desktop-linux \
--old-image "$OLD_IMAGE" \
--new-image "$NEW_IMAGE" \
--output /tmp/opencode-cold-recovery-result.json
```
The command accepts only full `sha256:` image IDs and a local Unix Docker
endpoint. It never pulls an image. It creates isolated PostgreSQL data and two
owned containers. The containers have no network, a read-only image filesystem,
resource limits, and a private fixture OS account. The synthetic model runs on
container loopback. No external model credential or API is used.
The command uses production workspace save/restore, encrypted checkpoint and
harness storage, artifact publication, admission, SDK transport, Rust Runner,
and actual OpenCode. It destroys the original allocation's account and controller
files, changes the image and allocation lease, and restores the same provider
session. It requires the useful workspace file and earlier model context to
survive. Exactly one model request is allowed in each turn. The result records
image IDs, pack digests, checkout revision and dirty state, lease identities,
byte counts, session continuity, and useful output. The command removes only its
own containers and PostgreSQL server. It retains its temporary evidence directory.
The independent object-store fixture uses the real StorageService and encrypted
streams with an owned local directory outside both destroyed allocations. This
qualifies the cold restore contract. It does not qualify a hosted S3 service,
a task's HTTP/actor workflow, or completion of an existing hosted task.