## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents in remote sandboxes must have a saved, verified copy of their launch files before their first turn (#19622, #21287) > - The first runs of a company share one copy of these files per server. Other runs wait for it > - In a restart chaos wave, the shared copy out of one sandbox stalled for 330 s. Seven other runs waited silently for it. Two runs then failed before their first turn with the cause `unknown` > - The copy speed out of a sandbox differs from one sandbox to the next. When the first copy failed, the other sandboxes copied the same 314 MB in 15 to 24 s > - This pull request lets every waiting run use the first copy that any run saves, starts a second copy when the first one is slow, and names the cause of a stalled copy > - The benefit is a shorter cold start for the first runs of a company, no failed run when another run already saved the files, and a named cause instead of `unknown` ## Linked Issues or Issue Description Refs #19622 Refs #21287 **What happened?** A restart chaos wave started 15 OpenCode runs on new Daytona sandboxes (1 vCPU) in a new company. The first run (A) started the shared copy of the 314 MB launch files out of its sandbox. The copy did not finish. The plugin call timed out after 330 s, and run A failed before its first turn. The other runs waited for that copy without any run event, so the task looked dead for 284 to 343 s. When run A failed, every waiting run started its own copy at the same time. Five copies finished in 15 to 24 s. One copy (run B) stalled again and failed after 330 s, although a saved copy existed after 15 s. One copy (run C) took 5 min and started its turn 10 min after the run started, although it could have used the saved copy 5 min earlier. Both failures had the technical cause `unknown`. Each failure also logged "native result could not be marked workspace_failed; the reconciliation loop will retry persisted results", but the run had no result, so there was nothing to retry. **Expected behavior** A run uses the first saved copy as soon as one exists. A slow copy does not hold all first runs. A stalled copy out of the sandbox has a named cause. A run with no result does not log a result-finalization failure. **Steps to reproduce** 1. Use a company with remote OpenCode agents on Daytona sandboxes with durable storage and no saved launch-file copy. 2. Start 8 or more runs together on new sandboxes. 3. Make the copy out of the first sandbox stall (for example, a slow sandbox link). 4. See that all other runs wait for the full plugin timeout, and see the failures with the cause `unknown`. **Agent adapter(s) involved** OpenCode (native runtime, remote sandbox). ## What Changed - `shareNativeProviderArtifactCapture` (`native-provider-artifact-capture.ts`): - The first capture that saves a ready copy ends the wait for every run. A run whose own capture is still copying uses the saved copy and aborts its own copy. - A run whose own capture fails uses a saved copy when one exists, and fails only when none exists. - When the running capture is slower than 90 s, one waiting run starts one more capture from its own sandbox. At most two captures run at once. A capture keeps its slot until its copy out of the sandbox ends. - The start decision is owned by one `SharedArtifactCapture` coordinator (`tryStart()` returns a release lease). - When the shared capture fails, one waiting run starts the next capture. The other runs keep waiting for it. Before, every waiting run started its own copy at the same time. - The capture callback now gets its own abort signal, so a superseded copy stops before upload. - A failed copy out of the sandbox is now `NativeProviderArtifactExportError` with the code `native_provider_artifacts_export_timeout` (plugin call timed out) or `native_provider_artifacts_export_failed`. The run's technical cause shows this code instead of `unknown`. A timeout shows "Setting up the computer took too long ...". - `runtime-execution.ts`: do not try to mark the workspace as failed on a native result when the run has no result. Before, this always threw `native_finalization_missing` and logged a false "will retry" warning. - `doc/opencode-cold-recovery.md`: describe the new sharing, the second capture and the new cause codes. ## Verification - Red/green: with the old `native-provider-artifact-capture.ts`, these 5 new tests fail. With this change, they pass: - `native-provider-artifact-gate.test.ts`: "starts a run's turn from another run's saved artifact while its own capture is still copying", "uses a saved artifact when the run's own capture fails after another run saved one", "starts one more capture when the shared capture runs past the hedge delay, not one per waiting run", "does not start a capture per waiting run when the shared capture fails", "names a stalled copy out of the sandbox instead of an unknown cause". - Red/green: `heartbeat-native-predispatch-retry.test.ts` "continues on a fresh ... sandbox when the provider artifacts fail to save before any turn" now also asserts that no "could not be marked workspace_failed" warning is logged. It fails before the `runtime-execution.ts` change and passes after it. - `cd server && npx vitest run src/services/native-runtime/native-provider-artifact-gate.test.ts src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts src/services/native-runtime/native-provider-artifact-capture.test.ts src/services/heartbeat/runtime-execution.test.ts src/services/task-interruption-presentation.test.ts src/__tests__/heartbeat-native-predispatch-retry.test.ts`: all pass. - `npx tsc --noEmit -p server`: pass. ## Risks - A slow first capture can now cause one more 314 MB copy after 90 s. At most two copies run at once for one company and manifest. A run whose saved copy cannot be reused (for example, an ownership fence mismatch) starts its own copy at once. Before, a failed burst started one copy per waiting run at once, which was more. - A superseded copy never uploads, but the provider call that copies files out of the sandbox continues until it ends or times out. Its local temporary directory is removed when it ends. - Sharing is still per server. Two servers can still copy the same files. - No change to the rule from #19622: a verified saved copy still exists before the first turn. - No runner or image change. No migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, test runs). Default context window; extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
15 KiB
OpenCode launch artifacts and cold recovery
OpenCode records the absolute paths and hashes of its Node executable, proxy, and native executable in its durable provider state. An image upgrade can change these files. A saved session must keep its admitted files. The Runner still rejects a different launch profile.
For a new managed remote OpenCode session with durable object storage, Core puts
these three qualified files in the runtime account's private, content-addressed
launch directory. Core reads the home and UID from the runtime OS account
database. It does not infer ownership from HOME, a workspace path, or a provider
response. The launch receipt records this UID, the absolute directory, the pack
manifest, and the encrypted artifact identity.
Core saves the files before it releases the first provider turn. Artifact capture uses the current native coordinator claim and the recorded Core admission. A brief admission lock waits for at most one second and checks the current owner after it acquires the lock. A stale, revoked, or unknown owner cannot publish. The artifact operation does not grant execution or tool authority.
The archive is limited to 512 MiB of launch files. Core streams files and checks hashes and file identity before and after capture. Pack-internal package links can resolve to regular source files. Links outside the selected pack are refused. The published launch directory contains regular files and private, owned parents. Node and OpenCode must be qualified native ELF executables. A script wrapper or a changed account fails closed. The selected runtime must still supply the qualified native executables' platform and system dependencies.
The artifact object uses authenticated encryption and a company-bound key wrapper. A company can reuse a ready object with the same file descriptor. Canonical run and harness-backup references keep the object alive. Large launch files are excluded from each later harness archive. A pending upload records its exact encrypted object reference before upload, so an interrupted reply can be retired without guessing an object key.
A ready company closure is reused. Before it copies anything, admission looks for a ready, verified company object whose descriptor has the same file paths and SHA-256 digests as the selected manifest. The database matches the target, paths and digests, so the lookup reads only matching objects, not every version the company keeps. When one exists, admission pins it for the run and records the run receipt in one transaction. No file leaves the sandbox, and nothing is encrypted, uploaded or downloaded. A durable artifact therefore still exists before the first turn. Concurrent first admissions on one server with the same company and manifest share one in-flight capture. The other admissions wait for it, then reuse its ready object under their own ownership fence. If the shared capture fails, one waiting admission starts the next capture and the others keep waiting for it. When the running capture has not finished after 90 seconds, one more waiting admission starts a capture from its own sandbox, because the copy speed out of a sandbox differs from one sandbox to the next. At most two captures run at once for one company and manifest; a capture counts until its copy out of the sandbox ends. A run that finds a saved object it cannot reuse starts its own capture as soon as a slot is free, without waiting for the 90 seconds. The first capture that saves a ready object ends the wait for every admission: an admission whose own capture is still copying reuses that object and abandons its capture, and an admission whose own capture fails reuses it too. An abandoned capture never uploads or publishes. Its copy out of the sandbox is not interrupted: that provider call runs until it ends or reaches its own timeout, and its local files are then removed. Admissions on different servers may still upload the same closure independently.
Each capture has the 512 MiB limit; a run can retain at most three pending
uploads. There is no single deadline for the whole capture. Each network step
has its own bound instead: each export command has 120 seconds, and the sandbox
transfer keeps its provider's command timeouts. When the copy out of the
sandbox fails, the failure is native_provider_artifacts_export_timeout if the
provider call timed out, and native_provider_artifacts_export_failed
otherwise. The upload and the download that
verifies it each have the larger of 120 seconds and the time for the object at
256 KiB per second. When an upload or download step exceeds its bound, the
failure is native_provider_artifacts_transfer_timeout and names the step. The
step is then retried from the local copy, up to three attempts in all, without
copying the files out of the sandbox again. Stop interrupts the capture and the
wait for a shared capture. Publication selects one verified ready object under
the closure lock. A loser pins that winner before deleting only its own object.
This bounds each admitted run, not total simultaneous company egress. A burst can
therefore transfer duplicate bytes; distributed upload election is not included.
On cold recovery, Core restores the canonical Runner checkpoint, workspace snapshot, and native harness data through their normal paths. It also verifies and decrypts the recorded launch files. The runtime must have the same owned account UID and absolute home. Core stages the exact files at the recorded content-addressed paths. It does not replace files in the current image.
Published workspaces retain their recorded directory basename across lease changes. A replacement allocation with the same provider base therefore keeps the native session's absolute cwd. The workspace service can restore files to a different base, but that alone does not authorize changing a native session's cwd. This change adds no directory aliases and makes no change to the Rust profile guard.
An explicitly governed continuity break archives the controller launch cache with the prior session state, receipt, and backups. The cache is not a new execution authority and must not remain active for the fresh provider session.
Live Runner reattachment does not download or replace the launch files. Historical sessions that recorded image-rooted paths keep those paths. A governed fresh-session transition can use a new pack after existing effects and receipts are reconciled. It must not replay a previous turn or treat a missing result as proof that no effect occurred.
Historical provider-pack code sources
An older session can retain its exact pack manifest while the current server
image contains a different pack. When neither the staged nor the preinstalled
pack verifies, the normal uploader resolves the selected historical code from
ghcr.io/paperclipai/paperclip. It uses the manifest's recorded source revision
and Linux architecture. The full source-SHA image tag is a lookup hint; a tag
name alone does not prove the code identity.
The resolver verifies the OCI index, selected platform manifest, image config, source revision, and dedicated provider-pack transport layer. It then verifies the selected canonical pack manifest, compressed layer hashes, tar paths and links, and each file layer's plaintext bytes. The final installed pack still passes the normal verification against the original selected manifest. The OpenCode version alone cannot admit a different proxy or contract. This source selection also serves other providers that use the full provider-pack uploader; it does not change their compatibility or qualification rules.
The code source is downloaded and verified before the uploader can clear or transfer anything to the target. A historical source requires the existing qualified archive transport. An unavailable transport leaves the target intact. Missing images, changed bytes, invalid source identity, and unsupported packs fail closed. The resolver never resets provider state, substitutes the current pack, changes the stored launch paths, or starts an old application image.
The temporary cache contains public code and source provenance only. It uses private, process-account-owned directories and exact manifest keys. An empty host can reconstruct the cache from the registry. A cached source is checked again before upload. Imports stream data with a four-GiB code limit and a three-minute deadline, allow two active imports or cache verifications, and bound queued distinct sources to 32. Concurrent callers share verification of the same cached pack. Import waits and downloads respond to cancellation; one cancelled caller does not cancel a shared import still needed by another caller. Inactive instances perform no registry reads or cache scans. Each import requires disk headroom before writing; completed cached code remains until temporary storage is removed.
The normal per-turn startup owns the source wait, including OpenCode starts without a warm execution context. Stop, execution-lease loss, and startup completion cancel that wait. An existing execution context can cancel it sooner. The source deadline applies before Runner admission. It does not increase the separate cold-admission or provider-turn time limits.
The resolver uses only an anonymous token scoped to reading this public code repository. It does not use database, sandbox, model, or object-store credentials. Its cache and registry dependency are separate from the encrypted S3 backup of provider conversation state and workspace files.
The Docker publication workflow publishes full source-SHA tags. There is no image-deletion workflow in this repository, but this is not a permanent registry retention guarantee. GitHub package administrators can delete package versions. Historical continuation therefore depends on retaining the corresponding code image in GHCR, or retaining a verified source cache. Deleting both keeps the session blocked with its original state intact; an S3 provider-state backup alone does not reconstruct legacy image-owned executable code.
Artifact cleanup uses the existing bounded checkpoint and backup expiration owners. It requires closed work, settled native ownership, and exact stopped remote execution receipts. Backup retirement releases only that backup reference. The separate run reference lasts until fenced checkpoint expiration, including when an interrupted backup upload is retired while a resume handle still exists. A remaining backup or run reference keeps the object. A failed external delete keeps its receipt and reference for a later normal expiration attempt. There is no new idle scan or timer.
Checkpoint expiration checks current resume handles under its existing locks. It then fences old generations and atomically replaces the expired header with a versioned, company/run-scoped retirement marker. An earlier handle keeps the original checkpoint and files. After retirement, readers and header writers refuse the old checkpoint. Source consumers also check retirement in their existing ownership query, so a cached old snapshot cannot bypass the barrier. The marker stays in the existing header index until external artifact deletion succeeds, including after a server restart. No transaction waits for a network delete. Families with more than 100 related runs keep every pin and checkpoint. Missing object storage keeps the marker and pins for a later normal cleanup attempt.
The capture wait responds to Stop, startup completion, and the execution deadline. Native bulk transfer does not offer a cancellation signal. Core keeps that actual transfer registered as pending startup work after it stops waiting. The late operation can write only its private local scratch files. It cannot publish after the context closes. It removes its scratch files and releases its worker responsibility only after actual settlement. Stop does not imply that this transport has finished.
Local qualification
Run the focused server suites with the repository's supported pnpm version:
pnpm --filter @paperclipai/server exec vitest run --config vitest.config.ts \
src/services/native-runtime/native-harness-backup-storage.integration.test.ts \
src/services/native-runtime/native-harness-backup-archive.test.ts \
src/services/native-runtime/remote-opencode-artifacts.test.ts \
src/services/native-runtime/native-provider-pack-admission.test.ts \
src/services/native-runtime/native-provider-artifact-capture.test.ts \
src/services/native-runtime/native-runner-checkpoint-retirement.test.ts \
src/services/native-runtime/native-session-executor.test.ts \
src/__tests__/native-harness-backup-expiration.test.ts \
src/__tests__/native-harness-backup-expiration-config.test.ts
The opt-in Linux qualification requires two locally cached Paperclip images with
full immutable image IDs and different OpenCode proxy hashes. Both images must
have the same Linux architecture and the standard qualified provider pack. The
new image must include the packaged Runner binary at
/app/server/dist/vendor/paperclip-runner/bin/paperclip-runnerd.
Build the current checkout's Runner TypeScript SDK first. Use the local Docker
context that has the cached images:
pnpm --filter @paperclipai/paperclip-runner build:typescript
pnpm --filter @paperclipai/server exec tsx scripts/qualify-opencode-cold-recovery.mts \
--context desktop-linux \
--old-image "$OLD_IMAGE" \
--new-image "$NEW_IMAGE" \
--output /tmp/opencode-cold-recovery-result.json
The command accepts only full sha256: image IDs and a local Unix Docker
endpoint. It never pulls an image. It creates isolated PostgreSQL data and two
owned containers. The containers have no network, a read-only image filesystem,
resource limits, and a private fixture OS account. The synthetic model runs on
container loopback. No external model credential or API is used.
The command uses production workspace save/restore, encrypted checkpoint and harness storage, artifact publication, admission, SDK transport, Rust Runner, and actual OpenCode. It destroys the original allocation's account and controller files, changes the image and allocation lease, and restores the same provider session. It requires the useful workspace file and earlier model context to survive. Exactly one model request is allowed in each turn. The result records image IDs, pack digests, checkout revision and dirty state, lease identities, byte counts, session continuity, and useful output. The command removes only its own containers and PostgreSQL server. It retains its temporary evidence directory.
The independent object-store fixture uses the real StorageService and encrypted streams with an owned local directory outside both destroyed allocations. This qualifies the cold restore contract. It does not qualify a hosted S3 service, a task's HTTP/actor workflow, or completion of an existing hosted task.