Files
PaperClipAI/packages/plugins/sandbox-providers/daytona
DottaandPaperclip 3a2249dd57 fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution must retain its admitted model and execution
authority during recovery.
> - A sandbox provider can reject Runner endpoint acquisition before the
Runner gets a credential.
> - The old error path loses the structured HTTP status and leaves that
startup failure ambiguous.
> - A zero counter in an old checkpoint cannot prove that no credential
was issued.
> - This pull request adds a versioned producer proof and carries closed
provider status through the existing RPC.
> - A proved transient startup failure uses the existing retry budget. A
terminal proved-unused run can settle its accounting.

## Linked Issues or Issue Description

Refs #18018 and #18934. Those changes restore existing checkpoints and
ready retained environments. This change covers endpoint acquisition
before Runner dispatch. A search of related public PRs found no
duplicate implementation.

**What happened?**

A provider HTTP 502 during Runner endpoint acquisition becomes an
untyped worker error. Native recovery treats it as ambiguous and stops
without retrying. The run's native usage receipt also stays pending
after startup ends without model work.

**Expected behavior**

Retry a structured transient endpoint failure only when the complete
current controller checkpoint proves that this exact run never issued a
Runner credential. Preserve authentication, configuration, Stop,
ownership and provider-progress gates. Settle zero usage only under the
existing terminal accounting owner.

**Steps to reproduce**

1. Use the controlled endpoint in the Daytona ingress integration test
to return HTTP 502.
2. Register the actual native Runner driver and persist its bootstrap
journal through the encrypted checkpoint writer.
3. Observe the old terminal ambiguous disposition instead of bounded
startup recovery.

**Paperclip version or commit**

Reproduced on the pre-change master source. The original regression used
the production native driver, a real HTTP fault, a real durable control
plane and local Postgres. The new contract additionally tests the pinned
Daytona SDK and real worker RPC.

## What Changed

- The controller owns a v2 persisted state with retained origin and
credential history. Current readers still accept v1. Older writers
reject v2. Restore and checkpoint decoding do not add missing history.
- The proof validates the complete bootstrap journal, all six identity
fields and every delivery, credential and progress control. The server
rechecks its fingerprint under the current controller claim before retry
or accounting changes.
- Daytona and the SDK carry a closed acquisition phase and HTTP status
through the existing ingress RPC. Both public error wrappers omit
response bodies and causes, including Pino cause serialization.
- Only structured HTTP 429 or 5xx ingress failures can use the existing
three-attempt startup budget. HTTP 401/403, configuration and unknown
errors do not gain retry authority.
- A terminal proved-unused run wakes existing accounting and records
zero provider work. It does not create a semantic result, prove physical
termination or release an unknown process/workspace hold.
- The normal automatic second attempt rechecks its exact claim, frozen
execution and matching complete local/encrypted journal before admitting
absent remote Runner state. It keeps the original pending bootstrap
command IDs. A proved unused Codex home can rematerialize sealed skills.
Full raw local state is checked before streaming-event or policy
projection.
- Normal saved-state readers and the existing E2E journal evidence
matcher accept the closed v1/v2 set. The separate legacy no-launch
witness is unchanged. Technical docs describe the contracts.

## Verification

Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta
adds only fixture artifact preparation). Production files are unchanged
from the independently reviewed `93c8a33520` source. Full repository
typecheck and build passed at `96417acb58`; the affected server
typecheck also passes after the final two-line fixture delta. Focused
protocol, raw-proof, real retry, SDK and RPC acceptance pass on this
source. Earlier broad suite counts are listed with their limits below:

- Durable Runner protocol: 187 tests.
- Native executor, driver, startup/accounting and redaction: 1,211
tests, including real Postgres and encrypted checkpoint persistence.
- Normal saved-state restore/projection compatibility: 21 focused cases.
- SDK: 209 tests.
- Standalone Daytona: 544 tests; 8 opt-in live tests skipped.
- Explicit standalone `pnpm -C
packages/plugins/sandbox-providers/daytona exec vitest run --config
vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and
worker RPC ingress acceptance. These include lookup 502/401/403/400,
private-preview 502 after successful lookup/refresh, missing credentials
and private-response exclusion.
- E2E journal evidence: 4 cases across both supported formats, with
wrong ownership, missing stimulus and unknown-schema controls.
- The exact historical controller source rejects a new v2 checkpoint at
decode and restore. A v1 control is accepted. This verifies the
downgrade fence with the actual older parser.
- Startup controls cover issued and pruned credentials, old/truncated
journals, provider progress, exhausted retry accounting, and
checkpoint/Stop/controller-claim races. Race cases retain the current
execution and pending usage.
- New exact admission controls: 19 cases through encrypted persistence
and cold restore. They cover current/stale/expired claims, Stop, pause,
model/trust/account changes, prior progress, issued/pruned credentials,
adopted/old/truncated state, local/canonical fingerprint mismatch,
hidden delivery, unsupported work and company scope. Two additional
red/green cases prove that raw local stream progress and missing event
policy are rejected before lossy checkpoint projection. No case mutates
run, claim, result or accounting state.
- Actual native executor, scheduler and Rust Runner retry: 2 cases.
Codex uses its fake app server. OpenCode uses the real proxy and fake
OpenCode HTTP server with the frozen DeepSeek model and allow permission
configuration. Each path launches no Runner on attempt 1, starts one
Runner and provider on attempt 2, reuses both original command IDs,
commits one accepted result, validates the persisted local workspace,
records nonzero synthetic native usage and settles accounting once. No
zero-work accounting flag survives.

The controlled SDK and protocol tests use local source, isolated
PostgreSQL, a private HTTPS preview and fake provider servers on macOS
arm64 with Node 24.22.2. They create no hosted sandbox and make no model
call. The root workspace intentionally omits standalone provider
dependencies; the explicit SDK target fails if its pinned dependency is
absent and has no conditional skip. They do not qualify hosted rollout
or repair an old unmarked checkpoint. The first full local typecheck
found missing checkout SDK links from an install with scripts disabled;
the repository link helper repaired that setup.

Current-head Linux CI typecheck and build pass. Independent final review
checked 12 source scopes and six raw-proof controls with no remaining
finding. Fresh Greptile review is 5/5 with no unresolved thread.

The full Runner CI suite found two older control-peer test doubles
missing the new proof method. Both now return no proof, with all 27
permission, ordering and cancellation assertions retained and passing.
The new Cargo-dependent native retry suite uses the existing Rust-cached
CI lane. All 39 lane coverage controls pass and prove complete,
duplicate-free suite coverage. The first full test run stopped at an
admitted chat Retry polling failure. CI also found a stale
retained-allocation fixture, now repaired on master by #18984, and a
canceled runner job. A real SDK suite initially relied on a dependency
absent from the root workspace; the explicit standalone target now owns
that acceptance.

The separate Product E2E support run had 12 failures in four untouched
files. All 12 reproduce on the original clean base with independent
dependencies, compiled base source and verified workspace resolution.
These concern workflow expectations, cleanup timeout arguments, a
profile snapshot and operating receipts. They are not passing
acceptance.

The later full local repository test command stopped at a 15-second
timeout in the unchanged GitHub pre-allocation retry `controller` case.
That exact case passed in isolation with unchanged assertions. The broad
Runner run initially omitted `test:typescript:prep` and lacked debug
fake-provider binaries. Required preparation then passed 370 + 44
package contract cases (11 skipped); the prepared full package run
reported 4,932 passed, 9 failed and 13 skipped. All failed families then
passed with unchanged assertions in isolation: 8 ACPX controller tests
and 2 credential controls. The paired original-base ACPX control also
passed. The full-run failures remain recorded; they are not passing
full-suite evidence.

The first Rust-cached native CI run found a missing generated OpenCode
proxy in a fresh checkout. The fixture now runs the existing verified
provider-entrypoint builder. Removing the generated proxy reproduces the
exact ENOENT before the fix; both real retry cases pass after the fix
from the same absent-artifact state.

All current-head CI gates pass, including the aggregate verification
check. Linux native CI passes all 10 suites: 177 tests pass and 14
opt-in cases skip. Both real provider retry cases pass. The full Runner
package passes 297 suites and 4,950 tests, with 1 suite and 10 tests
skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI
attempt had Runner shutdowns in the Runner package, Canary and three
general server jobs. One unchanged CLI shard also hit child-server
startup and dense ZIP deadlines. Both CLI files passed unchanged in
isolation: 18 tests. One bounded rerun of unsuccessful jobs and their
dependents passed. The original failures remain recorded.

## Risks

- Older controllers reject newly written v2 state on downgrade. Current
controllers retain v1 restore support, but old or unmarked state cannot
gain this new proof.
- An older plugin that omits structured ingress evidence remains held.
The server and provider must use the new producer contract for this
recovery path. The Docker `cloud` target builds the SDK, server/Runner
and standalone Daytona producer from one source tree. The default
`production` target does not build the standalone producer. Hosted
rollout must confirm the new image and active image-owned plugin worker;
a registry version alone does not prove its source. An npm-installed
producer needs an explicit matching artifact upgrade.
- This prevents future proved startup failures. It does not release
historical accounting based on an old checkpoint, change retry budgets,
replace a model or weaken sandbox isolation.
- Zero model usage is separate from process termination, saved files and
resource release. Those owners keep their existing gates.

## Model Used

OpenAI Codex with tool use and code execution. This runtime does not
expose the exact backend model ID or context window to the agent.
Read-only independent agents reviewed checkpoint compatibility and
SDK/RPC error boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full
local run limits are recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 12:16:51 -06:00
..

@paperclipai/plugin-daytona

Published Daytona sandbox provider plugin for Paperclip.

This package lives in the Paperclip monorepo, but it is intentionally excluded from the root pnpm workspace and shaped to publish and install like a standalone npm package. That lets operators install it from the Plugins page by package name without introducing root lockfile churn for Daytona's SDK dependencies.

Install

From a Paperclip instance, install:

@paperclipai/plugin-daytona

The host plugin installer runs npm install into the managed plugin directory, so transitive dependencies such as @daytonaio/sdk are pulled in during installation.

Configuration

Configure Daytona from Instance Settings -> Environments, not from the plugin's plugin page.

  • Put the Daytona API key on the sandbox environment itself.
  • When you save an environment, Paperclip stores pasted API keys as company secrets.
  • DAYTONA_API_KEY remains an optional host-level fallback when an environment omits the key.
  • Optional apiUrl and target settings map directly to the Daytona SDK/client configuration. If apiUrl is omitted, the Daytona SDK uses its default endpoint.

Notes:

  • The current published Daytona SDK package is @daytonaio/sdk.
  • The driver supports both snapshot-based and image-based sandbox creation. If both are set, validation rejects the config as ambiguous.
  • Optional ttlMinutes sets a positive whole-minute wall-clock lifetime in the creation request. It applies to image and snapshot sandboxes, including probes and interactive setup. The driver requires the provider to confirm the expiry before it returns the sandbox. It preserves an earlier creation expiry when a caller supplies a later deadline. Daytona destroys the sandbox and its files at expiry, even while active, stopped, paused, or archived. Omit this setting to keep existing lifetime behavior. This is separate from inactivity cleanup.
  • Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
  • For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records pending_cleanup before retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path.
  • Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved target does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens.
  • Idle sleep tracks pending hooks, underlying provider requests, open terminals, and sandbox allocations. A successful stop or delete clears only that sandbox's company, environment, and account scope. Every sandbox deletion waits for completed remote destruction within its existing timeout; acceptance alone cannot clear idle evidence. A timeout does not settle the underlying request. Cleanup that overlaps other work on the same sandbox needs a later quiet stop/delete receipt or read-only observation of the exact stopped or missing sandbox. Proof is captured before the provider observation. Other work that starts or finishes before its reply invalidates that proof, even after the other request has settled. An unobserved creation name is never absence proof. Active resources and uncertain cleanup continue to prevent idle sleep; verified cleanup no longer blocks sleep for the worker's entire lifetime.
  • A command that has already exited returns its final separated log snapshot without opening a follow socket. The initial status check is bounded to 500ms and the caller's command deadline. Each follow connection also makes up to four early status reads, with 25/50/100/200ms delays and the same observation/deadline bounds. A confirmed exit still requires the final separated snapshot, but does not wait for the follow socket to close. Unconfirmed or unavailable status keeps streaming; the early reads stop after their bounded attempt window. The command is dispatched only once.
  • A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts.
  • The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit.
  • A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.
  • Directory export normally builds a sandbox archive and downloads it. If archive creation exhausts disk space, quota, or the file-size limit, export streams the archive through a raw terminal instead. Each bounded chunk waits for a host write acknowledgement. Import still requires successful tar completion and the existing archive/path validation. The total archive cannot exceed the provider-reported sandbox disk allocation, and each host write must leave a 1 GiB free-space reserve. This fallback writes no sandbox archive or command log, preserves source files, and retains the configured transfer timeout. Terminal cleanup has a separate five-second limit and cannot hold the host archive open. It requires Bash and GNU tar, dd, and base64 in the sandbox. It does not repair a provider session that already failed to save its state or increase the sandbox's disk allocation.
  • Single-file export normally downloads a guarded sandbox copy. If that copy exhausts disk space, quota, or the file-size limit, the provider removes partial copies and streams the source through the same bounded sandbox-user channel. The stream rechecks canonical confinement and regular-file identity, refuses a symlink at open, and publishes host files only after a successful transfer receipt. It never hands a mutable source to the privileged download API. Permission, confinement, and transport failures still fail; a failed stream leaves prior host files intact. This removes the second archive-sized copy needed to download an already captured workspace archive. Creating the workspace archive itself still needs sandbox space.

Local development

cd packages/plugins/sandbox-providers/daytona
pnpm install --ignore-workspace --no-lockfile
pnpm build
pnpm test
pnpm exec vitest run --config vitest.sdk-ingress.config.ts
pnpm typecheck

These commands assume the repo root has already been installed once so the local @paperclipai/plugin-sdk workspace package is available to the compiler during development.

The dedicated vitest.sdk-ingress.config.ts requires this package's installed, pinned Daytona SDK. It sends local HTTP faults through the real SDK and worker RPC without a provider account or sandbox. The root workspace does not install standalone provider dependencies; its normal Daytona tests use mocked SDK boundaries. This target fails if the SDK is missing and runs all ingress acceptance cases without a conditional skip.

The capacity regression uses Python 3 for a real local PTY, Bash, and GNU core utilities (gtar, gdd, and gbase64 on macOS). It skips when those prerequisites are unavailable or their version checks do not match. With DAYTONA_API_KEY set, run pnpm test src/file-sync-capacity.live.test.ts to exercise an actual stopped and resumed Daytona sandbox, tar and file-copy writes to /dev/full, validated exports, and second exports after reconnect. Each case creates and deletes one disposable 1 GiB Ubuntu sandbox; it does not fill a disk or modify existing sandboxes.

Package layout

  • src/manifest.ts declares the sandbox-provider driver metadata
  • src/plugin.ts implements the environment lifecycle hooks
  • paperclipPlugin.manifest and paperclipPlugin.worker point the host at the built plugin entrypoints in dist/

Interrupted run-lease creation

The host commits a company-scoped creation operation before acquiring a new run lease. The operation UUID becomes paperclip-create-<uuid> and the paperclip-create-attempt label. The existing inspection hook can resolve that name and validate company, environment and run labels without starting a sandbox. Recovery persists the observed provider ID before the existing cleanup worker deletes it. A missing creation name is an uncertain outcome, not proof of removal. The host retains that operation for manual reconciliation and never repeats the creation request blindly. Credential identity must still match admission.

This capability is declared as creationOperationVersion: 1 in manifest version 0.1.10. It covers normal run/ad-hoc lease acquisition and plugin task replacement. Legacy provider environmentProbe and custom-image interactive setup have separate lifecycles and are outside this lease-orphan fallback replacement. The normal Test path which acquires a runtime lease is covered. Unit fixtures exercise lookup and deletion; live provider qualification requires an explicit disposable test account.