## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs use sandbox provider plugins. Daytona is one of
them.
> - After a failed run, Paperclip stops the sandbox and keeps its files
("stop-only cleanup"). A task can continue only when every remote lease
of the failed run has a provider stop receipt.
> - Daytona rejects Stop for a sandbox in its `error` state. A sandbox
that filled its disk went to that state. The plugin called Stop on it,
so every cleanup sweep failed with the same generic error.
> - The lease stayed in pending cleanup without a stop receipt. Each
Retry or comment waited forever with reason `remote_cleanup`. A manual
Daytona Recover was the only fix.
> - This pull request makes the Daytona stop-only hook recover a
recoverable sandbox once, with a bound, and then stop it. When it cannot
stop the sandbox, it returns a typed failure. The server escalates a
terminal failure at once instead of retrying it like a transient error.
> - The benefit is that a full-disk sandbox no longer blocks its task,
and a sandbox that cannot recover is visible to operators on the first
attempt.
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A task sandbox filled its 10 GB disk. Daytona then reported the sandbox
as `state: error`, `desiredState: started` and `recoverable: true`. Its
`errorReason` was a Docker `no space left on device` error. The Daytona
`stopLease` hook calls Stop on any sandbox that is not `stopped` or
`archived`. Daytona rejects Stop in the `error` state. The pending
cleanup sweep ran about every 30 minutes, and it failed on every pass.
The lease stayed `pending_cleanup` with `cleanup_status=failed` and no
stop receipt. Continuation admission
(`server/src/services/explicit-native-continuation.ts`) requires a stop
receipt on every remote lease of the failed run. So each Retry and
comment was deferred with "Waiting for the previous environment to stop.
Your message will start automatically." It never started.
A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The
sandboxes reached `started` in about one minute. The next sweep stopped
them, recorded the receipt, and the tasks continued.
**Expected behavior**
Stop-only cleanup recovers a recoverable errored sandbox and then stops
it. When the sandbox cannot be recovered, cleanup returns a clear, typed
failure that the server can tell apart from a transient error. Cleanup
never deletes the retained sandbox, and it never loops on Recover.
**Steps to reproduce**
1. Run a task in a Daytona sandbox until the sandbox fills its disk and
Daytona shows `state: error`, `recoverable: true`.
2. Let the run fail, so the server records a `sandboxStopAndRetain`
intent and calls `environmentStopLease`.
3. Before this change, Stop fails with `Sandbox failed to stop with
status: error` on every sweep, and the task cannot continue.
4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox
whose Stop fails from `error`.
**Paperclip version or commit**
Observed on a hosted deployment. Reproduced in unit tests on `master` at
`6452e964a36`.
**Deployment mode**
Managed Daytona sandbox execution with plugin-backed stop-only cleanup.
Related: #18172 (Daytona exports under disk pressure) and #19050
(session reset on every Daytona restart path). This PR does not change
the resume path in `ensureSandboxStarted`, which already recovers a
recoverable sandbox.
## What Changed
- **Daytona plugin
(`packages/plugins/sandbox-providers/daytona/src/plugin.ts`)**: after
`stopLease` confirms the retention policy, a sandbox in `error` goes
through `recoverSandboxForStop`.
- Recoverable: the hook calls `sandbox.recover()` once. The SDK waits
for `started`. The plugin bounds the wait to 20 s, plus the existing
liveness margin for a hung connection. Then the hook stops the sandbox
normally and returns the usual `stopped` receipt.
- Not recoverable: the hook makes no Recover or Stop call. It throws
`PluginEnvironmentStopError("resource_unrecoverable")`.
- Recover fails: the hook reads the provider state once. A sandbox back
in `error` gives `recovery_failed`, or `resource_unrecoverable` if
Daytona now marks it so. A sandbox that is still starting, or a failed
read, gives `recovery_incomplete`.
- The recovered sandbox fails again before Stop finishes (for example,
the disk fills again): a fresh read that shows `error` gives
`recovery_failed`. The hook never calls Recover a second time in the
same call.
- The hook never deletes or archives the sandbox. The non-error stop
path is unchanged.
- **Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`,
new)**: `PluginEnvironmentStopError`, `environmentStopErrorData` and
`readEnvironmentStopFailure`. This follows the existing
`PluginEnvironmentRunnerIngressError` pattern. The envelope holds only
one closed reason. The reader validates the reason and derives
`terminal` from it. `resource_unrecoverable` and `recovery_failed` are
terminal. `recovery_incomplete` is not.
- **Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`)**:
forwards that envelope as JSON-RPC error `data` for
`environmentStopLease` only.
- **Pending cleanup sweep
(`server/src/services/heartbeat/recovery.ts`)**: the catch reads the
validated envelope.
- On a terminal failure, it escalates on the first attempt. It writes
the one-time "needs operator attention" warning and uses the existing 30
minute retry interval. Before, this happened only at the fifth attempt.
- The failure log uses `errorKind: "provider_stop_terminal"` and
includes the closed `providerStopFailure` reason. The sweep still never
logs exception fields.
- The lease stays `pending_cleanup`. Automatic cleanup continues, and an
explicit Retry still bypasses the cooldown. Untyped and transient
failures keep their current behavior.
- **Docs**: the SDK README ("Stop failures"), the Daytona README,
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and
`doc/native-workspace-finalization-recovery.md` describe the new
behavior.
Shared contract note (module boundaries): the SDK envelope is the
smallest addition that lets the host tell a terminal provider stop
failure from a transient one. The worker host drops all other error data
across the RPC boundary, so the existing contracts cannot carry the
signal. The only consumer is the pending cleanup sweep. When a provider
does not throw the new error, or an older host ignores it, behavior is
unchanged. This PR does not touch
`server/src/services/workspace-recovery.ts` or
`sandbox-stop-and-retain.ts`.
## Verification
All commands ran in a clean worktree from `master`. Vitest ran from the
real path.
- Red on `master` (only the new tests added):
- Daytona, with `plugin.ts` from `master`: `npx vitest run --config
vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1.
All 7 new tests fail. The main case fails with `Sandbox failed to stop
with status: error`, which is the production error.
- SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run
tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the
typed case fails).
- Server, with `recovery.ts` from `master`: `npx vitest run
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal
provider stop failure"` exits 1. All 3 cases fail (`errorKind:
"destroy_failed"`, no escalation).
- Green on this branch:
- `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config
vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm
typecheck` exits 0. `pnpm build` exits 0.
- `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm
--filter @paperclipai/plugin-sdk typecheck` exits 0.
- `server`: 15 related suites exit 0 (1498 passed): pending cleanup
sweep, `heartbeat/recovery`, environment runtime, environment cleanup
work, native startup cleanup, native remote failure retention, observed
workspace preservation, retained native session cleanup, idle sleep
safety, explicit native continuation, task retry, workspace recovery,
native failed and cancelled workspace preservation, and Daytona close in
the plugin worker manager.
- `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm
check:module-boundaries` exits 0.
- New tests:
- Daytona: a recoverable errored sandbox is recovered once
(`recover(20)`) after the retention check and before Stop, then stopped.
An unrecoverable one returns `resource_unrecoverable` with no Recover,
Stop, start, delete or archive call. A failed recovery makes exactly one
Recover call for each outcome (`recovery_failed`,
`resource_unrecoverable`, `recovery_incomplete`). A Recover call that
never settles is cut off by the bound. A recovered sandbox that fails
again before Stop gives `recovery_failed` with no second Recover.
- SDK: the envelope holds only the reason. A forged or non-SDK error
gives no data. Unknown reasons, accessor fields and a sender-supplied
`terminal` flag are not trusted. The worker host forwards the typed
failure and drops a plain error's detail.
- Server: terminal reasons escalate on attempt 1 with the long retry and
one operator warning. `recovery_incomplete` keeps the short retry. The
lease stays `pending_cleanup`, and the provider message never reaches
the logs.
- Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`
and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live
Daytona probe ran. The mocks model the Daytona SDK `recover()` contract:
it waits for `started` and rejects when the sandbox returns to `error`.
## Risks
- Time budget: the host gives `environmentStopLease` at most 60 s. A
recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus
short reads. A slower recovery returns `recovery_incomplete`. The SDK
call can continue at Daytona after that, so the next sweep finds the
sandbox started and stops it normally. If the host deadline fires while
the worker still stops the sandbox, the next sweep sees `stopped` and
records the receipt.
- Classification: after a failed Recover, the plugin cannot tell a
rejected request from a sandbox that started and failed again. Both give
`recovery_failed` (terminal) when Daytona still shows `error`. A hung
Recover call with a sandbox still in `error` is also terminal. Terminal
failures still retry every 30 minutes, and each retry makes at most one
Recover call. An explicit Retry also makes one attempt. There is no
tight loop. A sandbox that fails Recover every time gets about two
Recover calls an hour until an operator acts.
- Recover restarts the sandbox container before Stop. The plugin runs no
command in it, and the retention policy is confirmed before Recover.
- The continuation message for a terminally failed lease still says the
message "will start automatically". This PR does not change admission. A
follow-up can use the typed reason there.
- Compatibility: a host without this change ignores the new error data
and keeps the old retry behavior. A plugin without this change sends no
typed data.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, code execution for tests and typechecks).
The runtime does not expose the context window size to the agent.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full repo
runs are left to CI, as noted above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
12 KiB
Sandbox provider capability contract
A sandbox provider plugin declares an environment driver with
kind: "sandbox_provider". Each driver can declare a set of optional sandbox
capabilities. This document is the contract for a third-party provider author.
It states what to declare, which worker methods each capability needs, what an
omitted key means, and when the host narrows or denies a capability.
Read Sandbox file-sync lifecycle hooks for the native file-transfer hooks. Read the driver declaration shape in the plugin specification.
Fresh lease acquisition timeout
A driver can declare defaultAcquireTimeoutMs as a positive integer of at most
86,400,000 milliseconds. The host uses it for environmentAcquireLease when the
resolved config has no positive finite numeric timeoutMs. A positive numeric
bridgeRequestTimeoutMs can extend that budget. The host adds 30 seconds for RPC
overhead. This applies to both sandbox providers and generic plugin drivers.
Daytona declares 300,000 milliseconds for the entire fresh acquisition. Creation,
workspace setup, shell detection, expiry setup, and inline failure cleanup share
that budget. Its host RPC can therefore wait 330 seconds instead of the worker's
normal 30 seconds. On timeout the provider returns the attempt's scoped cleanup
record for durable host reconciliation. Late SDK results cannot admit the lease
or start the next setup phase.
Drivers that omit the declaration keep the existing fallback. The host does not
infer this budget from config-schema defaults: a field named timeoutMs can
describe sandbox lifetime instead of the time needed to acquire it.
For reusable sandbox acquisition, the host also uses this budget for
environmentResumeLease, with the same explicit-config precedence, bridge
extension, and RPC margin. Resume attempts and transient retries share one
deadline. Without a configured or declared budget, resume keeps its one-minute
host default. Generic plugin-driver resume keeps its existing behavior.
This declaration does not change provider config, lease expiry, or other lifecycle calls. Providers must still enforce their operation timeouts and report uncertain allocations for cleanup. A bundled plugin must bump its manifest version when adding the field so existing installations receive it.
How the host resolves an effective capability
The host never trusts a declaration alone. For every run it resolves each capability as the intersection of three inputs:
effective = verified ∩ declared ∩ narrowing
- verified — the methods the live worker advertised in
InitializeResult.supportedMethods. The host maps each capability to the worker methods it needs (see the table below). A capability is verified only when the worker advertises every required method. An empty or missing method list verifies nothing, so every capability resolvesfalse. - declared — the values in the driver declaration. A declared key is optional (see the next section).
- narrowing — a per-run restriction from the lease policy or the resolved provider config. A narrowing can only remove a capability, never add one.
A capability is effective only when all three allow it. A declaration therefore never grants a capability that the worker did not verify.
The declaration is optional and partial
Declare capabilities through the nested sandboxCapabilities object on the
driver declaration:
environmentDrivers: [
{
driverKey: "my-provider",
kind: "sandbox_provider",
displayName: "My Provider",
configSchema: { type: "object", properties: {} },
sandboxCapabilities: {
reusableLeases: true,
nativeSyncIn: true,
nativeSyncOut: true,
persistentProcessSessions: false,
independentControlCommands: false,
incrementalSessionOutput: false,
},
},
]
Every key is optional. For a valid, identified provider each key has one of three states:
- Omitted — the host defers to verified worker discovery. The capability is
effective when the worker advertises the required methods and no narrowing
removes it. Omission is the correct default for a provider that follows the
standard method contract.
reusableLeasesandincrementalSessionOutputare the two exceptions: an omitted key never grants either capability. Both are opt-in (see the next two sections). false— the host narrows the capability to off. The capability is never effective, even when the worker advertises the required methods.true— the host still requires the verified prerequisites. Atruevalue never grants a capability without them. It documents intent and lets the host present the capability, but the worker must still advertise the required methods.
Reusable leases need an explicit opt-in
Reusable leases are the exception to the omission rule above. The host grants
reusable-lease acquisition only when the declaration sets reusableLeases to
true. The provider opts in through one of two fields:
- the nested
sandboxCapabilities.reusableLeases: true, or - the legacy
supportsReusableLeases: true.
An omitted key leaves reusableLeases unset. An unset key does not make the
provider eligible for reusable-lease acquisition, and it does not advertise
provider-level reusable support. The host then always creates an ephemeral lease.
The opt-in never removes the other prerequisites. The worker must still verify
all three lifecycle methods, environmentResumeLease, environmentReleaseLease,
and environmentDestroyLease, and per-run narrowing still applies.
The two opt-in fields have a fixed precedence. The host keeps the legacy
supportsReusableLeases field for backward compatibility, and it folds the field
into sandboxCapabilities.reusableLeases.
- When only
supportsReusableLeasesis present, the host reads it asreusableLeases. - When both
supportsReusableLeasesandsandboxCapabilities.reusableLeasesare present, the nested value wins.
A manifest with legacy true and nested false therefore resolves to false.
Prefer the nested sandboxCapabilities.reusableLeases in a new manifest.
Incremental session output needs an explicit opt-in
Incremental session output is the second exception to the omission rule. The host
selects the session-output streaming path only when the declaration sets
incrementalSessionOutput to true. An omitted key resolves the capability to
false, so the host keeps the output-file poll path.
The reason is that this key is a behavioral guarantee, not a worker-method
property. A generic one-shot provider can keep persistent process sessions and run
independent control commands, yet it never emits incremental stdout and stderr
from a live session. The two broad capabilities do not imply incremental output,
so the host requires the provider to declare the behavior. A provider that streams
incremental session output declares sandboxCapabilities.incrementalSessionOutput: true; every other provider omits the key and keeps the poll path.
The opt-in never removes the prerequisites. The worker must still verify
environmentExecute, and per-run narrowing still applies. A config-resolution
failure narrows the capability to off (see Failure behavior).
The capabilities and their worker-method prerequisites
| Capability | Required worker methods | Meaning |
|---|---|---|
reusableLeases |
environmentResumeLease, environmentReleaseLease, and environmentDestroyLease |
The host retains a provider lease and resumes it across runs. |
nativeSyncIn |
environmentSyncIn |
The host transfers files into the sandbox through the native inbound hook. |
nativeSyncOut |
environmentSyncOut |
The host transfers files out of the sandbox through the native outbound hook. |
persistentProcessSessions |
environmentExecute |
The provider keeps a persistent process session open across commands. |
independentControlCommands |
environmentExecute |
The provider runs a one-shot control command beside a long-lived command. |
incrementalSessionOutput |
environmentExecute |
The provider streams incremental stdout and stderr from a live session. Opt-in: an omitted key resolves false. |
Reusable leases need all three lifecycle methods. The host resumes a lease with
environmentResumeLease, ends it with environmentReleaseLease, and tears down a
stale lease with environmentDestroyLease. The reuse path destroys a stale lease
when a resume fails, so a provider that cannot destroy a lease would strand it. A
worker that omits any of the three methods never gets reusable leases, even with a
positive declaration. The host then always creates an ephemeral lease.
The host advertises and consumes the two native sync methods as a pair. Define both or neither. See Sandbox file-sync lifecycle hooks.
Target and config narrowing
A narrowing removes a capability that the provider verified and declared but that this run cannot use.
- Ephemeral lease policy. A lease that the host does not retain never reuses.
The host narrows
reusableLeasesto off for an ephemeral lease and keeps it on only for a reuse-by-environment lease. - Kubernetes Job backend. A Job-backed lease, or a lease the host marked as
unable to run native file sync, disables native sync. The host narrows
nativeSyncInandnativeSyncOutto off and keeps the base64-over-exec fallback.
Failure behavior
The host fails closed on two failure states. It never grants a capability from an unknown state.
- Config-resolution failure. A provider whose config the host cannot resolve
is untrusted. The host narrows
persistentProcessSessionsandincrementalSessionOutputto off instead of allowing either through an empty config. - Exact-plugin identity failure. A retained lease pins the exact plugin that
acquired it. When that plugin is absent, or when it no longer declares this
provider key with the
sandbox_providerkind, the host cannot establish the declaration. It resolves every effective capability tofalse, no matter what methods a stale worker still advertises. An omittedsandboxCapabilitiesobject on a valid, identified plugin is a different state; the host defers to verified discovery for that case.
Concurrency capabilities are not part of the contract
Earlier drafts listed two concurrency keys, concurrentSyncAndExec and
concurrentSyncOperations. The runtime never exposed a scheduling choice that
read either key, so a declaration had no effect. The host removed both keys. The
strict capability validator now rejects them as unknown keys. The host can
reintroduce a concurrency capability when a runtime path enforces it.
Preserving an unexported workspace
Native export recovery requires the separately discovered environmentStopLease RPC (onEnvironmentStopLease). It stops the exact allocation and preserves files regardless of its normal release policy. A failed or unconfirmed stop must throw and must never fall back to deletion. The host uses the recorded plugin ID and defers when that worker does not advertise the hook; a generic release method is insufficient evidence of stop-only support. Normal successful ephemeral release remains destructive after verified copyback.
Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment.
A provider can refuse Stop while the allocation is in a provider error state. A stop-only hook may run one bounded provider recovery and then stop. When it cannot stop the allocation, it throws PluginEnvironmentStopError with a closed reason. The host treats resource_unrecoverable and recovery_failed as terminal: it escalates the pending cleanup at once and retries at its slowest interval. Deletion is never a fallback.