Files
PaperClipAI/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md
T
DottaandClaude Opus 5.5 57f1afdf19 fix(daytona): retry a sandbox create that timed out before any agent work (#20828)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On a Daytona sandbox environment, every new task run first creates a
fresh sandbox
> - When one Daytona create did not start, the plugin waited the full
300 s acquisition budget
> - The run then failed `setup_failed` with `lease_acquire_failed`. No
agent work had started, but nothing retried it
> - On staging this hit two agents in 15 minutes. Each attempt used
about 5 minutes, and the task stopped until a person pressed Retry. The
manual Retry then got a sandbox in 28 s
> - This pull request ends a stuck create early, adopts a create that
started late, and routes the timeout to the bounded bootstrap retry that
already exists for transient provider refusals
> - The benefit is that a stuck create costs about 90 s and recovers by
itself. The task shows Retry only after the bounded attempts are used up

## Linked Issues or Issue Description

Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait
out a Daytona state change), Refs #20609 (force a stuck stop).

**What happened?**
A run on a Daytona sandbox environment failed after 307 s with `Failed
to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona
lease acquisition timed out; allocation cleanup is pending`. The run
recorded `errorCode: setup_failed`, `failureCause:
lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. The task became blocked with Retry. No
automatic retry was scheduled.

**Steps to reproduce**
1. Use a Daytona sandbox environment with the default `timeoutMs`
(300,000).
2. Start a run while Daytona keeps the new sandbox in `creating`.
3. See that the run waits 300 s, fails `setup_failed`, and is not
retried.

**Expected behavior**
A create that has not started before provider work must not cost 5
minutes. The run must retry by itself within bounds. A late sandbox that
has started must be used. A sandbox that was abandoned must be removed.

**Root cause**
- `packages/plugins/sandbox-providers/daytona/src/plugin.ts`
(`onEnvironmentAcquireLease`): the only bound on the create step was the
whole acquisition budget (`timeoutMs`, default 300 s).
- `server/src/services/environment-provider-unavailable-error.ts`
(`readTransientEnvironmentProviderFailure`): only an HTTP status or
errno on the create call counted as transient. The timeout envelope
carries neither, so the run fell through to generic `setup_failed`,
which `isTransientWorkspacePreparationCode` does not retry.

## What Changed

- Daytona: one fresh acquisition is now capped at 180 s. A create that
has not returned after 90 s is looked up once by its creation name. A
sandbox in `started` is adopted, and setup continues. A sandbox in
`pending_build`, `building_snapshot`, or `pulling_snapshot` keeps
waiting inside the cap. Any other state ends the attempt with the
existing cleanup envelope and an acquisition diagnostic whose elapsed
time reached the expired budget.
- Daytona: a late SDK failure no longer runs the compensating delete for
a sandbox that was adopted.
- Host: `readTransientEnvironmentProviderFailure` reads the validated
acquisition diagnostic. An acquisition whose first failure was its own
deadline is transient. The run fails with
`environment_provider_unavailable`, category `timeout`, and uses the
bounded bootstrap retry (30 s, then 60 s). An earlier setup failure
stays `setup_failed`. Message text is never parsed.
- Host: the task reason for this case is plain: "The sandbox for ... did
not finish starting in time. No agent work had started. Paperclip
retries automatically ...".
- Docs: `doc/execution-semantics.md` and
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`.

## Verification

Red before the fix, green after:
- `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh
sandbox create that timed out through bounded bootstrap recovery" uses
the staging error shape through the orchestrator. Red: `errorCode` was
`setup_failed`. Green: `environment_provider_unavailable`, category
`timeout`, bounded retry scheduled. The `started` and `early_failure`
cases stay `setup_failed` with no retry.
- `server/src/services/environment-provider-unavailable-error.test.ts`:
deadline diagnostic is transient (red, then green). An early failure is
not. The task message is plain.
- `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends
a create still pending at 90 seconds" (red: still pending at 90 s;
green) and "adopts a create that started by 90 seconds" (red; green, and
a late SDK rejection does not delete it). "keeps waiting for a create
that is still preparing its snapshot" is a safety case.
- Existing tests that encoded the 300 s acquisition budget now use the
180 s cap.

Commands run locally:
- `npx vitest run src/plugin.test.ts` (Daytona): 414 passed.
- `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/environment-provider-unavailable-error.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts
src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The
run-completion file reports an `afterAll` embedded-Postgres teardown
timeout. The same timeout occurs on unchanged `origin/master`, so it is
a local platform fault.
- Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass.
- `pnpm verify`: asked for broader CI coverage, because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

Upgrade safety: nothing new is persisted. A task already blocked by this
failure keeps `canRetry: true`. Its Retry starts a new run with a new
creation record, and that run uses the new path. On staging, the manual
Retry of the affected task got a fresh lease in 28 s. The creation
record of the earlier attempt stays with the creation reconciler.

Not verified: a live Daytona create that stays in `creating`. That
cannot be forced on demand.

## Risks

- A legitimately slow create that stays in `creating` or `starting` for
more than 90 s now ends the attempt and retries. Snapshot preparation
states still wait inside the 180 s cap. Staging creates normally take
about 20 to 30 s.
- An abandoned create is removed by the creation reconciler. It can stay
alive until that reconciler settles the record (the existing settle
window). This PR does not change that window.
- Other sandbox providers get the host classification only when they
send the same acquisition diagnostic.

## Model Used

Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code
with tool use (shell, file edits, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:04:17 -06:00

226 lines
13 KiB
Markdown

# Sandbox provider capability contract
A sandbox provider plugin declares an environment driver with
`kind: "sandbox_provider"`. Each driver can declare a set of optional sandbox
capabilities. This document is the contract for a third-party provider author.
It states what to declare, which worker methods each capability needs, what an
omitted key means, and when the host narrows or denies a capability.
Read [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md) for the
native file-transfer hooks. Read the driver declaration shape in
[the plugin specification](./PLUGIN_SPEC.md).
## Fresh lease acquisition timeout
A driver can declare `defaultAcquireTimeoutMs` as a positive integer of at most
86,400,000 milliseconds. The host uses it for `environmentAcquireLease` when the
resolved config has no positive finite numeric `timeoutMs`. A positive numeric
`bridgeRequestTimeoutMs` can extend that budget. The host adds 30 seconds for RPC
overhead. This applies to both sandbox providers and generic plugin drivers.
Daytona declares 300,000 milliseconds, and its host RPC can therefore wait 330
seconds instead of the worker's normal 30 seconds. The worker itself caps one
fresh acquisition at 180 seconds. Creation, workspace setup, shell detection,
expiry setup, and inline failure cleanup share that cap. A create that has not
returned after 90 seconds is looked up once by its creation name: a sandbox that
has started is adopted and setup continues, one still preparing its snapshot
keeps waiting within the cap, and any other state ends the attempt. On timeout
the provider returns the attempt's scoped cleanup record and an acquisition
diagnostic whose elapsed time has reached the expired budget. The host treats
that diagnostic as a transient provider failure (`environment_provider_unavailable`),
so the run is retried with the bounded bootstrap budget while the creation
reconciler removes the abandoned allocation. Late SDK results cannot admit the
lease or start the next setup phase, and a late SDK failure never deletes an
adopted sandbox.
Drivers that omit the declaration keep the existing fallback. The host does not
infer this budget from config-schema defaults: a field named `timeoutMs` can
describe sandbox lifetime instead of the time needed to acquire it.
For reusable sandbox acquisition, the host also uses this budget for
`environmentResumeLease`, with the same explicit-config precedence, bridge
extension, and RPC margin. Resume attempts and transient retries share one
deadline. Without a configured or declared budget, resume keeps its one-minute
host default. Generic plugin-driver resume keeps its existing behavior.
This declaration does not change provider config, lease expiry, or other
lifecycle calls. Providers must still enforce their operation timeouts and
report uncertain allocations for cleanup. A bundled plugin must bump its
manifest version when adding the field so existing installations receive it.
## How the host resolves an effective capability
The host never trusts a declaration alone. For every run it resolves each
capability as the intersection of three inputs:
```
effective = verified ∩ declared ∩ narrowing
```
- **verified** — the methods the live worker advertised in
`InitializeResult.supportedMethods`. The host maps each capability to the
worker methods it needs (see the table below). A capability is verified only
when the worker advertises every required method. An empty or missing method
list verifies nothing, so every capability resolves `false`.
- **declared** — the values in the driver declaration. A declared key is
optional (see the next section).
- **narrowing** — a per-run restriction from the lease policy or the resolved
provider config. A narrowing can only remove a capability, never add one.
A capability is effective only when all three allow it. A declaration therefore
never grants a capability that the worker did not verify.
## The declaration is optional and partial
Declare capabilities through the nested `sandboxCapabilities` object on the
driver declaration:
```ts
environmentDrivers: [
{
driverKey: "my-provider",
kind: "sandbox_provider",
displayName: "My Provider",
configSchema: { type: "object", properties: {} },
sandboxCapabilities: {
reusableLeases: true,
nativeSyncIn: true,
nativeSyncOut: true,
persistentProcessSessions: false,
independentControlCommands: false,
incrementalSessionOutput: false,
},
},
]
```
Every key is optional. For a valid, identified provider each key has one of three
states:
- **Omitted** — the host defers to verified worker discovery. The capability is
effective when the worker advertises the required methods and no narrowing
removes it. Omission is the correct default for a provider that follows the
standard method contract. `reusableLeases` and `incrementalSessionOutput` are
the two exceptions: an omitted key never grants either capability. Both are
opt-in (see the next two sections).
- **`false`** — the host narrows the capability to off. The capability is never
effective, even when the worker advertises the required methods.
- **`true`** — the host still requires the verified prerequisites. A `true`
value never grants a capability without them. It documents intent and lets the
host present the capability, but the worker must still advertise the required
methods.
## Reusable leases need an explicit opt-in
Reusable leases are the exception to the omission rule above. The host grants
reusable-lease acquisition only when the declaration sets `reusableLeases` to
`true`. The provider opts in through one of two fields:
- the nested `sandboxCapabilities.reusableLeases: true`, or
- the legacy `supportsReusableLeases: true`.
An omitted key leaves `reusableLeases` unset. An unset key does not make the
provider eligible for reusable-lease acquisition, and it does not advertise
provider-level reusable support. The host then always creates an ephemeral lease.
The opt-in never removes the other prerequisites. The worker must still verify
all three lifecycle methods, `environmentResumeLease`, `environmentReleaseLease`,
and `environmentDestroyLease`, and per-run narrowing still applies.
The two opt-in fields have a fixed precedence. The host keeps the legacy
`supportsReusableLeases` field for backward compatibility, and it folds the field
into `sandboxCapabilities.reusableLeases`.
- When only `supportsReusableLeases` is present, the host reads it as
`reusableLeases`.
- When both `supportsReusableLeases` and `sandboxCapabilities.reusableLeases` are
present, the nested value wins.
A manifest with legacy `true` and nested `false` therefore resolves to `false`.
Prefer the nested `sandboxCapabilities.reusableLeases` in a new manifest.
## Incremental session output needs an explicit opt-in
Incremental session output is the second exception to the omission rule. The host
selects the session-output streaming path only when the declaration sets
`incrementalSessionOutput` to `true`. An omitted key resolves the capability to
`false`, so the host keeps the output-file poll path.
The reason is that this key is a behavioral guarantee, not a worker-method
property. A generic one-shot provider can keep persistent process sessions and run
independent control commands, yet it never emits incremental stdout and stderr
from a live session. The two broad capabilities do not imply incremental output,
so the host requires the provider to declare the behavior. A provider that streams
incremental session output declares `sandboxCapabilities.incrementalSessionOutput:
true`; every other provider omits the key and keeps the poll path.
The opt-in never removes the prerequisites. The worker must still verify
`environmentExecute`, and per-run narrowing still applies. A config-resolution
failure narrows the capability to off (see [Failure behavior](#failure-behavior)).
## The capabilities and their worker-method prerequisites
| Capability | Required worker methods | Meaning |
| --- | --- | --- |
| `reusableLeases` | `environmentResumeLease`, `environmentReleaseLease`, **and** `environmentDestroyLease` | The host retains a provider lease and resumes it across runs. |
| `nativeSyncIn` | `environmentSyncIn` | The host transfers files into the sandbox through the native inbound hook. |
| `nativeSyncOut` | `environmentSyncOut` | The host transfers files out of the sandbox through the native outbound hook. |
| `persistentProcessSessions` | `environmentExecute` | The provider keeps a persistent process session open across commands. |
| `independentControlCommands` | `environmentExecute` | The provider runs a one-shot control command beside a long-lived command. |
| `incrementalSessionOutput` | `environmentExecute` | The provider streams incremental stdout and stderr from a live session. Opt-in: an omitted key resolves `false`. |
Reusable leases need all three lifecycle methods. The host resumes a lease with
`environmentResumeLease`, ends it with `environmentReleaseLease`, and tears down a
stale lease with `environmentDestroyLease`. The reuse path destroys a stale lease
when a resume fails, so a provider that cannot destroy a lease would strand it. A
worker that omits any of the three methods never gets reusable leases, even with a
positive declaration. The host then always creates an ephemeral lease.
The host advertises and consumes the two native sync methods as a pair. Define
both or neither. See [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md).
## Target and config narrowing
A narrowing removes a capability that the provider verified and declared but that
this run cannot use.
- **Ephemeral lease policy.** A lease that the host does not retain never reuses.
The host narrows `reusableLeases` to off for an ephemeral lease and keeps it on
only for a reuse-by-environment lease.
- **Kubernetes Job backend.** A Job-backed lease, or a lease the host marked as
unable to run native file sync, disables native sync. The host narrows
`nativeSyncIn` and `nativeSyncOut` to off and keeps the base64-over-exec
fallback.
## Failure behavior
The host fails closed on two failure states. It never grants a capability from an
unknown state.
- **Config-resolution failure.** A provider whose config the host cannot resolve
is untrusted. The host narrows `persistentProcessSessions` and
`incrementalSessionOutput` to off instead of allowing either through an empty
config.
- **Exact-plugin identity failure.** A retained lease pins the exact plugin that
acquired it. When that plugin is absent, or when it no longer declares this
provider key with the `sandbox_provider` kind, the host cannot establish the
declaration. It resolves every effective capability to `false`, no matter what
methods a stale worker still advertises. An omitted `sandboxCapabilities`
object on a valid, identified plugin is a different state; the host defers to
verified discovery for that case.
## Concurrency capabilities are not part of the contract
Earlier drafts listed two concurrency keys, `concurrentSyncAndExec` and
`concurrentSyncOperations`. The runtime never exposed a scheduling choice that
read either key, so a declaration had no effect. The host removed both keys. The
strict capability validator now rejects them as unknown keys. The host can
reintroduce a concurrency capability when a runtime path enforces it.
### Preserving an unexported workspace
Native export recovery requires the separately discovered `environmentStopLease` RPC (`onEnvironmentStopLease`). It stops the exact allocation and preserves files regardless of its normal release policy. A failed or unconfirmed stop must throw and must never fall back to deletion. The host uses the recorded plugin ID and defers when that worker does not advertise the hook; a generic release method is insufficient evidence of stop-only support. Normal successful ephemeral release remains destructive after verified copyback.
Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment.
A provider can refuse Stop while the allocation is in a provider error state. A stop-only hook may run one bounded provider recovery and then stop. When it cannot stop the allocation, it throws `PluginEnvironmentStopError` with a closed reason. The host treats `resource_unrecoverable` and `recovery_failed` as terminal: it escalates the pending cleanup at once and retries at its slowest interval. Deletion is never a fallback.