mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On a Daytona sandbox environment, every new task run first creates a fresh sandbox > - When one Daytona create did not start, the plugin waited the full 300 s acquisition budget > - The run then failed `setup_failed` with `lease_acquire_failed`. No agent work had started, but nothing retried it > - On staging this hit two agents in 15 minutes. Each attempt used about 5 minutes, and the task stopped until a person pressed Retry. The manual Retry then got a sandbox in 28 s > - This pull request ends a stuck create early, adopts a create that started late, and routes the timeout to the bounded bootstrap retry that already exists for transient provider refusals > - The benefit is that a stuck create costs about 90 s and recovers by itself. The task shows Retry only after the bounded attempts are used up ## Linked Issues or Issue Description Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait out a Daytona state change), Refs #20609 (force a stuck stop). **What happened?** A run on a Daytona sandbox environment failed after 307 s with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending`. The run recorded `errorCode: setup_failed`, `failureCause: lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. The task became blocked with Retry. No automatic retry was scheduled. **Steps to reproduce** 1. Use a Daytona sandbox environment with the default `timeoutMs` (300,000). 2. Start a run while Daytona keeps the new sandbox in `creating`. 3. See that the run waits 300 s, fails `setup_failed`, and is not retried. **Expected behavior** A create that has not started before provider work must not cost 5 minutes. The run must retry by itself within bounds. A late sandbox that has started must be used. A sandbox that was abandoned must be removed. **Root cause** - `packages/plugins/sandbox-providers/daytona/src/plugin.ts` (`onEnvironmentAcquireLease`): the only bound on the create step was the whole acquisition budget (`timeoutMs`, default 300 s). - `server/src/services/environment-provider-unavailable-error.ts` (`readTransientEnvironmentProviderFailure`): only an HTTP status or errno on the create call counted as transient. The timeout envelope carries neither, so the run fell through to generic `setup_failed`, which `isTransientWorkspacePreparationCode` does not retry. ## What Changed - Daytona: one fresh acquisition is now capped at 180 s. A create that has not returned after 90 s is looked up once by its creation name. A sandbox in `started` is adopted, and setup continues. A sandbox in `pending_build`, `building_snapshot`, or `pulling_snapshot` keeps waiting inside the cap. Any other state ends the attempt with the existing cleanup envelope and an acquisition diagnostic whose elapsed time reached the expired budget. - Daytona: a late SDK failure no longer runs the compensating delete for a sandbox that was adopted. - Host: `readTransientEnvironmentProviderFailure` reads the validated acquisition diagnostic. An acquisition whose first failure was its own deadline is transient. The run fails with `environment_provider_unavailable`, category `timeout`, and uses the bounded bootstrap retry (30 s, then 60 s). An earlier setup failure stays `setup_failed`. Message text is never parsed. - Host: the task reason for this case is plain: "The sandbox for ... did not finish starting in time. No agent work had started. Paperclip retries automatically ...". - Docs: `doc/execution-semantics.md` and `doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`. ## Verification Red before the fix, green after: - `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh sandbox create that timed out through bounded bootstrap recovery" uses the staging error shape through the orchestrator. Red: `errorCode` was `setup_failed`. Green: `environment_provider_unavailable`, category `timeout`, bounded retry scheduled. The `started` and `early_failure` cases stay `setup_failed` with no retry. - `server/src/services/environment-provider-unavailable-error.test.ts`: deadline diagnostic is transient (red, then green). An early failure is not. The task message is plain. - `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends a create still pending at 90 seconds" (red: still pending at 90 s; green) and "adopts a create that started by 90 seconds" (red; green, and a late SDK rejection does not delete it). "keeps waiting for a create that is still preparing its snapshot" is a safety case. - Existing tests that encoded the 300 s acquisition budget now use the 180 s cap. Commands run locally: - `npx vitest run src/plugin.test.ts` (Daytona): 414 passed. - `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/environment-provider-unavailable-error.test.ts src/__tests__/environment-acquisition-diagnostics.test.ts src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The run-completion file reports an `afterAll` embedded-Postgres teardown timeout. The same timeout occurs on unchanged `origin/master`, so it is a local platform fault. - Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass. - `pnpm verify`: asked for broader CI coverage, because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. Upgrade safety: nothing new is persisted. A task already blocked by this failure keeps `canRetry: true`. Its Retry starts a new run with a new creation record, and that run uses the new path. On staging, the manual Retry of the affected task got a fresh lease in 28 s. The creation record of the earlier attempt stays with the creation reconciler. Not verified: a live Daytona create that stays in `creating`. That cannot be forced on demand. ## Risks - A legitimately slow create that stays in `creating` or `starting` for more than 90 s now ends the attempt and retries. Snapshot preparation states still wait inside the 180 s cap. Staging creates normally take about 20 to 30 s. - An abandoned create is removed by the creation reconciler. It can stay alive until that reconciler settles the record (the existing settle window). This PR does not change that window. - Other sandbox providers get the host classification only when they send the same acquisition diagnostic. ## Model Used Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code with tool use (shell, file edits, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
226 lines
13 KiB
Markdown
226 lines
13 KiB
Markdown
# Sandbox provider capability contract
|
|
|
|
A sandbox provider plugin declares an environment driver with
|
|
`kind: "sandbox_provider"`. Each driver can declare a set of optional sandbox
|
|
capabilities. This document is the contract for a third-party provider author.
|
|
It states what to declare, which worker methods each capability needs, what an
|
|
omitted key means, and when the host narrows or denies a capability.
|
|
|
|
Read [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md) for the
|
|
native file-transfer hooks. Read the driver declaration shape in
|
|
[the plugin specification](./PLUGIN_SPEC.md).
|
|
|
|
## Fresh lease acquisition timeout
|
|
|
|
A driver can declare `defaultAcquireTimeoutMs` as a positive integer of at most
|
|
86,400,000 milliseconds. The host uses it for `environmentAcquireLease` when the
|
|
resolved config has no positive finite numeric `timeoutMs`. A positive numeric
|
|
`bridgeRequestTimeoutMs` can extend that budget. The host adds 30 seconds for RPC
|
|
overhead. This applies to both sandbox providers and generic plugin drivers.
|
|
|
|
Daytona declares 300,000 milliseconds, and its host RPC can therefore wait 330
|
|
seconds instead of the worker's normal 30 seconds. The worker itself caps one
|
|
fresh acquisition at 180 seconds. Creation, workspace setup, shell detection,
|
|
expiry setup, and inline failure cleanup share that cap. A create that has not
|
|
returned after 90 seconds is looked up once by its creation name: a sandbox that
|
|
has started is adopted and setup continues, one still preparing its snapshot
|
|
keeps waiting within the cap, and any other state ends the attempt. On timeout
|
|
the provider returns the attempt's scoped cleanup record and an acquisition
|
|
diagnostic whose elapsed time has reached the expired budget. The host treats
|
|
that diagnostic as a transient provider failure (`environment_provider_unavailable`),
|
|
so the run is retried with the bounded bootstrap budget while the creation
|
|
reconciler removes the abandoned allocation. Late SDK results cannot admit the
|
|
lease or start the next setup phase, and a late SDK failure never deletes an
|
|
adopted sandbox.
|
|
Drivers that omit the declaration keep the existing fallback. The host does not
|
|
infer this budget from config-schema defaults: a field named `timeoutMs` can
|
|
describe sandbox lifetime instead of the time needed to acquire it.
|
|
|
|
For reusable sandbox acquisition, the host also uses this budget for
|
|
`environmentResumeLease`, with the same explicit-config precedence, bridge
|
|
extension, and RPC margin. Resume attempts and transient retries share one
|
|
deadline. Without a configured or declared budget, resume keeps its one-minute
|
|
host default. Generic plugin-driver resume keeps its existing behavior.
|
|
|
|
This declaration does not change provider config, lease expiry, or other
|
|
lifecycle calls. Providers must still enforce their operation timeouts and
|
|
report uncertain allocations for cleanup. A bundled plugin must bump its
|
|
manifest version when adding the field so existing installations receive it.
|
|
|
|
## How the host resolves an effective capability
|
|
|
|
The host never trusts a declaration alone. For every run it resolves each
|
|
capability as the intersection of three inputs:
|
|
|
|
```
|
|
effective = verified ∩ declared ∩ narrowing
|
|
```
|
|
|
|
- **verified** — the methods the live worker advertised in
|
|
`InitializeResult.supportedMethods`. The host maps each capability to the
|
|
worker methods it needs (see the table below). A capability is verified only
|
|
when the worker advertises every required method. An empty or missing method
|
|
list verifies nothing, so every capability resolves `false`.
|
|
- **declared** — the values in the driver declaration. A declared key is
|
|
optional (see the next section).
|
|
- **narrowing** — a per-run restriction from the lease policy or the resolved
|
|
provider config. A narrowing can only remove a capability, never add one.
|
|
|
|
A capability is effective only when all three allow it. A declaration therefore
|
|
never grants a capability that the worker did not verify.
|
|
|
|
## The declaration is optional and partial
|
|
|
|
Declare capabilities through the nested `sandboxCapabilities` object on the
|
|
driver declaration:
|
|
|
|
```ts
|
|
environmentDrivers: [
|
|
{
|
|
driverKey: "my-provider",
|
|
kind: "sandbox_provider",
|
|
displayName: "My Provider",
|
|
configSchema: { type: "object", properties: {} },
|
|
sandboxCapabilities: {
|
|
reusableLeases: true,
|
|
nativeSyncIn: true,
|
|
nativeSyncOut: true,
|
|
persistentProcessSessions: false,
|
|
independentControlCommands: false,
|
|
incrementalSessionOutput: false,
|
|
},
|
|
},
|
|
]
|
|
```
|
|
|
|
Every key is optional. For a valid, identified provider each key has one of three
|
|
states:
|
|
|
|
- **Omitted** — the host defers to verified worker discovery. The capability is
|
|
effective when the worker advertises the required methods and no narrowing
|
|
removes it. Omission is the correct default for a provider that follows the
|
|
standard method contract. `reusableLeases` and `incrementalSessionOutput` are
|
|
the two exceptions: an omitted key never grants either capability. Both are
|
|
opt-in (see the next two sections).
|
|
- **`false`** — the host narrows the capability to off. The capability is never
|
|
effective, even when the worker advertises the required methods.
|
|
- **`true`** — the host still requires the verified prerequisites. A `true`
|
|
value never grants a capability without them. It documents intent and lets the
|
|
host present the capability, but the worker must still advertise the required
|
|
methods.
|
|
|
|
## Reusable leases need an explicit opt-in
|
|
|
|
Reusable leases are the exception to the omission rule above. The host grants
|
|
reusable-lease acquisition only when the declaration sets `reusableLeases` to
|
|
`true`. The provider opts in through one of two fields:
|
|
|
|
- the nested `sandboxCapabilities.reusableLeases: true`, or
|
|
- the legacy `supportsReusableLeases: true`.
|
|
|
|
An omitted key leaves `reusableLeases` unset. An unset key does not make the
|
|
provider eligible for reusable-lease acquisition, and it does not advertise
|
|
provider-level reusable support. The host then always creates an ephemeral lease.
|
|
|
|
The opt-in never removes the other prerequisites. The worker must still verify
|
|
all three lifecycle methods, `environmentResumeLease`, `environmentReleaseLease`,
|
|
and `environmentDestroyLease`, and per-run narrowing still applies.
|
|
|
|
The two opt-in fields have a fixed precedence. The host keeps the legacy
|
|
`supportsReusableLeases` field for backward compatibility, and it folds the field
|
|
into `sandboxCapabilities.reusableLeases`.
|
|
|
|
- When only `supportsReusableLeases` is present, the host reads it as
|
|
`reusableLeases`.
|
|
- When both `supportsReusableLeases` and `sandboxCapabilities.reusableLeases` are
|
|
present, the nested value wins.
|
|
|
|
A manifest with legacy `true` and nested `false` therefore resolves to `false`.
|
|
Prefer the nested `sandboxCapabilities.reusableLeases` in a new manifest.
|
|
|
|
## Incremental session output needs an explicit opt-in
|
|
|
|
Incremental session output is the second exception to the omission rule. The host
|
|
selects the session-output streaming path only when the declaration sets
|
|
`incrementalSessionOutput` to `true`. An omitted key resolves the capability to
|
|
`false`, so the host keeps the output-file poll path.
|
|
|
|
The reason is that this key is a behavioral guarantee, not a worker-method
|
|
property. A generic one-shot provider can keep persistent process sessions and run
|
|
independent control commands, yet it never emits incremental stdout and stderr
|
|
from a live session. The two broad capabilities do not imply incremental output,
|
|
so the host requires the provider to declare the behavior. A provider that streams
|
|
incremental session output declares `sandboxCapabilities.incrementalSessionOutput:
|
|
true`; every other provider omits the key and keeps the poll path.
|
|
|
|
The opt-in never removes the prerequisites. The worker must still verify
|
|
`environmentExecute`, and per-run narrowing still applies. A config-resolution
|
|
failure narrows the capability to off (see [Failure behavior](#failure-behavior)).
|
|
|
|
## The capabilities and their worker-method prerequisites
|
|
|
|
| Capability | Required worker methods | Meaning |
|
|
| --- | --- | --- |
|
|
| `reusableLeases` | `environmentResumeLease`, `environmentReleaseLease`, **and** `environmentDestroyLease` | The host retains a provider lease and resumes it across runs. |
|
|
| `nativeSyncIn` | `environmentSyncIn` | The host transfers files into the sandbox through the native inbound hook. |
|
|
| `nativeSyncOut` | `environmentSyncOut` | The host transfers files out of the sandbox through the native outbound hook. |
|
|
| `persistentProcessSessions` | `environmentExecute` | The provider keeps a persistent process session open across commands. |
|
|
| `independentControlCommands` | `environmentExecute` | The provider runs a one-shot control command beside a long-lived command. |
|
|
| `incrementalSessionOutput` | `environmentExecute` | The provider streams incremental stdout and stderr from a live session. Opt-in: an omitted key resolves `false`. |
|
|
|
|
Reusable leases need all three lifecycle methods. The host resumes a lease with
|
|
`environmentResumeLease`, ends it with `environmentReleaseLease`, and tears down a
|
|
stale lease with `environmentDestroyLease`. The reuse path destroys a stale lease
|
|
when a resume fails, so a provider that cannot destroy a lease would strand it. A
|
|
worker that omits any of the three methods never gets reusable leases, even with a
|
|
positive declaration. The host then always creates an ephemeral lease.
|
|
|
|
The host advertises and consumes the two native sync methods as a pair. Define
|
|
both or neither. See [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md).
|
|
|
|
## Target and config narrowing
|
|
|
|
A narrowing removes a capability that the provider verified and declared but that
|
|
this run cannot use.
|
|
|
|
- **Ephemeral lease policy.** A lease that the host does not retain never reuses.
|
|
The host narrows `reusableLeases` to off for an ephemeral lease and keeps it on
|
|
only for a reuse-by-environment lease.
|
|
- **Kubernetes Job backend.** A Job-backed lease, or a lease the host marked as
|
|
unable to run native file sync, disables native sync. The host narrows
|
|
`nativeSyncIn` and `nativeSyncOut` to off and keeps the base64-over-exec
|
|
fallback.
|
|
|
|
## Failure behavior
|
|
|
|
The host fails closed on two failure states. It never grants a capability from an
|
|
unknown state.
|
|
|
|
- **Config-resolution failure.** A provider whose config the host cannot resolve
|
|
is untrusted. The host narrows `persistentProcessSessions` and
|
|
`incrementalSessionOutput` to off instead of allowing either through an empty
|
|
config.
|
|
- **Exact-plugin identity failure.** A retained lease pins the exact plugin that
|
|
acquired it. When that plugin is absent, or when it no longer declares this
|
|
provider key with the `sandbox_provider` kind, the host cannot establish the
|
|
declaration. It resolves every effective capability to `false`, no matter what
|
|
methods a stale worker still advertises. An omitted `sandboxCapabilities`
|
|
object on a valid, identified plugin is a different state; the host defers to
|
|
verified discovery for that case.
|
|
|
|
## Concurrency capabilities are not part of the contract
|
|
|
|
Earlier drafts listed two concurrency keys, `concurrentSyncAndExec` and
|
|
`concurrentSyncOperations`. The runtime never exposed a scheduling choice that
|
|
read either key, so a declaration had no effect. The host removed both keys. The
|
|
strict capability validator now rejects them as unknown keys. The host can
|
|
reintroduce a concurrency capability when a runtime path enforces it.
|
|
|
|
### Preserving an unexported workspace
|
|
|
|
Native export recovery requires the separately discovered `environmentStopLease` RPC (`onEnvironmentStopLease`). It stops the exact allocation and preserves files regardless of its normal release policy. A failed or unconfirmed stop must throw and must never fall back to deletion. The host uses the recorded plugin ID and defers when that worker does not advertise the hook; a generic release method is insufficient evidence of stop-only support. Normal successful ephemeral release remains destructive after verified copyback.
|
|
|
|
Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment.
|
|
|
|
A provider can refuse Stop while the allocation is in a provider error state. A stop-only hook may run one bounded provider recovery and then stop. When it cannot stop the allocation, it throws `PluginEnvironmentStopError` with a closed reason. The host treats `resource_unrecoverable` and `recovery_failed` as terminal: it escalates the pending cleanup at once and retries at its slowest interval. Deletion is never a fallback.
|