mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
## Thinking Path > - Paperclip manages AI agents and their execution environments. > - Follow-up runs reuse saved sandbox computers to keep their workspace. > - Starting a stopped or archived computer can take more than one minute. > - Acquisition already uses the provider's startup budget, but reuse imposed a separate one-minute deadline. > - This pull request gives resume the configured or declared provider startup budget. > - The benefit is that a healthy slow start can continue on the same computer. ## Linked Issues or Issue Description **What happened?** A follow-up run failed during setup with `ReusableSandboxResumeError`. The saved computer became ready just after the host stopped waiting. The environment had no explicit timeout. Its provider declared a five-minute startup default, which the resume path ignored. **Expected behavior** Resume must honor the configured startup budget or the provider's declared default. It must retain the same lease. Failed or uncertain resumes must keep the saved computer. **Steps to reproduce** 1. Use a reusable sandbox provider with a startup budget longer than one minute. 2. Retain its lease and start a follow-up run. 3. Make provider resume finish after 61 seconds, within that budget. 4. Observe that the old host rejects at 60 seconds. **Paperclip version or commit** Reproduced on `aaeff418cf72341c674ff3393cd94f40dc4855f6`. **Deployment mode** Hosted sandbox execution. The regression uses a local isolated PostgreSQL database and controlled worker RPC timers. Searched open and closed PRs and open issues for sandbox resume and timeout fixes. No duplicate fix was found. Related acquisition timeout handling already exists on the base branch. ## What Changed - Use the provider's declared acquisition timeout when no explicit startup timeout exists. - Apply the resolved RPC budget to the shared resume/retry deadline instead of 60 seconds. - Add two deterministic 61-second resume regressions for declared and explicit startup budgets. Assert exact RPC budgets, including the 30-second margin, and check that the same lease remains active without replacement. - Document the timeout behavior, provider-author contract, and verification recipe. ## Verification - Red: both new cases failed before the production change with `ReusableSandboxResumeError`, caused by `Plugin RPC timed out`. - Green: `pnpm exec vitest run --fsModuleCache server/src/__tests__/environment-runtime.test.ts` passed all 163 cases. The final review follow-up also passed both slow-resume cases with exact RPC-budget assertions (330 seconds by declaration; 150 seconds by explicit config). - An earlier full run passed the 161 existing cases but timed out in the two new tests because their fake clock also captured database scheduling. The corrected test restores real timers before lease persistence. The complete rerun then passed. - `node scripts/check-maps.mjs` passed. - `pnpm -r typecheck` and `pnpm build` passed locally. - Started `pnpm test:run`, then stopped the duplicate serial run in favor of complete parallel PR CI. The repository timing data totals about 126 minutes for server suites alone. Full parallel CI passed on the final head. - CI attempt 1 had one unrelated failure: `FileViewerSheet.copy.test.tsx` expected the asynchronous `Copied link` confirmation before React rendered it. That test and component are unchanged from the base branch. All six cases passed unchanged locally. A single rerun of the failed shard passed; attempt 2 of the full workflow succeeded. The timing-race diagnosis is an inference from the test helper, not a claim that this PR repairs that UI test. - Greptile scored the final head 5/5. Both review findings are fixed and both threads are resolved. - The consequential assumption is that provider startup may exceed one minute. The test preserves the RPC timeout race with controlled timers and uses real lease persistence. Read-only provider evidence also showed the retained computer ready just after the failed run ended. The patch has not been deployed to that instance. ## Risks - A slow or unavailable provider may hold setup for its configured startup budget instead of one minute. Retries still share one finite deadline. Providers with no explicit or declared timeout keep the one-minute default. - Lease identity, company scope, backup validation, and replacement decisions use the existing paths. No schema or dependency changes are required. ## Model Used - OpenAI Codex, GPT-6. This session supports reasoning, tool use, code editing, and test execution. The exact runtime variant and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
216 lines
12 KiB
Markdown
216 lines
12 KiB
Markdown
# Sandbox provider capability contract
|
|
|
|
A sandbox provider plugin declares an environment driver with
|
|
`kind: "sandbox_provider"`. Each driver can declare a set of optional sandbox
|
|
capabilities. This document is the contract for a third-party provider author.
|
|
It states what to declare, which worker methods each capability needs, what an
|
|
omitted key means, and when the host narrows or denies a capability.
|
|
|
|
Read [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md) for the
|
|
native file-transfer hooks. Read the driver declaration shape in
|
|
[the plugin specification](./PLUGIN_SPEC.md).
|
|
|
|
## Fresh lease acquisition timeout
|
|
|
|
A driver can declare `defaultAcquireTimeoutMs` as a positive integer of at most
|
|
86,400,000 milliseconds. The host uses it for `environmentAcquireLease` when the
|
|
resolved config has no positive finite numeric `timeoutMs`. A positive numeric
|
|
`bridgeRequestTimeoutMs` can extend that budget. The host adds 30 seconds for RPC
|
|
overhead. This applies to both sandbox providers and generic plugin drivers.
|
|
|
|
Daytona declares 300,000 milliseconds for the entire fresh acquisition. Creation,
|
|
workspace setup, shell detection, expiry setup, and inline failure cleanup share
|
|
that budget. Its host RPC can therefore wait 330 seconds instead of the worker's
|
|
normal 30 seconds. On timeout the provider returns the attempt's scoped cleanup
|
|
record for durable host reconciliation. Late SDK results cannot admit the lease
|
|
or start the next setup phase.
|
|
Drivers that omit the declaration keep the existing fallback. The host does not
|
|
infer this budget from config-schema defaults: a field named `timeoutMs` can
|
|
describe sandbox lifetime instead of the time needed to acquire it.
|
|
|
|
For reusable sandbox acquisition, the host also uses this budget for
|
|
`environmentResumeLease`, with the same explicit-config precedence, bridge
|
|
extension, and RPC margin. Resume attempts and transient retries share one
|
|
deadline. Without a configured or declared budget, resume keeps its one-minute
|
|
host default. Generic plugin-driver resume keeps its existing behavior.
|
|
|
|
This declaration does not change provider config, lease expiry, or other
|
|
lifecycle calls. Providers must still enforce their operation timeouts and
|
|
report uncertain allocations for cleanup. A bundled plugin must bump its
|
|
manifest version when adding the field so existing installations receive it.
|
|
|
|
## How the host resolves an effective capability
|
|
|
|
The host never trusts a declaration alone. For every run it resolves each
|
|
capability as the intersection of three inputs:
|
|
|
|
```
|
|
effective = verified ∩ declared ∩ narrowing
|
|
```
|
|
|
|
- **verified** — the methods the live worker advertised in
|
|
`InitializeResult.supportedMethods`. The host maps each capability to the
|
|
worker methods it needs (see the table below). A capability is verified only
|
|
when the worker advertises every required method. An empty or missing method
|
|
list verifies nothing, so every capability resolves `false`.
|
|
- **declared** — the values in the driver declaration. A declared key is
|
|
optional (see the next section).
|
|
- **narrowing** — a per-run restriction from the lease policy or the resolved
|
|
provider config. A narrowing can only remove a capability, never add one.
|
|
|
|
A capability is effective only when all three allow it. A declaration therefore
|
|
never grants a capability that the worker did not verify.
|
|
|
|
## The declaration is optional and partial
|
|
|
|
Declare capabilities through the nested `sandboxCapabilities` object on the
|
|
driver declaration:
|
|
|
|
```ts
|
|
environmentDrivers: [
|
|
{
|
|
driverKey: "my-provider",
|
|
kind: "sandbox_provider",
|
|
displayName: "My Provider",
|
|
configSchema: { type: "object", properties: {} },
|
|
sandboxCapabilities: {
|
|
reusableLeases: true,
|
|
nativeSyncIn: true,
|
|
nativeSyncOut: true,
|
|
persistentProcessSessions: false,
|
|
independentControlCommands: false,
|
|
incrementalSessionOutput: false,
|
|
},
|
|
},
|
|
]
|
|
```
|
|
|
|
Every key is optional. For a valid, identified provider each key has one of three
|
|
states:
|
|
|
|
- **Omitted** — the host defers to verified worker discovery. The capability is
|
|
effective when the worker advertises the required methods and no narrowing
|
|
removes it. Omission is the correct default for a provider that follows the
|
|
standard method contract. `reusableLeases` and `incrementalSessionOutput` are
|
|
the two exceptions: an omitted key never grants either capability. Both are
|
|
opt-in (see the next two sections).
|
|
- **`false`** — the host narrows the capability to off. The capability is never
|
|
effective, even when the worker advertises the required methods.
|
|
- **`true`** — the host still requires the verified prerequisites. A `true`
|
|
value never grants a capability without them. It documents intent and lets the
|
|
host present the capability, but the worker must still advertise the required
|
|
methods.
|
|
|
|
## Reusable leases need an explicit opt-in
|
|
|
|
Reusable leases are the exception to the omission rule above. The host grants
|
|
reusable-lease acquisition only when the declaration sets `reusableLeases` to
|
|
`true`. The provider opts in through one of two fields:
|
|
|
|
- the nested `sandboxCapabilities.reusableLeases: true`, or
|
|
- the legacy `supportsReusableLeases: true`.
|
|
|
|
An omitted key leaves `reusableLeases` unset. An unset key does not make the
|
|
provider eligible for reusable-lease acquisition, and it does not advertise
|
|
provider-level reusable support. The host then always creates an ephemeral lease.
|
|
|
|
The opt-in never removes the other prerequisites. The worker must still verify
|
|
all three lifecycle methods, `environmentResumeLease`, `environmentReleaseLease`,
|
|
and `environmentDestroyLease`, and per-run narrowing still applies.
|
|
|
|
The two opt-in fields have a fixed precedence. The host keeps the legacy
|
|
`supportsReusableLeases` field for backward compatibility, and it folds the field
|
|
into `sandboxCapabilities.reusableLeases`.
|
|
|
|
- When only `supportsReusableLeases` is present, the host reads it as
|
|
`reusableLeases`.
|
|
- When both `supportsReusableLeases` and `sandboxCapabilities.reusableLeases` are
|
|
present, the nested value wins.
|
|
|
|
A manifest with legacy `true` and nested `false` therefore resolves to `false`.
|
|
Prefer the nested `sandboxCapabilities.reusableLeases` in a new manifest.
|
|
|
|
## Incremental session output needs an explicit opt-in
|
|
|
|
Incremental session output is the second exception to the omission rule. The host
|
|
selects the session-output streaming path only when the declaration sets
|
|
`incrementalSessionOutput` to `true`. An omitted key resolves the capability to
|
|
`false`, so the host keeps the output-file poll path.
|
|
|
|
The reason is that this key is a behavioral guarantee, not a worker-method
|
|
property. A generic one-shot provider can keep persistent process sessions and run
|
|
independent control commands, yet it never emits incremental stdout and stderr
|
|
from a live session. The two broad capabilities do not imply incremental output,
|
|
so the host requires the provider to declare the behavior. A provider that streams
|
|
incremental session output declares `sandboxCapabilities.incrementalSessionOutput:
|
|
true`; every other provider omits the key and keeps the poll path.
|
|
|
|
The opt-in never removes the prerequisites. The worker must still verify
|
|
`environmentExecute`, and per-run narrowing still applies. A config-resolution
|
|
failure narrows the capability to off (see [Failure behavior](#failure-behavior)).
|
|
|
|
## The capabilities and their worker-method prerequisites
|
|
|
|
| Capability | Required worker methods | Meaning |
|
|
| --- | --- | --- |
|
|
| `reusableLeases` | `environmentResumeLease`, `environmentReleaseLease`, **and** `environmentDestroyLease` | The host retains a provider lease and resumes it across runs. |
|
|
| `nativeSyncIn` | `environmentSyncIn` | The host transfers files into the sandbox through the native inbound hook. |
|
|
| `nativeSyncOut` | `environmentSyncOut` | The host transfers files out of the sandbox through the native outbound hook. |
|
|
| `persistentProcessSessions` | `environmentExecute` | The provider keeps a persistent process session open across commands. |
|
|
| `independentControlCommands` | `environmentExecute` | The provider runs a one-shot control command beside a long-lived command. |
|
|
| `incrementalSessionOutput` | `environmentExecute` | The provider streams incremental stdout and stderr from a live session. Opt-in: an omitted key resolves `false`. |
|
|
|
|
Reusable leases need all three lifecycle methods. The host resumes a lease with
|
|
`environmentResumeLease`, ends it with `environmentReleaseLease`, and tears down a
|
|
stale lease with `environmentDestroyLease`. The reuse path destroys a stale lease
|
|
when a resume fails, so a provider that cannot destroy a lease would strand it. A
|
|
worker that omits any of the three methods never gets reusable leases, even with a
|
|
positive declaration. The host then always creates an ephemeral lease.
|
|
|
|
The host advertises and consumes the two native sync methods as a pair. Define
|
|
both or neither. See [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md).
|
|
|
|
## Target and config narrowing
|
|
|
|
A narrowing removes a capability that the provider verified and declared but that
|
|
this run cannot use.
|
|
|
|
- **Ephemeral lease policy.** A lease that the host does not retain never reuses.
|
|
The host narrows `reusableLeases` to off for an ephemeral lease and keeps it on
|
|
only for a reuse-by-environment lease.
|
|
- **Kubernetes Job backend.** A Job-backed lease, or a lease the host marked as
|
|
unable to run native file sync, disables native sync. The host narrows
|
|
`nativeSyncIn` and `nativeSyncOut` to off and keeps the base64-over-exec
|
|
fallback.
|
|
|
|
## Failure behavior
|
|
|
|
The host fails closed on two failure states. It never grants a capability from an
|
|
unknown state.
|
|
|
|
- **Config-resolution failure.** A provider whose config the host cannot resolve
|
|
is untrusted. The host narrows `persistentProcessSessions` and
|
|
`incrementalSessionOutput` to off instead of allowing either through an empty
|
|
config.
|
|
- **Exact-plugin identity failure.** A retained lease pins the exact plugin that
|
|
acquired it. When that plugin is absent, or when it no longer declares this
|
|
provider key with the `sandbox_provider` kind, the host cannot establish the
|
|
declaration. It resolves every effective capability to `false`, no matter what
|
|
methods a stale worker still advertises. An omitted `sandboxCapabilities`
|
|
object on a valid, identified plugin is a different state; the host defers to
|
|
verified discovery for that case.
|
|
|
|
## Concurrency capabilities are not part of the contract
|
|
|
|
Earlier drafts listed two concurrency keys, `concurrentSyncAndExec` and
|
|
`concurrentSyncOperations`. The runtime never exposed a scheduling choice that
|
|
read either key, so a declaration had no effect. The host removed both keys. The
|
|
strict capability validator now rejects them as unknown keys. The host can
|
|
reintroduce a concurrency capability when a runtime path enforces it.
|
|
|
|
### Preserving an unexported workspace
|
|
|
|
Native export recovery requires the separately discovered `environmentStopLease` RPC (`onEnvironmentStopLease`). It stops the exact allocation and preserves files regardless of its normal release policy. A failed or unconfirmed stop must throw and must never fall back to deletion. The host uses the recorded plugin ID and defers when that worker does not advertise the hook; a generic release method is insufficient evidence of stop-only support. Normal successful ephemeral release remains destructive after verified copyback.
|
|
|
|
Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment.
|