0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-23 00:53:38 +02:00

Files

T

DottaandPaperclip 1dbffb4c04 fix(daytona): clean up sandbox allocation after create failure (#13979 )

## Thinking Path

> - Paperclip manages AI agents and their execution environments.
> - The Daytona provider creates sandboxes before it returns a lease.
> - Daytona can allocate a sandbox, then time out while waiting for it
to start.
> - The SDK throws without returning the allocated sandbox handle.
> - Paperclip previously lost the resource identity and could leave a
running sandbox behind.
> - This pull request keeps an exact creation identity and deletes a
matching failed allocation.

## Linked Issues or Issue Description

Related: #13882. Searches for Daytona create cleanup and orphan fixes
found no duplicate for this path.

**What happened?**

In [Grok qualification campaign
36080870743](https://github.com/paperclipai/paperclip/actions/runs/36080870743),
Daytona sandbox creation exceeded 300 seconds. No native runner or
provider session started. The SDK threw, but an ownership-filtered
provider lookup found the allocated sandbox still running. The
disposable resource was then deleted separately. The original attempt
and its failure remain retained.

**Expected behavior**

A failed create must retain enough identity to clean up an allocated
sandbox. Cleanup must never delete another attempt's resource. If the
provider cannot confirm cleanup during sandbox lease acquisition, the
host must retain a durable cleanup record and retry the exact owned
allocation after restart.

**Steps to reproduce**

1. Have Daytona allocate a sandbox for an environment acquisition.
2. Make the SDK throw while waiting for startup, before it returns the
sandbox handle.
3. Before this fix, acquisition fails without deleting the allocated
sandbox.
4. The regression tests reproduce this failure and verify exact-owner
deletion.

**Paperclip version or commit**

Observed at `1b0551bb7c8de3c54f4bee64dbe2c88328b3645e`. The same
creation path exists on master; this fix starts from `efce9356b`.

## What Changed

- Assign each cold create a unique provider name and creation-attempt
label.
- On create failure, look up that exact name and verify every ownership
label before deletion.
- Wait for provider deletion with bounded lookup and deletion calls.
- Preserve the original create error after successful cleanup. Report
both errors and the provider name when cleanup cannot be confirmed.
- Treat a missing lookup after an uncertain create as unconfirmed
cleanup: a delayed provider request might still create the sandbox.
- Transfer only validated ownership fields through the acquire-lease RPC
error; provider exceptions and arbitrary error data are not serialized.
- Persist a pending-cleanup lease before the host retries deletion.
Reuse the existing cleanup sweep and durable spool, retaining the
original environment scope after its row is deleted.
- Fence retries by company, environment, run, provider account, unique
creation name, and every ownership label. Missing provisional-name
lookups remain unresolved.
- Journal an ownership-verified provider ID before host-driven deletion.
Its absence then confirms cleanup after a lost deletion reply or final
database update; failed journal writes block deletion.
- Keep ownership labels separate from the mutable input Daytona modifies
during create.
- Report confirmed host-side deletion accurately while still rejecting
the failed acquisition.
- Add protocol, malformed-evidence, cross-scope denial,
controller-restart, and deleted-environment regressions alongside the
original bounded cleanup tests.

## Verification

- SDK suite: 92 passed. Daytona plugin suite: 277 passed; six opt-in
live tests skipped. Environment-runtime suite: 103 passed, then all four
focused observation/recovery variants passed after adding a
foreign-observation case. SDK and provider TypeScript checks and `git
diff --check` pass.
- Regressions failed before the respective fixes: missing durable
ownership, misleading successful-cleanup message, and SDK mutation of
the ownership labels.
- Controlled real-Daytona proof at
`d2ac1c95b32ca64daf039c0427137657f351c0e0`: create a real sandbox;
inject an SDK failure and a failed immediate lookup; persist the
validated ownership envelope; have a child journal the observed provider
ID before deleting; deliberately drop its successful deletion receipt;
reconcile absence from another fresh process and confirm it with a
separate provider read. Passed, with no manual cleanup and zero model
calls. The live proof uses an owner-only envelope file; actual host
database/spool recovery is tested separately in the environment-runtime
suite.
- The preceding controlled attempt failed because the real SDK mutated
the labels object. That failure is retained. The harness deleted its
exact owned sandbox, and a fresh lookup confirmed absence. The fix
copies the SDK input labels separately from the ownership snapshot.
- [Repository
CI](https://github.com/paperclipai/paperclip/actions/runs/36095925399)
passes on this exact head, with a clean 5/5 review. The unchanged Cursor
remote-command test passed its one bounded rerun after a 10-second
timeout; the same test had already passed on the combined tree. The
original failure is retained. Earlier `199f0852` CI and
immediate-deletion proof are retained, without being promoted to the new
source.
- Combined-source verification uses temporary PR #13990 against the Grok
feature branch. No Docker or Rust build runs on the developer laptop.

## Risks

Creation failures now add up to ten seconds for lookup and fifteen
seconds for deletion confirmation. The SDK still owns the initial
creation timeout. If a request materializes only after the failed
lookup, its durable pending record remains eligible for later cleanup; a
never-observed creation name is never reported deleted from a 404. A
previously observed provider ID can be reconciled as deleted. The host
and SDK must be deployed together. Durable handoff applies to
plugin-backed sandbox lease acquisition used by heartbeat and runner
login. Probe and custom-image interactive setup retain immediate cleanup
and explicit failure reporting, but do not use this lease-recovery path.
A worker crash before delivering the failure envelope still cannot be
recovered through this mechanism. A cleanup error never returns a lease.
Successful creation behavior is unchanged except for the provider name
and ownership label.

## Model Used

OpenAI GPT-6 through Codex, with repository tools and code execution.
The exact serving identifier and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-09-25 10:06:57 -05:00

3.9 KiB

Raw Blame History

`@paperclipai/plugin-daytona`

Published Daytona sandbox provider plugin for Paperclip.

This package lives in the Paperclip monorepo, but it is intentionally excluded from the root pnpm workspace and shaped to publish and install like a standalone npm package. That lets operators install it from the Plugins page by package name without introducing root lockfile churn for Daytona's SDK dependencies.

Install

From a Paperclip instance, install:

@paperclipai/plugin-daytona

The host plugin installer runs npm install into the managed plugin directory, so transitive dependencies such as @daytonaio/sdk are pulled in during installation.

Configuration

Configure Daytona from Instance Settings -> Environments, not from the plugin's plugin page.

Put the Daytona API key on the sandbox environment itself.
When you save an environment, Paperclip stores pasted API keys as company secrets.
DAYTONA_API_KEY remains an optional host-level fallback when an environment omits the key.
Optional apiUrl and target settings map directly to the Daytona SDK/client configuration. If apiUrl is omitted, the Daytona SDK uses its default endpoint.

Notes:

The current published Daytona SDK package is @daytonaio/sdk.
The driver supports both snapshot-based and image-based sandbox creation. If both are set, validation rejects the config as ambiguous.
Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records pending_cleanup before retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path.
Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved target does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens.
A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.

Local development

cd packages/plugins/sandbox-providers/daytona
pnpm install --ignore-workspace --no-lockfile
pnpm build
pnpm test
pnpm typecheck

These commands assume the repo root has already been installed once so the local @paperclipai/plugin-sdk workspace package is available to the compiler during development.

Package layout

src/manifest.ts declares the sandbox-provider driver metadata
src/plugin.ts implements the environment lifecycle hooks
paperclipPlugin.manifest and paperclipPlugin.worker point the host at the built plugin entrypoints in dist/

3.9 KiB Raw Blame History

@paperclipai/plugin-daytona