Files
Devin FoleyandPaperclip 4e52463203 fix(daytona): recover output from stalled log streams (#14889)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Daytona driver streams sandbox command output to the host.
> - A log socket can stop delivering bytes without closing or rejecting.
> - Missing permission requests and cancellation results can leave a run
active and block queued work.
> - This pull request switches an idle log stream to saved-output
polling for the same command.
> - The host can receive the missing output without another command
dispatch.

## Linked Issues or Issue Description

**What happened?**

The driver waits for the SDK log-stream promise before it can start
recovery. If that promise never settles, the host can miss new output
that is already in the provider's saved logs. A run can remain active
after a watch completes. Interrupt can then time out with “Execution is
still stopping; termination has not been verified.”

**Expected behavior**

Recover command observation when the live stream stalls. Forward new
permission requests and cancellation results. Require a recorded command
exit before reporting completion. Keep quiet commands running under the
caller's existing lifetime controls.

**Steps to reproduce**

1. Start a session command and leave the callback log promise pending.
2. Put new output in the snapshot API without invoking the stream
callback.
3. Keep the command running until the host receives that output, then
expose its cancellation output and exit code.
4. Verify that the host receives each byte once and dispatches the
command once.

**Paperclip version or commit**

Base commit `479554120d`.

**Agent adapter(s) involved**

Daytona session commands, including sandbox ACP agent sessions.

Related: #14799 handles closed streams; this change handles sockets that
never close. #14485 handles input delivery retries. #13262 adds
permission diagnostics.

## What Changed

- After 15 seconds with no stdout or stderr, switch directly to the
existing status and log-snapshot polling path.
- Ignore callbacks and delayed failures from the abandoned stream. Clear
its idle timer on every exit path.
- Preserve byte-offset deduplication, one command dispatch, and
independent timeouts for recovery reads.
- Add regressions for an initial stream stall, a stall after UTF-8
output, cancellation output, late callbacks, hung recovery reads, and
quiet commands that outlive an operation timeout.
- Update the provider documentation and keep hour-long healthy-stream
coverage active with periodic output.

## Verification

- `pnpm vitest run
packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: 245
passed.
- `pnpm exec vitest run --project @paperclipai/plugin-daytona`: 339
passed; 14 gated live tests skipped.
- Both new stalled-stream regressions fail on the unchanged base driver
because it never starts snapshot recovery. Both pass with this change.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- `pnpm test:run`: the local run did not pass. It was stopped after
confirmed local skill-path and macOS skill-cache failures, once complete
PR CI was green. Four chat/email tests could not load connector skill
files from an ancestor directory outside the checkout. Three
company-skills tests hit macOS `EACCES` during cache publication. One
unrelated wakeup test timed out in the full run and passed on a focused
rerun (`1 passed`, `27 skipped`). No source or test assertions were
changed for these failures. This is not a complete local-suite pass.
- Complete PR CI on `11ac4e030b`: 53 successful checks, 2 expected
skips, no pending or failed checks. The clean CI run includes the full
test suite.
- Greptile reviewed `11ac4e030b` at 5/5 with no findings or unresolved
threads. The branch has no merge conflicts with `master`.
- `git diff --check` and a local scan for secrets and private
identifiers passed.

## Risks

- Quiet healthy commands also switch to polling. Full snapshots can
increase bandwidth as output grows; polling remains limited to one
snapshot per second.
- The SDK exposes no stream cancellation handle. The old socket remains
owned by session teardown, and its callbacks cannot publish after
fallback.
- This change recovers a stalled output stream. It does not claim to
identify every cause of an unanswered permission request or change the
requirement to verify termination before releasing work.
- No schema migration or command replay.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact
serving model ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run local change-specific tests; the full suite passes in
CI, with local-suite limitations documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-01 21:04:41 -07:00
..

@paperclipai/plugin-daytona

Published Daytona sandbox provider plugin for Paperclip.

This package lives in the Paperclip monorepo, but it is intentionally excluded from the root pnpm workspace and shaped to publish and install like a standalone npm package. That lets operators install it from the Plugins page by package name without introducing root lockfile churn for Daytona's SDK dependencies.

Install

From a Paperclip instance, install:

@paperclipai/plugin-daytona

The host plugin installer runs npm install into the managed plugin directory, so transitive dependencies such as @daytonaio/sdk are pulled in during installation.

Configuration

Configure Daytona from Instance Settings -> Environments, not from the plugin's plugin page.

  • Put the Daytona API key on the sandbox environment itself.
  • When you save an environment, Paperclip stores pasted API keys as company secrets.
  • DAYTONA_API_KEY remains an optional host-level fallback when an environment omits the key.
  • Optional apiUrl and target settings map directly to the Daytona SDK/client configuration. If apiUrl is omitted, the Daytona SDK uses its default endpoint.

Notes:

  • The current published Daytona SDK package is @daytonaio/sdk.
  • The driver supports both snapshot-based and image-based sandbox creation. If both are set, validation rejects the config as ambiguous.
  • Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
  • For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records pending_cleanup before retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path.
  • Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved target does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens.
  • A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts.
  • The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit.
  • A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.

Local development

cd packages/plugins/sandbox-providers/daytona
pnpm install --ignore-workspace --no-lockfile
pnpm build
pnpm test
pnpm typecheck

These commands assume the repo root has already been installed once so the local @paperclipai/plugin-sdk workspace package is available to the compiler during development.

Package layout

  • src/manifest.ts declares the sandbox-provider driver metadata
  • src/plugin.ts implements the environment lifecycle hooks
  • paperclipPlugin.manifest and paperclipPlugin.worker point the host at the built plugin entrypoints in dist/