mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
DottaandClaude Opus 5.5
c656b95b6b
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox after a turn, so a follow-up
turn resumes the same sandbox and its files
> - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes
by default). The warm native idle window is also 15 minutes
> - A follow-up that reaches the sandbox during that stop gets a
provider refusal. The host keeps the sandbox ("the lease was preserved
and no replacement was created"), which is correct
> - But setup then records a generic `setup_failed`. No retry is
scheduled, no cause is recorded, and the reply that woke the agent is
effectively dropped
> - This pull request gives that refusal its own failure code and routes
it through the existing bounded bootstrap retry. The retry resumes the
same recorded sandbox
> - The benefit is that a follow-up continues a few seconds later on the
same saved sandbox, instead of stopping the task with no next action
## Linked Issues or Issue Description
Refs #18852 (the bounded bootstrap retry that this change reuses).
Related: #19152 and #19187 (copy-back causes), #19178 (Daytona
errored-sandbox recovery before stop-only cleanup), #19182 and #19180
(recovery holds). This change does not overlap with them: they cover
cleanup, copy-back, and recovery after a run. This change covers a
resume refusal before provider work starts.
**What happened?**
On a staging soak, two follow-up turns failed at setup after a
successful first turn:
- A board reply woke the agent. The run failed after 4.9 seconds with
`setup_failed`: `Reusable sandbox lease "…" could not be resumed; the
lease was preserved and no replacement was created.` The issue stayed in
review with no recovery action. The reply was not delivered.
- An earlier run on another task failed with the same message.
Both resume attempts started 15 minutes 8–25 seconds after the previous
turn released the sandbox. The first turns used `keep_running` with a
warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval`
is also 15 minutes. So the resume reached the sandbox while the provider
was stopping it. The provider refused within seconds; this was not a
long resume timeout. The provider cause was not recorded, so this timing
is the strongest evidence. The exact provider state remains an
inference.
**Expected behavior**
A refused resume before provider dispatch is a temporary infrastructure
condition. The host retries it with a small, bounded budget. Each retry
resumes the same recorded sandbox. A refusal does not prove that the
sandbox is lost, so the host never destroys it or creates a replacement.
Only an explicit provider not-found or a workspace identity mismatch
with a verified backup permits replacement (unchanged). The run records
why the provider refused, without provider text.
**Steps to reproduce**
1. Run a warm remote turn that keeps its sandbox.
2. Make the provider's `environmentResumeLease` throw a non-transient
error (for example, a 400/409 while the sandbox stops).
3. Wake the agent with a reply.
4. Before this change: the run fails with `setup_failed`, no retry is
scheduled, and the error has no cause.
**Paperclip version or commit**
Seen on staging at `5b99bdae`. The touched code is unchanged on current
`master` (`f7a4c5678f`).
## What Changed
- Move `ReusableSandboxResumeError` to
`server/src/services/reusable-sandbox-resume-error.ts` (re-exported from
`environment-runtime.ts`). Give it the code
`reusable_sandbox_resume_failed`.
- Add the provider refusal to the error message by closed labels only:
`(cause=<label> status=<http> code=<errno>)`. The labels come from the
existing transport classifier, now exported as
`sandboxProviderFailureLabel`. Provider text, paths, and IDs are never
copied.
- Carry the provider status across the plugin worker RPC. A JSON-RPC
error has only a message and a numeric RPC code. The worker host now
attaches a validated `paperclip/environment-resume-error/v1` envelope
(HTTP status and an allowlisted errno only) to `environmentResumeLease`
failures. The host re-validates it and adds it to the recorded cause.
- In setup-failure settlement, find this error through the orchestrator
wrapper. Record `reusable_sandbox_resume_failed` only when provider
dispatch did not start and the run has no native session or runner
identity (the same guard as `remote_workspace_prepare_timeout`).
- Add the code to `isTransientWorkspacePreparationCode`. The existing
scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop,
pause, and dependency checks.
- Exclude the code from generic continuation recovery and from legacy
reconciliation (no unknown provider actions; bootstrap evidence proves
none started), like the template.
- Add a specific stranded-recovery notice: "Saved sandbox could not be
resumed".
- Document the contract in `doc/execution-semantics.md`.
## Verification
- Red (before the fix, same tests): 6 failures. The `fresh` case
recorded `setup_failed` and did not schedule a retry. The error had no
cause. Legacy reconciliation held the run. Generic recovery granted
another budget.
- Green: `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery/
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/environment-runtime.test.ts
src/services/remote-workspace-transport.test.ts
src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass.
Also `src/services/recovery/stranded-notice` and
`src/services/remote-workspace*`: 411 pass.
- Safety regressions:
- `run-completion.test.ts`: only a pre-dispatch legacy failure with no
native identity gets the code and a retry. `started`, `retained`
(native), and an untyped copy of the message stay `setup_failed` with no
retry. The recorded error contains the closed cause and not the provider
path.
- `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent
schedulers across a restart create exactly one successor per attempt.
The budget ends after two attempts. Stop, pause, and a new dependency
cancel the scheduled retry. Generic stranded recovery creates no extra
run after exhaustion.
- `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through
the real worker RPC, the `JsonRpcCallError` has no status, and the
envelope gives 409 and the errno. Provider text does not cross. Other
methods get no envelope. Malformed envelopes are rejected.
- `environment-runtime.test.ts`: a throwing resume still preserves the
exact lease. The runtime does not call destroy or acquire. The resume
throws a `JsonRpcCallError` whose message has no status. The thrown
message names `cause=provider_error status=409` from the envelope, and
not the provider text.
- `node scripts/typecheck-low-memory.mjs` passes. `git diff --check`
passes.
- Not verified: a live staging run at this revision. The original tasks
were not retried. This PR is not deployed.
## Risks
- Low. The change affects only runs that fail with this typed error
before provider dispatch. Each retry resumes the same lease and inherits
the source wake context, so a reply is delivered once and no second
executor starts.
- If the provider cannot start the sandbox for longer than about 90
seconds, the two retries end and the task gets the specific recovery
notice. The sandbox and its saved files are kept.
- The error message now has a parenthetical cause before the semicolon.
No code parses this message.
- An issue in review does not get the immediate stranded notice
(unchanged existing behavior). The retry, not the notice, delivers the
reply in that case.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads of staging diagnostics).
Extended reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>