mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - On staging, Daytona left two idle sandboxes in the `archiving` state for more than three hours, so no resume call could succeed > - The native runner leases had no verified harness backup, so the run never replaced the sandbox, even though the task files were saved > - Each retry spent six minutes on the same stuck sandbox, the bounded retries ran out, and the task waited for a person to press Retry > - This pull request continues on a new sandbox when the provider has proven it cannot start the old one and the workspace is saved, and it keeps the old sandbox > - The benefit is that the task recovers by itself and a sandbox that may hold unsaved runner state is never destroyed ## Linked Issues or Issue Description Refs #20626, #21113 **What happened?** Eight Codie runs on staging failed in one hour with `reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` They came in two clusters (05:59 to 06:00Z and 06:32 to 06:38Z). Each run took about six minutes, so the 330-second idle resume budget from #21113 was applied. All runs were already `transient_failure_retry` wakes. Investigation of one instance (lease `c23d6120…`, run `d16c272e…`): - The lease was `released` at 02:37Z with a confirmed stop-and-retain. No live run held it, so the lease was idle and the extended budget applied. - A read-only Daytona API read at 06:5xZ showed the sandbox in state `archiving` (desired state `archived`), last updated 03:38Z, one hour after the stop (the auto-archive interval is 60 minutes). The second sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does not start a sandbox while it archives, so every resume call timed out. More time or a new HTTP connection could not help. - The task workspace was clean and had a committed saved snapshot. - The agent uses `paperclip_runner`. The last run on the sandbox failed while it saved the runner session, so the lease had no `nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires a verified harness backup for a runner lease, so the run never replaced the sandbox. It failed strictly, the bounded retries (2) ran out, and the task waited for a manual Retry. **Expected behavior** When an idle sandbox never starts through the whole provider start deadline and the task files are saved, the run continues on a new sandbox automatically. A sandbox that may hold unsaved runner state is not destroyed. **Steps to reproduce** 1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the last run end without a harness backup (for example, the runner exits while it saves the session). The workspace is clean with a committed snapshot. 2. Let Daytona start to archive the stopped sandbox and get stuck in `archiving`. 3. Wake the task. Every `environmentResumeLease` call times out, the run fails after about 330 seconds with `reusable_sandbox_resume_failed`, and the retries fail the same way. **Paperclip version or commit** Staging Core `0c04ff76eb` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `paperclip_runner` (native runner) on a Daytona sandbox environment. **Root cause** `server/src/services/reusable-sandbox-resume-replacement.ts`: a runner lease without a verified harness backup was never replaceable, for any failure. The only other path was to retry the same sandbox, which cannot work while the provider is stuck. Replacement also always retired the old lease to `pending_cleanup`, so allowing it without a backup would have destroyed the only copy of the runner session state. ## What Changed - `reusable-sandbox-resume-replacement.ts`: `reusableSandboxReplacementIsSafe` becomes `reusableSandboxReplacementPlan`. It returns `retire` (same rules as before), `keep`, or `null`. It returns `keep` for a runner lease without a verified harness backup only when the caller passes `providerCannotStart`. All other conditions still apply: idle lease, same workspace, clean workspace, committed snapshot, this run is the writer, and no access refusal. - Add `keepReplacedReusableSandboxLease`. After the replacement lease is recorded, one guarded write marks the old lease with `reusableSandboxKeptAfterReplacement`. The lease keeps its status and its sandbox. Nothing destroys it. - `environment-runtime.ts`: set `providerCannotStart` only when the idle start deadline was extended and the run then used all of it on timeouts. Lease selection skips a lease with the keep marker, so later runs do not spend time on it and obsolete-lease cleanup does not destroy it. - The replacement records `replacement` / `resume_failed` with no harness backup. The native executor already starts a fresh session in a new sandbox in this case (`freshReplacement`), and workspace preparation restores the committed snapshot. - The keep plan needs every resume call to time out. A rate limit, a reset, or any other failure in between keeps the strict behavior. - `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard. Every automatic teardown path skips a marked lease: retained-orphan listing and settle, `destroyReusableSandboxLease` (obsolete cleanup, expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace teardown), recovered-workspace cleanup, and abandoned-file release. Only an operator-consented environment delete still destroys it. - `doc/execution-semantics.md`: document the rule. ## Verification - Red before the fix: the new staging-shape test in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (released runner lease, no harness backup, clean committed snapshot, every resume times out) failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Review round: "keeps an idle sandbox with no harness backup when some resume calls failed for another reason", "never destroys a sandbox kept after replacement through the run or workspace teardown paths", and the retained-orphan case "keeps a warm sandbox while a replacement kept it" each failed before the fix and pass after it. `environment-retained-orphan-sandbox.test.ts`, `abandoned-workspace-files-hold.test.ts`, `workspace-recovery-preparation-gates.test.ts`, `workspace-recovery-admission.test.ts`: all pass. - Green after the fix: 22/22 pass in the replacement suite. The run gets a new sandbox within 330 seconds, makes no destroy call, the old lease stays `released` with the keep marker, and the next run resumes the new sandbox only. With unsaved work or no saved snapshot, the run still fails strictly and does not mark the old lease. Earlier tests (a refusal without a backup, a live owner, a dirty workspace with a timeout) are unchanged. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/__tests__/reusable-leases-default.test.ts`: 215 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - Not run: a live replacement on staging. It is verified after deploy on the next wake of the affected tasks. ## Risks - A runner task that hits this path starts a fresh runner session. The files come from the saved snapshot, and the task, comments, and run history are kept. Before this change, the task stopped and waited for a person. - The kept sandbox stays with the provider (archived storage) until someone removes it. This is the cost of never destroying unsaved runner state. - The rule needs the full extended start deadline (about 330 seconds) of timeouts, so a slow but healthy start still resumes the same sandbox. - No schema change. Old leases follow the same rule. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>