mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, and the workspace files live in those sandboxes until a save > - When a run cannot resume exactly, the fresh continuation fallback (#21135) abandons the old workspace epoch and keeps its unsaved files "recorded and unapplied" on the old sandbox > - The abandonment clears the dirty flag, the writer and the location, so the workspace looks clean. No code reads the abandonment receipt > - A later failed resume can then replace the sandbox, and cleanup can destroy it. That sandbox holds the only copy of the unsaved files > - This pull request adds one shared hold. The task continues on a new sandbox, and the old sandbox is parked until its files are saved, discarded, or kept for 7 days > - The benefit is that the fallback keeps its promise. Abandoned files are not lost ## Linked Issues or Issue Description Refs #21135 Refs #21297 **What happened?** After the fresh continuation fallback abandoned a workspace epoch, the receipt in `remoteWorkspaceAbandonedEpochs` was write-only. `reusableSandboxReplacementIsSafe` saw a clean workspace with a committed save and allowed replacement. `retireReplacedReusableSandboxLease` then moved the old lease to `pending_cleanup`, and the cleanup sweep destroyed the sandbox with the unsaved files. `nativeWorkspaceExportPendingForLease` (the check that the sweep, the retained orphan settlement and the scoped destroy paths use) also answered "nothing pending". **What did you expect?** A sandbox that holds abandoned, unsaved files stays until the files are saved, a person discards them, or a stated retention period ends. **Steps to reproduce** 1. Abandon a remote workspace epoch whose location names sandbox L. 2. Let the next run fail to resume L. 3. The replacement decision says safe, L goes to `pending_cleanup`, and the sweep destroys it. ## What Changed - New `server/src/services/remote-workspace-abandoned-files-hold.ts` holds the single predicate for "this lease holds abandoned files". It is one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a workspace in its company has an abandoned-epoch receipt whose `location.providerLeaseId` equals the lease's provider lease id, and that receipt has no `released` object. A receipt written by the current release (no new fields) counts as unreleased. - Replace and park. A failed resume of a held sandbox still lets the run replace it, so the task continues at once on a new sandbox restored from the last committed save. `retireReplacedReusableSandboxLease` parks the old lease as `retained` with the `abandonedWorkspaceFilesParked` marker (failure reason `abandoned_workspace_files_held`) instead of `pending_cleanup`. `reusableLeaseCanBeResumed` and `reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never select it again. The check is a metadata read with no query, so the run path does no extra work. - Every path that can destroy a sandbox now honours the hold: - `nativeWorkspaceExportPendingForLease` answers pending for a held lease. This covers the pending cleanup sweep (defer), the orphaned active lease sweep, the retained orphan settlement `heldBy`, `destroyClaimedSandboxLease` (terminal workspace reaper `destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the environment delete path, and run lease release with `destroy`), the `destroyRunLease` dispatcher, and the plugin sandbox teardown retry. - The run lease settlement (`environment_run_lease_active`) is the one caller that ignores the hold (`{ destructive: false }`). It never destroys a sandbox. It stops the sandbox and keeps it, or it moves a lease without a sandbox to `pending_cleanup`, where the sweep parks a held lease. If it waited for the hold, a dead run's lease would stay `active` and block the retry for the whole retention period. - `destroyReusableSandboxLease` parks the lease and returns without a provider call. This covers obsolete reusable lease cleanup, the destroy after an expired or mismatched resume, release with `expired`, and the sandbox driver's `destroyRunLease`. - The `retryPendingSandboxTeardown` dispatcher refuses a destructive teardown of a held lease. Stop-only cleanup still runs, because a stop keeps the filesystem. - The retained orphan listing (`alwaysHeldSql`) uses the same SQL fragment, so held orphans are not polled. - Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released` with a reason (`recovered`, `discarded`, `retention_expired`) and an actor. It moves the parked leases for that sandbox to `pending_cleanup` when no other unreleased receipt holds them, and it writes a `workspace.abandoned_files_released` activity entry on the task in plain language. No export path for abandoned epochs exists today. The `recovered` reason is the hook for one; this pull request adds no new export mechanism. - Retention expiry. New receipts carry `retainUntil` (7 days). An old receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt` includes the earliest retention end of any unreleased receipt. The shared cleanup worker therefore wakes at that real deadline, and a restart rebuilds it from the database. At the deadline the sweep runs `expireAbandonedWorkspaceFileHolds`. This releases each due receipt as an attributed system decision (actor `workspace-retention`) and logs the held receipts and parked leases for each company. Each expiry always writes a task entry: "Unsaved files from an interrupted run were kept for N days, and their retention period has ended." The entry adds "They are scheduled for removal." only when a parked lease was queued for cleanup. A separate entry, "... kept for N days and have now been removed.", is written only after the provider confirms the delete. Each receipt is released in its own transaction. A failed receipt is logged and skipped, and it is retried after a backoff (5 minutes, doubling, at most 6 hours). The other receipts are still processed. The entries reference a run or a task only if it still exists. If the task was deleted, the decision goes to the server log only. - Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that an unreleased receipt holds goes back to `retained` and is parked by the sweep. A sandbox that a release before this change already destroyed cannot be recovered. Its receipt ends at its retention time like any other receipt. - Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch` trims only released receipts and keeps the last 20 of them. It never drops an unreleased receipt. A new abandonment of the same sandbox and root replaces the unreleased receipt for that root. So a workspace holds at most one unreleased receipt for each distinct sandbox and root abandoned in the last 7 days. ## Verification - `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19 tests). It follows the lease through the real acquisition path with a failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`: - `gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone` - `pulls a held lease already queued for cleanup back to retained, and leaves it on the provider` - `treats a receipt written before release fields existed as unreleased` - `records a retention expiry on the receipt` - `hands the parked lease to normal cleanup once the files are recovered` / `... discarded` - `removes the files only after the retention period, and says removed only after the provider confirms it` - `does not report removal when the provider delete fails` - `records the end of retention without scheduling removal when another receipt still holds the sandbox` - `records the end of retention when it expires before the lease was ever parked` - `wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet` - `releases a receipt whose run was deleted and records it without the run reference` - `releases a receipt whose task was deleted without writing a task entry` - `keeps expiring later receipts when one fails, and retries the failed one after a backoff` - `keeps a held sandbox when the retained orphan settlement reviews it` - `lists the sandbox for the retained orphan settlement again once the receipt is released` - `keeps holding the first sandbox after many later abandonments on another sandbox` - `replaces and cleans the sandbox as before when the receipt names a different sandbox` - `replaces and cleans the sandbox as before when no epoch was abandoned` - Red on the base revision (62d673cc3f): the park, upgrade, old-receipt, expiry, many-abandonments and scheduling tests fail. The controls pass. - Green: 19 of 19. Also `environment-runtime-reusable-resume-replacement.test.ts` 17/17, `environment-retained-orphan-sandbox.test.ts` 32/32, `heartbeat-pending-cleanup-sweep.test.ts` 51/51, `fresh-continuation-fallback.test.ts` 58/58. - `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new test file: clean. `environment-cleanup-work.test.ts` + `heartbeat-pending-cleanup-sweep.test.ts`: 80/80. - No existing test expectation was changed. Three CI failures onbb43f52came from this change. Two came from the hold blocking the run lease settlement: `ownership-reconciliation-report.test.ts > releases the lease and frees the workspace for the retry once a preparation-only epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a retry prepare the workspace again after its first writer was killed during preparation`. The third, `native-session-executor.test.ts > restores a warm backup into a created lease only when a settled restore hold retained the stopped source`, came from a mocked database that has no `execute`; the predicate now runs through `select`. All three pass locally now. So do `heartbeat-orphaned-active-lease-sweep`, `environment-run-lease-ownership`, `native-workspace-warm-reuse` and `remote-workspaces.integration`. ## Risks - Cost of a parked sandbox. Parking makes no provider call and changes only the lease row. A sandbox that could not be resumed is not running. A reusable sandbox that was released is already stopped and kept. In both cases the cost is stopped storage for up to 7 days. One case is different: if a destructive path finds a held warm sandbox that is still running (`retained` after a keep-running release), the sandbox stays running until the receipt is released. The expiry log line counts parked leases for each company, so this cost can be seen. - Run lease release for an `ephemeral` lease calls the provider's release. Some providers delete the sandbox on that call. A held sandbox normally has a reusable lease, so this pull request does not gate that call. - No schema change. The receipt is JSON in `execution_workspaces.metadata`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>