fix(workspaces): restore a saved workspace into one copy of sandbox disk (#20695)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task on a remote (Daytona) sandbox continues from its saved
workspace. Before the agent starts, the server uploads the saved archive
into the sandbox and a sandbox script extracts it.
> - The extraction wrote a full second copy of the workspace beside the
uploaded archive. A 7.5 GB workspace on a fresh sandbox then needed
about 15 GB, and the restore failed with `stage=extract code=ENOSPC
disk=full`.
> - The failure was classified as retryable, so every automatic retry
and every Retry repeated it on a same-size sandbox. The task could not
continue.
> - On a retained sandbox it is worse: the old root, the archive, and
the new tree all share the disk.
> - This pull request makes the restore need about one copy of the
workspace. The sandbox removes only data that control-plane storage
already holds durably, and it never removes a root whose save the server
did not prove.
> - The benefit is that a workspace that fits once on the sandbox disk
restores. A disk that is too small for one copy now gets a named failure
that does not loop.

## Linked Issues or Issue Description

**What happened?**
A saved remote workspace of about 7.5 GB could not be restored on a new
Daytona sandbox. The run failed with `remote_workspace_restore_failed:
step=sandbox_restore; remote_workspace_operation_failed:
operation=restore exit=1; remote_workspace_restore_failed: stage=extract
code=ENOSPC disk=full exit=1`. The run marked the failure as retryable,
and each retry failed the same way.

**Steps to reproduce**
1. Save a remote workspace whose archive is larger than half of the free
sandbox disk.
2. Start the next run on another sandbox, so that the server restores
the saved archive.
3. The upload succeeds. The extraction writes the second copy and fails
with ENOSPC.

**Expected behavior**
A workspace that fits once on the sandbox disk restores. A disk that
cannot hold one copy gives a specific failure that is not retried
automatically, and Retry works after the sandbox has more space.

**What the disk holds during a restore (from the code)**
- The uploaded archive
(`<parent>/.paperclip-runtime/workspace-snapshots/<operation>/workspace.pcws`,
about the workspace size, because the archive is not compressed).
- The extracted tree (`<root>.restoring-*`, the workspace size).
- On a retained sandbox only: the old root, which stayed until after the
new root was published.
- After a restore that died: its partial tree and its archive, which no
later attempt removed.

Refs #20678 (the save side of the same full-disk class).

## What Changed

- `workspaceArchive` restore has a `consumeArchive` mode. It creates the
directories and symlinks first, then writes the file bodies from the
last to the first. After it verifies each file against the manifest
hash, it truncates the archive behind that file. The archive and the
tree together need the workspace size plus the largest file. The default
mode, which the server uses for local restores, does not change.
- The sandbox restore script uses that mode on a private rename of the
uploaded archive. A later upload to the same path cannot change the file
that is being consumed. The script removes the consumed file when it
stops.
- The helper's own staging is now inside the root-specific
`<root>.restoring-*` folder, not in a shared `.workspace-restore-*`
sibling.
- While it holds the root's restore lock, and after the epoch fence, the
script removes leftovers of an earlier attempt on the same root that
died: `<root>.restoring-*` folders, `.consuming` archives in its own
staging, and archive staging folders that a `restore-target.json` binds
to the same root. Each restore writes that binding after the reclaim
step.
- When the server proved that a retained root is its own earlier, fully
saved allocation (`replaceRetiredRoot` and a Paperclip receipt, as
before), the script removes the root's files before the extraction, not
after the publish. The receipt and `.paperclip-runtime` stay, so an
interrupted restore proves the same root again and continues.
- A full disk at any restore stage reports `reason=sandbox_disk_full`.
The existing named-refusal rule makes it non-retryable, so it goes to
the board recovery action and does not loop.
- `doc/execution-semantics.md` describes the disk need, the reclaim, the
early removal, and the new reason.

## Verification

- `cd server && npx vitest run
src/services/remote-workspace-transport.test.ts
src/services/remote-workspace-snapshot-archive.test.ts` (287 passed).
- `cd server && npx vitest run
src/services/remote-workspaces.integration.test.ts
src/services/remote-workspace-lifecycle.test.ts
src/services/remote-workspace-archive-worker.test.ts
src/services/remote-workspace-snapshot-archive-environments.test.ts`
(122 passed, 16 skipped).
- `cd server && npx tsc --noEmit -p tsconfig.json` (pass).
- The new tests run the real sandbox scripts against a folder with a
byte capacity. A `--require` preload makes each file write fail with
ENOSPC when the folder would exceed the capacity.
- Red on `origin/master`, green with this change:
- Fresh sandbox, room for the archive plus one file: master fails with
the exact production error `stage=extract code=ENOSPC disk=full exit=1`.
The fix restores every byte and leaves only the restored files.
- Retained root that the server proved, room for the old root, the
archive, and one file: master fails with ENOSPC. The fix restores and
keeps the runner state.
- Too small for one copy: the failure is `stage=extract
reason=sandbox_disk_full`, with no partial tree or archive left. A retry
with more space then restores.
- Both sandbox processes are killed mid-extraction (a restart): a retry
with the same identity and a retry with a new identity and epoch both
reclaim the leftovers and restore into the same capacity.
- The command dies right after it removed a proven root's files: the
next restore proves the root again from its kept receipt, restores, and
keeps the runner state.
- Integration test through `prepareRemoteWorkspace`: a full disk gives
`retryable: false` with `reason=sandbox_disk_full`. On master the same
case is `retryable: true`.
- Guards (pass on master and with the fix): a retained root without the
server proof, or without a Paperclip receipt, is refused with
`reason=occupied_root`, and none of its files are removed.
- Previous-release story: the archive format and the saved state do not
change. A partial `<root>.restoring-*` tree from the previous release is
reclaimed (test). Archive staging from the previous release has no
binding, so it is kept (test). The previous release's shared
`.workspace-restore-*` siblings cannot be bound to a root, so they are
kept.

## Risks

- The consumed archive cannot be read again. Each restore uploads a new
copy from control-plane storage, and the early-return path for an
already restored root does not read the archive. Every file is still
checked against the manifest, and the manifest is checked against the
saved digest.
- The reclaim step removes data only under the root's restore lock and
after the epoch fence. It removes only names that a restore of the same
root creates. Staging that names another root, or no root, is kept.
- The early removal of a retained root depends on the existing server
proof (`isRetiredOwnAllocation`) and the receipt check. If a restore
fails after the removal, the saved version in storage is the source of
truth, and the next restore proves the same root again.
- Files in a lower overlay layer of a resumed sandbox image do not free
space when they are removed. Such a sandbox can still be full.
- A workspace that is larger than the free disk after the upload, plus
its largest file, still fails. It now fails with the named,
non-retryable `reason=sandbox_disk_full`.
- No change to `packages/paperclip-runner`, the database schema, or the
API.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

doc/execution-semantics.md

@@ -1449,6 +1449,22 @@ folder), or an exhausted budget leaves a board-owned recovery action. Stranded-w
 recovery names this failure for the statuses it scans; for other statuses, such as
 a `blocked` task woken by a reply, the failure path records the action directly.
 
+A restore needs about one workspace of sandbox disk, not two. The uploaded
+archive is a disposable copy of a saved version that control-plane storage
+already holds, and the sandbox checks every file against that version's
+manifest. Extraction writes the files from the last to the first and shortens the
+archive behind each one, so the archive and the restored tree together need the
+workspace size plus the largest file. Before the upload, a reclaim step takes the
+root's restore lock and checks the epoch fence. It removes what an earlier attempt
+on that root left when it died: a `<root>.restoring-*` partial tree, an archive it
+was uploading or consuming, and archive staging that names the same root. Each
+attempt names its staging before its upload starts. Staging that an earlier
+release left without that name is kept. A sandbox that still cannot hold one copy
+fails with `reason=sandbox_disk_full` at the stage that ran out of space (`helper`,
+`upload`, or a restore script stage). A retry on the same disk would fail again,
+so it is not retried automatically. The board recovery action's Retry works once
+the sandbox has room for the workspace.
+
 A retained sandbox can still hold a workspace's earlier root after the workspace
 moved to another allocation, for example when child tasks that an older release
 bound to a parent's workspace saved newer work on their own sandboxes. When the parent wakes on its
@@ -1458,7 +1474,12 @@ recorded use of that root on this sandbox is an earlier epoch that ended in a
 quiesced, committed final save, and the folder carries a Paperclip preparation or
 restore receipt. Its files are then already part of the saved lineage being
 restored. Any other occupant (no receipt, a later unsaved use, or a sandbox that
-never saved that root) is still refused with `reason=occupied_root`. The replaced
+never saved that root) is still refused with `reason=occupied_root`, and none of
+its files are removed. After that proof, the reclaim step removes the root's files
+before the saved archive is uploaded, so the old copy never shares the disk with
+the new one. Its receipt and `.paperclip-runtime` stay until the restored root is
+published, so a restore interrupted after the removal proves the same root again
+and finishes from the saved version. The replaced
 root's `.paperclip-runtime` state, such as a retained runner session, is never in
 a snapshot and belongs to the sandbox, so it moves into the restored root. Only the
 old preparation or restore receipt and export staging are dropped. A retained