Files
PaperClipAI/doc/recoverable-disk-cleanup.md
T
Devin FoleyandPaperclip 785d54636e Delete re-homed isolated clones when their contents are reproducible (#17410)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Isolated task workspaces are Git worktrees on the server disk, and
the terminal workspace reaper keeps that disk usage bounded
> - #17369 moves old workspaces off their private per-issue clones onto
the project's shared clone and then deletes the private clone
> - On one instance, the first live sweep moved 12 workspaces but
deleted none of their clones (about 600 MB each), and 3 workspaces could
not move because their restore point was "too large"
> - The kept clones are not referenced by any workspace, so no other
cleanup removes them
> - This pull request replaces the clone check with a real
reproducibility check, sweeps the orphaned clones, lifts the restore
point limits for local transfers, and fixes a false
`source_identity_changed`
> - The benefit is that the disk space #17369 was meant to reclaim is
reclaimed, without deleting unique work

## Linked Issues or Issue Description

Refs #17369. Bug report described in-PR (bug report template).

**What happened?**
On one instance, with `PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true`
and dry run off, the first sweep re-homed 12 workspaces, skipped 5 with
`live_or_unresolved_consumer` and 3 with `restore_point_too_large`. No
clone was removed: 11 logged `cloneKeptReason:
"clone_checkout_has_files"` and 1 logged `"source_identity_changed"`.
Each clone's own checkout was clean (`git status --porcelain
--untracked-files=all` empty). Its only history that the shared clone
lacked was remote branches and release tags that the shared clone never
fetched, which exist on the remote. An operator removed 11 clones by
hand after verifying them.

**Expected behavior**
A re-homed clone whose checkout is clean and whose local history is in
the shared clone or on the remote is deleted. A clone with real unique
work is kept. Workspaces whose changes exceed the upload limits still
move, because the transfer is local.

**Steps to reproduce**
1. Use a project repository whose `.gitignore` (or the clone's
`info/exclude`) contains `.paperclip/`, with a release tag on a commit
that is not on any branch.
2. Re-home an isolated workspace of that repository with #17369.
3. The old worktree is removed, the empty `.paperclip/worktrees`
directory stays, and `git ls-files --others --ignored --directory` lists
`.paperclip/`. The clone is kept with `clone_checkout_has_files`.
Without the ignore line, the local release tag keeps it with
`unique_clone_history`.
4. Add a file larger than 64 MiB to a workspace. The move skips with
`restore_point_too_large`.

**Paperclip version or commit:** `master` at aa0037a137 (rebased on
ee29e2850d, after #17407).
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- New clone reproducibility check (`cloneDisposable` in
`server/src/services/isolated-repository-rehome.ts`), used before
deleting a re-homed clone:
- `git worktree prune`, then only the clone's own worktree may remain
(`base_has_worktrees`);
- no tracked changes and no untracked files
(`clone_checkout_has_files`); the only untracked exception is
Paperclip's compaction trash;
- ignored files only inside regenerable directories (`node_modules`,
`dist`, ...) or Paperclip's compaction trash. Ignored entries are listed
with `--no-empty-directory`, and a non-regenerable ignored directory is
expanded file by file, so empty directories such as the old worktree's
`.paperclip/worktrees` parent no longer count;
- only `refs/heads`, `refs/tags` and `refs/remotes` (no stash, notes or
replacement refs: `unsupported_clone_refs`);
- every commit reachable from local branches, local tags, non-remote
reflogs, `ORIG_HEAD` and `HEAD` (so a detached checkout counts) must be
on a remote-tracking ref, present in the shared clone, or on the remote.
The remote check reuses `hasCommitsMissingUpstream` (now exported),
which fetches the remote's branches and tags into a temporary store, and
it runs only when some commit is in neither place. A local tag is
accepted when its commit is on the remote. Otherwise the clone is kept
(`unique_clone_history`), or kept with
`managed_checkout_upstream_unavailable` when the remote cannot be read;
- existing 5,000 budgets for refs, reflog entries and unique commits
(`clone_history_budget`), plus at most 1,000 ignored directories outside
regenerable ones, read in one Git call (`clone_file_budget`);
- the log entry now includes `cloneKeptPaths`: up to five paths,
worktrees, refs or commits that kept the clone.
- Cleanup also checks that the clone's `origin` still matches the
workspace `repoUrl` before deleting it.
- Old worktree ownership (#4): it is proven by `git rev-parse
--git-common-dir`, or, when that fails, by the worktree's `.git` pointer
into `<clone>/.git/worktrees/`.
- Orphan sweep (`sweepOrphanedIsolatedRepositorySources`): at the first
reaper rotation after server start (feature on), it walks
`isolated-repository-sources/<company>/<issue>/checkout`. It holds the
isolated source lock and skips clones that a workspace still needs (an
open workspace with matching `isolatedRemoteSource`, a closed one that
is not expired with its worktree removed, a project workspace or a
running service), clones with a pending move, and clones whose re-homed
workspace still has its own cleanup pending. The repository URL comes
only from the database: the issue's workspaces must record exactly one
URL and the clone's `origin` must equal it (`unproven_repository` or
`source_identity_changed` otherwise), so a rewritten `origin` is never
fetched. It then applies the same check (using the shared clone of a
workspace of the same issue and repository when one exists), renames to
trash and deletes. It logs `event:
"execution_workspace.isolated_source_orphan_cleanup"` and removes
leftover trash directories from interrupted removals. It inspects at
most eight issue directories per reaper sweep and resumes from the saved
position in the next sweep, so slow remotes cannot hold up other
cleanup. When a full walk kept any orphan, it registers a deadline one
day later in the reaper's existing deadline set. Otherwise it registers
nothing until the next server start. No new timer.
- Restore point limits for re-home (#3):
`ISOLATED_REHOME_RESTORE_POINT_LIMITS` (500,000 files, 1 GiB per file
and in total, 1.5 GiB bundle, 1,000,000 commits) replaces the upload
defaults for this local transfer. Verification is unchanged (exact
snapshot equality). In `git-restore-point.ts`, `applyRestorePoint` now
streams `git diff --binary` into `git apply` instead of buffering it
(and skips the pipe when there is no diff, so no new Git version is
needed). On a failure it kills both processes and waits for both. The
ignored-file read sizes its buffer from the tree instead of the default
limit.
- Docs: `doc/recoverable-disk-cleanup.md`.

### What I found for `source_identity_changed` (#4)

In #17369, cleanup ran `git rev-parse --git-common-dir` in the old
worktree and used `""` when the command failed. Then it compared that to
the clone's `.git` path, so any Git failure in the old worktree was
reported as `source_identity_changed`. Git fails there when the old
worktree's registration in the clone (`<clone>/.git/worktrees/<name>`)
is gone while the directory still exists. Its `.git` file then points to
a missing directory. I reproduced this by deleting the registration
after a re-home: the old code keeps the clone with
`source_identity_changed`, the new code removes the old worktree and the
clone. I could not inspect the instance, so the exact event that removed
that one registration is not confirmed. The check was a false positive
in that case, and the fix proves ownership from the `.git` pointer
instead. A worktree whose pointer leads outside the clone is still kept.

## Verification

- `pnpm exec vitest run
src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 28
passed (real Git repositories, embedded Postgres). New cases:
- a clean clone whose only extra history is a remote branch and a
release tag on a commit outside any branch, both absent from the shared
clone, with `.paperclip/` ignored in the clone: deleted;
- a clone with a local branch commit that is neither upstream nor in the
shared clone: kept (`unique_clone_history`, the commit is in
`cloneKeptPaths`);
- a clone with a tracked change, then an ignored non-regenerable file
next to an ignored `node_modules`, then an untracked `build/notes.txt`:
each kept (`clone_checkout_has_files` with the path), then deleted after
cleanup;
- a clone with a second registered worktree: kept
(`base_has_worktrees`);
- a clone whose detached `HEAD` holds the only copy of a commit (reflogs
expired): kept (`unique_clone_history`);
- an old worktree whose registration was pruned: old worktree and clone
removed;
- an orphan whose `origin` was rewritten is kept with
`source_identity_changed`, and with `unproven_repository` once no
workspace records its repository;
  - 1,001 ignored directories keep a clone (`clone_file_budget`);
- orphan sweep: a batch of one returns a resume position and the next
batch finishes the walk; a dry run changes nothing. Then an orphan whose
history is in the shared clone or on the remote is deleted, and an
orphan with no workspace rows and an unpushed commit is kept, with the
next deadline one day later;
- a workspace with 65 MiB untracked and ignored files (default capture
refuses them with `restore_point_too_large`) moves with an exact
snapshot match.
- After rebasing on #17407 (nested clone migration, which reuses helpers
from the re-home module): `isolated-repository-rehome`,
`nested-clone-migration`, `execution-workspaces-service`,
`execution-workspaces-routes`, `execution-workspace-reopen`,
`execution-workspace-compaction`, `execution-workspace-policy`,
`execution-workspace-runtime-control-conflict`,
`execution-workspace-runtime-lease-route`, `git-workspace-expiration`,
`git-restore-point`, `git-restore-point-expiration`: 405 passed;
typecheck passes.
- Related suites in `server/` (before the rebase):
`git-workspace-expiration`, `git-restore-point`,
`git-restore-point-expiration`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`,
`git-reproducible-managed-checkouts`,
`isolated-remote-workspace-source`, with the new suite: 398 passed.
- `pnpm run typecheck` in `server/` passes.
- Mutation checks, each failing a test:
- remote proof replaced by "always unique": the unfetched-history and
orphan tests fail;
- remote proof replaced by "never unique": the local-commit and orphan
tests fail;
  - `.git` pointer fallback removed: the pruned-registration test fails;
  - re-home limits removed: the large-file test fails;
  - orphan sweep removed from the reaper: the orphan test fails;
  - worktree count check removed: the second-worktree test fails;
- regenerable-directory exception applied to untracked files: the
dirty-checkout test fails;
  - orphan batch limit removed: the orphan test fails;
- orphan URL taken from the clone again: the rewritten-origin test
fails;
  - ignored directory budget removed: the many-directory test fails;
- `HEAD` left out of the checked commits: the detached-HEAD test fails.
- Dropping only `--no-empty-directory` does not fail a test, because the
per-directory expansion also accepts an empty ignored directory. The two
mechanisms are redundant on purpose.
- Not run: the full server suite, or the change on a live instance.

## Risks

- Deleting a clone now depends on reading the remote when some local
commit is neither on a remote-tracking ref nor in the shared clone. If
the remote cannot be read, the clone is kept.
- Commits that only remote-tracking refs hold (for example a remote
branch deleted upstream after the clone fetched it) are treated as not
unique, as before. They were pushed once.
- The orphan sweep deletes clones that no workspace refers to. A closed
workspace that is expired with its worktree removed does not count as a
user, because reopening it rebuilds from the shared clone or clones
again.
- Re-home transfers can now use up to about 1.5 GiB of memory for a
single restore point payload, so a very large workspace uses more memory
during its move than before.
- The orphan sweep runs once per server start and then only while it
keeps an orphan, so a clone that becomes orphaned later waits for the
next restart.
- No migration. No schema change.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, Git, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 09:17:28 -07:00

36 KiB

Recoverable disk cleanup

The server materializes immutable skill versions under <instance>/skills/<company>/__versions__/<skill>/<version>. These are disposable disk copies: the company-scoped version and repository snapshot rows retain the contents, including binary files and executable flags. Normal skill resolution rebuilds a missing copy from the database without contacting its upstream source.

Skill version caches

An hourly background sweep considers at most 50 directories per batch, yielding between batches. It starts after server readiness, and respects the existing worker shutdown, standby, and idle-drain controls. A non-current version becomes eligible after one hour without use. Current versions, active-run pins and unsettled recovery pins are retained. If an active run's version selection is unknown, that company's caches remain. Non-versioned skill directories and database snapshots are never removed by this worker.

Last-use records live separately under __version_access__. Older installations without those records receive a full one-hour observation period. Preparation, pin persistence and eviction share a company-scoped database lock across server processes. Snapshot validation and byte counting run outside that lock; cleanup rechecks current versions, run pins, last use, and directory identity before rename. Audits hold the lock through their file reads. Eviction atomically moves a cache into unique .trash-<UUID> directories; trash deletion uses its own lock so normal preparation can restore files while deletion finishes. A later sweep finishes interrupted trash removal without deleting a newly materialized version.

Controls:

Environment variable Default Effect
PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED true Set false to stop future sweeps; restoration stays available
PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN false Set true to report candidates without changing cache or access records

The internal skillVersionRetentionService supports company-scoped diagnosis, bounded continuations, cancellation between entries, and dry run. Explicit agent pins stay cached. Operators can also retain newest versions with PAPERCLIP_SKILL_VERSION_RETENTION_KEEP (default 0), or change the one-hour window with PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES (default 60). Each batch reports candidate and reclaimed allocated bytes, eviction counts, and skip reasons in the server log. Compare volume usage before and after to verify physical savings; concurrent writes and filesystem allocation can affect the net change. A dry run does not start the legacy observation period.

Roll out with dry run enabled, inspect candidates/skips, then enable removal. Recovery archives and runner backups have separate ownership and retention; this worker never scans or removes them.

Native session failover backups

After each remote turn, the server copies the sandbox's harness state (runner state plus the provider's session home, such as codex-home) to <instance>/runtime/paperclip-runner/durable-sessions/<scope>/failover-backups/current. A replacement sandbox restores the session from this copy, and an ephemeral sandbox restores it at the start of every turn. Each session keeps one copy, and a single copy can reach several gigabytes. Codex copies omit credentials, scratch directories, and data Codex downloads again on startup: remote plugin bundles under plugins/cache/<remote marketplace>, the remote plugin catalog cache, and plugin install staging. After a restore, remote plugins can be unavailable to a turn until Codex finishes downloading them.

An hourly background sweep expires the copy once the session can only resume by reopening closed work. It checks at most 200 sessions and expires at most 50 per batch, starting two minutes after readiness. A copy expires when all of these hold:

  • every issue the session ran on is done or cancelled, and closed before the window;
  • the copy itself was written before the window;
  • no run of the session is queued, running, or unfinished;
  • every native run finalization of the session is settled, with no recovery in progress;
  • no active, released, retained, or pending-cleanup environment lease references the copy;
  • the session is not executing, initializing, warm, or closing in this process, and no cleanup activation is pending in its directory.

A session whose control-plane state is missing, unreadable, or larger than 4 MB keeps its copy, because the sweep cannot identify its runs.

The sweep checks and locks these rows in one transaction, and removes the agent task session that points at the session there. A reopened issue therefore starts a fresh native session instead of resuming the expired one. The copy is atomically renamed to failover-backups/.retiring-<UUID>, and a final check looks for a run of the session. If one appeared, or the check fails, the rename is undone. Only a confirmed copy is renamed to .expired-<UUID> and deleted asynchronously. A later sweep restores an unconfirmed .retiring-* copy when current is missing, drops it when a newer current exists, and finishes an interrupted deletion. The rest of the session directory, including its control-plane state, is kept.

Environment variable Default Effect
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED true Set false to stop expiring copies
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN false Set true to report candidates without deleting copies or task sessions
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS 7 Days an issue must be closed before its sessions' copies expire (minimum 1)

Each batch that finds candidates logs them with candidate and reclaimed allocated bytes, cleared task sessions, and skip reasons.

Native workspace sync folders

Each native run that syncs a workspace with a remote sandbox writes a folder at <instance>/native-workspace-sync/<run>/ with its descriptor, manifest, and the workspace and git seed archives. A seed can reach more than a gigabyte. Only the same run reads the folder again: restart recovery, finalization export, board export retry, and retained workspace recovery. Later runs adopt warm workspaces from lease metadata and the shared generations cache, which this worker never touches.

A retention worker on the shared delivery-work scheduler removes a run's folder once nothing can read it. It runs at startup and at real deadlines: the earliest settled run's finish (or an orphan folder's newest file) plus the retention window, a recheck one window later for folders it had to keep, and a 15-minute retry after a failure, a held finalization lock or a leftover it could not delete. With no folders it schedules nothing; when a run then creates its folder in this server process, the worker wakes and schedules that folder's deadline. A folder named for a run id, that is a real directory and whose newest file is older than the window, is removed when one of these holds:

  • no run with that id exists;
  • the run has no valid workspace sync reference and is settled: it succeeded, failed, was cancelled, timed_out or interrupted, finished before the window, has no live process, no live controller lease and no finalization owner receipt;
  • the run is settled as above, and also:
    • its native run finalization, if any, has no lease owner or scheduled retry and is committed, or terminal_failure with no recovery state or blocked;
    • no active or escalated workspace export or owner-unverified recovery action names the run;
    • no lease of the run carries nativeWorkspaceExportResume or a cleanup in flight;
    • its referenced lease was destroyed, or no retained workspace recovery of the run is scheduled, running, awaiting evidence, queued or recovering;
    • historical recovery adoption has no pending work for the run.

Failed native runs can keep their recorded process ID after they finish. A recorded process blocks removal only while it can still be the run's own process: the run has a local lease or no lease, and the process ID or process group still exists on this host with the recorded start time (or a start time that cannot be read). A process ID from a remote sandbox, or one the operating system reused, does not block removal.

Each folder is checked in one transaction that holds the run's finalization ownership lock, so it never races a finalization or export of the same run. A held lock skips the folder. Inside that transaction the folder is renamed to .expired-<UUID>, and it is deleted after commit. The next sweep finishes an interrupted deletion. Each batch tries at most 50 leftovers, then checks at most 200 folders and removes at most 50. Leftovers and folders each continue from where the previous batch stopped, so a leftover that cannot be deleted does not block others. Deadline discovery reads run state 200 folders at a time and stops when the server drains or shuts down.

Environment variable Default Effect
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED true Set false to stop removing folders
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN false Set true to report candidates without removing folders
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS 7 Days a run must be finished, and its folder untouched, before removal (minimum 1)

Each batch that finds candidates logs them with candidate and reclaimed allocated bytes and skip reasons.

Git workspace restore points

Git cache expiration (PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true) normally removes a closed issue's worktree only when its remote can reproduce it exactly. With PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true, it can also remove a worktree that has unpushed commits, uncommitted or untracked files, or history-only commits, once its issue tree has been closed for PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS (default 3). It also removes the worktree of an issue tree that is still open once nothing has touched it for PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS (default 7, 0 turns this off): no issue in the tree or bound to the workspace was updated and no run for those issues or the workspace was created, started or finished in that time, and no run is still live. An interrupted removal is finished by later sweeps while the issue stays open. The workspace reaper sets a deadline for that moment, so open trees cost no recurring checks before then. When a run later finds its issue's workspace removed this way, it reopens the workspace, restore point included, before using it; if the restore fails, the run fails instead of starting in a new workspace. A comment or an agent update on the issue reopens it the same way. Before removal it saves a restore point:

  • a git bundle of commits the remote does not have, against a remote-verified base, including commits only reachable from the reflog or ORIG_HEAD;
  • deterministic snapshots of the index and working tree, including untracked files and non-default file modes. Untracked .env and .env.* files are left out, because provisioning recreates them with current secrets;
  • a separate snapshot of ignored files, such as build caches or files an agent kept out of git. It leaves out .env files, nested .paperclip-repositories checkouts (they get their own restore points), .claude/settings.local.json (saved with the workspace contents) and anything inside a regenerable directory such as node_modules or dist. A restore point with ignored files uses payload schema v2, which older servers reject instead of restoring without them. Restoring never overwrites an ignored file that already exists, and an ignored nested repository keeps the worktree.

The restore point is encrypted with a per-restore-point key wrapped by the instance secrets key, uploaded to file storage, read back and hash-checked, then activated in the same transaction that archives the workspace with a versioned recipe (paperclip.git-cache-restoration.v2). Older servers reject that recipe instead of restoring without the saved work. Under the expiration locks, the worktree must still match the captured snapshot exactly; otherwise expiration rolls back, keeps the worktree, and drops the restore point.

Nested managed checkouts under .paperclip-repositories/ get their own restore points when their checked-out branch has unpushed commits or uncommitted, untracked or history-only work. Each is stored under its checkout name, recorded in the recipe's contents, and verified against its own snapshot under the expiration locks. A nested checkout still blocks expiration when another local branch or tag has commits its remote lacks, when any reflog holds commits that are neither on the remote nor part of the checked-out history the restore point bundles, when it has objects reachable from nothing, or when its COMMIT_EDITMSG holds a draft that matches no commit in its history. A COMMIT_EDITMSG that matches a commit is accepted for every nested checkout. A nested checkout's remote-tracking refs (refs/remotes/origin/*) and their reflogs can point at commits the remote has since dropped, for example a branch deleted after a squash merge. Strict expiration keeps such a checkout. In restore point mode those commits are saved in the checkout's restore point instead, as history-only objects (the remote-tracking refs themselves are not recreated), and any change to the remote-tracking refs after capture rolls expiration back. Each restore point also records a digest of its HEAD reflog and ORIG_HEAD; under the expiration locks that history must be unchanged too, so a commit followed by a reset after capture rolls expiration back. Reopening re-clones the checkout, imports its bundle, checks out the saved commit and applies the restore point. Submodules, special git state and ignored files other than .env still keep a worktree from expiring. Restore points are refused when the file storage is s3 and the secrets master key does not come from PAPERCLIP_SECRETS_MASTER_KEY.

Nested project repository worktrees

Extra project repositories that have no configured local folder are no longer full clones inside every task. Each task's .paperclip-repositories/<name>-<key> checkout is a Git worktree of one shared clone per company, project and repository URL: the managed project checkout for that URL under the instance's projects/ directory (the same clone that main task worktrees use when the repository is a task's primary source). Objects are stored once in that clone, so a new task adds only its working files. Repositories with a configured local folder are still seeded as independent copies. Existing standalone nested clones keep working unchanged until the optional conversion below turns them into worktrees of the shared clone.

Each nested worktree checks out its own branch, paperclip/<id>/<branch>, where <id> derives from the worktree path and <branch> is the remote's current default branch (refreshed before each new worktree) or the configured branch ref, tracking origin/<branch>. Reopening derives the upstream from the saved source ref, not from the branch name. A tag or commit ref is checked out detached, as before. Branch names therefore never collide between tasks. When a checkout kept under .paperclip-runtime/detached-repositories/ still holds that branch, the new worktree uses paperclip/<id>/r<n>/<branch> instead, so the retained work is never reset. git pull works as before; pushing needs an explicit destination such as git push origin HEAD:<branch>. Adding the worktree, fetching and removing registrations are serialized per shared clone. Each new worktree first fetches with --prune, so a branch deleted upstream is rejected as a fresh clone would reject it. Server-side Git commands on the shared clone run with hooks disabled, so hooks an agent plants there never run in the server.

Branches and reflogs of a worktree live partly in the shared clone. Expiration and restore points therefore scope their checks to what belongs to the nested worktree: its HEAD, HEAD reflog, ORIG_HEAD, index and working tree (stored per worktree) and every branch under its own paperclip/<id>/ prefix together with that branch's reflog. Those must be on the remote or saved in a restore point exactly as for a standalone checkout. Checking out any other branch, stashes, unsupported refs or custom local config in the shared clone still block expiration. Branches outside the prefix, other tasks' branches and their history stay in the shared clone and never block or get deleted. Commits that existed only in the nested worktree's HEAD reflog are written back into the restored worktree's HEAD reflog, so they stay protected after reopening.

Expiring a task removes the nested worktree's registration from the shared clone and deletes its own branch only at the verified recorded commit. Reopening recreates the worktree from the shared clone (re-cloning the shared clone if it is gone) on the same branch and commit and applies its restore point. Deleting or archiving a task removes the registration of each of its nested worktrees and deletes their branches only when every commit is on a remote-tracking ref; an unpushed branch stays in the shared clone. Moving a nested worktree to .paperclip-runtime/detached-repositories/ when its repository leaves the project repairs its registration.

Converting existing standalone nested clones

Tasks prepared before nested worktrees existed still hold a full standalone clone in each .paperclip-repositories/<name>-<key> checkout, often of the same large repository many times over. With PAPERCLIP_NESTED_CLONE_MIGRATION_ENABLED=true, the terminal workspace reaper converts each such checkout in an open task into a worktree of the project's shared clone and deletes the standalone clone. It holds the source lock and workspace lifecycle lock that git cache expiration holds, the per-shared-clone lock that nested worktree creation holds, and the expiration lock files (index.lock, HEAD.lock and the current branch's ref lock) in the clone, so no run, expiration, reopen, restore or other conversion against the same shared clone can interleave. A checkout is converted only when:

  • no run, native finalization, environment lease or runtime service is still live for the workspace, and it is not waiting for a reopen to be consumed (the issue may be open);
  • its name is the managed checkout name of one of the project's repository workspaces (unknown_managed_checkout otherwise), that repository has no configured local folder (local_source_checkout), and the clone's origin is exactly the repository URL (foreign_origin);
  • the project has a verified shared clone for that URL (no_shared_base, shared_base_repo_mismatch) whose local config runs no filters or commands (unsupported_base_git_config);
  • the clone holds only branches, tags and origin remote-tracking refs, no stash (nested_clone_has_stash; stashes are not converted, because refs/stash would be shared by every worktree of the shared clone), no worktrees of its own, no custom hooks, no local config beyond the stock keys, origin and branch tracking, no info/exclude pattern the shared clone lacks (unsupported_info_exclude), no in-progress merge, rebase or similar state, no hidden index flags, and no commit that only another branch's reflog holds (unique_branch_reflog_commits);
  • nothing a restore point refuses (submodules, filtered paths, special files, untracked nested repositories, oversized contents or history).

The conversion fetches the shared clone, then brings the clone's history in under the checkout's own prefix: the current branch becomes paperclip/<id>/<branch> at the same commit, tracking origin/<branch> (or the branch's configured origin upstream); every other local branch whose tip the shared clone's remote-tracking refs do not contain becomes paperclip/<id>/<branch>, every such tag becomes paperclip/<id>/tags/<tag> at the tagged commit, and every such origin remote-tracking ref (for example one whose branch was deleted or force-pushed upstream) becomes paperclip/<id>/remotes/origin/<branch>. <id> derives from the final checkout path, so later expiration and release treat those branches exactly like a nested worktree's own branches: they block expiration until pushed or saved and are never deleted while unpushed. A restore point of the clone, captured against commits both repositories share and preserving those branch tips, its HEAD reflog and ORIG_HEAD, is imported into the shared clone. Other tasks' branches and worktrees are never touched; a name that already exists in the shared clone skips the checkout (nested_clone_ref_conflict).

The new worktree is added at <name>.clone-migrating beside the clone, the restore point is applied (index, worktree changes, untracked and ignored files, file modes), the HEAD reflog and ORIG_HEAD are copied, and regenerable directories, untracked .env files and .claude/settings.local.json are moved in by rename. The two must then match exactly (HEAD, index, worktree, ignored files, modes, reflog, branch and every created ref). Any difference or failure before the swap moves the items back, removes the new worktree and the created branches, and leaves the clone untouched. The swap renames the clone to <name>.clone-standalone, moves the new worktree into <name> with git worktree move and runs git worktree repair in the shared clone. Then one database update records the worktree in a saved restoration recipe that lists the checkout, and the old clone is renamed to <name>.clone-trash and deleted.

Before it changes the shared clone, the conversion records nestedCloneMigrationPending.<name> in workspace metadata (shared clone, HEAD, created refs, moved items and phase). The next pass uses it after a crash: before the swap it moves items back, removes the new worktree and the created branches, then converts again; once the clone has been renamed aside it finishes the swap and deletes the old clone, so work done in the converted checkout after a crash is kept. A state that matches neither (for example a new checkout created at <name> while the clone was aside) is left alone with nested_migration_recovery_conflict. The reaper also finishes a pending record after the workspace is archived; if the task worktree is already gone, it forgets the record, removes the stale registrations and deletes only created branches whose commits are on the remote. Expiration treats the .clone- names as unexplained files and keeps the workspace until the conversion settles. A dry run reports a pending record without acting on it.

Each attempt logs event: "execution_workspace.nested_clone_migration" with the checkout name, outcome (migrated, recovered, rolled_back, dry_run or skipped), skip reason, new branch, candidate bytes (the clone's .git) and reclaimed bytes. A workspace is inspected again at most hourly, and a retry deadline it registered survives later sweeps during that hour. A retry deadline is registered only while a pending record exists or the skip comes from a live consumer, a busy lock or a changed workspace row; other skips, including a change caught by verification, wait for a later sweep. The reaper looks only at checkouts named after the project's configured repositories (plus pending records), never at arbitrary directories, and attempts at most 8 per workspace per sweep. Pending recoveries go first, and the rest rotate across sweeps. While a rotation has not yet visited every checkout it registers a retry deadline; after a full rotation without progress it waits for a later sweep, until a conversion succeeds or the set of candidate checkouts changes. PAPERCLIP_NESTED_CLONE_MIGRATION_DRY_RUN=true reports candidates without changing anything and registers no deadlines.

Reopening the issue, or the next run that recreates the worktree, imports the bundle, re-adds the worktree at the saved commit, applies the restore point and verifies the result. The bundle is always imported, so history-only commits are present again as objects in the repository; they are not recreated as branches or reflog entries, and the restore point keeps them until it expires. A draft commit message left by an aborted commit still keeps the worktree from expiring, because restore points do not save it. A worktree that already matches is only marked applied, and a worktree that diverged is never overwritten: the restore fails closed and can be retried. A missing or corrupt restore point also fails closed.

When a restore point has expired, through retention or on a nested checkout, reopening starts fresh instead: the worktree is recreated on its branch (the branch tip if the repository still has it, otherwise a new branch from the base ref), the project's nested repositories are cloned fresh before the provision command runs, the saved recipe is dropped, and a system comment on the issue warns that unpushed and uncommitted work from before the cleanup is no longer available. The same warning is returned with the reopen result and recorded on the execution_workspace.reopened activity.

Restore points are encrypted saves of the parts of a git worktree that its remote cannot reproduce: unpushed commits and uncommitted files. They let a closed issue's worktree be removed and restored later. The table execution_workspace_restore_points tracks each one with its storage key, hash, size and status (pending, active, expiring, superseded, delete_requested, deleted). The objects live in the instance's configured file storage (local_disk or s3).

An active restore point's delete_after is either its expiry time or empty while its source issue tree is open. A database trigger on issues keeps this current: when an issue's status or parent changes, or an issue is deleted, it marks every active restore point whose source issue is that issue, its old or new parent, or any of their ancestors as due now, in the same transaction. A retention worker on the shared delivery-work scheduler runs at startup, when issue changes settle while restore points exist, and at real deadlines: the earliest expiry, the end of an abandoned capture's grace period, or a failed deletion's retry. With nothing due, it schedules nothing, and open trees cause no recurring checks. At review it checks the source issue tree:

  • Tree still open: clear delete_after and wait for the next issue change.
  • Tree done or cancelled: expire PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS (default 90, minimum 1) after the tree's latest close, immediately if that time has passed. Expiry first moves the row to expiring, runs the expiry hook, and only then requests deletion. An expiring row's object is never deleted; after a restart or a hook failure the hook is retried, after 15 minutes when it failed.
  • Source issue gone: delete.

Deleting an issue or a company marks its restore points delete_requested in the same transaction and notifies the worker, so their objects are removed promptly. A pending restore point older than one hour was abandoned mid-capture and is deleted. superseded and delete_requested restore points have their stored object removed, then become deleted once the storage provider confirms the object is gone. A storage failure is recorded in last_error and retried after 15 minutes. Work is processed in batches of 50; a full batch continues immediately.

Rows have no foreign keys, so they outlive a deleted workspace, issue or company long enough to remove the stored object. Each run that changes anything logs its counts.

Isolated repository sources

For a while, workspace preparation gave some isolated git_worktree workspaces their own full clone of the project repository under instances/<id>/isolated-repository-sources/<companyId>/<issueId>/checkout, with the issue's worktree inside it. Workspace metadata marks these with isolatedRemoteSource. New workspaces no longer get one; they are worktrees of the project's shared managed clone. Each old clone still holds a full .git and an unused default-branch checkout.

With PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true (and git cache expiration enabled), the terminal workspace reaper moves each such open workspace onto the shared clone and then deletes its private clone. It runs under the same source lock, workspace lifecycle lock and Git lock files as git cache expiration, so no run, expiration, reopen or restore can interleave. A workspace is moved only when:

  • no run, native finalization, environment lease or runtime service is still live for it, and it is not waiting for a reopen to be consumed (the issue may be open);
  • the project has a shared managed clone for the same repoUrl;
  • its worktree sits at the standard .paperclip/worktrees/<branch> location, and the branch is absent from the shared clone, points to an ancestor of the worktree's HEAD there, and is not checked out in another of its worktrees;
  • the shared clone's local config sets no filter drivers, includes, attribute files, worktree config or command settings (core.fsmonitor, core.hooksPath, core.sshCommand and similar), and it has no info/attributes, because the move checks out files there;
  • its Git state is transferable: no stash or replacement refs, no in-progress merge, rebase or other worktree-specific state, no hidden index flags, no shallow history, and nothing a restore point refuses (submodules, filtered paths, special files, untracked nested repositories). The transfer is local, so it uses much larger restore point limits than uploads do: 500,000 files, 1 GiB per file and in total, a 1.5 GiB bundle and 1,000,000 commits.

The move captures a restore point of the old worktree against commits both repositories share, imports it into the shared clone, creates or fast-forwards the issue branch, adds the new worktree at the same HEAD and applies the restore point (index, worktree changes, untracked and ignored files, file modes). The HEAD reflog and ORIG_HEAD are copied. What restore points leave out is moved by rename on the same filesystem: regenerable directories such as node_modules, untracked .env and .env.* files, .claude/settings.local.json, and each nested .paperclip-repositories/* checkout as a whole. A standalone nested clone moves as is. A nested checkout that is a worktree of its own shared clone moves too, and the move runs git worktree repair in that clone so its registration follows the new path. Its paperclip/<hash>/ branch is renamed to the prefix of the new path, because that prefix proves the checkout owns the branch (a rollback moves it back, repairs it and renames the branch back). Items inside a moved nested checkout, such as its node_modules, move with it. Any other ignored file that a restore point does not save keeps the workspace in place.

The new worktree must then match the old one exactly (HEAD, index, worktree, ignored files, modes and reflog), and every moved item must exist. Only then does one database update point the workspace at the new path, drop isolatedRemoteSource, rewrite a saved restoration recipe to the shared clone, and record isolatedRepositoryRehome ({ at, from, to, cleanedAt }). Any failure before that update removes the new worktree, moves items back, restores the branch, and leaves the old worktree untouched. Before it changes the shared clone, the move records isolatedRepositoryRehomePending (target, branch, previous branch tip and planned moves). If the server stops before the update, the next attempt uses that record to move items back, remove the new worktree and restore the branch, then starts again. A dry run reports such a record without undoing it. If an item exists in both places (for example a process recreated .env in the old worktree), the rollback stops with rehome_rollback_conflict and keeps the new worktree and the record, so neither copy is lost; later attempts retry the rollback once the conflict is resolved. A new worktree is removed only when the move created it: a target that appears from elsewhere is never deleted.

After the update, the old worktree is removed (its ownership is proven by git rev-parse or, when its registration in the clone was already pruned, by its .git pointer into the clone's worktrees directory). The clone is then renamed to trash and deleted when nothing else needs it and everything in it is reproducible:

  • no other workspace or project workspace refers to it (a workspace already expired with its worktree removed does not count) and no runtime service runs in it;
  • after git worktree prune, it has no worktree besides its own checkout;
  • its own checkout has no tracked changes and no untracked files (only Paperclip's compaction trash is allowed, even inside a directory with a regenerable name), and its ignored files are only inside regenerable directories (such as node_modules) or Paperclip's compaction trash. Empty directories, such as the .paperclip/worktrees parent left by the old worktree, do not count;
  • it has only branch, tag and remote-tracking refs (no stash, notes or replacement refs);
  • every commit that its local branches, local tags, reflogs (other than remote-tracking ones) or ORIG_HEAD hold is on one of its remote-tracking refs, present in the shared clone, or found on the remote. The remote is read fresh (branches and tags) with the company's Git identity, falling back to anonymous access, and only when some commit is in neither place. HEAD itself counts too, so a detached checkout is covered. Remote branches and release tags that the shared clone never fetched do not keep a clone.

A clone with more than 5,000 refs, reflog entries or such commits is kept (clone_history_budget), and so is one with more than 1,000 ignored directories outside regenerable ones (clone_file_budget); those directories are read in one Git call. The log entry names up to five paths, worktrees, refs or commits that kept a clone (cloneKeptPaths). A clone another workspace still uses is kept (cloneRetained); that workspace's own move removes it later. If the server stops after the update, a later sweep finishes the removal and sets cleanedAt, also after the workspace has been archived.

The reaper also sweeps orphaned clones under isolated-repository-sources/ once per server start: clones that no execution workspace refers to (no open workspace with matching isolatedRemoteSource, and no closed one that would need the clone to reopen) and that no pending move or cleanup still owns. The repository must be proven by the database: the workspaces of that issue must record exactly one repository URL, and the clone's origin must equal it (unproven_repository or source_identity_changed otherwise), so a rewritten origin is never fetched or trusted. It applies the same reproducibility check, using the shared clone of a workspace of the same issue and repository when one exists, and logs event: "execution_workspace.isolated_source_orphan_cleanup". It inspects at most eight issue directories per reaper sweep and continues from that position in the next sweep, so a slow remote cannot hold up other workspace cleanup. When the walk ends and it kept any orphan, it looks again a day later; otherwise it registers no further work. It also deletes leftover trash directories from interrupted removals.

Each attempt logs event: "execution_workspace.isolated_source_rehome" with the outcome, skip reason, candidate bytes and reclaimed bytes. A workspace that is skipped, or whose clone is kept for a reason that can change, is retried an hour later through the reaper's existing deadlines; nothing runs while there is no candidate. PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN=true reports candidates without changing anything.

Re-homed workspaces get a new path. An agent that resumes an older session may still remember the old path; the run's working directory comes from the updated workspace.

Reopening or restoring an expired workspace whose recipe came from a private clone no longer recreates that clone when the project has a shared managed clone for the same repository, its branch there is absent or at the saved commit, and the shared clone has every commit the saved restore point bundle needs as a base. A recipe with worktree-backed nested checkouts on branches keeps the old behavior, because their branch names are bound to the old path. The workspace is rebuilt as a worktree of the shared clone at the standard location, with the saved commit and restore point imported and applied as before. Without a shared clone, the old behavior applies.