## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Isolated task workspaces are Git worktrees on the server disk, and the terminal workspace reaper keeps that disk usage bounded > - #17369 moves old workspaces off their private per-issue clones onto the project's shared clone and then deletes the private clone > - On one instance, the first live sweep moved 12 workspaces but deleted none of their clones (about 600 MB each), and 3 workspaces could not move because their restore point was "too large" > - The kept clones are not referenced by any workspace, so no other cleanup removes them > - This pull request replaces the clone check with a real reproducibility check, sweeps the orphaned clones, lifts the restore point limits for local transfers, and fixes a false `source_identity_changed` > - The benefit is that the disk space #17369 was meant to reclaim is reclaimed, without deleting unique work ## Linked Issues or Issue Description Refs #17369. Bug report described in-PR (bug report template). **What happened?** On one instance, with `PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` and dry run off, the first sweep re-homed 12 workspaces, skipped 5 with `live_or_unresolved_consumer` and 3 with `restore_point_too_large`. No clone was removed: 11 logged `cloneKeptReason: "clone_checkout_has_files"` and 1 logged `"source_identity_changed"`. Each clone's own checkout was clean (`git status --porcelain --untracked-files=all` empty). Its only history that the shared clone lacked was remote branches and release tags that the shared clone never fetched, which exist on the remote. An operator removed 11 clones by hand after verifying them. **Expected behavior** A re-homed clone whose checkout is clean and whose local history is in the shared clone or on the remote is deleted. A clone with real unique work is kept. Workspaces whose changes exceed the upload limits still move, because the transfer is local. **Steps to reproduce** 1. Use a project repository whose `.gitignore` (or the clone's `info/exclude`) contains `.paperclip/`, with a release tag on a commit that is not on any branch. 2. Re-home an isolated workspace of that repository with #17369. 3. The old worktree is removed, the empty `.paperclip/worktrees` directory stays, and `git ls-files --others --ignored --directory` lists `.paperclip/`. The clone is kept with `clone_checkout_has_files`. Without the ignore line, the local release tag keeps it with `unique_clone_history`. 4. Add a file larger than 64 MiB to a workspace. The move skips with `restore_point_too_large`. **Paperclip version or commit:** `master` ataa0037a137(rebased onee29e2850d, after #17407). **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed - New clone reproducibility check (`cloneDisposable` in `server/src/services/isolated-repository-rehome.ts`), used before deleting a re-homed clone: - `git worktree prune`, then only the clone's own worktree may remain (`base_has_worktrees`); - no tracked changes and no untracked files (`clone_checkout_has_files`); the only untracked exception is Paperclip's compaction trash; - ignored files only inside regenerable directories (`node_modules`, `dist`, ...) or Paperclip's compaction trash. Ignored entries are listed with `--no-empty-directory`, and a non-regenerable ignored directory is expanded file by file, so empty directories such as the old worktree's `.paperclip/worktrees` parent no longer count; - only `refs/heads`, `refs/tags` and `refs/remotes` (no stash, notes or replacement refs: `unsupported_clone_refs`); - every commit reachable from local branches, local tags, non-remote reflogs, `ORIG_HEAD` and `HEAD` (so a detached checkout counts) must be on a remote-tracking ref, present in the shared clone, or on the remote. The remote check reuses `hasCommitsMissingUpstream` (now exported), which fetches the remote's branches and tags into a temporary store, and it runs only when some commit is in neither place. A local tag is accepted when its commit is on the remote. Otherwise the clone is kept (`unique_clone_history`), or kept with `managed_checkout_upstream_unavailable` when the remote cannot be read; - existing 5,000 budgets for refs, reflog entries and unique commits (`clone_history_budget`), plus at most 1,000 ignored directories outside regenerable ones, read in one Git call (`clone_file_budget`); - the log entry now includes `cloneKeptPaths`: up to five paths, worktrees, refs or commits that kept the clone. - Cleanup also checks that the clone's `origin` still matches the workspace `repoUrl` before deleting it. - Old worktree ownership (#4): it is proven by `git rev-parse --git-common-dir`, or, when that fails, by the worktree's `.git` pointer into `<clone>/.git/worktrees/`. - Orphan sweep (`sweepOrphanedIsolatedRepositorySources`): at the first reaper rotation after server start (feature on), it walks `isolated-repository-sources/<company>/<issue>/checkout`. It holds the isolated source lock and skips clones that a workspace still needs (an open workspace with matching `isolatedRemoteSource`, a closed one that is not expired with its worktree removed, a project workspace or a running service), clones with a pending move, and clones whose re-homed workspace still has its own cleanup pending. The repository URL comes only from the database: the issue's workspaces must record exactly one URL and the clone's `origin` must equal it (`unproven_repository` or `source_identity_changed` otherwise), so a rewritten `origin` is never fetched. It then applies the same check (using the shared clone of a workspace of the same issue and repository when one exists), renames to trash and deletes. It logs `event: "execution_workspace.isolated_source_orphan_cleanup"` and removes leftover trash directories from interrupted removals. It inspects at most eight issue directories per reaper sweep and resumes from the saved position in the next sweep, so slow remotes cannot hold up other cleanup. When a full walk kept any orphan, it registers a deadline one day later in the reaper's existing deadline set. Otherwise it registers nothing until the next server start. No new timer. - Restore point limits for re-home (#3): `ISOLATED_REHOME_RESTORE_POINT_LIMITS` (500,000 files, 1 GiB per file and in total, 1.5 GiB bundle, 1,000,000 commits) replaces the upload defaults for this local transfer. Verification is unchanged (exact snapshot equality). In `git-restore-point.ts`, `applyRestorePoint` now streams `git diff --binary` into `git apply` instead of buffering it (and skips the pipe when there is no diff, so no new Git version is needed). On a failure it kills both processes and waits for both. The ignored-file read sizes its buffer from the tree instead of the default limit. - Docs: `doc/recoverable-disk-cleanup.md`. ### What I found for `source_identity_changed` (#4) In #17369, cleanup ran `git rev-parse --git-common-dir` in the old worktree and used `""` when the command failed. Then it compared that to the clone's `.git` path, so any Git failure in the old worktree was reported as `source_identity_changed`. Git fails there when the old worktree's registration in the clone (`<clone>/.git/worktrees/<name>`) is gone while the directory still exists. Its `.git` file then points to a missing directory. I reproduced this by deleting the registration after a re-home: the old code keeps the clone with `source_identity_changed`, the new code removes the old worktree and the clone. I could not inspect the instance, so the exact event that removed that one registration is not confirmed. The check was a false positive in that case, and the fix proves ownership from the `.git` pointer instead. A worktree whose pointer leads outside the clone is still kept. ## Verification - `pnpm exec vitest run src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 28 passed (real Git repositories, embedded Postgres). New cases: - a clean clone whose only extra history is a remote branch and a release tag on a commit outside any branch, both absent from the shared clone, with `.paperclip/` ignored in the clone: deleted; - a clone with a local branch commit that is neither upstream nor in the shared clone: kept (`unique_clone_history`, the commit is in `cloneKeptPaths`); - a clone with a tracked change, then an ignored non-regenerable file next to an ignored `node_modules`, then an untracked `build/notes.txt`: each kept (`clone_checkout_has_files` with the path), then deleted after cleanup; - a clone with a second registered worktree: kept (`base_has_worktrees`); - a clone whose detached `HEAD` holds the only copy of a commit (reflogs expired): kept (`unique_clone_history`); - an old worktree whose registration was pruned: old worktree and clone removed; - an orphan whose `origin` was rewritten is kept with `source_identity_changed`, and with `unproven_repository` once no workspace records its repository; - 1,001 ignored directories keep a clone (`clone_file_budget`); - orphan sweep: a batch of one returns a resume position and the next batch finishes the walk; a dry run changes nothing. Then an orphan whose history is in the shared clone or on the remote is deleted, and an orphan with no workspace rows and an unpushed commit is kept, with the next deadline one day later; - a workspace with 65 MiB untracked and ignored files (default capture refuses them with `restore_point_too_large`) moves with an exact snapshot match. - After rebasing on #17407 (nested clone migration, which reuses helpers from the re-home module): `isolated-repository-rehome`, `nested-clone-migration`, `execution-workspaces-service`, `execution-workspaces-routes`, `execution-workspace-reopen`, `execution-workspace-compaction`, `execution-workspace-policy`, `execution-workspace-runtime-control-conflict`, `execution-workspace-runtime-lease-route`, `git-workspace-expiration`, `git-restore-point`, `git-restore-point-expiration`: 405 passed; typecheck passes. - Related suites in `server/` (before the rebase): `git-workspace-expiration`, `git-restore-point`, `git-restore-point-expiration`, `execution-workspace-reopen`, `execution-workspaces-service`, `execution-workspace-compaction`, `heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`, `git-reproducible-managed-checkouts`, `isolated-remote-workspace-source`, with the new suite: 398 passed. - `pnpm run typecheck` in `server/` passes. - Mutation checks, each failing a test: - remote proof replaced by "always unique": the unfetched-history and orphan tests fail; - remote proof replaced by "never unique": the local-commit and orphan tests fail; - `.git` pointer fallback removed: the pruned-registration test fails; - re-home limits removed: the large-file test fails; - orphan sweep removed from the reaper: the orphan test fails; - worktree count check removed: the second-worktree test fails; - regenerable-directory exception applied to untracked files: the dirty-checkout test fails; - orphan batch limit removed: the orphan test fails; - orphan URL taken from the clone again: the rewritten-origin test fails; - ignored directory budget removed: the many-directory test fails; - `HEAD` left out of the checked commits: the detached-HEAD test fails. - Dropping only `--no-empty-directory` does not fail a test, because the per-directory expansion also accepts an empty ignored directory. The two mechanisms are redundant on purpose. - Not run: the full server suite, or the change on a live instance. ## Risks - Deleting a clone now depends on reading the remote when some local commit is neither on a remote-tracking ref nor in the shared clone. If the remote cannot be read, the clone is kept. - Commits that only remote-tracking refs hold (for example a remote branch deleted upstream after the clone fetched it) are treated as not unique, as before. They were pushed once. - The orphan sweep deletes clones that no workspace refers to. A closed workspace that is expired with its worktree removed does not count as a user, because reopening it rebuilds from the shared clone or clones again. - Re-home transfers can now use up to about 1.5 GiB of memory for a single restore point payload, so a very large workspace uses more memory during its move than before. - The orphan sweep runs once per server start and then only while it keeps an orphan, so a clone that becomes orphaned later waits for the next restart. - No migration. No schema change. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file editing, Git, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
36 KiB
Recoverable disk cleanup
The server materializes immutable skill versions under
<instance>/skills/<company>/__versions__/<skill>/<version>. These are disposable
disk copies: the company-scoped version and repository snapshot rows retain the
contents, including binary files and executable flags. Normal skill resolution
rebuilds a missing copy from the database without contacting its upstream source.
Skill version caches
An hourly background sweep considers at most 50 directories per batch, yielding between batches. It starts after server readiness, and respects the existing worker shutdown, standby, and idle-drain controls. A non-current version becomes eligible after one hour without use. Current versions, active-run pins and unsettled recovery pins are retained. If an active run's version selection is unknown, that company's caches remain. Non-versioned skill directories and database snapshots are never removed by this worker.
Last-use records live separately under __version_access__. Older installations
without those records receive a full one-hour observation period. Preparation,
pin persistence and eviction share a company-scoped database lock across server
processes. Snapshot validation and byte counting run outside that lock; cleanup
rechecks current versions, run pins, last use, and directory identity before rename. Audits hold the lock through their file reads. Eviction atomically
moves a cache into unique .trash-<UUID> directories; trash deletion uses its own lock so normal
preparation can restore files while deletion finishes. A later sweep finishes
interrupted trash removal without deleting a newly materialized version.
Controls:
| Environment variable | Default | Effect |
|---|---|---|
PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED |
true |
Set false to stop future sweeps; restoration stays available |
PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN |
false |
Set true to report candidates without changing cache or access records |
The internal skillVersionRetentionService supports company-scoped diagnosis,
bounded continuations, cancellation between entries, and dry run. Explicit agent
pins stay cached. Operators can also retain newest versions with
PAPERCLIP_SKILL_VERSION_RETENTION_KEEP (default 0), or change the one-hour
window with PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES (default 60). Each batch reports
candidate and reclaimed allocated bytes, eviction counts, and skip reasons in
the server log. Compare volume usage before and after to verify physical savings;
concurrent writes and filesystem allocation can affect the net change. A dry run
does not start the legacy observation period.
Roll out with dry run enabled, inspect candidates/skips, then enable removal. Recovery archives and runner backups have separate ownership and retention; this worker never scans or removes them.
Native session failover backups
After each remote turn, the server copies the sandbox's harness state (runner
state plus the provider's session home, such as codex-home) to
<instance>/runtime/paperclip-runner/durable-sessions/<scope>/failover-backups/current.
A replacement sandbox restores the session from this copy, and an ephemeral sandbox
restores it at the start of every turn. Each session keeps one copy, and a single
copy can reach several gigabytes. Codex copies omit credentials, scratch
directories, and data Codex downloads again on startup: remote plugin bundles under
plugins/cache/<remote marketplace>, the remote plugin catalog cache, and plugin
install staging. After a restore, remote plugins can be unavailable to a turn until
Codex finishes downloading them.
An hourly background sweep expires the copy once the session can only resume by reopening closed work. It checks at most 200 sessions and expires at most 50 per batch, starting two minutes after readiness. A copy expires when all of these hold:
- every issue the session ran on is
doneorcancelled, and closed before the window; - the copy itself was written before the window;
- no run of the session is queued, running, or unfinished;
- every native run finalization of the session is settled, with no recovery in progress;
- no active, released, retained, or pending-cleanup environment lease references the copy;
- the session is not executing, initializing, warm, or closing in this process, and no cleanup activation is pending in its directory.
A session whose control-plane state is missing, unreadable, or larger than 4 MB keeps its copy, because the sweep cannot identify its runs.
The sweep checks and locks these rows in one transaction, and removes the agent task
session that points at the session there. A reopened issue therefore starts a fresh
native session instead of resuming the expired one. The copy is atomically renamed to
failover-backups/.retiring-<UUID>, and a final check looks for a run of the session.
If one appeared, or the check fails, the rename is undone. Only a confirmed copy is
renamed to .expired-<UUID> and deleted asynchronously. A later sweep restores an
unconfirmed .retiring-* copy when current is missing, drops it when a newer
current exists, and finishes an interrupted deletion.
The rest of the session directory, including its control-plane state, is kept.
| Environment variable | Default | Effect |
|---|---|---|
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED |
true |
Set false to stop expiring copies |
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN |
false |
Set true to report candidates without deleting copies or task sessions |
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS |
7 |
Days an issue must be closed before its sessions' copies expire (minimum 1) |
Each batch that finds candidates logs them with candidate and reclaimed allocated bytes, cleared task sessions, and skip reasons.
Native workspace sync folders
Each native run that syncs a workspace with a remote sandbox writes a folder at
<instance>/native-workspace-sync/<run>/ with its descriptor, manifest, and the
workspace and git seed archives. A seed can reach more than a gigabyte. Only the
same run reads the folder again: restart recovery, finalization export, board export
retry, and retained workspace recovery. Later runs adopt warm workspaces from lease
metadata and the shared generations cache, which this worker never touches.
A retention worker on the shared delivery-work scheduler removes a run's folder once nothing can read it. It runs at startup and at real deadlines: the earliest settled run's finish (or an orphan folder's newest file) plus the retention window, a recheck one window later for folders it had to keep, and a 15-minute retry after a failure, a held finalization lock or a leftover it could not delete. With no folders it schedules nothing; when a run then creates its folder in this server process, the worker wakes and schedules that folder's deadline. A folder named for a run id, that is a real directory and whose newest file is older than the window, is removed when one of these holds:
- no run with that id exists;
- the run has no valid workspace sync reference and is settled: it
succeeded,failed, wascancelled,timed_outorinterrupted, finished before the window, has no live process, no live controller lease and no finalization owner receipt; - the run is settled as above, and also:
- its native run finalization, if any, has no lease owner or scheduled retry and is
committed, orterminal_failurewith no recovery state orblocked; - no active or escalated workspace export or owner-unverified recovery action names the run;
- no lease of the run carries
nativeWorkspaceExportResumeor a cleanup in flight; - its referenced lease was destroyed, or no retained workspace recovery of the run is scheduled, running, awaiting evidence, queued or recovering;
- historical recovery adoption has no pending work for the run.
- its native run finalization, if any, has no lease owner or scheduled retry and is
Failed native runs can keep their recorded process ID after they finish. A recorded process blocks removal only while it can still be the run's own process: the run has a local lease or no lease, and the process ID or process group still exists on this host with the recorded start time (or a start time that cannot be read). A process ID from a remote sandbox, or one the operating system reused, does not block removal.
Each folder is checked in one transaction that holds the run's finalization
ownership lock, so it never races a finalization or export of the same run. A held
lock skips the folder. Inside that transaction the folder is renamed to
.expired-<UUID>, and it is deleted after commit. The next sweep finishes an
interrupted deletion. Each batch tries at most 50 leftovers, then checks at most 200
folders and removes at most 50. Leftovers and folders each continue from where the
previous batch stopped, so a leftover that cannot be deleted does not block others. Deadline discovery reads
run state 200 folders at a time and stops when the server drains or shuts down.
| Environment variable | Default | Effect |
|---|---|---|
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED |
true |
Set false to stop removing folders |
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN |
false |
Set true to report candidates without removing folders |
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS |
7 |
Days a run must be finished, and its folder untouched, before removal (minimum 1) |
Each batch that finds candidates logs them with candidate and reclaimed allocated bytes and skip reasons.
Git workspace restore points
Git cache expiration (PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true) normally removes a
closed issue's worktree only when its remote can reproduce it exactly. With
PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true, it can also remove a worktree that has
unpushed commits, uncommitted or untracked files, or history-only commits, once its issue
tree has been closed for PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS (default 3).
It also removes the worktree of an issue tree that is still open once nothing has
touched it for PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS (default 7, 0 turns this off):
no issue in the tree or bound to the workspace was updated and no run for those issues or
the workspace was created, started or finished in that time, and no run is still live. An
interrupted removal is finished by later sweeps while the issue stays open. The workspace reaper sets a
deadline for that moment, so open trees cost no recurring checks before then. When a run
later finds its issue's workspace removed this way, it reopens the workspace, restore
point included, before using it; if the restore fails, the run fails instead of starting
in a new workspace. A comment or an agent update on the issue reopens it the same way.
Before removal it saves a restore point:
- a git bundle of commits the remote does not have, against a remote-verified base,
including commits only reachable from the reflog or
ORIG_HEAD; - deterministic snapshots of the index and working tree, including untracked files and
non-default file modes. Untracked
.envand.env.*files are left out, because provisioning recreates them with current secrets; - a separate snapshot of ignored files, such as build caches or files an agent kept out of
git. It leaves out
.envfiles, nested.paperclip-repositoriescheckouts (they get their own restore points),.claude/settings.local.json(saved with the workspace contents) and anything inside a regenerable directory such asnode_modulesordist. A restore point with ignored files uses payload schemav2, which older servers reject instead of restoring without them. Restoring never overwrites an ignored file that already exists, and an ignored nested repository keeps the worktree.
The restore point is encrypted with a per-restore-point key wrapped by the instance
secrets key, uploaded to file storage, read back and hash-checked, then activated in
the same transaction that archives the workspace with a versioned recipe
(paperclip.git-cache-restoration.v2). Older servers reject that recipe instead of
restoring without the saved work. Under the expiration locks, the worktree must still
match the captured snapshot exactly; otherwise expiration rolls back, keeps the
worktree, and drops the restore point.
Nested managed checkouts under .paperclip-repositories/ get their own restore points
when their checked-out branch has unpushed commits or uncommitted, untracked or
history-only work. Each is stored under its checkout name, recorded in the recipe's
contents, and verified against its own snapshot under the expiration locks. A nested
checkout still blocks expiration when another local branch or tag has commits its
remote lacks, when any reflog holds commits that are neither on the remote nor part of
the checked-out history the restore point bundles, when it has objects reachable from
nothing, or when its COMMIT_EDITMSG holds a draft that matches no commit in its
history. A COMMIT_EDITMSG that matches a commit is accepted for every nested checkout.
A nested checkout's remote-tracking refs (refs/remotes/origin/*) and their reflogs can
point at commits the remote has since dropped, for example a branch deleted after a squash
merge. Strict expiration keeps such a checkout. In restore point mode those commits are
saved in the checkout's restore point instead, as history-only objects (the remote-tracking
refs themselves are not recreated), and any change to the remote-tracking refs after capture
rolls expiration back.
Each restore point also records a digest of its HEAD reflog and ORIG_HEAD; under the
expiration locks that history must be unchanged too, so a commit followed by a reset
after capture rolls expiration back. Reopening
re-clones the checkout, imports its bundle, checks out the saved commit and applies the
restore point. Submodules, special git state and ignored files other than .env
still keep a worktree from expiring.
Restore points are refused when the file storage is s3 and the secrets master key
does not come from PAPERCLIP_SECRETS_MASTER_KEY.
Nested project repository worktrees
Extra project repositories that have no configured local folder are no longer full
clones inside every task. Each task's .paperclip-repositories/<name>-<key> checkout is a
Git worktree of one shared clone per company, project and repository URL: the managed
project checkout for that URL under the instance's projects/ directory (the same clone
that main task worktrees use when the repository is a task's primary source). Objects are
stored once in that clone, so a new task adds only its working files. Repositories with a
configured local folder are still seeded as independent copies. Existing standalone
nested clones keep working unchanged until the optional conversion below turns them into
worktrees of the shared clone.
Each nested worktree checks out its own branch, paperclip/<id>/<branch>, where <id>
derives from the worktree path and <branch> is the remote's current default branch
(refreshed before each new worktree) or the configured branch ref, tracking
origin/<branch>. Reopening derives the upstream from the saved source ref, not from the
branch name. A tag or commit ref is checked out detached, as
before. Branch names therefore never collide between tasks. When a checkout kept under
.paperclip-runtime/detached-repositories/ still holds that branch, the new worktree
uses paperclip/<id>/r<n>/<branch> instead, so the retained work is never reset. git pull works as before;
pushing needs an explicit destination such as git push origin HEAD:<branch>. Adding the
worktree, fetching and removing registrations are serialized per shared clone. Each new
worktree first fetches with --prune, so a branch deleted upstream is rejected as a fresh
clone would reject it. Server-side Git commands on the shared clone run with hooks
disabled, so hooks an agent plants there never run in the server.
Branches and reflogs of a worktree live partly in the shared clone. Expiration and restore
points therefore scope their checks to what belongs to the nested worktree: its HEAD,
HEAD reflog, ORIG_HEAD, index and working tree (stored per worktree) and every branch
under its own paperclip/<id>/ prefix together with that branch's reflog. Those must be on
the remote or saved in a restore point exactly as for a standalone checkout. Checking out
any other branch, stashes, unsupported refs or custom local config in the shared clone
still block expiration. Branches outside the prefix, other tasks' branches and their
history stay in the shared clone and never block or get deleted. Commits that existed only
in the nested worktree's HEAD reflog are written back into the restored worktree's HEAD
reflog, so they stay protected after reopening.
Expiring a task removes the nested worktree's registration from the shared clone and
deletes its own branch only at the verified recorded commit. Reopening recreates the
worktree from the shared clone (re-cloning the shared clone if it is gone) on the same
branch and commit and applies its restore point. Deleting or archiving a task removes the
registration of each of its nested worktrees and deletes their branches only when every
commit is on a remote-tracking ref; an unpushed branch stays in the shared clone. Moving a
nested worktree to .paperclip-runtime/detached-repositories/ when its repository leaves
the project repairs its registration.
Converting existing standalone nested clones
Tasks prepared before nested worktrees existed still hold a full standalone clone in each
.paperclip-repositories/<name>-<key> checkout, often of the same large repository many
times over. With PAPERCLIP_NESTED_CLONE_MIGRATION_ENABLED=true, the terminal workspace
reaper converts each such checkout in an open task into a worktree of the project's shared
clone and deletes the standalone clone. It holds the source lock and workspace lifecycle
lock that git cache expiration holds, the per-shared-clone lock that nested worktree
creation holds, and the expiration lock files (index.lock, HEAD.lock and the current
branch's ref lock) in the clone, so no run,
expiration, reopen, restore or other conversion against the same shared clone can
interleave. A checkout is converted only when:
- no run, native finalization, environment lease or runtime service is still live for the workspace, and it is not waiting for a reopen to be consumed (the issue may be open);
- its name is the managed checkout name of one of the project's repository workspaces
(
unknown_managed_checkoutotherwise), that repository has no configured local folder (local_source_checkout), and the clone'soriginis exactly the repository URL (foreign_origin); - the project has a verified shared clone for that URL (
no_shared_base,shared_base_repo_mismatch) whose local config runs no filters or commands (unsupported_base_git_config); - the clone holds only branches, tags and
originremote-tracking refs, no stash (nested_clone_has_stash; stashes are not converted, becauserefs/stashwould be shared by every worktree of the shared clone), no worktrees of its own, no custom hooks, no local config beyond the stock keys,originand branch tracking, noinfo/excludepattern the shared clone lacks (unsupported_info_exclude), no in-progress merge, rebase or similar state, no hidden index flags, and no commit that only another branch's reflog holds (unique_branch_reflog_commits); - nothing a restore point refuses (submodules, filtered paths, special files, untracked nested repositories, oversized contents or history).
The conversion fetches the shared clone, then brings the clone's history in under the
checkout's own prefix: the current branch becomes paperclip/<id>/<branch> at the same
commit, tracking origin/<branch> (or the branch's configured origin upstream); every
other local branch whose tip the shared clone's remote-tracking refs do not contain becomes
paperclip/<id>/<branch>, every such tag becomes paperclip/<id>/tags/<tag> at the
tagged commit, and every such origin remote-tracking ref (for example one whose branch was
deleted or force-pushed upstream) becomes paperclip/<id>/remotes/origin/<branch>. <id> derives from the final checkout path, so later expiration and release
treat those branches exactly like a nested worktree's own branches: they block expiration
until pushed or saved and are never deleted while unpushed. A restore point of the clone,
captured against commits both repositories share and preserving those branch tips, its
HEAD reflog and ORIG_HEAD, is imported into the shared clone. Other tasks' branches and
worktrees are never touched; a name that already exists in the shared clone skips the
checkout (nested_clone_ref_conflict).
The new worktree is added at <name>.clone-migrating beside the clone, the restore point
is applied (index, worktree changes, untracked and ignored files, file modes), the HEAD
reflog and ORIG_HEAD are copied, and regenerable directories, untracked .env files and
.claude/settings.local.json are moved in by rename. The two must then match exactly
(HEAD, index, worktree, ignored files, modes, reflog, branch and every created ref). Any
difference or failure before the swap moves the items back, removes the new worktree and
the created branches, and leaves the clone untouched. The swap renames the clone to
<name>.clone-standalone, moves the new worktree into <name> with git worktree move
and runs git worktree repair in the shared clone. Then one database update records the
worktree in a saved restoration recipe that lists the checkout, and the old clone is
renamed to <name>.clone-trash and deleted.
Before it changes the shared clone, the conversion records
nestedCloneMigrationPending.<name> in workspace metadata (shared clone, HEAD, created
refs, moved items and phase). The next pass uses it after a crash: before the swap it moves
items back, removes the new worktree and the created branches, then converts again; once
the clone has been renamed aside it finishes the swap and deletes the old clone, so work done
in the converted checkout after a crash is kept. A state that matches neither (for example a
new checkout created at <name> while the clone was aside) is left alone with
nested_migration_recovery_conflict. The reaper also finishes a pending record after the
workspace is archived; if the task worktree is already gone, it forgets the record, removes
the stale registrations and deletes only created branches whose commits are on the remote.
Expiration treats the .clone- names as unexplained
files and keeps the workspace until the conversion settles. A dry run reports a pending
record without acting on it.
Each attempt logs event: "execution_workspace.nested_clone_migration" with the checkout
name, outcome (migrated, recovered, rolled_back, dry_run or skipped), skip reason,
new branch, candidate bytes (the clone's .git) and reclaimed bytes. A workspace is
inspected again at most hourly, and a retry deadline it registered survives later sweeps
during that hour. A retry deadline is registered only while a pending record
exists or the skip comes from a live consumer, a busy lock or a changed workspace row; other
skips, including a change caught by verification, wait for a later sweep. The reaper looks
only at checkouts named after the project's configured repositories (plus pending records),
never at arbitrary directories, and attempts at most 8 per workspace per sweep. Pending
recoveries go first, and the rest rotate across sweeps. While a rotation has not yet visited
every checkout it registers a retry deadline; after a full rotation without progress it waits
for a later sweep, until a conversion succeeds or the set of candidate checkouts changes. PAPERCLIP_NESTED_CLONE_MIGRATION_DRY_RUN=true reports candidates
without changing anything and registers no deadlines.
Reopening the issue, or the next run that recreates the worktree, imports the bundle, re-adds the worktree at the saved commit, applies the restore point and verifies the result. The bundle is always imported, so history-only commits are present again as objects in the repository; they are not recreated as branches or reflog entries, and the restore point keeps them until it expires. A draft commit message left by an aborted commit still keeps the worktree from expiring, because restore points do not save it. A worktree that already matches is only marked applied, and a worktree that diverged is never overwritten: the restore fails closed and can be retried. A missing or corrupt restore point also fails closed.
When a restore point has expired, through retention or on a nested checkout, reopening
starts fresh instead: the worktree is recreated on its branch (the branch tip if the
repository still has it, otherwise a new branch from the base ref), the project's nested
repositories are cloned fresh before the provision command runs, the saved recipe is
dropped, and a system comment on the issue warns that unpushed and uncommitted work from
before the cleanup is no longer available. The same warning is returned with the reopen
result and recorded on the execution_workspace.reopened activity.
Restore points are encrypted saves of the parts of a git worktree that its remote
cannot reproduce: unpushed commits and uncommitted files. They let a closed issue's
worktree be removed and restored later. The table execution_workspace_restore_points
tracks each one with its storage key, hash, size and status
(pending, active, expiring, superseded, delete_requested, deleted). The objects live in
the instance's configured file storage (local_disk or s3).
An active restore point's delete_after is either its expiry time or empty while its
source issue tree is open. A database trigger on issues keeps this current: when an
issue's status or parent changes, or an issue is deleted, it marks every active restore
point whose source issue is that issue, its old or new parent, or any of their ancestors
as due now, in the same transaction. A retention worker on the shared delivery-work
scheduler runs at startup, when issue changes settle while restore points exist, and at
real deadlines: the earliest expiry, the end of an abandoned capture's grace period, or
a failed deletion's retry. With nothing due, it schedules nothing, and open trees cause
no recurring checks. At review it checks the source issue tree:
- Tree still open: clear
delete_afterand wait for the next issue change. - Tree
doneorcancelled: expirePAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS(default90, minimum1) after the tree's latest close, immediately if that time has passed. Expiry first moves the row toexpiring, runs the expiry hook, and only then requests deletion. Anexpiringrow's object is never deleted; after a restart or a hook failure the hook is retried, after 15 minutes when it failed. - Source issue gone: delete.
Deleting an issue or a company marks its restore points delete_requested in the same
transaction and notifies the worker, so their objects are removed promptly. A
pending restore point older than one hour was abandoned mid-capture and is deleted.
superseded and delete_requested restore points have their stored object removed,
then become deleted once the storage provider confirms the object is gone. A
storage failure is recorded in last_error and retried after 15 minutes. Work is
processed in batches of 50; a full batch continues immediately.
Rows have no foreign keys, so they outlive a deleted workspace, issue or company long enough to remove the stored object. Each run that changes anything logs its counts.
Isolated repository sources
For a while, workspace preparation gave some isolated git_worktree workspaces their
own full clone of the project repository under
instances/<id>/isolated-repository-sources/<companyId>/<issueId>/checkout, with the
issue's worktree inside it. Workspace metadata marks these with isolatedRemoteSource.
New workspaces no longer get one; they are worktrees of the project's shared managed
clone. Each old clone still holds a full .git and an unused default-branch checkout.
With PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true (and git cache expiration enabled),
the terminal workspace reaper moves each such open workspace onto the shared clone and
then deletes its private clone. It runs under the same source lock, workspace lifecycle
lock and Git lock files as git cache expiration, so no run, expiration, reopen or
restore can interleave. A workspace is moved only when:
- no run, native finalization, environment lease or runtime service is still live for it, and it is not waiting for a reopen to be consumed (the issue may be open);
- the project has a shared managed clone for the same
repoUrl; - its worktree sits at the standard
.paperclip/worktrees/<branch>location, and the branch is absent from the shared clone, points to an ancestor of the worktree's HEAD there, and is not checked out in another of its worktrees; - the shared clone's local config sets no filter drivers, includes, attribute files, worktree
config or command settings (
core.fsmonitor,core.hooksPath,core.sshCommandand similar), and it has noinfo/attributes, because the move checks out files there; - its Git state is transferable: no stash or replacement refs, no in-progress merge, rebase or other worktree-specific state, no hidden index flags, no shallow history, and nothing a restore point refuses (submodules, filtered paths, special files, untracked nested repositories). The transfer is local, so it uses much larger restore point limits than uploads do: 500,000 files, 1 GiB per file and in total, a 1.5 GiB bundle and 1,000,000 commits.
The move captures a restore point of the old worktree against commits both repositories
share, imports it into the shared clone, creates or fast-forwards the issue branch, adds
the new worktree at the same HEAD and applies the restore point (index, worktree changes,
untracked and ignored files, file modes). The HEAD reflog and ORIG_HEAD are copied.
What restore points leave out is moved by rename on the same filesystem: regenerable
directories such as node_modules, untracked .env and .env.* files,
.claude/settings.local.json, and each nested .paperclip-repositories/* checkout as a
whole. A standalone nested clone moves as is. A nested checkout that is a worktree of its own
shared clone moves too, and the move runs git worktree repair in that clone so its
registration follows the new path. Its paperclip/<hash>/ branch is renamed to the prefix of
the new path, because that prefix proves the checkout owns the branch (a rollback moves it
back, repairs it and renames the branch back). Items inside a moved nested checkout, such as
its node_modules, move with it. Any other
ignored file that a restore point does not save keeps the workspace in
place.
The new worktree must then match the old one exactly (HEAD, index, worktree, ignored files,
modes and reflog), and every moved item must exist. Only then does one database update
point the workspace at the new path, drop isolatedRemoteSource, rewrite a saved
restoration recipe to the shared clone, and record isolatedRepositoryRehome
({ at, from, to, cleanedAt }). Any failure before that update removes the new worktree,
moves items back, restores the branch, and leaves the old worktree untouched. Before it
changes the shared clone, the move records isolatedRepositoryRehomePending (target, branch,
previous branch tip and planned moves). If the server stops before the update, the next
attempt uses that record to move items back, remove the new worktree and restore the
branch, then starts again. A dry run reports such a record without undoing it. If an item
exists in both places (for example a process recreated .env in the old worktree), the
rollback stops with rehome_rollback_conflict and keeps the new worktree and the record, so
neither copy is lost; later attempts retry the rollback once the conflict is resolved. A new worktree is removed only when the move created it: a
target that appears from elsewhere is never deleted.
After the update, the old worktree is removed (its ownership is proven by git rev-parse
or, when its registration in the clone was already pruned, by its .git pointer into the
clone's worktrees directory). The clone is then renamed to trash and deleted when nothing
else needs it and everything in it is reproducible:
- no other workspace or project workspace refers to it (a workspace already expired with its worktree removed does not count) and no runtime service runs in it;
- after
git worktree prune, it has no worktree besides its own checkout; - its own checkout has no tracked changes and no untracked files (only Paperclip's
compaction trash is allowed, even inside a directory with a regenerable name), and its
ignored files are only inside regenerable directories (such as
node_modules) or Paperclip's compaction trash. Empty directories, such as the.paperclip/worktreesparent left by the old worktree, do not count; - it has only branch, tag and remote-tracking refs (no stash, notes or replacement refs);
- every commit that its local branches, local tags, reflogs (other than remote-tracking
ones) or
ORIG_HEADhold is on one of its remote-tracking refs, present in the shared clone, or found on the remote. The remote is read fresh (branches and tags) with the company's Git identity, falling back to anonymous access, and only when some commit is in neither place.HEADitself counts too, so a detached checkout is covered. Remote branches and release tags that the shared clone never fetched do not keep a clone.
A clone with more than 5,000 refs, reflog entries or such commits is kept
(clone_history_budget), and so is one with more than 1,000 ignored directories outside
regenerable ones (clone_file_budget); those directories are read in one Git call. The log entry names up to five paths, worktrees, refs or commits
that kept a clone (cloneKeptPaths). A clone another workspace still uses is kept
(cloneRetained); that workspace's own move removes it later. If the server stops after
the update, a later sweep finishes the removal and sets cleanedAt, also after the
workspace has been archived.
The reaper also sweeps orphaned clones under isolated-repository-sources/ once per server
start: clones that no execution workspace refers to (no open workspace with matching
isolatedRemoteSource, and no closed one that would need the clone to reopen) and that no
pending move or cleanup still owns. The repository must be proven by the database: the
workspaces of that issue must record exactly one repository URL, and the clone's origin
must equal it (unproven_repository or source_identity_changed otherwise), so a rewritten
origin is never fetched or trusted. It applies the same reproducibility check, using the
shared clone of a workspace of the same issue and repository when one exists, and logs
event: "execution_workspace.isolated_source_orphan_cleanup". It inspects at most eight
issue directories per reaper sweep and continues from that position in the next sweep, so a
slow remote cannot hold up other workspace cleanup. When the walk ends and it kept any
orphan, it looks again a day later; otherwise it registers no further work. It also deletes leftover
trash directories from interrupted removals.
Each attempt logs event: "execution_workspace.isolated_source_rehome" with the outcome,
skip reason, candidate bytes and reclaimed bytes. A workspace that is skipped, or whose
clone is kept for a reason that can change, is retried an hour later through the reaper's
existing deadlines; nothing runs while there is no candidate.
PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN=true reports candidates without changing
anything.
Re-homed workspaces get a new path. An agent that resumes an older session may still remember the old path; the run's working directory comes from the updated workspace.
Reopening or restoring an expired workspace whose recipe came from a private clone no longer recreates that clone when the project has a shared managed clone for the same repository, its branch there is absent or at the saved commit, and the shared clone has every commit the saved restore point bundle needs as a base. A recipe with worktree-backed nested checkouts on branches keeps the old behavior, because their branch names are bound to the old path. The workspace is rebuilt as a worktree of the shared clone at the standard location, with the saved commit and restore point imported and applied as before. Without a shared clone, the old behavior applies.