PaperClipAI/doc/recoverable-disk-cleanup.md

Recoverable disk cleanup

The server materializes immutable skill versions under <instance>/skills/<company>/__versions__/<skill>/<version>. These are disposable disk copies: the company-scoped version and repository snapshot rows retain the contents, including binary files and executable flags. Normal skill resolution rebuilds a missing copy from the database without contacting its upstream source.

Skill version caches

An hourly background sweep considers at most 50 directories per batch, yielding between batches. It starts after server readiness, and respects the existing worker shutdown, standby, and idle-drain controls. A non-current version becomes eligible after one hour without use. Current versions, active-run pins and unsettled recovery pins are retained. If an active run's version selection is unknown, that company's caches remain. Non-versioned skill directories and database snapshots are never removed by this worker.

Last-use records live separately under __version_access__. Older installations without those records receive a full one-hour observation period. Preparation, pin persistence and eviction share a company-scoped database lock across server processes. Snapshot validation and byte counting run outside that lock; cleanup rechecks current versions, run pins, last use, and directory identity before rename. Audits hold the lock through their file reads. Eviction atomically moves a cache into unique .trash-<UUID> directories; trash deletion uses its own lock so normal preparation can restore files while deletion finishes. A later sweep finishes interrupted trash removal without deleting a newly materialized version.

Controls:

Environment variable	Default	Effect
PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED	true	Set false to stop future sweeps; restoration stays available
PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN	false	Set true to report candidates without changing cache or access records

The internal skillVersionRetentionService supports company-scoped diagnosis, bounded continuations, cancellation between entries, and dry run. Explicit agent pins stay cached. Operators can also retain newest versions with PAPERCLIP_SKILL_VERSION_RETENTION_KEEP (default 0), or change the one-hour window with PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES (default 60). Each batch reports candidate and reclaimed allocated bytes, eviction counts, and skip reasons in the server log. Compare volume usage before and after to verify physical savings; concurrent writes and filesystem allocation can affect the net change. A dry run does not start the legacy observation period.

Roll out with dry run enabled, inspect candidates/skips, then enable removal. Recovery archives and runner backups have separate ownership and retention; this worker never scans or removes them.

Native session failover backups

After each remote turn, the server copies the sandbox's harness state (runner state plus the provider's session home, such as codex-home) to <instance>/runtime/paperclip-runner/durable-sessions/<scope>/failover-backups/current. A replacement sandbox restores the session from this copy, and an ephemeral sandbox restores it at the start of every turn. Each session keeps one copy, and a single copy can reach several gigabytes. Codex copies omit credentials, scratch directories, and data Codex downloads again on startup: remote plugin bundles under plugins/cache/<remote marketplace>, the remote plugin catalog cache, and plugin install staging. After a restore, remote plugins can be unavailable to a turn until Codex finishes downloading them.

An hourly background sweep expires the copy once the session can only resume by reopening closed work. It checks at most 200 sessions and expires at most 50 per batch, starting two minutes after readiness. A copy expires when all of these hold:

• every issue the session ran on is done or cancelled, and closed before the window;
• the copy itself was written before the window;
• no run of the session is queued, running, or unfinished;
• every native run finalization of the session is settled, with no recovery in progress;
• no active, released, retained, or pending-cleanup environment lease references the copy;
• the session is not executing, initializing, warm, or closing in this process, and no cleanup activation is pending in its directory.

A session whose control-plane state is missing, unreadable, or larger than 4 MB keeps its copy, because the sweep cannot identify its runs.

The sweep checks and locks these rows in one transaction, and removes the agent task session that points at the session there. A reopened issue therefore starts a fresh native session instead of resuming the expired one. The copy is atomically renamed to failover-backups/.retiring-<UUID>, and a final check looks for a run of the session. If one appeared, or the check fails, the rename is undone. Only a confirmed copy is renamed to .expired-<UUID> and deleted asynchronously. A later sweep restores an unconfirmed .retiring-* copy when current is missing, drops it when a newer current exists, and finishes an interrupted deletion. The rest of the session directory, including its control-plane state, is kept.

Environment variable	Default	Effect
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED	true	Set false to stop expiring copies
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN	false	Set true to report candidates without deleting copies or task sessions
PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS	7	Days an issue must be closed before its sessions' copies expire (minimum 1)
PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS	7	Days continuity-break archives and quarantined session state are kept (minimum 1)

Each batch that finds candidates logs them with candidate and reclaimed allocated bytes, cleared task sessions, and skip reasons.

Continuity-break archives and quarantined session state

The same sweep also removes two kinds of evidence that native sessions leave behind under the same root:

• <session>/continuity-breaks/<epoch ms>-<UUID>/, written when a session starts a fresh provider session and moves the retired session's state aside. Nothing reads these archives again.
• quarantine/<session>.<reason>.<epoch ms>.<pid>.<UUID>/, written when a session's state loses authority. Recovery and failed-run retry still consult these entries.

An entry's age comes from the epoch in its name, or from its modification time when that epoch is in the future. An entry is removed once it is older than PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS and its session is not executing, initializing, warm, or closing in this process. Quarantine entries created by native cleanup (<session>.identity_indeterminate.cleanup.<UUID>) are referenced from recovery history and kept. Symlinks are never followed or removed. Once a quarantined entry is removed, recovery and failed-run retry no longer see it, so a later run of that session may start a fresh provider session instead of refusing.

The evidence check runs on the sweep only when something is due: once at startup, then when the oldest retained entry, or an entry the executor has written since, reaches the window, and on the next sweep after a busy session, a failed removal, or a full batch. It only reads the filesystem and never queries the database. A dry run reports each candidate once until newly written evidence comes due.

An entry is renamed to .expired-<UUID> in its folder before it is deleted, and a later sweep finishes an interrupted deletion. An empty continuity-breaks folder is removed. Archives, quarantine entries, and leftovers count toward the batch limits of 200 entries and 50 removals shared with backup copies, and PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED and _DRY_RUN apply to both.

Native workspace sync folders

Each native run that syncs a workspace with a remote sandbox writes a folder at <instance>/native-workspace-sync/<run>/ with its descriptor, manifest, and the workspace and git seed archives. A seed can reach more than a gigabyte. Only the same run reads the folder again: restart recovery, finalization export, board export retry, and retained workspace recovery. Later runs adopt warm workspaces from lease metadata and the shared generations cache, which this worker never touches.

A retention worker on the shared delivery-work scheduler removes a run's folder once nothing can read it. It runs at startup and at real deadlines: the earliest settled run's finish (or an orphan folder's newest file) plus the retention window, a recheck one window later for folders it had to keep, and a 15-minute retry after a failure, a held finalization lock or a leftover it could not delete. With no folders it schedules nothing; when a run then creates its folder in this server process, the worker wakes and schedules that folder's deadline. A folder named for a run id, that is a real directory and whose newest file is older than the window, is removed when one of these holds:

• no run with that id exists;
• the run has no valid workspace sync reference and is settled: it succeeded, failed, was cancelled, timed_out or interrupted, finished before the window, has no live process, no live controller lease and no finalization owner receipt;
• the run is settled as above, and also:
  • its native run finalization, if any, has no lease owner or scheduled retry and is committed, or terminal_failure with no recovery state or blocked;
  • no active or escalated workspace export or owner-unverified recovery action names the run;
  • no lease of the run carries nativeWorkspaceExportResume or a cleanup in flight;
  • its referenced lease was destroyed, or no retained workspace recovery of the run is scheduled, running, awaiting evidence, queued or recovering;
  • historical recovery adoption has no pending work for the run.

Failed native runs can keep their recorded process ID after they finish. A recorded process blocks removal only while it can still be the run's own process: the run has a local lease or no lease, and the process ID or process group still exists on this host with the recorded start time (or a start time that cannot be read). A process ID from a remote sandbox, or one the operating system reused, does not block removal.

Each folder is checked in one transaction that holds the run's finalization ownership lock, so it never races a finalization or export of the same run. A held lock skips the folder. Inside that transaction the folder is renamed to .expired-<UUID>, and it is deleted after commit. The next sweep finishes an interrupted deletion. Each batch tries at most 50 leftovers, then checks at most 200 folders and removes at most 50. Leftovers and folders each continue from where the previous batch stopped, so a leftover that cannot be deleted does not block others. Deadline discovery reads run state 200 folders at a time and stops when the server drains or shuts down.

Environment variable	Default	Effect
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED	true	Set false to stop removing folders
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN	false	Set true to report candidates without removing folders
PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS	7	Days a run must be finished, and its folder untouched, before removal (minimum 1)

Each batch that finds candidates logs them with candidate and reclaimed allocated bytes and skip reasons.

Git workspace restore points

Git cache expiration (PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true) normally removes a closed issue's worktree only when its remote can reproduce it exactly. With PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true, it can also remove a worktree that has unpushed commits, uncommitted or untracked files, or history-only commits, once its issue tree has been closed for PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS (default 3). For this, only the issues in the tree that use the workspace count: the source issue, and each descendant that is bound to it (execution_workspace_id, which is how a child that inherits the parent's workspace with reuse_existing is recorded) or that selects it as its current or pending existing workspace. The delay runs from the latest close time among those issues. A closed issue's worktree can therefore be removed while descendants that work in their own workspaces stay open. Any issue still bound to the workspace, and any live run, finalization, lease, recovery action or runtime service of an issue in the tree or of the workspace, still keeps it. If an open descendant later binds to the removed workspace, its next run or a comment reopens it from the restore point, as below. The workspace reaper does not compact or archive such a worktree by its normal rules while descendants are open; only expiration removes it. It also removes the worktree of an issue tree that is still open once nothing has touched it for PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS (default 7, 0 turns this off): no issue in the tree or bound to the workspace was updated and no run for those issues or the workspace was created, started or finished in that time, and no run is still live. An interrupted removal is finished by later sweeps while the issue stays open. The workspace reaper sets a deadline for that moment, so open trees cost no recurring checks before then. When a run later finds its issue's workspace removed this way, it reopens the workspace, restore point included, before using it; if the restore fails, the run fails instead of starting in a new workspace. A comment or an agent update on the issue reopens it the same way. Before removal it saves a restore point:

• a git bundle of commits the remote does not have, against a remote-verified base, including commits only reachable from the reflog or ORIG_HEAD;
• deterministic snapshots of the index and working tree, including untracked files and non-default file modes. Untracked .env and .env.* files are left out, because provisioning recreates them with current secrets;
• a separate snapshot of ignored files, such as build caches or files an agent kept out of git. It leaves out .env files, nested .paperclip-repositories checkouts (they get their own restore points), .claude/settings.local.json (saved with the workspace contents) and anything inside a regenerable directory such as node_modules or dist. A restore point with ignored files uses payload schema v2, which older servers reject instead of restoring without them. Restoring never overwrites an ignored file that already exists, and an ignored nested repository keeps the worktree.

The restore point is encrypted with a per-restore-point key wrapped by the instance secrets key, uploaded to file storage, read back and hash-checked, then activated in the same transaction that archives the workspace with a versioned recipe (paperclip.git-cache-restoration.v2). Older servers reject that recipe instead of restoring without the saved work. Under the expiration locks, the worktree must still match the captured snapshot exactly; otherwise expiration rolls back, keeps the worktree, and drops the restore point.

Nested managed checkouts under .paperclip-repositories/ get their own restore points when their checked-out branch has unpushed commits or uncommitted, untracked or history-only work. Each is stored under its checkout name, recorded in the recipe's contents, and verified against its own snapshot under the expiration locks. A nested checkout still blocks expiration when another local branch or tag has commits its remote lacks, when any reflog holds commits that are neither on the remote nor part of the checked-out history the restore point bundles, when it has objects reachable from nothing, or when its COMMIT_EDITMSG holds a draft that matches no commit in its history. A COMMIT_EDITMSG that matches a commit is accepted for every nested checkout. A nested checkout's remote-tracking refs (refs/remotes/origin/*) and their reflogs can point at commits the remote has since dropped, for example a branch deleted after a squash merge. Strict expiration keeps such a checkout. In restore point mode those commits are saved in the checkout's restore point instead, as history-only objects (the remote-tracking refs themselves are not recreated), and any change to the remote-tracking refs after capture rolls expiration back. Each restore point also records a digest of its HEAD reflog and ORIG_HEAD; under the expiration locks that history must be unchanged too, so a commit followed by a reset after capture rolls expiration back. Reopening re-clones the checkout, imports its bundle, checks out the saved commit and applies the restore point. Submodules, special git state and ignored files other than .env still keep a worktree from expiring. Restore points are refused when the file storage is s3 and the secrets master key does not come from PAPERCLIP_SECRETS_MASTER_KEY.

Isolated task directories

Tasks without a project repository run in a plain (non-Git) directory at <instance>/isolated-workspaces/<companyId>/<issueId>/. Its execution workspace row has strategy project_primary, provider local_fs and that exact path as cwd, in isolated_workspace or shared_workspace mode. Agents can leave the only copy of their work there, such as reports or generated files, because nothing pushes it anywhere.

Before this cleanup existed nothing removed these directories. The terminal workspace reaper archives a workspace only after proving delivery through Git (a merged pull request or a merged branch); for a directory that is not a Git checkout the status inspection fails, so the row was always skipped as undelivered. A workspace archived by hand keeps its directory too: artifact cleanup removes a local_fs directory only when the row says the runtime created it, and task directory rows never say so. Deleting them on archive would discard work silently, so retention saves the files first instead.

With PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED=true the terminal workspace reaper handles an open task directory row once every issue in its tree has been done or cancelled for PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS (default 3), the same delay git restore points use. It registers that moment as its deadline, so a closed tree costs no checks before then. It removes the directory only when all of these hold, checked under the issue's source lock and the workspace lifecycle lock:

• the source issue is still bound to the workspace (issue_not_bound otherwise), because reopening restores through that binding;
• no run of the tree or the workspace is queued, running or unfinished, no native run finalization is unsettled, no environment lease or runtime service is live, no recovery action is unresolved and the workspace is not waiting for a reopen to be consumed (live_or_unresolved_consumer); these are the checks git cache expiration uses;
• no other open workspace or project workspace points into the directory (closed rows for the same issue do not count);
• the directory is a real directory, not a symbolic link (unsafe_task_directory).

It then saves a restore point of the directory: every file with its mode, every directory and every symbolic link (the link itself; links are never followed). A node_modules directory next to a package.json is left out, because installing the package recreates it. Sockets, devices and other special files keep the directory (unsupported_filesystem_entry), and so does a directory over the restore point limits (task_directory_too_many_files, task_directory_file_too_large, task_directory_too_large; 20,000 files, 64 MiB per file, 256 MiB in total). The captured snapshot is decoded once before anything is removed; a directory whose snapshot could not be restored stays (task_directory_not_restorable). The restore point uses the git restore point ledger, storage and encryption: it is sealed with a per-restore-point key wrapped by the instance secrets key, uploaded to file storage under git-restore-points/<workspace>/<generation>/task-directory/, read back and hash-checked, and activated in the same transaction that archives the workspace and records the isolatedTaskDirectoryRetention state in its metadata. It follows the git restore point retention (PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS, default 90), counted from the later of the tree's close and the removal, so a task that closed long ago still keeps its files for the full period. Restore points must be enabled (PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true, restore_points_disabled otherwise), and they are refused when the file storage is s3 and the secrets master key does not come from PAPERCLIP_SECRETS_MASTER_KEY.

Right before deletion the directory is read again. If anything changed since the capture, the workspace is made active again, the restore point is dropped and the directory stays (task_directory_changed). Otherwise the directory is renamed to a recorded .expired-<random> name beside it, no longer than the original name, and deleted. The archive and the removal are recorded as execution_workspace.task_directory_archived and execution_workspace.task_directory_removed activity. A removal the server could not finish is finished by a later sweep.

Reopening the workspace restores the directory before anything uses it: a comment or an agent update on an issue bound to a closed isolated_workspace, or the next run of the issue in either mode. The files are written to a .restoring-<random> staging directory beside it, whose name is no longer than the original, verified against the saved digest and renamed into place. An empty directory in the way is replaced; a non-empty one that does not match keeps the workspace closed and fails the reopen instead of overwriting it. After a successful reopen the restore point is superseded and removed; the next retention saves a new one. When the restore point has already expired, the reopen starts with an empty directory and posts a system comment on the issue saying the earlier files are no longer available.

Orphaned task directories

The reaper also walks isolated-workspaces/ once per server start, at most 8 issue directories per sweep, continuing from that position in the next sweep. It never follows symbolic links. For each directory:

• an open execution workspace pointing at or into it keeps it (the reaper handles that row as above);
• a closed workspace row that its issue is still bound to is handled like an open row: the directory is saved in a restore point on that row and removed after the terminal delay, and reopening restores it (outcome preserved_and_removed);
• any other directory is an orphan: its workspace row is missing, or archived or cleanup_failed without the issue bound to it. An orphan is removed when its issue is gone, or every issue in the tree has been closed for PAPERCLIP_TASK_DIRECTORY_ORPHAN_GRACE_DAYS (default 7), and nothing in it changed during that period, and no project workspace, live runtime service, unsettled run, finalization, live lease or unresolved recovery action of the issue refers to it. Leases count when they belong to a run of the issue tree, to an issue in the tree, or to any workspace row of the directory. An orphan is not saved first: no workspace row can reopen it. If its issue is reopened later without another workspace, its next run starts in an empty directory.

After that first walk, the reaper looks only at the directories that changes point to: a workspace row that becomes archived or cleanup_failed, or an issue that becomes done or cancelled, sends the reaper to that issue's directory and the directories of its ancestors, whose trees may have just closed. A directory it kept is checked again at its own deadline: the end of the grace period or terminal delay, or an hour later when a lock was busy or a live run, lease, service, finalization or recovery action still used it. A directory whose issue is open waits for the issue to close. With nothing pending it registers no work. The walk also deletes leftover .expired-* and .restoring-* directories older than an hour; when deleting an orphan's trash fails, another walk is scheduled an hour later.

Controls and logs

Environment variable	Default	Effect
PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED	false	Set true to let the reaper handle task directories
PAPERCLIP_TASK_DIRECTORY_RETENTION_DRY_RUN	true	Set false to save and remove; a dry run only reports candidates and needs no restore points
PAPERCLIP_TASK_DIRECTORY_ORPHAN_GRACE_DAYS	7	Days an orphan's issue tree must be closed and its contents unchanged before removal (minimum 1)

Each open or pending workspace logs event: "execution_workspace.task_directory_retention" with its outcome (removed, dry_run or skipped), skip reason, candidate bytes and reclaimed bytes. Each directory the walk inspects logs event: "execution_workspace.task_directory_cleanup" with its outcome (removed, preserved_and_removed, dry_run or skipped), reason and bytes. Each reopen that restores a directory logs event: "execution_workspace.task_directory_restore". A sweep saves and removes at most 8 open task directories; the rest continue in the next sweep. A workspace that was skipped is inspected again only when it or its issue changes, or at its retry deadline, so skipped workspaces do not use up that limit. Roll out with the dry run, inspect the candidates and skip reasons, then set the dry run to false.

Nested project repository worktrees

Extra project repositories that have no configured local folder are no longer full clones inside every task. Each task's .paperclip-repositories/<name>-<key> checkout is a Git worktree of one shared clone per company, project and repository URL: the managed project checkout for that URL under the instance's projects/ directory (the same clone that main task worktrees use when the repository is a task's primary source). Objects are stored once in that clone, so a new task adds only its working files. Repositories with a configured local folder are still seeded as independent copies. Existing standalone nested clones keep working unchanged until the optional conversion below turns them into worktrees of the shared clone.

Each nested worktree checks out its own branch, paperclip/<id>/<branch>, where <id> derives from the worktree path and <branch> is the remote's current default branch (refreshed before each new worktree) or the configured branch ref, tracking origin/<branch>. Reopening derives the upstream from the saved source ref, not from the branch name. A tag or commit ref is checked out detached, as before. Branch names therefore never collide between tasks. When a checkout kept under .paperclip-runtime/detached-repositories/ still holds that branch, the new worktree uses paperclip/<id>/r<n>/<branch> instead, so the retained work is never reset. git pull works as before; pushing needs an explicit destination such as git push origin HEAD:<branch>. Adding the worktree, fetching and removing registrations are serialized per shared clone. Each new worktree first fetches with --prune, so a branch deleted upstream is rejected as a fresh clone would reject it. Server-side Git commands on the shared clone run with hooks disabled, so hooks an agent plants there never run in the server.

Branches and reflogs of a worktree live partly in the shared clone. Expiration and restore points therefore scope their checks to what belongs to the nested worktree: its HEAD, HEAD reflog, ORIG_HEAD, index and working tree (stored per worktree) and every branch under its own paperclip/<id>/ prefix together with that branch's reflog. Those must be on the remote or saved in a restore point exactly as for a standalone checkout. Checking out any other branch, stashes, unsupported refs or custom local config in the shared clone still block expiration. Branches outside the prefix, other tasks' branches and their history stay in the shared clone and never block or get deleted. Commits that existed only in the nested worktree's HEAD reflog are written back into the restored worktree's HEAD reflog, so they stay protected after reopening.

Expiring a task removes the nested worktree's registration from the shared clone and deletes its own branch only at the verified recorded commit. Reopening recreates the worktree from the shared clone (re-cloning the shared clone if it is gone) on the same branch and commit and applies its restore point. Deleting or archiving a task removes the registration of each of its nested worktrees and deletes their branches only when every commit is on a remote-tracking ref; an unpushed branch stays in the shared clone. Moving a nested worktree to .paperclip-runtime/detached-repositories/ when its repository leaves the project repairs its registration.

Converting existing standalone nested clones

Tasks prepared before nested worktrees existed still hold a full standalone clone in each .paperclip-repositories/<name>-<key> checkout, often of the same large repository many times over. With PAPERCLIP_NESTED_CLONE_MIGRATION_ENABLED=true, the terminal workspace reaper converts each such checkout in an open task into a worktree of the project's shared clone and deletes the standalone clone. It holds the source lock and workspace lifecycle lock that git cache expiration holds, the per-shared-clone lock that nested worktree creation holds, and the expiration lock files (index.lock, HEAD.lock and the current branch's ref lock) in the clone, so no run, expiration, reopen, restore or other conversion against the same shared clone can interleave. A checkout is converted only when:

• no run, native finalization, environment lease or runtime service is still live for the workspace, and it is not waiting for a reopen to be consumed (the issue may be open);
• its name is the managed checkout name of one of the project's repository workspaces (unknown_managed_checkout otherwise), that repository has no configured local folder (local_source_checkout), and the clone's origin is exactly the repository URL (foreign_origin);
• the project has a verified shared clone for that URL (no_shared_base, shared_base_repo_mismatch) whose local config runs no filters or commands (unsupported_base_git_config);
• the clone holds only branches, tags and origin remote-tracking refs, no stash (nested_clone_has_stash; stashes are not converted, because refs/stash would be shared by every worktree of the shared clone), no worktrees of its own, no custom hooks, no local config beyond the stock keys, origin and branch tracking, no info/exclude pattern the shared clone lacks (unsupported_info_exclude), no in-progress merge, rebase or similar state, no hidden index flags, and no commit that only another branch's reflog holds (unique_branch_reflog_commits);
• nothing a restore point refuses (submodules, filtered paths, special files, untracked nested repositories, oversized contents or history).

The conversion fetches the shared clone, then brings the clone's history in under the checkout's own prefix: the current branch becomes paperclip/<id>/<branch> at the same commit, tracking origin/<branch> (or the branch's configured origin upstream); every other local branch whose tip the shared clone's remote-tracking refs do not contain becomes paperclip/<id>/<branch>, every such tag becomes paperclip/<id>/tags/<tag> at the tagged commit, and every such origin remote-tracking ref (for example one whose branch was deleted or force-pushed upstream) becomes paperclip/<id>/remotes/origin/<branch>. <id> derives from the final checkout path, so later expiration and release treat those branches exactly like a nested worktree's own branches: they block expiration until pushed or saved and are never deleted while unpushed. A restore point of the clone, captured against commits both repositories share and preserving those branch tips, its HEAD reflog and ORIG_HEAD, is imported into the shared clone. Other tasks' branches and worktrees are never touched; a name that already exists in the shared clone skips the checkout (nested_clone_ref_conflict).

The new worktree is added at <name>.clone-migrating beside the clone, the restore point is applied (index, worktree changes, untracked and ignored files, file modes), the HEAD reflog and ORIG_HEAD are copied, and regenerable directories, untracked .env files and .claude/settings.local.json are moved in by rename. The two must then match exactly (HEAD, index, worktree, ignored files, modes, reflog, branch and every created ref). Any difference or failure before the swap moves the items back, removes the new worktree and the created branches, and leaves the clone untouched. The swap renames the clone to <name>.clone-standalone, moves the new worktree into <name> with git worktree move and runs git worktree repair in the shared clone. Then one database update records the worktree in a saved restoration recipe that lists the checkout, and the old clone is renamed to <name>.clone-trash and deleted.

Before it changes the shared clone, the conversion records nestedCloneMigrationPending.<name> in workspace metadata (shared clone, HEAD, created refs, moved items and phase). The next pass uses it after a crash: before the swap it moves items back, removes the new worktree and the created branches, then converts again; once the clone has been renamed aside it finishes the swap and deletes the old clone, so work done in the converted checkout after a crash is kept. A state that matches neither (for example a new checkout created at <name> while the clone was aside) is left alone with nested_migration_recovery_conflict. The reaper also finishes a pending record after the workspace is archived; if the task worktree is already gone, it forgets the record, removes the stale registrations and deletes only created branches whose commits are on the remote. Expiration treats the .clone- names as unexplained files and keeps the workspace until the conversion settles. A dry run reports a pending record without acting on it.

Each attempt logs event: "execution_workspace.nested_clone_migration" with the checkout name, outcome (migrated, recovered, rolled_back, dry_run or skipped), skip reason, new branch, candidate bytes (the clone's .git) and reclaimed bytes. A workspace is inspected again at most hourly, and a retry deadline it registered survives later sweeps during that hour. A retry deadline is registered only while a pending record exists or the skip comes from a live consumer, a busy lock or a changed workspace row; other skips, including a change caught by verification, wait for a later sweep. The reaper looks only at checkouts named after the project's configured repositories (plus pending records), never at arbitrary directories, and attempts at most 8 per workspace per sweep. Pending recoveries go first, and the rest rotate across sweeps. While a rotation has not yet visited every checkout it registers a retry deadline; after a full rotation without progress it waits for a later sweep, until a conversion succeeds or the set of candidate checkouts changes. PAPERCLIP_NESTED_CLONE_MIGRATION_DRY_RUN=true reports candidates without changing anything and registers no deadlines.

Reopening the issue, or the next run that recreates the worktree, imports the bundle, re-adds the worktree at the saved commit, applies the restore point and verifies the result. The bundle is always imported, so history-only commits are present again as objects in the repository; they are not recreated as branches or reflog entries, and the restore point keeps them until it expires. A draft commit message left by an aborted commit still keeps the worktree from expiring, because restore points do not save it. A worktree that already matches is only marked applied, and a worktree that diverged is never overwritten: the restore fails closed and can be retried. A missing or corrupt restore point also fails closed.

When a restore point has expired, through retention or on a nested checkout, reopening starts fresh instead: the worktree is recreated on its branch (the branch tip if the repository still has it, otherwise a new branch from the base ref), the project's nested repositories are cloned fresh before the provision command runs, the saved recipe is dropped, and a system comment on the issue warns that unpushed and uncommitted work from before the cleanup is no longer available. The same warning is returned with the reopen result and recorded on the execution_workspace.reopened activity.

Restore points are encrypted saves of the parts of a git worktree that its remote cannot reproduce: unpushed commits and uncommitted files. They let a closed issue's worktree be removed and restored later. The table execution_workspace_restore_points tracks each one with its storage key, hash, size and status (pending, active, expiring, superseded, delete_requested, deleted). The objects live in the instance's configured file storage (local_disk or s3).

An active restore point's delete_after is either its expiry time or empty while its source issue tree is open. A database trigger on issues keeps this current: when an issue's status or parent changes, or an issue is deleted, it marks every active restore point whose source issue is that issue, its old or new parent, or any of their ancestors as due now, in the same transaction. A retention worker on the shared delivery-work scheduler runs at startup, when issue changes settle while restore points exist, and at real deadlines: the earliest expiry, the end of an abandoned capture's grace period, or a failed deletion's retry. With nothing due, it schedules nothing, and open trees cause no recurring checks. At review it checks the source issue tree:

• Tree still open: clear delete_after and wait for the next issue change.
• Tree done or cancelled: expire PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS (default 90, minimum 1) after the tree's latest close, immediately if that time has passed. Expiry first moves the row to expiring, runs the expiry hook, and only then requests deletion. An expiring row's object is never deleted; after a restart or a hook failure the hook is retried, after 15 minutes when it failed.
• Source issue gone: delete.

Deleting an issue or a company marks its restore points delete_requested in the same transaction and notifies the worker, so their objects are removed promptly. A pending restore point older than one hour was abandoned mid-capture and is deleted. superseded and delete_requested restore points have their stored object removed, then become deleted once the storage provider confirms the object is gone. A storage failure is recorded in last_error and retried after 15 minutes. Work is processed in batches of 50; a full batch continues immediately.

Rows have no foreign keys, so they outlive a deleted workspace, issue or company long enough to remove the stored object. Each run that changes anything logs its counts.

Isolated repository sources

For a while, workspace preparation gave some isolated git_worktree workspaces their own full clone of the project repository under instances/<id>/isolated-repository-sources/<companyId>/<issueId>/checkout, with the issue's worktree inside it. Workspace metadata marks these with isolatedRemoteSource. New workspaces no longer get one; they are worktrees of the project's shared managed clone. Each old clone still holds a full .git and an unused default-branch checkout.

With PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true (and git cache expiration enabled), the terminal workspace reaper moves each such open workspace onto the shared clone and then deletes its private clone. It runs under the same source lock, workspace lifecycle lock and Git lock files as git cache expiration, so no run, expiration, reopen or restore can interleave. A workspace is moved only when:

• no run, native finalization, environment lease or runtime service is still live for it, and it is not waiting for a reopen to be consumed (the issue may be open);
• the project has a shared managed clone for the same repoUrl;
• its worktree sits at the standard .paperclip/worktrees/<branch> location, and the branch is absent from the shared clone, points to an ancestor of the worktree's HEAD there, and is not checked out in another of its worktrees;
• the shared clone's local config sets no filter drivers, includes, attribute files, worktree config or command settings (core.fsmonitor, core.hooksPath, core.sshCommand and similar), and it has no info/attributes, because the move checks out files there;
• its Git state is transferable: no stash or replacement refs, no in-progress merge, rebase or other worktree-specific state, no hidden index flags, no shallow history, and nothing a restore point refuses (submodules, filtered paths, special files, untracked nested repositories). The transfer is local, so it uses much larger restore point limits than uploads do: 500,000 files, 1 GiB per file and in total, a 1.5 GiB bundle and 1,000,000 commits.

The move captures a restore point of the old worktree against commits both repositories share, imports it into the shared clone, creates or fast-forwards the issue branch, adds the new worktree at the same HEAD and applies the restore point (index, worktree changes, untracked and ignored files, file modes). The HEAD reflog and ORIG_HEAD are copied. What restore points leave out is moved by rename on the same filesystem: regenerable directories such as node_modules, untracked .env and .env.* files, .claude/settings.local.json, and each nested .paperclip-repositories/* checkout as a whole. A standalone nested clone moves as is. A nested checkout that is a worktree of its own shared clone moves too, and the move runs git worktree repair in that clone so its registration follows the new path. Its paperclip/<hash>/ branch is renamed to the prefix of the new path, because that prefix proves the checkout owns the branch (a rollback moves it back, repairs it and renames the branch back). Items inside a moved nested checkout, such as its node_modules, move with it. Any other ignored file that a restore point does not save keeps the workspace in place.

The new worktree must then match the old one exactly (HEAD, index, worktree, ignored files, modes and reflog), and every moved item must exist. Only then does one database update point the workspace at the new path, drop isolatedRemoteSource, rewrite a saved restoration recipe to the shared clone, and record isolatedRepositoryRehome ({ at, from, to, cleanedAt }). Any failure before that update removes the new worktree, moves items back, restores the branch, and leaves the old worktree untouched. Before it changes the shared clone, the move records isolatedRepositoryRehomePending (target, branch, previous branch tip and planned moves). If the server stops before the update, the next attempt uses that record to move items back, remove the new worktree and restore the branch, then starts again. A dry run reports such a record without undoing it. If an item exists in both places (for example a process recreated .env in the old worktree), the rollback stops with rehome_rollback_conflict and keeps the new worktree and the record, so neither copy is lost; later attempts retry the rollback once the conflict is resolved. A new worktree is removed only when the move created it: a target that appears from elsewhere is never deleted.

After the update, the old worktree is removed (its ownership is proven by git rev-parse or, when its registration in the clone was already pruned, by its .git pointer into the clone's worktrees directory). The clone is then renamed to trash and deleted when nothing else needs it and everything in it is reproducible:

• no other workspace or project workspace refers to it (a workspace already expired with its worktree removed does not count) and no runtime service runs in it;
• after git worktree prune, it has no worktree besides its own checkout;
• its own checkout has no tracked changes and no untracked files (only Paperclip's compaction trash is allowed, even inside a directory with a regenerable name), and its ignored files are only inside regenerable directories (such as node_modules) or Paperclip's compaction trash. Empty directories, such as the .paperclip/worktrees parent left by the old worktree, do not count;
• it has only branch, tag and remote-tracking refs (no stash, notes or replacement refs);
• every commit that its local branches, local tags, reflogs (other than remote-tracking ones) or ORIG_HEAD hold is on one of its remote-tracking refs, present in the shared clone, or found on the remote. The remote is read fresh (branches and tags) with the company's Git identity, falling back to anonymous access, and only when some commit is in neither place. HEAD itself counts too, so a detached checkout is covered. Remote branches and release tags that the shared clone never fetched do not keep a clone.

A clone with more than 5,000 refs, reflog entries or such commits is kept (clone_history_budget), and so is one with more than 1,000 ignored directories outside regenerable ones (clone_file_budget); those directories are read in one Git call. The log entry names up to five paths, worktrees, refs or commits that kept a clone (cloneKeptPaths). A clone another workspace still uses is kept (cloneRetained); that workspace's own move removes it later. If the server stops after the update, a later sweep finishes the removal and sets cleanedAt, also after the workspace has been archived.

The reaper also sweeps orphaned clones under isolated-repository-sources/ once per server start: clones that no execution workspace refers to (no open workspace with matching isolatedRemoteSource, and no closed one that would need the clone to reopen) and that no pending move or cleanup still owns. The repository must be proven by the database: the workspaces of that issue must record exactly one repository URL, and the clone's origin must equal it (unproven_repository or source_identity_changed otherwise), so a rewritten origin is never fetched or trusted. It applies the same reproducibility check, using the shared clone of a workspace of the same issue and repository when one exists, and logs event: "execution_workspace.isolated_source_orphan_cleanup". It inspects at most eight issue directories per reaper sweep and continues from that position in the next sweep, so a slow remote cannot hold up other workspace cleanup. When the walk ends and it kept any orphan, it looks again a day later; otherwise it registers no further work. It also deletes leftover trash directories from interrupted removals.

Each attempt logs event: "execution_workspace.isolated_source_rehome" with the outcome, skip reason, candidate bytes and reclaimed bytes. A workspace that is skipped, or whose clone is kept for a reason that can change, is retried an hour later through the reaper's existing deadlines; nothing runs while there is no candidate. PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN=true reports candidates without changing anything.

Re-homed workspaces get a new path. An agent that resumes an older session may still remember the old path; the run's working directory comes from the updated workspace.

Reopening or restoring an expired workspace whose recipe came from a private clone no longer recreates that clone when the project has a shared managed clone for the same repository, its branch there is absent or at the saved commit, and the shared clone has every commit the saved restore point bundle needs as a base. A recipe with worktree-backed nested checkouts on branches keeps the old behavior, because their branch names are bound to the old path. The workspace is rebuilt as a worktree of the shared clone at the standard location, with the saved commit and restore point imported and applied as before. Without a shared clone, the old behavior applies.