## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each task runs in a git worktree. Extra project repositories sit in
the task under `.paperclip-repositories/`.
> - #17382 made new nested checkouts git worktrees of one shared clone
per project repository. Existing standalone nested clones stayed as full
clones.
> - Long-lived tasks keep those clones for a long time. Many tasks hold
the same large repository, so most of the disk is duplicate `.git` data.
> - #17369 added a locked and verified move of isolated workspaces onto
the shared clone, with a reaper sweep behind a setting.
> - This pull request adds the same kind of locked and verified
conversion for standalone nested clones in open tasks, and runs it from
the same reaper path behind a new setting.
> - The benefit is that the duplicate clones go away without losing any
state or local history, and the disk stops filling.
## Linked Issues or Issue Description
Refs #17382#17369. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 44 open task worktrees in one project each hold a
standalone nested clone of the same large repository. Together they use
about 25 GB. About 16 GB of that is duplicate `.git` data. Another
project has 33 smaller clones. The clones are removed only when their
tasks expire, and many tasks stay open for a long time. The disk keeps
filling.
**Expected behavior**
Existing standalone nested clones in open tasks become worktrees of the
project's shared clone, with the same state and local history, and the
standalone clones are deleted.
**Steps to reproduce**
1. Prepare a task with a nested project repository before #17382, or
without a shared clone, so that `.paperclip-repositories/<name>` is a
standalone clone.
2. Create the project's shared clone, as #17382 does for new tasks.
3. Observe that the old checkout stays a full clone until the task
expires.
**Paperclip version or commit:** `master` at 8549ecaf9a.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- New `server/src/services/nested-clone-migration.ts`. It converts one
standalone nested clone in an open execution workspace into a worktree
of the shared clone:
- It holds the source lock and workspace lifecycle lock (as git cache
expiration does), the per-shared-clone lock from #17382, and the
expiration lock files (`index.lock`, `HEAD.lock`, current branch ref
lock) in the clone.
- It requires the #17369 no-live-consumer eligibility (no live runs,
leased finalizations, unreleased leases, running services, or reopen
flag). It checks again before the swap.
- It identifies the repository by managed checkout name, project
workspace rows and the origin URL, and it requires a verified shared
clone.
- It brings local history into the shared clone under the checkout's own
`paperclip/<id>/` prefix. The current branch becomes
`paperclip/<id>/<branch>`, tracking `origin/<branch>`. Unpushed local
branches become `paperclip/<id>/<branch>`. Unpushed tags become
`paperclip/<id>/tags/<tag>`. A restore point carries the commits, the
`HEAD` reflog and `ORIG_HEAD`.
- It stages the worktree at `<name>.clone-migrating`, applies the
restore point, copies the `HEAD` reflog and `ORIG_HEAD`, and moves
regenerable dirs, untracked `.env` files and
`.claude/settings.local.json` by rename. Old and new must then match
exactly (snapshot, history, branch, created refs, moved items).
- Swap: it renames the clone to `<name>.clone-standalone`, runs `git
worktree move` and `git worktree repair`, updates a saved restoration
recipe that lists the checkout, and then renames the old clone to trash
and deletes it.
- A `nestedCloneMigrationPending.<name>` metadata record makes recovery
possible. A crash before the swap rolls back (moves items back, removes
the staged worktree and the created branches). A crash after the swap
rolls forward.
- Skip reasons (nothing changes): `live_or_unresolved_consumer`,
`unknown_managed_checkout`, `local_source_checkout`, `foreign_origin`,
`no_shared_base`, `shared_base_repo_mismatch`,
`unsupported_base_git_config`, `nested_clone_has_stash` (stashes are
skipped, because `refs/stash` would be shared by every worktree),
`nested_clone_has_worktrees`, `nested_clone_has_hooks`,
`unsupported_local_git_config`, `unsupported_local_git_state`,
`unsupported_info_exclude`, `unsupported_nested_clone_refs`,
`unsupported_index_flags`, `unique_branch_reflog_commits`,
`nested_clone_ref_conflict`, `snapshot_mismatch`, `history_mismatch`,
restore point limits, and `nested_migration_recovery_conflict`.
- Reaper: `sweepTerminalWorkspaces` runs the conversion for open
workspaces when `PAPERCLIP_NESTED_CLONE_MIGRATION_ENABLED=true` (default
off). `PAPERCLIP_NESTED_CLONE_MIGRATION_DRY_RUN=true` only reports. Each
attempt logs `event: "execution_workspace.nested_clone_migration"` with
outcome, reason, branch, candidate bytes and reclaimed bytes. There is
no new timer. A retry deadline is registered on the existing reaper
deadlines only for a pending record or a transient skip (live consumer,
busy lock, changed row). A workspace is attempted again at most hourly.
A workspace without standalone clones costs one directory listing.
- `isolated-repository-rehome.ts`: I exported the shared pieces
(eligibility, locks, safe base config check, transfer plan, history
boundary, snapshot and history keys, item moves, rollback and
verification) instead of copying them. Re-home behavior does not change.
- `nested-repository-worktrees.ts`: I exported the stale registration
cleanup and added the `moveNestedRepositoryWorktree` (move and repair)
and `trackNestedWorktreeUpstream` helpers.
- Docs: `doc/recoverable-disk-cleanup.md` (new section about the
conversion) and `doc/DEVELOPING.md` (settings).
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/nested-clone-migration.test.ts`: 19 passed. It uses
embedded Postgres and real git repositories, and covers:
- Conversion with an unpushed commit, a second branch with unpushed
commits, a lightweight tag and an annotated tag, staged and unstaged
edits, a deleted file, an untracked file with mode 0600, an ignored
file, `node_modules`, `.env` and the settings file. The result is a
worktree of the shared clone (`--git-common-dir`), with the same HEAD,
status, staged content and reflog, the owned branch tracking
`origin/main`, and the history under the task's prefix. No other branch
in the shared clone changes. The old clone is gone and reclaimed bytes
are reported.
- Skips with nothing changed: stash, live run, foreign origin, unknown
checkout, no shared clone, local excludes, and a change between capture
and verification (hook).
- Two tasks convert concurrently against one shared clone. The test
holds the shared-clone lock and shows that neither conversion advances.
Then they run one at a time (peak 1), and each keeps its own branches.
- Crash recovery at `afterAside`, `afterMove` and `afterFinalize` (roll
forward), and before the swap (roll back, then convert). A dry run
reports the pending record and changes nothing.
- Expiration and restore of a converted checkout work afterwards.
Unpushed branches in a converted checkout block expiration
(`unpushed_managed_checkout_branches`).
- Reaper: disabled by default; dry run changes nothing and registers no
deadline; a live run registers a retry deadline that survives a second
sweep; enabled conversion leaves no deadline.
- Commits that only the clone's `origin` remote-tracking refs hold are
kept as `paperclip/<id>/remotes/origin/<branch>`.
- A pending conversion is finished after the workspace is archived, and
forgotten (keeping unpushed branches) when the worktree is gone.
- A filter planted in the shared clone right before the worktree add is
refused (`unsupported_base_git_config`) and never runs.
- Only checkouts named after configured project repositories are
considered; at most 8 per sweep, pending first, rotating (unit test).
- Related suites (run before review fixes; the nested, re-home and
execution workspace service suites were rerun after them): `pnpm exec
vitest run` on execution-workspaces-routes,
execution-workspaces-service, git-reproducible-managed-checkouts,
git-restore-point (all 5 files), git-workspace-contents,
git-workspace-expiration, git-workspace-expiration-eligibility,
heartbeat-git-restore-point-resume, heartbeat-project-repositories,
isolated-repository-rehome, nested-clone-migration: 15 files, 393 tests
passed.
- `cd server && pnpm run typecheck` (`tsc --noEmit`): passes.
- Mutation checks. Each mutation makes the targeted test fail, and I
reverted each one:
- Remove both eligibility checks: the live run test fails.
- Remove the snapshot comparison: the capture/verify change test fails.
- Disable roll forward after the swap: the recovery tests fail.
- Drop the `paperclip/<id>/` prefix from transferred branches: the
conversion and concurrency tests fail.
- Remove the shared-clone lock: the concurrency test fails.
- Drop remote-tracking refs from the plan, the deadline restore during
backoff, archived recovery, or the config check before the worktree add:
each targeted test fails.
- Not run: the full server suite and a run on a live instance.
## Risks
- The setting is off by default. Enable a dry run first and read the
`execution_workspace.nested_clone_migration` logs.
- A clone that a run writes to between verification and the swap rename
can lose those writes. This window is the same as in #17369. Locks, the
eligibility check and the git lock files make it small.
- A crash in the short time between the rename aside and `git worktree
move` can collide with a run that recreates `<name>`. In that case the
conversion stops with `nested_migration_recovery_conflict` and keeps
both copies. Expiration keeps such a workspace because of the `.clone-`
names.
- Stashes, annotated tag messages (the tagged commit is kept as a
branch), custom hooks, custom local config and per-branch reflog-only
history are not converted. Clones with them are skipped with a named
reason. Only unpushed branches and tags get a name in the shared clone.
A pushed local branch name is not kept, but its commits stay on the
remote.
- The conversion fetches the shared clone first. A failed fetch is
ignored, and then the restore point can be larger.
- Pushing from a converted checkout needs an explicit destination, as
for all #17382 nested worktrees.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can attach extra repositories, and every task worktree
gets a managed checkout of each one under
`.paperclip-repositories/<name>`
> - Each of these nested checkouts is a full clone. The clone uses
`--dissociate`, so no objects are shared between tasks
> - On one instance, 44 task worktrees held nested checkouts of the same
repository: 24 GB of 27 GB of worktrees. Each new task added about 0.5
GB, and the disk filled twice in one day
> - Main task worktrees already avoid this, because they are git
worktrees of one managed project clone
> - This pull request makes nested checkouts git worktrees of one shared
clone per project and repository, and adapts expiration, restore points,
restore and cleanup to them without weaker safety checks
> - The benefit is that repository objects are stored once, and a new
task adds only its working files
## Linked Issues or Issue Description
No public issue exists. This is the in-PR bug report.
**What happened?**
Every task in a project with extra repositories got a full, independent
clone of each extra repository under `.paperclip-repositories/`. On one
instance this used 24 GB of 27 GB of task worktrees (16 GB of it `.git`
data) and filled the disk twice in one day.
**Expected behavior**
Nested checkouts of the same repository share one object store, like
main task worktrees share the managed project clone. Disk use per new
task is only its working files.
**Steps to reproduce**
1. Create a project with a primary repository and a second repository
that has a repository URL and no local folder.
2. Run several tasks in isolated git worktrees in that project.
3. Look at `<task worktree>/.paperclip-repositories/<name>/.git`: each
one is a full directory with its own copy of every object.
**Paperclip version or commit:** `master` at ff835a3fc6.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
Related: #15662 (SSH restores skip nested worktrees) touches a different
path; this PR does not change adapter sync.
## What Changed
- New `server/src/services/nested-repository-worktrees.ts`. It finds the
shared clone (the managed project checkout for the repository URL, from
`ensureManagedProjectWorkspace`), adds nested worktrees, restores them,
removes their registrations, and serializes all of these operations and
fetches per shared clone with an in-process lock.
- `prepareProjectRepositoryWorkspaces` takes a `sharedBase` (company and
project). For a repository without a configured local folder, it fetches
the shared clone and adds a worktree at the same
`.paperclip-repositories/<name>-<key>` path. Each worktree gets its own
branch `paperclip/<id>/<branch>` that tracks `origin/<branch>` (`<id>`
derives from the worktree path), or a detached HEAD for a tag or commit
ref. An existing worktree must belong to the project's shared clone.
Configured local-folder copies and existing standalone clones are
unchanged. Moving a worktree to `detached-repositories` repairs its
registration. Heartbeat preparation and fresh-start reopening pass
`sharedBase`.
- `assertReproducibleManagedCheckout` and `provenManagedCheckoutHistory`
accept worktree-backed checkouts. They verify the worktree link, its
admin directory, the shared clone identity and origin, and that HEAD is
detached or on a branch under the worktree's own prefix. History checks
scope to HEAD, the per-worktree HEAD reflog, ORIG_HEAD and the owned
branches with their reflogs. Other tasks' branches are not inspected.
Stashes, unsupported refs and custom config in the shared clone still
block. The upstream proof reads objects from the common git directory.
- Restoration contents record `worktree: { base, admin }` for a
worktree-backed checkout. Validation checks its shape and the owned
branch name, and every use checks the base against the project's
expected clone paths.
- Expiration locks for a worktree-backed checkout sit in the shared
clone (`.git/worktrees/<admin>/index.lock`, `HEAD.lock`, and the owned
branch ref lock). `gitCacheExpirationLockLocations` is now async.
- `finishGitCacheTrash` removes the expired worktree's registration and
deletes its owned branch only at the recorded commit.
- `restoreGitWorkspaceContents` recreates a worktree-backed checkout
from the shared clone, and clones the shared clone again if it is gone.
It imports the restore point into the shared clone, adds the worktree on
the saved branch and commit, writes reflog-only commits back into the
new HEAD reflog, and applies the restore point.
- `cleanupExecutionWorkspaceArtifacts` (terminal reaper archival, close
and delete) collects the task's nested worktrees first. After the task
directory is gone, it removes their registrations and deletes owned
branches only when every commit is on a remote-tracking ref. An unpushed
branch stays in the shared clone.
- Review follow-ups. Owned branches always track origin, including names
with dots, and the config check accepts their tracking keys. A ref held
by a retained detached checkout gets `paperclip/<id>/r<n>/<branch>`.
Restore derives the upstream from the saved source ref. Each new
worktree fetches with `--prune`, refreshes `origin/HEAD` when no ref is
configured, and strips `refs/heads/` from branch refs. Git commands on
the shared clone run with hooks disabled. The worktree `gitdir` pointer
is read through a bounded, non-following, non-blocking handle, and more
than 64 owned branches stop the proof before reflog reads.
- Docs: new "Nested project repository worktrees" section in
`doc/recoverable-disk-cleanup.md`, and updates in `doc/DEVELOPING.md`.
## Verification
- `pnpm -r --filter "./packages/**" build`, then `pnpm run typecheck` in
`server/`: passes.
- `pnpm exec vitest run` in `server/` for `git-workspace-expiration`,
`git-reproducible-managed-checkouts`, `git-workspace-contents`,
`git-restore-point*`, `heartbeat-project-repositories`,
`execution-workspaces-service`, `execution-workspaces-routes`,
`execution-workspace-compaction`, `workspace-compaction`,
`heartbeat-git-restore-point-resume`, `managed-repository-checkout`,
`task-workspace-repositories*`, `native-workspace-sync*`,
`heartbeat-managed-clone-credentials`,
`services/heartbeat/workspace-preparation`,
`project-repositories-persistence`, `git-workspace-base-storage`,
`agent-directory-probe`, `workspace-runtime`,
`workspace-preparation-history` and `native-completion-feedback`: 866
passed, 2 failed. The 2 failures are in `workspace-runtime.test.ts`
(`rejects explicit main on a master-only remote...` and `does not
diagnose a missing remote branch...`). They fail the same way on
unmodified `master` on this machine.
- New tests (embedded Postgres, real git repositories) in
`git-workspace-expiration.test.ts`, section "nested checkouts as
worktrees of a shared base":
- Two tasks get worktrees of one shared clone. `--git-common-dir` points
to it, each task has a distinct `paperclip/<id>/main` branch, upstream
is `origin/main` and the remote is the repository URL.
- Concurrent preparation of three tasks succeeds, and preparation waits
while the shared clone lock is held.
- A legacy standalone clone is still adopted, and no shared clone is
created for it.
- Expiring a clean nested worktree removes only its registration and
branch. The sibling worktree and branch stay. Reopening recreates the
worktree from the shared clone on the same branch and commit, with its
local settings.
- Reopening clones a deleted shared clone again.
- Unpushed commits and uncommitted files block plain expiration, are
saved in a restore point and are restored exactly.
- Reflog-only history is saved and written back into the restored HEAD
reflog, and it survives `gc --prune=now` of the shared clone.
- Stashes, a checked-out branch outside the worktree's prefix, and
unpushed owned branches or owned reflog entries outside HEAD history
block expiration.
- Another task's unpushed branch does not block, and it stays after
expiration.
- A worktree of a foreign clone at the nested path is rejected by
preparation and by the prover.
- Expiration locks in the shared clone are recovered after a process
exit.
- Deleting a task removes only its nested registrations, keeps its
unpushed branch and deletes the sibling's pushed branch when the sibling
is deleted.
- Follow-up tests: dotted branch tracking, a ref switched away and back
with a retained checkout, a changed remote default branch, a renumbered
branch restored next to a same-named remote branch, a fully qualified
branch ref after the remote advances, a branch deleted upstream, planted
`post-checkout` and `reference-transaction` hooks, more than 64 owned
branches, and a FIFO, oversized or symlinked `gitdir` pointer. Each fix
was mutation-checked.
- `heartbeat-project-repositories.test.ts`: a repository-URL-only
sibling is now a worktree of the shared clone on an owned branch, and
its detached copy is still a valid checkout after its repository leaves
the project. In `workspace-preparation.test.ts`, the publication guard
now runs before any worktree exists.
- Mutation checks. I reverted one guard at a time and confirmed that a
test fails: the owned-HEAD check, the owned-branch check in restore
point mode, the owned-reflog check, the shared-clone identity check, the
pushed-only branch deletion on cleanup, the cleanup release hook, the
registration removal on expiration, the reflog-only history write-back
on restore, and the shared clone lock. The owned-branch-tip check alone
is also covered by the owned-reflog check, so removing only one of the
two keeps the tests green.
- Not run: the full server test suite, the UI and e2e suites, and a live
instance migration.
## Risks
- Behavior change for agents: a nested checkout of a URL-only repository
is now on `paperclip/<id>/<branch>`, not `<branch>`. `git pull` and `git
status` work as before. A plain `git push` now fails because the branch
names differ. Agents must push with an explicit destination, for example
`git push origin HEAD:<branch>`. Branches in the shared clone are
visible to all tasks of the project, like main task worktree branches.
- Agent-created branches outside the worktree's prefix, stashes, and
local config live in the shared clone. Stashes and custom config block
expiration of every worktree of that clone (fail-closed). Other branches
stay in the shared clone after a task is deleted.
- The shared clone is the managed project checkout for that repository
URL. Fetches into it update its remote-tracking refs. A concurrent
restore point capture of a sibling can then roll back and retry on the
next sweep.
- The base lock is per process. Across processes, git's own ref and
index locks apply.
- No migration in this PR. Existing standalone nested clones keep their
disk use until their tasks are removed or expired. A safe migration
helper is a follow-up. Task repositories (`task-repo-*`) and configured
local-folder copies stay standalone clones.
- Restoration recipes now can carry `contents.checkouts[].worktree`. An
older server rejects such a recipe as invalid (fail-closed) and does not
restore without it.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
