## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner sessions keep durable state under
`<instance>/runtime/paperclip-runner/durable-sessions/`
> - When a session starts a fresh provider session, the retired state is
moved into `<session>/continuity-breaks/<epoch>-<uuid>/`; when state
loses authority, it is moved into
`quarantine/<session>.<reason>.<epoch>.<pid>.<uuid>/`
> - Nothing ever removes either, and each can hold a full Codex home of
several gigabytes
> - A sleep-aware sweep already expires failover backup copies in the
same tree
> - This pull request extends that sweep to remove both kinds of
evidence after a retention window, while keeping anything recovery may
still read
> - The benefit is that native runner disk use stops growing without
bound
## Linked Issues or Issue Description
Bug report described in-PR (bug report template).
**What happened?**
On one instance, one day of native runs produced 20 continuity-break
archives totaling 4.8 GB, one session alone holding 4 GB. The quarantine
folder held 12 entries totaling 2.1 GB, the oldest a month old. Neither
is read again by normal operation (continuity-break archives are written
by `archiveContinuityState` and never read), and nothing deletes them.
They were removed by hand to free space and will regrow.
**Expected behavior**
Evidence older than a bounded window is removed automatically, as
failover backup copies already are, without removing anything a live
session or recovery path still needs.
**Steps to reproduce**
1. Run a native Codex session that hits a continuity break (fresh
provider session), or whose state is quarantined (for example
`identity_indeterminate`).
2. Inspect `durable-sessions/<session>/continuity-breaks/` and
`durable-sessions/quarantine/`.
3. Wait any length of time: the entries stay forever.
**Paperclip version or commit:** `master` at e335265568.
**Deployment mode:** any with native runner sessions.
**Agent adapter(s) involved:** native runner (Codex, and any provider
whose state lands in these folders).
## What Changed
-
`server/src/services/native-runtime/native-harness-backup-expiration.ts`:
the existing tick also retires evidence.
- **Continuity-break archives** (`<epoch ms>-<uuid>` under each
session): removed when older than the window. Age is the epoch in the
name, or the folder mtime when that epoch is in the future.
`archiveContinuityState` refuses to archive state with warm-transition
evidence, and no code path reads `continuity-breaks`.
- **Quarantine entries**
(`<session>.<identity_indeterminate|identity_mismatch>.<epoch
ms>.<pid>.<uuid>`): removed by the same age rule. Native cleanup source
archives (`<session>.identity_indeterminate.cleanup.<uuid>`) are
referenced from recovery history, have no epoch, and keep their original
inode's mtime, so they are always kept.
- Every entry is skipped while its session is executing, initializing,
warm, or closing in this process; this is checked again just before
removal. Symlinked entries and symlinked
`continuity-breaks`/`quarantine` folders are reported as `unsafe_path`
and never followed.
- Removal renames to `.expired-<uuid>` in the same folder and then
deletes; later sweeps finish interrupted deletions. An empty
`continuity-breaks` folder is removed.
- Archives, quarantine entries, and leftover deletions count toward
`maxEntries` and `maxRemovals`, and stop on `shouldContinue`. Paging
resumes inside a session's archives and then in quarantine, so a bounded
or dry-run batch always makes progress and never recounts a session's
backup.
- New result fields: `evidenceCandidates`, `evidenceCandidateBytes`,
`evidenceRemoved`, `evidenceReclaimedBytes`, and skip reasons
`recent_evidence` and `cleanup_archive`. The service also exposes
`noteEvidence(writtenAt)`.
- `server/src/services/native-runtime/native-runner-evidence-signal.ts`
(new): a process-local signal. `native-session-executor.ts` raises it
with the entry's epoch right after `archiveContinuityState` creates an
archive and after `quarantineRunnerdStateRoot` moves state into
quarantine; `server/src/index.ts` subscribes the sweep and unsubscribes
on shutdown.
- `server/src/config.ts`:
`PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` (default `7`, minimum
`1`).
- `server/src/index.ts`: passes the window and logs a batch when it
found evidence candidates.
- `doc/recoverable-disk-cleanup.md`, `doc/DEVELOPING.md`: document the
variable and behavior.
**Sleep-aware background work**
(`rules/sleep-aware-background-work.mdc`):
- *Trigger:* each entry's own expiry deadline (its epoch plus the
window), plus a direct call from the executor when it writes new
evidence. No timer, interval, or polling loop is added; the pass runs
inside the existing sweep's tick and keeps its drain, warm-standby, and
shutdown handling.
- *Source of truth:* the filesystem entries themselves. After a restart
the first sweep reconstructs the next deadline from them.
- *Pending-work predicate / next deadline:* after a full pass the sweep
keeps the earliest due time of the entries it retained; ticks before
that time skip the evidence pass entirely (no readdir, no SQL). A busy
session, a failed removal, or a full batch makes it due on the next
sweep while that obligation remains.
- *Quiet condition:* no evidence, or only evidence not yet due. The pass
never queries the database, so retained evidence never causes recurring
SQL. Dry run reports each candidate once rather than every hour.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/native-harness-backup-expiration`
(in `server/`, embedded Postgres): 52 passed. New cases:
- old archives and quarantine entries are removed, recent ones kept; the
empty `continuity-breaks` folder is removed;
- age comes from the name, with mtime used for a future epoch;
non-archive folders are ignored;
- a busy session keeps its evidence, including when it becomes busy
after the first check, and it is retried on the next tick;
- dry run reports candidates and bytes, changes nothing (including
leftovers), and reports each candidate once until new evidence comes
due;
- symlinked entries and symlinked evidence folders are skipped without
touching the target;
- an old quarantine entry is removed even when its session's issue is
still open; cleanup source archives are kept;
- the pass stays quiet until retained evidence is due, and a
`noteEvidence` signal schedules newly written evidence;
- inspected archives and leftovers count against `maxEntries`, and
leftover removal stops on `shouldContinue` and resumes next tick;
- a due pass stopped by drain before any session stays pending;
one-entry batches finish without recounting a resumed session;
- interrupted `.expired-*` deletions are finished; paging across ticks
with `maxRemovals`, in normal and dry run, with no recount of the backup
when resuming.
- `native-session-executor.test.ts`: quarantining a legacy root raises
the evidence signal with the epoch in the entry name.
- `native-harness-backup-expiration-config`: default, explicit, minimum,
and fallback parsing.
- **Mutation checks**, each failing at least one test: removing the age
check, the future-epoch fallback, either busy check, the busy retry
deadline, the recent-entry deadline, the due gate, the `noteEvidence`
update, merging the pass deadline, the entry and both folder symlink
checks, the dry-run return, the dry-run leftover guard, leftover
removal, the leftover budget check and its retry deadline, counting
archives against `maxEntries`, the cleanup-archive skip, the
empty-folder removal, the resume cursor, the per-archive budget check,
the resume skip of the backup pass, not charging a resumed session,
keeping the pass active across drain, and the executor's quarantine
signal.
- `pnpm exec vitest run config native-harness
native-workspace-sync-retention native-session-executor server-startup`
(in `server/`): 1272 passed on the previous commit; `config
native-harness native-workspace-sync-retention server-startup`: 450
passed on the head commit.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke. The continuity-archive signal
call is exercised only by a runner integration test that needs the
runnerd binary.
## Risks
- **Quarantine as a fail-closed fence:** recovery
(`recoverQuiescentRunnerdState`) and failed-run retry
(`nativeFailedRunRetryStateIsSafe`) treat a quarantined entry for a
session as a reason not to bootstrap fresh. Once such an entry ages out,
a later run of that session can start a fresh provider session instead
of failing with `quarantine_owner_unverified`. This applies only after
the window, matching what backup expiry already accepts for closed work.
- **Signal coverage:** the signal is process-local. Evidence written by
another process sharing the same state directory is picked up when the
next retained deadline is due or after a restart, not immediately.
- **Legacy-path quarantine:** entries quarantined from legacy roots
carry a legacy digest, so the in-process busy check cannot match them;
they are removed by age only.
- **Remote sandboxes:** archives moved aside on a remote runner
(`<root>.continuity-break-<token>`) are outside this sweep.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents do task work in execution workspaces on the instance disk
> - A task without a project repository runs in a plain (non-Git) task
directory under `<instance>/isolated-workspaces/<company>/<issue>/`
> - Nothing removes these directories after the work is done, so the
disk fills up
> - The directories can hold the only copy of agent work, so plain
deletion is not safe
> - This pull request saves each finished task directory in an encrypted
restore point, removes it, and restores it when the work reopens. It
also removes orphaned directories after a grace period
> - The benefit is that finished task directories stop using disk, and
no agent work is lost silently
## Linked Issues or Issue Description
**What happened?**
On one instance, `instances/<id>/isolated-workspaces/` held 251 task
directories and 12 GB, and it grew every day. 61 directories belonged to
open workspace rows whose issues closed more than 7 days before (3.9
GB). 12 belonged to issues closed 3 to 7 days before (0.7 GB). 4.0 GB
were directories with no open workspace row (the row was archived or
missing).
The cause has two parts:
1. The terminal workspace reaper archives a workspace only when it
proves delivery through Git (a merged pull request or a merged branch).
A task directory is not a Git checkout, so the Git status inspection
fails and the reaper always skips the row as "undelivered". Git cache
expiration handles only `git_worktree` workspaces.
2. When a person archives such a workspace by hand, artifact cleanup
removes a `local_fs` directory only when `metadata.createdByRuntime` is
true. Task directory rows never set it, so the directory stays and the
row becomes `cleanup_failed`.
**Expected behavior**
Paperclip reclaims the task directories of finished work by itself. It
must not lose files that only exist in those directories.
**Steps to reproduce**
1. Run an agent on an issue without a project repository (for example a
low-trust sandbox task, or an issue with no project), so it works in
`isolated-workspaces/<company>/<issue>/`.
2. Let the agent write files there, then mark the issue `done`.
3. Wait any number of days. The directory and its workspace row stay.
The reaper logs the row as skipped (undelivered).
4. Archive the workspace by hand. The directory still stays.
**Paperclip version or commit**
`master` at fe85fd7ac4
**Deployment mode**
Self-hosted, local disk storage.
**Agent adapter(s) involved**
Any adapter that runs in a task directory (seen with local and sandbox
runs).
## What Changed
- Added `isolated-task-directory-snapshot.ts`: a deterministic snapshot
of a plain directory (files with modes, directories, symbolic links as
links). It never follows symbolic links, refuses special files, and
enforces the restore point limits (20,000 files, 64 MiB per file, 256
MiB in total). A `node_modules` directory next to a `package.json` is
left out. Restore writes to a staging directory, verifies the digest,
then renames it into place.
- Added `isolated-task-directory-retention.ts`:
- `expireIsolatedTaskDirectory` saves the directory in a restore point,
archives the workspace with the `isolatedTaskDirectoryRetention` state
in one transaction, checks again that the directory did not change,
renames it to trash and deletes it. A change after the capture rolls the
workspace back to active and drops the restore point. An interrupted
removal finishes on a later sweep.
- It uses the git cache expiration consumer checks
(`isGitWorkspaceExpirationEligible`): no live run, unsettled
finalization, live lease, live runtime service, unresolved recovery
action or reopen fence, and no other open workspace or project workspace
in the directory. It also requires the source issue to be bound to the
workspace, because reopen restores through that binding.
- `restoreIsolatedTaskDirectory` restores the directory on reopen. It
replaces an empty directory and fails closed on a non-empty directory
that does not match. When the restore point expired, it reopens an empty
directory and posts a system comment.
- `sweepIsolatedTaskDirectories` walks `isolated-workspaces/` in bounded
batches (8 per sweep, with a cursor). A closed workspace that its issue
is still bound to is saved and removed like an open one. Other folders
are orphans and are removed after a grace period when no live row,
project workspace, runtime service, run, finalization, lease or recovery
action refers to them.
- Reused the git restore point ledger, storage and encryption.
`storeSealedRestorePoint` and `readRestorePointObject` were extracted
from `storeGitRestorePoint`. `markGitRestorePointExpired` now also marks
a task directory restore point as expired.
- `isGitWorkspaceExpirationEligible` takes `closedSiblingsAllowed`, so
closed rows for the same issue directory do not block a task directory
forever. Git expiration keeps its old behavior.
- Terminal workspace reaper (`execution-workspaces.ts`): handles open
task directory rows after the terminal delay and registers that moment
as a deadline. It finishes pending removals, runs the folder walk at the
start of a rotation (once per server start, then only at real
deadlines), and saves at most 8 directories per sweep.
- Reopen accepts a closed `shared_workspace` row that holds task
directory state. The run path also reopens (and restores) a closed
workspace with that state before preparing the run.
- Settings: `PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED` (default
`false`), `PAPERCLIP_TASK_DIRECTORY_RETENTION_DRY_RUN` (default `true`),
`PAPERCLIP_TASK_DIRECTORY_ORPHAN_GRACE_DAYS` (default `7`, minimum `1`).
The delay is `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS`. Removal
needs `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- Logs: `execution_workspace.task_directory_retention`,
`execution_workspace.task_directory_cleanup` and
`execution_workspace.task_directory_restore`, with outcome, skip reason,
candidate bytes and reclaimed bytes.
- Documented the behavior in `doc/recoverable-disk-cleanup.md` and the
settings in `doc/DEVELOPING.md`.
**Preservation decision.** The code shows that these directories are not
scratch space. Nothing pushes or copies their contents, and the next run
of the same issue reuses the same path. So the files are saved before
removal, with the same ledger, encryption and retention as git restore
points, and restored on reopen. Orphans are not saved: no workspace row
can reopen them. The docs and the Risks section state this.
**Scheduling.** The work follows
`rules/sleep-aware-background-work.mdc`. It adds no timer and no
polling. Open rows use the existing reaper and its per-workspace
deadlines (the end of the terminal delay, or one hour after a retryable
skip). The folder walk runs once at startup. After that, workspace
archive and issue close notifications (the existing terminal workspace
hints) point the reaper at the affected folder, and a kept folder is
checked again only at its own deadline (end of a grace period or
terminal delay, or one hour after a busy lock or live consumer). When
the tracked state is full, the reaper schedules another walk instead of
dropping work. With nothing pending, it registers nothing.
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/isolated-task-directory-retention-config.test.ts
src/__tests__/heartbeat-task-directory-restore.test.ts` (embedded
Postgres): 48 tests pass. They cover:
- a closed issue's task directory is archived and removed after the
delay by the reaper, and reopen restores files, modes and symbolic links
exactly;
- the same through reopen for a `shared_workspace` row, and through a
real heartbeat run (the run restores the files and runs in them);
- the directory stays before the delay (and the deadline is the end of
the delay), while the issue is open, and when retention is off;
- the directory stays while each consumer check fails: live run, live
runtime service, reopen fence, unsettled finalization, live lease,
unresolved recovery action, issue bound elsewhere, live sibling
workspace in the directory;
- no removal without restore points, with a non-durable key, before
`notBefore`, or over the limits;
- dry run changes nothing; a change after the capture rolls back; an
interrupted removal finishes on the next sweep; an expired restore point
reopens empty with a warning; a diverged directory fails the reopen and
stays untouched;
- folder sweep: an orphan is removed after the grace period, kept when
recently modified, in dry run, when a live row points into it, and when
its issue is open; an archived unbound row's folder is removed; an
archived bound row's folder is saved, removed and restored on reopen;
the walk is bounded and continues from its cursor;
- review follow-ups: a file name with a backslash saves and restores;
after the startup walk, a workspace archived later is found through its
change hint; an orphan used by a live run is retried; skipped rows do
not starve later rows of the per-sweep budget; overflowing tracked
deadlines schedule another walk instead of being dropped; a closing
child sends the reaper back to its archived parent's folder; a pending
lease cleanup on a closed workspace keeps an orphan; a deep path at the
platform path limit restores; an expiry recorded during removal survives
and the archive and removal are recorded as activity; a task closed
beyond retention keeps its new restore point for the full period;
- symbolic links: a task directory that is a link, an orphan that is a
link, and a link inside a directory are never followed; the link targets
stay untouched.
- Mutation checks. Each change below made at least one of the tests
above fail, and the test passed again after revert:
- drop the run-path reopen condition; drop the skipped-row cache; drop
hint tracking; reject backslashes again; drop the live-consumer retry
for folders; evict overflowing folder deadlines; skip ancestor hints;
ignore workspace-attached leases for orphans; stage restores under the
old longer name; overwrite the whole metadata on the final write; retain
task directory restore points from the tree close only; drop the removal
activity; force `closedSiblingsAllowed` off; make live siblings not
block; skip the digest recheck before rename; skip the issue binding
check; remove both symbolic link refusals (task directory); remove the
orphan symbolic link refusal; ignore live rows in the folder walk; skip
the orphan modification grace; ignore dry run for orphans; skip the
restore on reopen; skip the terminal delay; skip the restore points
requirement; skip the orphan open-issue check.
- Removing only one of the two task directory symbolic link checks did
not fail a test, because the other check still refuses the link.
Removing both fails the test.
- Related suites: `pnpm exec vitest run execution-workspaces
isolated-task-directory workspace-runtime git-restore-point
git-workspace-expiration terminal-workspace workspace-reaper
heartbeat-git-restore heartbeat-task-directory
isolated-repository-rehome nested-clone-migration environment-selection`
in `server/`: 34 files, 749 tests pass, 4 skipped. Two
`workspace-runtime.test.ts` tests fail (`does not diagnose a missing
remote branch as a rejected credential`, `rejects explicit main on a
master-only remote ...`). They fail the same way on unchanged `master`
on the local machine, so they are not caused by this change.
- `pnpm run typecheck` in `server/`: passes.
- Not run: the full server suite, a live instance rollout, and a check
that the host disk usage drops after removal.
## Risks
- The feature is off by default, and it starts as a dry run when
enabled. It needs restore points to remove anything. Rollback: set
`PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED=false`. Saved restore points
and restore on reopen keep working.
- Orphan folders are removed without a restore point. This applies only
when no workspace row is bound to the issue at that path. If such an
issue is reopened later without another workspace, its next run starts
in an empty directory. The grace period (default 7 days of a closed tree
and no changes) and the dry run limit this risk.
- A directory over the restore point limits stays on disk (logged with
its reason). Special files keep a directory too.
- `node_modules` next to a `package.json` is not saved. A reopened task
must install its packages again.
- The restore point lives as long as git restore points (90 days by
default), counted from the later of the tree close and the removal. Only
task directory restore points use the removal time; git restore points
keep their retention. After that, reopen starts with an empty directory
and posts a warning comment.
- Reopen now accepts a closed `shared_workspace` row, but only when it
holds task directory state written by this feature.
- No schema migration. The ledger table and its columns are unchanged.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees and saves
restore points for unsaved work
> - It treated a workspace as closed only when the source issue and
every descendant were done or cancelled
> - A routine that closes a parent and leaves children open for days
therefore kept every parent worktree, although the children work in
their own workspaces
> - The idle path did not help, because the children are updated every
day
> - This pull request counts only the issues that use the workspace when
it decides that a closed workspace can expire
> - The benefit is that closed parent worktrees are removed while their
children work elsewhere, and a child that later binds to the workspace
gets it back from the restore point
## Linked Issues or Issue Description
Refs #17271. Bug report described in-PR (bug report template).
**What happened?**
On one instance, an hourly routine creates a parent issue, works it in
its own worktree, marks the parent done, and leaves child issues open
(`blocked`, `in_review` or `in_progress`) for days. The children work in
their own workspaces. They are not bound to the parent's workspace. Git
cache expiration kept every parent worktree (about 300 MB each), because
the closed-tree check and the workspace reaper required all descendants
to be closed. In one project, 18 of 19 long-closed parent worktrees were
kept only by open children that did not use them. The idle path never
ran, because the children were updated daily.
**Expected behavior**
A closed issue's worktree expires while descendants are open, as long as
no open descendant uses that workspace.
**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Close an issue that has an isolated git worktree, and leave a child
issue open that works in its own workspace (or has no workspace
binding).
3. Wait past the terminal delay and run the workspace reaper: the parent
worktree stays, and the reaper counts it as `skippedNonTerminalTree`.
**Paperclip version or commit:** `master` at 410e5c6eed.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- `isGitWorkspaceExpirationEligible` (closed path, not the idle path):
the "tree closed for one day" check now covers only the issues in the
tree that use the workspace. An issue uses the workspace when it is the
source issue, when its `execution_workspace_id` is the workspace, or
when its current `workspace_selection` or its
`workspace_pending_selection` selects it as the `existing` workspace.
Inheritance from a parent (`reuse_existing`) and `existing` selections
are both stored as `execution_workspace_id`, and heartbeat workspace
resolution reuses an existing workspace only through that binding or a
run's persisted native binding. An open descendant without one of these
references gets a new workspace on its next run, so it no longer keeps
the parent's worktree.
- All other checks stay the same: any open issue bound to the workspace,
live or unsettled runs of any issue in the tree or of the workspace
(including a run whose context names the workspace), finalizations,
leases, recovery actions, runtime services, shared cwd and the reopen
fence.
- Workspace reaper: when the source issue is closed, git cache
expiration applies, and no reopen is pending, the reaper routes the
workspace by the same rule. It takes the restore point due time from the
source issue and the descendants that use the workspace, also after
every descendant is closed. A recently closed child that used the
workspace still holds it for the terminal delay. The compaction delay
and the archive cooldown still use the whole tree when it is closed.
- Workspace reaper: compaction and the normal archive path keep their
behavior. While any descendant is open, the reaper does not compact the
worktree, and the expiration branch handles the workspace and continues.
The archive path still requires a closed subtree. With expiration off,
or a pending reopen, routing is unchanged.
- New exported helper `listIssuesUsingGitWorkspace` shares the SQL "uses
the workspace" predicate between eligibility and the reaper.
- Docs: `doc/recoverable-disk-cleanup.md` and the
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` entry in
`doc/DEVELOPING.md`.
- The existing `preserves live descendants and pending reopening` case
now binds its open child to the workspace. An unbound open child no
longer keeps the worktree by design.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/heartbeat-git-restore-point-resume.test.ts` in `server/`:
all pass.
- `pnpm exec vitest run src/__tests__/git-workspace-expiration
src/__tests__/git-restore-point src/__tests__/execution-workspace
src/__tests__/heartbeat-git-restore-point-resume
src/services/heartbeat/environment-selection
src/__tests__/isolated-repository` in `server/`: 15 of 17 files passed
in one parallel run. The other failures were the updated descendant case
(now fixed) and 45-second timeouts in restore point tests while 17
embedded Postgres files ran at the same time. Both files pass when run
on their own.
- `pnpm run typecheck` in `server/`: passes.
- New embedded Postgres cases with real git repositories:
- A closed parent with open `blocked`, `in_review`, `in_progress` and
`todo` children and a grandchild in their own workspaces (or with none):
the worktree expires with a restore point and the reopen restores it
exactly.
- The reaper archives it, timed from the issues that use the workspace.
A child closed one minute ago with its own workspace does not delay it.
- A recently closed child bound to the workspace holds it until its
terminal delay passes.
- When every child is closed and the most recent one used its own
workspace, the reaper does not restart the restore point delay.
- The worktree is kept when an open child is bound to it, when an open
grandchild under a child with its own workspace is bound to it, when a
child created later with `issueService.create` inherits it with
`reuse_existing`, when an open child selects it through
`workspace_selection` or `workspace_pending_selection`, when a child run
that references it is live, and when the parent was closed inside the
delay.
- The reaper never compacts the worktree while children are open, with
expiration on or off. A pending reopen keeps its current fence handling.
- After expiry, an open child that binds to the workspace reopens it
with the saved commits, index, working tree and untracked files.
- Heartbeat: a queued run for an open child that binds to the expired
workspace of a closed parent restores the workspace and runs the agent
in it with the saved state.
- Mutation checks (each run against the two restore point test files):
- Closed-tree check restored to the whole tree: 7 tests fail.
- `execution_workspace_id` removed from the rule: 3 fail.
- `workspace_selection` removed from the rule: 1 fails.
- `workspace_pending_selection` removed from the rule: 1 fails.
- Reaper `latestTerminalAt` from the whole tree: 3 fail.
- Reaper routing by the whole tree: 2 fail.
- Reaper restore point due time from the whole tree once it is closed: 1
fails.
- Pending-reopen gate removed: 1 fails.
- Compaction gate (`!treeOpen`) removed: no test fails. Compaction's own
guard under the lifecycle lock still rejects open trees, so the gate
only skips work that cannot succeed.
- Not run: the full server suite and the UI suites. CI covers them.
## Risks
- Closed parent worktrees now expire while their children are open. A
child that later binds to the workspace depends on the restore point and
on the run-side reopen from #17271. The new tests cover this path. When
restore points are off, only exactly reproducible worktrees expire.
- An open child with no workspace binding no longer keeps the parent's
worktree. Heartbeat resolution never reuses a workspace without a
binding, so this does not change where the child runs.
- The rule is conservative. A workspace selected only by an issue's
current or pending selection is kept, and live runs of any issue in the
tree still block.
- The reaper runs one extra indexed query for each closed workspace when
git cache expiration applies. No migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Isolated task workspaces are Git worktrees on the server disk, and
the terminal workspace reaper keeps that disk usage bounded
> - #17369 moves old workspaces off their private per-issue clones onto
the project's shared clone and then deletes the private clone
> - On one instance, the first live sweep moved 12 workspaces but
deleted none of their clones (about 600 MB each), and 3 workspaces could
not move because their restore point was "too large"
> - The kept clones are not referenced by any workspace, so no other
cleanup removes them
> - This pull request replaces the clone check with a real
reproducibility check, sweeps the orphaned clones, lifts the restore
point limits for local transfers, and fixes a false
`source_identity_changed`
> - The benefit is that the disk space #17369 was meant to reclaim is
reclaimed, without deleting unique work
## Linked Issues or Issue Description
Refs #17369. Bug report described in-PR (bug report template).
**What happened?**
On one instance, with `PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true`
and dry run off, the first sweep re-homed 12 workspaces, skipped 5 with
`live_or_unresolved_consumer` and 3 with `restore_point_too_large`. No
clone was removed: 11 logged `cloneKeptReason:
"clone_checkout_has_files"` and 1 logged `"source_identity_changed"`.
Each clone's own checkout was clean (`git status --porcelain
--untracked-files=all` empty). Its only history that the shared clone
lacked was remote branches and release tags that the shared clone never
fetched, which exist on the remote. An operator removed 11 clones by
hand after verifying them.
**Expected behavior**
A re-homed clone whose checkout is clean and whose local history is in
the shared clone or on the remote is deleted. A clone with real unique
work is kept. Workspaces whose changes exceed the upload limits still
move, because the transfer is local.
**Steps to reproduce**
1. Use a project repository whose `.gitignore` (or the clone's
`info/exclude`) contains `.paperclip/`, with a release tag on a commit
that is not on any branch.
2. Re-home an isolated workspace of that repository with #17369.
3. The old worktree is removed, the empty `.paperclip/worktrees`
directory stays, and `git ls-files --others --ignored --directory` lists
`.paperclip/`. The clone is kept with `clone_checkout_has_files`.
Without the ignore line, the local release tag keeps it with
`unique_clone_history`.
4. Add a file larger than 64 MiB to a workspace. The move skips with
`restore_point_too_large`.
**Paperclip version or commit:** `master` at aa0037a137 (rebased on
ee29e2850d, after #17407).
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- New clone reproducibility check (`cloneDisposable` in
`server/src/services/isolated-repository-rehome.ts`), used before
deleting a re-homed clone:
- `git worktree prune`, then only the clone's own worktree may remain
(`base_has_worktrees`);
- no tracked changes and no untracked files
(`clone_checkout_has_files`); the only untracked exception is
Paperclip's compaction trash;
- ignored files only inside regenerable directories (`node_modules`,
`dist`, ...) or Paperclip's compaction trash. Ignored entries are listed
with `--no-empty-directory`, and a non-regenerable ignored directory is
expanded file by file, so empty directories such as the old worktree's
`.paperclip/worktrees` parent no longer count;
- only `refs/heads`, `refs/tags` and `refs/remotes` (no stash, notes or
replacement refs: `unsupported_clone_refs`);
- every commit reachable from local branches, local tags, non-remote
reflogs, `ORIG_HEAD` and `HEAD` (so a detached checkout counts) must be
on a remote-tracking ref, present in the shared clone, or on the remote.
The remote check reuses `hasCommitsMissingUpstream` (now exported),
which fetches the remote's branches and tags into a temporary store, and
it runs only when some commit is in neither place. A local tag is
accepted when its commit is on the remote. Otherwise the clone is kept
(`unique_clone_history`), or kept with
`managed_checkout_upstream_unavailable` when the remote cannot be read;
- existing 5,000 budgets for refs, reflog entries and unique commits
(`clone_history_budget`), plus at most 1,000 ignored directories outside
regenerable ones, read in one Git call (`clone_file_budget`);
- the log entry now includes `cloneKeptPaths`: up to five paths,
worktrees, refs or commits that kept the clone.
- Cleanup also checks that the clone's `origin` still matches the
workspace `repoUrl` before deleting it.
- Old worktree ownership (#4): it is proven by `git rev-parse
--git-common-dir`, or, when that fails, by the worktree's `.git` pointer
into `<clone>/.git/worktrees/`.
- Orphan sweep (`sweepOrphanedIsolatedRepositorySources`): at the first
reaper rotation after server start (feature on), it walks
`isolated-repository-sources/<company>/<issue>/checkout`. It holds the
isolated source lock and skips clones that a workspace still needs (an
open workspace with matching `isolatedRemoteSource`, a closed one that
is not expired with its worktree removed, a project workspace or a
running service), clones with a pending move, and clones whose re-homed
workspace still has its own cleanup pending. The repository URL comes
only from the database: the issue's workspaces must record exactly one
URL and the clone's `origin` must equal it (`unproven_repository` or
`source_identity_changed` otherwise), so a rewritten `origin` is never
fetched. It then applies the same check (using the shared clone of a
workspace of the same issue and repository when one exists), renames to
trash and deletes. It logs `event:
"execution_workspace.isolated_source_orphan_cleanup"` and removes
leftover trash directories from interrupted removals. It inspects at
most eight issue directories per reaper sweep and resumes from the saved
position in the next sweep, so slow remotes cannot hold up other
cleanup. When a full walk kept any orphan, it registers a deadline one
day later in the reaper's existing deadline set. Otherwise it registers
nothing until the next server start. No new timer.
- Restore point limits for re-home (#3):
`ISOLATED_REHOME_RESTORE_POINT_LIMITS` (500,000 files, 1 GiB per file
and in total, 1.5 GiB bundle, 1,000,000 commits) replaces the upload
defaults for this local transfer. Verification is unchanged (exact
snapshot equality). In `git-restore-point.ts`, `applyRestorePoint` now
streams `git diff --binary` into `git apply` instead of buffering it
(and skips the pipe when there is no diff, so no new Git version is
needed). On a failure it kills both processes and waits for both. The
ignored-file read sizes its buffer from the tree instead of the default
limit.
- Docs: `doc/recoverable-disk-cleanup.md`.
### What I found for `source_identity_changed` (#4)
In #17369, cleanup ran `git rev-parse --git-common-dir` in the old
worktree and used `""` when the command failed. Then it compared that to
the clone's `.git` path, so any Git failure in the old worktree was
reported as `source_identity_changed`. Git fails there when the old
worktree's registration in the clone (`<clone>/.git/worktrees/<name>`)
is gone while the directory still exists. Its `.git` file then points to
a missing directory. I reproduced this by deleting the registration
after a re-home: the old code keeps the clone with
`source_identity_changed`, the new code removes the old worktree and the
clone. I could not inspect the instance, so the exact event that removed
that one registration is not confirmed. The check was a false positive
in that case, and the fix proves ownership from the `.git` pointer
instead. A worktree whose pointer leads outside the clone is still kept.
## Verification
- `pnpm exec vitest run
src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 28
passed (real Git repositories, embedded Postgres). New cases:
- a clean clone whose only extra history is a remote branch and a
release tag on a commit outside any branch, both absent from the shared
clone, with `.paperclip/` ignored in the clone: deleted;
- a clone with a local branch commit that is neither upstream nor in the
shared clone: kept (`unique_clone_history`, the commit is in
`cloneKeptPaths`);
- a clone with a tracked change, then an ignored non-regenerable file
next to an ignored `node_modules`, then an untracked `build/notes.txt`:
each kept (`clone_checkout_has_files` with the path), then deleted after
cleanup;
- a clone with a second registered worktree: kept
(`base_has_worktrees`);
- a clone whose detached `HEAD` holds the only copy of a commit (reflogs
expired): kept (`unique_clone_history`);
- an old worktree whose registration was pruned: old worktree and clone
removed;
- an orphan whose `origin` was rewritten is kept with
`source_identity_changed`, and with `unproven_repository` once no
workspace records its repository;
- 1,001 ignored directories keep a clone (`clone_file_budget`);
- orphan sweep: a batch of one returns a resume position and the next
batch finishes the walk; a dry run changes nothing. Then an orphan whose
history is in the shared clone or on the remote is deleted, and an
orphan with no workspace rows and an unpushed commit is kept, with the
next deadline one day later;
- a workspace with 65 MiB untracked and ignored files (default capture
refuses them with `restore_point_too_large`) moves with an exact
snapshot match.
- After rebasing on #17407 (nested clone migration, which reuses helpers
from the re-home module): `isolated-repository-rehome`,
`nested-clone-migration`, `execution-workspaces-service`,
`execution-workspaces-routes`, `execution-workspace-reopen`,
`execution-workspace-compaction`, `execution-workspace-policy`,
`execution-workspace-runtime-control-conflict`,
`execution-workspace-runtime-lease-route`, `git-workspace-expiration`,
`git-restore-point`, `git-restore-point-expiration`: 405 passed;
typecheck passes.
- Related suites in `server/` (before the rebase):
`git-workspace-expiration`, `git-restore-point`,
`git-restore-point-expiration`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`,
`git-reproducible-managed-checkouts`,
`isolated-remote-workspace-source`, with the new suite: 398 passed.
- `pnpm run typecheck` in `server/` passes.
- Mutation checks, each failing a test:
- remote proof replaced by "always unique": the unfetched-history and
orphan tests fail;
- remote proof replaced by "never unique": the local-commit and orphan
tests fail;
- `.git` pointer fallback removed: the pruned-registration test fails;
- re-home limits removed: the large-file test fails;
- orphan sweep removed from the reaper: the orphan test fails;
- worktree count check removed: the second-worktree test fails;
- regenerable-directory exception applied to untracked files: the
dirty-checkout test fails;
- orphan batch limit removed: the orphan test fails;
- orphan URL taken from the clone again: the rewritten-origin test
fails;
- ignored directory budget removed: the many-directory test fails;
- `HEAD` left out of the checked commits: the detached-HEAD test fails.
- Dropping only `--no-empty-directory` does not fail a test, because the
per-directory expansion also accepts an empty ignored directory. The two
mechanisms are redundant on purpose.
- Not run: the full server suite, or the change on a live instance.
## Risks
- Deleting a clone now depends on reading the remote when some local
commit is neither on a remote-tracking ref nor in the shared clone. If
the remote cannot be read, the clone is kept.
- Commits that only remote-tracking refs hold (for example a remote
branch deleted upstream after the clone fetched it) are treated as not
unique, as before. They were pushed once.
- The orphan sweep deletes clones that no workspace refers to. A closed
workspace that is expired with its worktree removed does not count as a
user, because reopening it rebuilds from the shared clone or clones
again.
- Re-home transfers can now use up to about 1.5 GiB of memory for a
single restore point payload, so a very large workspace uses more memory
during its move than before.
- The orphan sweep runs once per server start and then only while it
keeps an orphan, so a clone that becomes orphaned later waits for the
next restart.
- No migration. No schema change.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, Git, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each task runs in a git worktree. Extra project repositories sit in
the task under `.paperclip-repositories/`.
> - #17382 made new nested checkouts git worktrees of one shared clone
per project repository. Existing standalone nested clones stayed as full
clones.
> - Long-lived tasks keep those clones for a long time. Many tasks hold
the same large repository, so most of the disk is duplicate `.git` data.
> - #17369 added a locked and verified move of isolated workspaces onto
the shared clone, with a reaper sweep behind a setting.
> - This pull request adds the same kind of locked and verified
conversion for standalone nested clones in open tasks, and runs it from
the same reaper path behind a new setting.
> - The benefit is that the duplicate clones go away without losing any
state or local history, and the disk stops filling.
## Linked Issues or Issue Description
Refs #17382#17369. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 44 open task worktrees in one project each hold a
standalone nested clone of the same large repository. Together they use
about 25 GB. About 16 GB of that is duplicate `.git` data. Another
project has 33 smaller clones. The clones are removed only when their
tasks expire, and many tasks stay open for a long time. The disk keeps
filling.
**Expected behavior**
Existing standalone nested clones in open tasks become worktrees of the
project's shared clone, with the same state and local history, and the
standalone clones are deleted.
**Steps to reproduce**
1. Prepare a task with a nested project repository before #17382, or
without a shared clone, so that `.paperclip-repositories/<name>` is a
standalone clone.
2. Create the project's shared clone, as #17382 does for new tasks.
3. Observe that the old checkout stays a full clone until the task
expires.
**Paperclip version or commit:** `master` at 8549ecaf9a.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- New `server/src/services/nested-clone-migration.ts`. It converts one
standalone nested clone in an open execution workspace into a worktree
of the shared clone:
- It holds the source lock and workspace lifecycle lock (as git cache
expiration does), the per-shared-clone lock from #17382, and the
expiration lock files (`index.lock`, `HEAD.lock`, current branch ref
lock) in the clone.
- It requires the #17369 no-live-consumer eligibility (no live runs,
leased finalizations, unreleased leases, running services, or reopen
flag). It checks again before the swap.
- It identifies the repository by managed checkout name, project
workspace rows and the origin URL, and it requires a verified shared
clone.
- It brings local history into the shared clone under the checkout's own
`paperclip/<id>/` prefix. The current branch becomes
`paperclip/<id>/<branch>`, tracking `origin/<branch>`. Unpushed local
branches become `paperclip/<id>/<branch>`. Unpushed tags become
`paperclip/<id>/tags/<tag>`. A restore point carries the commits, the
`HEAD` reflog and `ORIG_HEAD`.
- It stages the worktree at `<name>.clone-migrating`, applies the
restore point, copies the `HEAD` reflog and `ORIG_HEAD`, and moves
regenerable dirs, untracked `.env` files and
`.claude/settings.local.json` by rename. Old and new must then match
exactly (snapshot, history, branch, created refs, moved items).
- Swap: it renames the clone to `<name>.clone-standalone`, runs `git
worktree move` and `git worktree repair`, updates a saved restoration
recipe that lists the checkout, and then renames the old clone to trash
and deletes it.
- A `nestedCloneMigrationPending.<name>` metadata record makes recovery
possible. A crash before the swap rolls back (moves items back, removes
the staged worktree and the created branches). A crash after the swap
rolls forward.
- Skip reasons (nothing changes): `live_or_unresolved_consumer`,
`unknown_managed_checkout`, `local_source_checkout`, `foreign_origin`,
`no_shared_base`, `shared_base_repo_mismatch`,
`unsupported_base_git_config`, `nested_clone_has_stash` (stashes are
skipped, because `refs/stash` would be shared by every worktree),
`nested_clone_has_worktrees`, `nested_clone_has_hooks`,
`unsupported_local_git_config`, `unsupported_local_git_state`,
`unsupported_info_exclude`, `unsupported_nested_clone_refs`,
`unsupported_index_flags`, `unique_branch_reflog_commits`,
`nested_clone_ref_conflict`, `snapshot_mismatch`, `history_mismatch`,
restore point limits, and `nested_migration_recovery_conflict`.
- Reaper: `sweepTerminalWorkspaces` runs the conversion for open
workspaces when `PAPERCLIP_NESTED_CLONE_MIGRATION_ENABLED=true` (default
off). `PAPERCLIP_NESTED_CLONE_MIGRATION_DRY_RUN=true` only reports. Each
attempt logs `event: "execution_workspace.nested_clone_migration"` with
outcome, reason, branch, candidate bytes and reclaimed bytes. There is
no new timer. A retry deadline is registered on the existing reaper
deadlines only for a pending record or a transient skip (live consumer,
busy lock, changed row). A workspace is attempted again at most hourly.
A workspace without standalone clones costs one directory listing.
- `isolated-repository-rehome.ts`: I exported the shared pieces
(eligibility, locks, safe base config check, transfer plan, history
boundary, snapshot and history keys, item moves, rollback and
verification) instead of copying them. Re-home behavior does not change.
- `nested-repository-worktrees.ts`: I exported the stale registration
cleanup and added the `moveNestedRepositoryWorktree` (move and repair)
and `trackNestedWorktreeUpstream` helpers.
- Docs: `doc/recoverable-disk-cleanup.md` (new section about the
conversion) and `doc/DEVELOPING.md` (settings).
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/nested-clone-migration.test.ts`: 19 passed. It uses
embedded Postgres and real git repositories, and covers:
- Conversion with an unpushed commit, a second branch with unpushed
commits, a lightweight tag and an annotated tag, staged and unstaged
edits, a deleted file, an untracked file with mode 0600, an ignored
file, `node_modules`, `.env` and the settings file. The result is a
worktree of the shared clone (`--git-common-dir`), with the same HEAD,
status, staged content and reflog, the owned branch tracking
`origin/main`, and the history under the task's prefix. No other branch
in the shared clone changes. The old clone is gone and reclaimed bytes
are reported.
- Skips with nothing changed: stash, live run, foreign origin, unknown
checkout, no shared clone, local excludes, and a change between capture
and verification (hook).
- Two tasks convert concurrently against one shared clone. The test
holds the shared-clone lock and shows that neither conversion advances.
Then they run one at a time (peak 1), and each keeps its own branches.
- Crash recovery at `afterAside`, `afterMove` and `afterFinalize` (roll
forward), and before the swap (roll back, then convert). A dry run
reports the pending record and changes nothing.
- Expiration and restore of a converted checkout work afterwards.
Unpushed branches in a converted checkout block expiration
(`unpushed_managed_checkout_branches`).
- Reaper: disabled by default; dry run changes nothing and registers no
deadline; a live run registers a retry deadline that survives a second
sweep; enabled conversion leaves no deadline.
- Commits that only the clone's `origin` remote-tracking refs hold are
kept as `paperclip/<id>/remotes/origin/<branch>`.
- A pending conversion is finished after the workspace is archived, and
forgotten (keeping unpushed branches) when the worktree is gone.
- A filter planted in the shared clone right before the worktree add is
refused (`unsupported_base_git_config`) and never runs.
- Only checkouts named after configured project repositories are
considered; at most 8 per sweep, pending first, rotating (unit test).
- Related suites (run before review fixes; the nested, re-home and
execution workspace service suites were rerun after them): `pnpm exec
vitest run` on execution-workspaces-routes,
execution-workspaces-service, git-reproducible-managed-checkouts,
git-restore-point (all 5 files), git-workspace-contents,
git-workspace-expiration, git-workspace-expiration-eligibility,
heartbeat-git-restore-point-resume, heartbeat-project-repositories,
isolated-repository-rehome, nested-clone-migration: 15 files, 393 tests
passed.
- `cd server && pnpm run typecheck` (`tsc --noEmit`): passes.
- Mutation checks. Each mutation makes the targeted test fail, and I
reverted each one:
- Remove both eligibility checks: the live run test fails.
- Remove the snapshot comparison: the capture/verify change test fails.
- Disable roll forward after the swap: the recovery tests fail.
- Drop the `paperclip/<id>/` prefix from transferred branches: the
conversion and concurrency tests fail.
- Remove the shared-clone lock: the concurrency test fails.
- Drop remote-tracking refs from the plan, the deadline restore during
backoff, archived recovery, or the config check before the worktree add:
each targeted test fails.
- Not run: the full server suite and a run on a live instance.
## Risks
- The setting is off by default. Enable a dry run first and read the
`execution_workspace.nested_clone_migration` logs.
- A clone that a run writes to between verification and the swap rename
can lose those writes. This window is the same as in #17369. Locks, the
eligibility check and the git lock files make it small.
- A crash in the short time between the rename aside and `git worktree
move` can collide with a run that recreates `<name>`. In that case the
conversion stops with `nested_migration_recovery_conflict` and keeps
both copies. Expiration keeps such a workspace because of the `.clone-`
names.
- Stashes, annotated tag messages (the tagged commit is kept as a
branch), custom hooks, custom local config and per-branch reflog-only
history are not converted. Clones with them are skipped with a named
reason. Only unpushed branches and tags get a name in the shared clone.
A pushed local branch name is not kept, but its commits stay on the
remote.
- The conversion fetches the shared clone first. A failed fetch is
ignored, and then the restore point can be larger.
- Pushing from a converted checkout needs an explicit destination, as
for all #17382 nested worktrees.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each isolated task workspace is a Git worktree on the server disk,
and the terminal workspace reaper and git cache expiration keep that
disk usage bounded
> - For some days, workspace preparation gave each isolated
`git_worktree` workspace its own full clone under
`isolated-repository-sources/`, with the worktree inside it. #16963
removed that creation path, but the existing clones stay, and restoring
an expired workspace still recreates its private clone
> - On one instance, 30 such clones use about 23 GB: 12.1 GB of `.git`
data, 5.2 GB of unused default-branch checkouts and 5.75 GB of real
worktrees. 28 of them back open workspaces, and the disk is nearly full
> - This pull request moves each such open workspace onto the project's
shared managed clone with its exact state, deletes the private clone,
and makes restore use the shared clone
> - The benefit is that disk use drops to one shared clone per project
repository, with no lost work and no change for workspaces that cannot
move safely
## Linked Issues or Issue Description
Refs #16963. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 30 per-issue clones under
`isolated-repository-sources/<companyId>/<issueId>/checkout` use about
23 GB. 28 of them back open workspaces. No process moves these
workspaces to the shared project clone that new workspaces use, and git
cache expiration does not remove a clone while its workspace is open.
Reopening an expired workspace whose recipe has `sourceKind:
"isolated_remote"` clones the full repository again. Separately, a
restore point with a non-default file mode cannot be restored: the
reference read back from jsonb has reordered `modes` keys, so
`loadGitRestorePoint` fails with `restore_point_manifest_mismatch`.
**Expected behavior**
An open workspace with a private clone moves to the shared project clone
with the same branch, HEAD, index, worktree, untracked and ignored
files, and file modes. The private clone is then deleted. Reopening an
expired workspace uses the shared clone when one exists. A restore point
with non-default file modes restores.
**Steps to reproduce**
1. Create an isolated `git_worktree` workspace with
`materializeIsolatedRemoteWorkspaceSource` and record
`isolatedRemoteSource` metadata, as preparation did before #16963.
2. Keep its issue open. The clone stays, because no sweep moves or
removes it.
3. Expire the workspace with a restore point that has a file with mode
`0600`, then reopen it. The private clone is created again, and the
restore fails with `restore_point_manifest_mismatch`.
**Paperclip version or commit:** `master` at ec2e0d6407.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- New `server/src/services/isolated-repository-rehome.ts` with
`rehomeIsolatedRepositoryWorkspace`. It holds the isolated source lock,
the `execution_workspace_lifecycle:<id>` advisory lock and
expiration-style Git lock files (`index.lock`, `HEAD.lock` and the
branch ref lock in the clone).
- Eligibility uses the safety checks of git cache expiration that matter
while a workspace stays open: no live or unfinished relevant runs, no
unsettled native finalizations, no unreleased environment leases, no
running runtime services, no pending reopen, and no other workspace on
the same path. It does not require a closed or idle issue tree. The
check runs again inside the update transaction.
- The target base is the project's existing shared managed clone for the
same `repoUrl` (the paths `ensureManagedProjectWorkspace` uses, verified
with the same origin and storage checks as restoration).
- The move captures a restore point of the old worktree. The upstream
boundary is commits that both repositories have, so the bundle is small.
It imports the bundle into the shared clone, creates
`refs/heads/<branch>` or fast-forwards it from an ancestor, and adds the
worktree at `<base>/.paperclip/worktrees/<branch>`. Then it applies the
restore point. It copies the `HEAD` reflog and `ORIG_HEAD`, copies the
branch's upstream config when it creates the branch, and moves
regenerable directories, untracked `.env`/`.env.*` files,
`.claude/settings.local.json` and nested `.paperclip-repositories/*`
checkouts by rename. Standalone nested clones move as is. Nested
checkouts that are worktrees of their own shared clone (#17382) move
too, followed by `git worktree repair` in that clone so the registration
follows the new path, and their `paperclip/<hash>/` branch is renamed to
the new path's prefix so ownership checks still pass. Entries inside a
moved nested checkout (for example its `node_modules`) move with it.
Rollback and interrupted-move recovery repair them again after moving
them back, and verification checks that each one still resolves to the
same shared clone.
- Verification: the restore point snapshot (HEAD, index, worktree,
ignored files, modes) of the new worktree must equal the old one before
and after the move. The reflog and `ORIG_HEAD` must match, and every
moved item must exist. Only then does one transaction set
`cwd`/`providerRef`, drop `isolatedRemoteSource` and
`gitCacheExpiration`, rewrite a saved active restoration recipe to
`managed_project`, and record `isolatedRepositoryRehome: { at, from, to,
cleanedAt }`. A failure before that removes the new worktree, moves
items back, restores the branch, and leaves the old worktree untouched.
- Safety of the checkout in the shared clone: the move skips with
`unsupported_base_git_config` when the shared clone's local config sets
filter drivers, includes, attribute files, worktree config or command
settings, or when it has `info/attributes`. It checks before planning
and again right before `git worktree add`.
- Crash safety before the update: the move records
`isolatedRepositoryRehomePending` (base, target, branch, head, previous
branch tip, planned moves) before the first change to the shared clone.
The next attempt undoes an interrupted move under the same locks: it
moves items back, removes the new worktree only when it is proven to be
this move's (a linked worktree of the shared clone on the issue branch
that no workspace row references), and restores the branch. A live
failure removes the target only after this attempt created it. When an
item exists in both places, rollback stops with
`rehome_rollback_conflict` and keeps the target and the record, so no
copy is lost.
- Cleanup after the update: remove the old worktree (rename to trash,
then delete, then `worktree prune`). Delete the clone (rename to trash,
then delete) only when no other workspace, project workspace or running
service uses it, it has no other worktree, its checkout has no changes
or non-regenerable files, its refs are only branches, tags and
remote-tracking refs, and every local commit is on a remote-tracking ref
or present in the shared clone. Cleanup reads at most 5,000 refs, reflog
entries and unique commits (`clone_history_budget`). `cleanedAt` records
completion. A crash after the update leaves a pending record that the
next pass finishes, also after the workspace is archived: the reaper
scan includes archived rows with unfinished re-home cleanup when the
feature is on.
- Reaper integration in `sweepTerminalWorkspaces`: behind
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED` (default `false`) and
`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN`, only when git cache
expiration is enabled. It logs `event:
"execution_workspace.isolated_source_rehome"` with outcome, reason,
candidate bytes and reclaimed bytes. A skipped workspace gets a retry
deadline one hour later in the reaper's existing deadline set. No new
timer.
- Restore: `reopenClosedIsolatedExecutionWorkspaceForIssue` calls
`rehomeExpiredIsolatedRestoration` first. For an expired workspace with
an `isolated_remote` recipe, it uses the shared clone when one exists
and the branch there is absent or at the saved commit. It keeps the old
behavior for recipes with worktree-backed nested checkouts on branches,
because those branch names are bound to the old path. It also requires
that the shared clone has every base commit (prerequisite) of the saved
restore point bundle. It rewrites the row and recipe to the shared clone
and the standard worktree path, then the existing restore runs
(`ensureGitRestorePointCommit`, `ensureSavedGitCommit`, apply). Without
a shared clone, the old behavior stays.
- Fix: restore point `modes` comparison in
`git-restore-point-storage.ts` compares by path and mode, so jsonb key
order does not matter.
- Small exports: `allocatedBytes`, `expirationScope`,
`managedSourcePaths`, `verifiedGitCacheSource`.
- Docs: new "Isolated repository sources" section in
`doc/recoverable-disk-cleanup.md`, new settings in `doc/DEVELOPING.md`.
Deviations from the requested design:
- The sweep does not create a missing shared clone. When none exists, it
skips with `no_shared_base`, because a new network clone in a cleanup
path can use more disk. Restore follows the same rule and keeps the old
behavior.
- History moves through the restore point bundle, not a separate fetch
from the clone. The bundle contains HEAD and every reflog/`ORIG_HEAD`
commit that the shared clone does not have. The move then checks that
all of them are present.
- The standard location is `<base>/.paperclip/worktrees/<branch>`. A
worktree outside the clone's standard location is skipped
(`nonstandard_worktree_location`).
- Lock files use a token derived from the workspace id, so a crashed
attempt's locks are recovered without a database marker.
- A clone that another open workspace still uses is kept and marked
`cloneRetained: "base_has_consumers"`. That workspace's own move removes
it later.
Skip reasons: `live_or_unresolved_consumer`,
`unsupported_base_git_config`, `rehome_rollback_conflict`,
`nested_branch_conflict`, `invalid_rehome_record`,
`interrupted_rehome_pending` (dry run only), `lifecycle_busy`,
`source_busy`, `workspace_changed`, `no_shared_base`,
`shared_base_repo_mismatch`, `missing_worktree`,
`nonstandard_worktree_location`, `source_identity_changed`,
`branch_identity_changed`, `unproven_worktree_admin`,
`unsupported_worktree_git_state`, `unique_worktree_refs`,
`unknown_worktree_git_logs`, `unsupported_local_git_state`,
`unique_git_refs`, `unsupported_index_flags`, `unsupported_branch_name`,
`invalid_restoration_recipe`, `unsupported_restoration_recipe`,
`target_worktree_exists`, `base_branch_checked_out`,
`base_branch_diverged`, `untransferable_ignored_files`,
`untransferable_managed_checkouts`, `worktree_git_busy`, `head_changed`,
`history_not_transferred`, `transfer_conflict`, `snapshot_mismatch`,
`history_mismatch`, `transfer_incomplete`, and restore point reasons
such as `untracked_nested_repository`, `submodules_unsupported`,
`filtered_paths`, `restore_point_too_large`. Clone kept reasons:
`base_has_consumers`, `base_has_live_services`, `base_has_worktrees`,
`clone_checkout_has_files`, `unsupported_clone_refs`,
`unique_clone_history`, `clone_history_budget`.
## Verification
- `pnpm exec vitest run
src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 18
passed. It uses real Git repositories and embedded Postgres. It covers:
- a move with an unpushed commit, staged and unstaged edits, a deletion,
an untracked file with mode `0600`, an ignored file, `node_modules`,
`.env`, `.claude/settings.local.json` and a nested
`.paperclip-repositories` checkout. Snapshots, branch, HEAD, `git
status` and reflog match, the row points to the new path without
`isolatedRemoteSource`, and the old worktree and clone are gone;
- skips with nothing changed on disk or in the database: live run,
branch in the shared clone at an unrelated commit, no shared clone, an
untracked nested repository, and a concurrent change between capture and
verify (hook). The new worktree is removed and the old one keeps its
moved items;
- the clone is kept while another workspace references it;
- a worktree-backed nested checkout moves with the task worktree, stays
registered in its shared clone at the new path (not prunable), and keeps
its changes, with its branch renamed to the new path's prefix. On
rollback it returns to the old path with its registration repaired and
its branch name restored. A standalone nested checkout moves together
with its own ignored `node_modules`;
- a file recreated in the old worktree before verification stops the
rollback and keeps both copies, then the move completes once the
conflict is gone;
- a hostile `filter.*.smudge` in the shared clone's config makes the
move skip without running it;
- a clone with more than 5,000 refs is kept, and is deleted once the
refs are gone;
- a move interrupted before the database update is reported (not undone)
by a dry run, then undone and completed on the next attempt;
- a target that another party creates right before `git worktree add` is
never deleted;
- a crash after the database update; after the row is archived, a later
reaper sweep removes the old worktree and clone;
- restore keeps the private-clone fallback when the shared clone lacks
the restore point's base commits;
- disabled by default (both flags needed), dry run changes nothing, and
the retry deadline is registered;
- reopening an expired `isolated_remote` workspace rebuilds it in the
shared clone with the restore point applied and no clone created.
- After the round 4 fixes, the related suites
(`git-workspace-expiration`, `git-restore-point-expiration`,
`execution-workspace-reopen`, `execution-workspaces-service`,
`heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`,
`isolated-remote-workspace-source`) with the new suite: 326 passed.
- After rebasing on #17382, the related suites (adding
`heartbeat-project-repositories`) with the new suite: 374 passed.
- After the first review fixes, the related suites
(`git-workspace-expiration`, `git-restore-point`,
`git-restore-point-expiration`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`) with the new suite: 301 passed.
- Related suites in `server/` (first commit): `pnpm exec vitest run` on
`git-workspace-expiration`, `git-workspace-expiration-eligibility`,
`git-restore-point`, `git-restore-point-expiration`,
`git-restore-point-expiration-config`, `git-restore-point-ledger`,
`git-restore-point-ledger-config`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`,
`isolated-remote-workspace-source-lock`, `git-workspace-base-storage`,
`git-reproducible-managed-checkouts`, `git-expiration-auth-fallback` and
`workspace-runtime`: 550 passed, 2 failed. The 2 failures are in
`workspace-runtime.test.ts` (base ref resolution) and also fail on
unmodified `master` on the same machine.
- `pnpm run typecheck` in `server/` passes (after `pnpm -r --filter
"./packages/**" build`).
- Mutation checks. Each reverted guard fails one test in the new suite:
- eligibility always true: "keeps a workspace with a live run" fails;
- snapshot verification removed: "removes the new worktree and keeps the
old one when the workspace changes during the move" fails;
- clone reference check removed: "keeps the clone while another
workspace still references it" fails;
- restore re-home removed from reopen: "restores an expired isolated
workspace into the shared project clone" fails;
- old `JSON.stringify` modes comparison: the same restore test fails;
- interrupted-move recovery removed: the interrupted-move test fails;
- target existence check before `git worktree add` replaced by assumed
ownership: the foreign-target test fails;
- bundle prerequisite check removed: the stale shared clone restore test
fails;
- archived-row cleanup scan removed: the crash test fails;
- dry run check before interrupted-move recovery removed: the
interrupted-move test fails;
- `git worktree repair` after the forward move removed: both nested
worktree tests fail. The repair after moving back removed: the rollback
test fails;
- nested branch rename forward or back removed: the matching nested test
fails;
- rollback conflict check removed: the reappearing-file test fails;
- covered-entry filter removed: the full re-home test fails;
- shared clone config check removed: the hostile filter test fails;
- ref budget removed: the many-ref test fails.
- Not run: the full server suite, and the move on a live instance.
## Risks
- Workspace paths change for re-homed workspaces. Agents that resume an
old session can remember the old path. The run's working directory comes
from the updated row, but text in a resumed conversation can still name
the old path.
- The feature is off by default. It runs only with
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` and git cache expiration
enabled. Dry run is available. The restore change is not behind the
flag: reopening an expired private-clone workspace now uses the shared
clone when one exists.
- A run can start between the last eligibility check and the database
update. The update transaction checks eligibility again, and the old
worktree is deleted only after another check. This is the same window
that git cache expiration accepts.
- Imported restore point objects stay in the shared clone as unreachable
objects after a failed attempt until Git garbage collection removes
them. Base expiration keeps a base with unreachable objects, so this
only delays base expiration.
- A workspace that cannot move (for example `no_shared_base`) is retried
each hour while the feature is on. Each retry runs a few Git commands
for that workspace.
- No migration. No schema change.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, Git, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can attach extra repositories, and every task worktree
gets a managed checkout of each one under
`.paperclip-repositories/<name>`
> - Each of these nested checkouts is a full clone. The clone uses
`--dissociate`, so no objects are shared between tasks
> - On one instance, 44 task worktrees held nested checkouts of the same
repository: 24 GB of 27 GB of worktrees. Each new task added about 0.5
GB, and the disk filled twice in one day
> - Main task worktrees already avoid this, because they are git
worktrees of one managed project clone
> - This pull request makes nested checkouts git worktrees of one shared
clone per project and repository, and adapts expiration, restore points,
restore and cleanup to them without weaker safety checks
> - The benefit is that repository objects are stored once, and a new
task adds only its working files
## Linked Issues or Issue Description
No public issue exists. This is the in-PR bug report.
**What happened?**
Every task in a project with extra repositories got a full, independent
clone of each extra repository under `.paperclip-repositories/`. On one
instance this used 24 GB of 27 GB of task worktrees (16 GB of it `.git`
data) and filled the disk twice in one day.
**Expected behavior**
Nested checkouts of the same repository share one object store, like
main task worktrees share the managed project clone. Disk use per new
task is only its working files.
**Steps to reproduce**
1. Create a project with a primary repository and a second repository
that has a repository URL and no local folder.
2. Run several tasks in isolated git worktrees in that project.
3. Look at `<task worktree>/.paperclip-repositories/<name>/.git`: each
one is a full directory with its own copy of every object.
**Paperclip version or commit:** `master` at ff835a3fc6.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
Related: #15662 (SSH restores skip nested worktrees) touches a different
path; this PR does not change adapter sync.
## What Changed
- New `server/src/services/nested-repository-worktrees.ts`. It finds the
shared clone (the managed project checkout for the repository URL, from
`ensureManagedProjectWorkspace`), adds nested worktrees, restores them,
removes their registrations, and serializes all of these operations and
fetches per shared clone with an in-process lock.
- `prepareProjectRepositoryWorkspaces` takes a `sharedBase` (company and
project). For a repository without a configured local folder, it fetches
the shared clone and adds a worktree at the same
`.paperclip-repositories/<name>-<key>` path. Each worktree gets its own
branch `paperclip/<id>/<branch>` that tracks `origin/<branch>` (`<id>`
derives from the worktree path), or a detached HEAD for a tag or commit
ref. An existing worktree must belong to the project's shared clone.
Configured local-folder copies and existing standalone clones are
unchanged. Moving a worktree to `detached-repositories` repairs its
registration. Heartbeat preparation and fresh-start reopening pass
`sharedBase`.
- `assertReproducibleManagedCheckout` and `provenManagedCheckoutHistory`
accept worktree-backed checkouts. They verify the worktree link, its
admin directory, the shared clone identity and origin, and that HEAD is
detached or on a branch under the worktree's own prefix. History checks
scope to HEAD, the per-worktree HEAD reflog, ORIG_HEAD and the owned
branches with their reflogs. Other tasks' branches are not inspected.
Stashes, unsupported refs and custom config in the shared clone still
block. The upstream proof reads objects from the common git directory.
- Restoration contents record `worktree: { base, admin }` for a
worktree-backed checkout. Validation checks its shape and the owned
branch name, and every use checks the base against the project's
expected clone paths.
- Expiration locks for a worktree-backed checkout sit in the shared
clone (`.git/worktrees/<admin>/index.lock`, `HEAD.lock`, and the owned
branch ref lock). `gitCacheExpirationLockLocations` is now async.
- `finishGitCacheTrash` removes the expired worktree's registration and
deletes its owned branch only at the recorded commit.
- `restoreGitWorkspaceContents` recreates a worktree-backed checkout
from the shared clone, and clones the shared clone again if it is gone.
It imports the restore point into the shared clone, adds the worktree on
the saved branch and commit, writes reflog-only commits back into the
new HEAD reflog, and applies the restore point.
- `cleanupExecutionWorkspaceArtifacts` (terminal reaper archival, close
and delete) collects the task's nested worktrees first. After the task
directory is gone, it removes their registrations and deletes owned
branches only when every commit is on a remote-tracking ref. An unpushed
branch stays in the shared clone.
- Review follow-ups. Owned branches always track origin, including names
with dots, and the config check accepts their tracking keys. A ref held
by a retained detached checkout gets `paperclip/<id>/r<n>/<branch>`.
Restore derives the upstream from the saved source ref. Each new
worktree fetches with `--prune`, refreshes `origin/HEAD` when no ref is
configured, and strips `refs/heads/` from branch refs. Git commands on
the shared clone run with hooks disabled. The worktree `gitdir` pointer
is read through a bounded, non-following, non-blocking handle, and more
than 64 owned branches stop the proof before reflog reads.
- Docs: new "Nested project repository worktrees" section in
`doc/recoverable-disk-cleanup.md`, and updates in `doc/DEVELOPING.md`.
## Verification
- `pnpm -r --filter "./packages/**" build`, then `pnpm run typecheck` in
`server/`: passes.
- `pnpm exec vitest run` in `server/` for `git-workspace-expiration`,
`git-reproducible-managed-checkouts`, `git-workspace-contents`,
`git-restore-point*`, `heartbeat-project-repositories`,
`execution-workspaces-service`, `execution-workspaces-routes`,
`execution-workspace-compaction`, `workspace-compaction`,
`heartbeat-git-restore-point-resume`, `managed-repository-checkout`,
`task-workspace-repositories*`, `native-workspace-sync*`,
`heartbeat-managed-clone-credentials`,
`services/heartbeat/workspace-preparation`,
`project-repositories-persistence`, `git-workspace-base-storage`,
`agent-directory-probe`, `workspace-runtime`,
`workspace-preparation-history` and `native-completion-feedback`: 866
passed, 2 failed. The 2 failures are in `workspace-runtime.test.ts`
(`rejects explicit main on a master-only remote...` and `does not
diagnose a missing remote branch...`). They fail the same way on
unmodified `master` on this machine.
- New tests (embedded Postgres, real git repositories) in
`git-workspace-expiration.test.ts`, section "nested checkouts as
worktrees of a shared base":
- Two tasks get worktrees of one shared clone. `--git-common-dir` points
to it, each task has a distinct `paperclip/<id>/main` branch, upstream
is `origin/main` and the remote is the repository URL.
- Concurrent preparation of three tasks succeeds, and preparation waits
while the shared clone lock is held.
- A legacy standalone clone is still adopted, and no shared clone is
created for it.
- Expiring a clean nested worktree removes only its registration and
branch. The sibling worktree and branch stay. Reopening recreates the
worktree from the shared clone on the same branch and commit, with its
local settings.
- Reopening clones a deleted shared clone again.
- Unpushed commits and uncommitted files block plain expiration, are
saved in a restore point and are restored exactly.
- Reflog-only history is saved and written back into the restored HEAD
reflog, and it survives `gc --prune=now` of the shared clone.
- Stashes, a checked-out branch outside the worktree's prefix, and
unpushed owned branches or owned reflog entries outside HEAD history
block expiration.
- Another task's unpushed branch does not block, and it stays after
expiration.
- A worktree of a foreign clone at the nested path is rejected by
preparation and by the prover.
- Expiration locks in the shared clone are recovered after a process
exit.
- Deleting a task removes only its nested registrations, keeps its
unpushed branch and deletes the sibling's pushed branch when the sibling
is deleted.
- Follow-up tests: dotted branch tracking, a ref switched away and back
with a retained checkout, a changed remote default branch, a renumbered
branch restored next to a same-named remote branch, a fully qualified
branch ref after the remote advances, a branch deleted upstream, planted
`post-checkout` and `reference-transaction` hooks, more than 64 owned
branches, and a FIFO, oversized or symlinked `gitdir` pointer. Each fix
was mutation-checked.
- `heartbeat-project-repositories.test.ts`: a repository-URL-only
sibling is now a worktree of the shared clone on an owned branch, and
its detached copy is still a valid checkout after its repository leaves
the project. In `workspace-preparation.test.ts`, the publication guard
now runs before any worktree exists.
- Mutation checks. I reverted one guard at a time and confirmed that a
test fails: the owned-HEAD check, the owned-branch check in restore
point mode, the owned-reflog check, the shared-clone identity check, the
pushed-only branch deletion on cleanup, the cleanup release hook, the
registration removal on expiration, the reflog-only history write-back
on restore, and the shared clone lock. The owned-branch-tip check alone
is also covered by the owned-reflog check, so removing only one of the
two keeps the tests green.
- Not run: the full server test suite, the UI and e2e suites, and a live
instance migration.
## Risks
- Behavior change for agents: a nested checkout of a URL-only repository
is now on `paperclip/<id>/<branch>`, not `<branch>`. `git pull` and `git
status` work as before. A plain `git push` now fails because the branch
names differ. Agents must push with an explicit destination, for example
`git push origin HEAD:<branch>`. Branches in the shared clone are
visible to all tasks of the project, like main task worktree branches.
- Agent-created branches outside the worktree's prefix, stashes, and
local config live in the shared clone. Stashes and custom config block
expiration of every worktree of that clone (fail-closed). Other branches
stay in the shared clone after a task is deleted.
- The shared clone is the managed project checkout for that repository
URL. Fetches into it update its remote-tracking refs. A concurrent
restore point capture of a sibling can then roll back and retry on the
next sweep.
- The base lock is per process. Across processes, git's own ref and
index locks apply.
- No migration in this PR. Existing standalone nested clones keep their
disk use until their tasks are removed or expired. A safe migration
helper is a follow-up. Task repositories (`task-repo-*`) and configured
local-folder copies stay standalone clones.
- Restoration recipes now can carry `contents.checkouts[].worktree`. An
older server rejects such a recipe as invalid (fail-closed) and does not
restore without it.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs sync their workspace with a remote sandbox through a
per-run folder under `<instance>/native-workspace-sync/<run>/`
> - The folder holds a descriptor, a manifest, and workspace and git
seed archives, and a seed can be more than a gigabyte
> - Only a few finalization commit paths delete the folder, so many runs
leave it on disk forever
> - Instance disks fill up with folders that no code will read again
> - This pull request adds a bounded, deadline-driven retention worker
that removes a folder only when no reader can still need it
> - The benefit is that instance disks stop growing with stale sync
folders, and recovery and export paths keep their files
## Linked Issues or Issue Description
**What happened?**
`cleanupNativeWorkspaceSync(runId)` is the only code that deletes
`native-workspace-sync/<run>/`. It runs only when finalization commits
through specific paths. Runs that end in `terminal_failure`, retained
recovery, reconstruction, superseded runs, commits through other paths,
and server crashes leave the folder forever. On one instance, 22 folders
older than 7 days held 6.8 GB. Single seed archives were up to 1.4 GB.
**Expected behavior**
A run's sync folder is removed after a retention window, once the run
and its finalization are settled and no recovery or export path can
still read it. The shared `generations` seed cache stays.
**Steps to reproduce**
1. Run a native agent with a remote sandbox and let the run end in
`terminal_failure` (or crash the server during finalization).
2. Wait more than 7 days.
3. Look in `<instance>/native-workspace-sync/`. The run's folder and its
seed archives are still there.
**Paperclip version or commit**
`master` at 837add0caa
**Deployment mode**
Any deployment that runs native agents with remote sandboxes.
**Agent adapter(s) involved**
Native runtime with remote sandbox workspace sync (for example Codex on
Daytona).
## What Changed
- Added
`server/src/services/native-runtime/native-workspace-sync-retention.ts`.
It removes `native-workspace-sync/<name>/` when `<name>` is a UUID, the
entry is a real directory, its newest file is older than the window, and
one of these holds:
- no `heartbeat_runs` row has that id (orphan);
- the run has no valid `nativeWorkspaceSync` reference and is settled:
terminal status, `finished_at` before the window, no live recorded
process (see below), no live controller lease, no
`nativeWorkspaceFinalizationOwner` receipt;
- the run is settled as above, its `native_run_finalizations` row (if
any) has no `lease_owner` and no `next_attempt_at` and is `committed`
(any recovery state) or `terminal_failure` with no or `blocked` recovery
state, no active/escalated export-cause or owner-unverified recovery
action names it, no lease carries `nativeWorkspaceExportResume` or
`pendingCleanupInFlight`, retained recovery cannot still use it
(referenced lease destroyed, or no scheduled/running/awaiting_evidence
system recovery and no queued/recovering workspace recovery), and
historical recovery adoption (`adoptHistoricalWorkspaceRecoveries`) has
no pending candidate for it.
- Each folder is decided in one transaction that holds
`pg_try_advisory_xact_lock` on the run's finalization ownership key. The
rule is evaluated while the lock is held. A held lock skips the folder.
The folder is renamed to `.expired-<uuid>` in that transaction and
deleted after commit. The next sweep removes leftover `.expired-*`
folders. `generations`, non-UUID names, symlinks and files are never
touched. Each tick tries at most 50 leftovers, then checks at most 200
folders and removes at most 50. Leftovers and folders each continue from
their own cursor, so undeletable leftovers do not block run folders. A
leftover that cannot be deleted schedules the 15-minute retry. Deadline
discovery reads run state 200 IDs at a time and stops when asked to. It
supports dry run and `shouldContinue`.
- A recorded `process_pid` / `process_group_id` blocks removal only
while it can still be the run's own process. This reuses the rule from
`conversation-continuation.ts`: PIDs are ignored when every lease of the
run is remote (they are sandbox PIDs), and a local PID or process group
blocks only while it exists on this host and its start time matches
`process_started_at`. An unreadable start time keeps the folder. A
recycled PID does not.
- Separate finding, not changed here: native runs do not clear
`process_pid` when they finish. It is cleared only by the status
arbiter's `release_run_resources` effect (cancellations and commits) and
by the recovery paths. A failed run gets a `schedule_retry` effect
instead, so its PID stays set. For remote targets the stored value is
the sandbox PID. Other readers already treat these PIDs as liveness
hints only, so this PR does not change that write path.
- Exported `nativeWorkspaceFinalizationLockKey` from
`native-workspace-finalization-ownership.ts` and used it in
`withNativeWorkspaceFinalizationOwnership` and the retention worker, so
both use the same key derivation.
- Exported `nativeWorkspaceSyncStateRoot()` from
`native-workspace-sync.ts`; `descriptorDirectory` now uses it.
- Registered the worker on the shared delivery-work coordinator
(`DELIVERY_QUEUES.nativeWorkspaceSyncRetention`) in
`server/src/index.ts`, next to the git restore point worker. It follows
`rules/sleep-aware-background-work.mdc`: no interval. It runs at startup
and at real deadlines: the earliest settled run's finish (or an orphan's
newest file) plus the window; a recheck one window later for folders it
kept; and a 15-minute retry after a failure or a held lock. With no
folders, `nextRunAt` is `null`. The run folder is now created through
one helper in `native-workspace-sync.ts` that notifies process-local
listeners; the worker wakes on that signal only while it has no
deadline, because any scheduled deadline is never later than a new
folder's own deadline. `index.ts` unsubscribes the listener on shutdown.
It pauses passive maintenance during idle drain.
- Added `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED` (default on
unless `false`), `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN`
(default off) and `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS`
(default `7`, minimum `1`) in `server/src/config.ts`. Documented them in
`doc/DEVELOPING.md` and added a "Native workspace sync folders" section
to `doc/recoverable-disk-cleanup.md`.
## Verification
- `pnpm exec vitest run
src/__tests__/native-workspace-sync-retention.test.ts
src/__tests__/native-workspace-sync-retention-config.test.ts` in
`server/`: 62 tests pass. They cover:
- deletes a settled committed run (also with a left-over
`resuming_session` recovery state), a blocked `terminal_failure` with no
recovery owner, an orphan, and an unreferenced settled run;
- keeps runs whose local process or process group is still alive, or
whose live process start time cannot be read; keeps a running run with a
PID; deletes terminal runs whose recorded local process exited, whose
PID was recycled, or whose PID came from a destroyed sandbox;
- keeps recent files, recently finished runs, unsettled runs, a live
controller lease, an owner receipt, `observed` and `retryable_failure`
phases, a live `lease_owner`, a scheduled `next_attempt_at`, and each
non-blocked recovery state on `terminal_failure`;
- keeps runs with an open export-cause or owner-unverified recovery
action, `nativeWorkspaceExportResume` or `pendingCleanupInFlight` lease
metadata, retained recovery in flight on a lease that was not destroyed,
or pending historical recovery adoption; deletes when the retained lease
was destroyed;
- never touches `generations`, non-UUID names, symlinks or files; dry
run deletes nothing; leftover `.expired-*` folders are removed; an
advisory lock held by another transaction makes it skip; ticks are
bounded and resume from a cursor;
- `nextDeadlineAt` returns `null` with no folders, now with leftovers,
the settled finish plus the window, the orphan mtime plus the window,
and one window later for kept folders; the registered worker goes quiet
with no folders, retries after a held lock or an undeletable leftover,
and the real delivery-work coordinator wakes again when the real
producer (`writeDescriptor`) creates a run folder; leftover removal is
bounded and undeletable leftovers do not block run folders.
- Mutation checks: I removed each guard one at a time (advisory lock,
owner receipt, process pid, finished_at window, lease owner, next
attempt, phase, recovery state, export recovery, owner-unverified,
export resume, retained recovery, backfill pending,
symlink/real-directory check, dry run, and both mtime checks together,
plus the folder-creation wake, the process liveness rule (restoring the
old `process_pid IS NULL` rule, the sandbox exemption, the process group
check, the recycled-PID check, the unreadable start time), the leftover
batch limit, the leftover cursor and the leftover retry delay). Each
mutation made at least one test fail. I restored the code after each
one. Removing only the first mtime check does not fail a test, because
the second check inside the transaction still keeps the folder.
- `pnpm exec vitest run` on related suites in `server/`:
`native-workspace-sync.test.ts`,
`native-workspace-sync-history.test.ts`,
`native-workspace-finalizer.test.ts`,
`native-workspace-finalizer-failure.test.ts`,
`native-harness-backup-expiration*.test.ts`,
`git-restore-point-ledger*.test.ts`,
`delivery-work-coordinator.test.ts`, `chat-delivery-work.test.ts`. All
pass.
- `pnpm run typecheck` in `server/` passes.
- Greptile review: earlier rounds; all findings fixed with tests; final
score 5/5 with no open threads.
- Not run: the full server test suite and an end-to-end run against a
real instance. I did not measure disk savings on a live instance.
## Risks
- Deleting a folder that a reader still needs would make that path fall
back or fail. Same-run resume and export run under the finalization
ownership lock or before the run settles, and the worker checks the rule
while it holds the same lock. Retained workspace recovery already
handles a missing folder (ENOENT) by snapshotting the sandbox. The rule
keeps the folder for every in-flight or reopenable recovery state listed
above.
- A committed finalization row with a stale recovery state and no lease
owner is treated as settled. Nothing reads the recovery state of
committed rows.
- The worker is on by default. Operators can set
`PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN=true` first to see
candidates in the log, or turn it off with
`PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED=false`.
- The first sweep on a large instance does more filesystem work. It is
bounded to 200 folders checked and 50 removed per tick.
- The folder-creation wake is process-local. A folder created by another
server process sharing the same instance directory is picked up at the
next deadline or restart.
- A stale `process_pid` no longer blocks removal. If a run's local
process is still alive with its recorded start time, the folder stays.
Live work is also still guarded by the run status, controller lease,
finalization lease and ownership lock checks.
- No schema migration.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees once their
remote, or a restore point, can reproduce them
> - A nested `.paperclip-repositories` checkout's remote-tracking refs
can point at commits the remote has since dropped, such as a branch
deleted after a squash merge
> - Strict expiration rightly keeps such a checkout, because it may hold
the only copy; but restore point mode also kept it forever, because
there was nothing else to save
> - This pull request saves those commits in the checkout's restore
point, so the workspace can be reclaimed without losing them
> - The benefit is that long-lived multi-repository workspaces are
reclaimed in restore point mode
## Linked Issues or Issue Description
Refs #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, four closed workspaces on one instance were
kept with `unpushed_managed_checkout`, after a dry run had counted them
as candidates. Each nested `paperclip` checkout was clean, on `main` at
a commit that is on GitHub `main`. But it held about 1,000
remote-tracking refs, and between 28 and 41 of them pointed at branches
the remote has since deleted or force-pushed (147 to 203 commits it no
longer has).
The relaxed check passed, so the checkout needed no restore point beyond
those commits. Capture saved only `HEAD` history, so it found nothing to
save and stored no restore point. The final check before deletion then
ran the strict check, which keeps stale remote-tracking commits, and
rolled back. The dry run never reaches that final check, so it
over-reported. Together these workspaces held about 2.4 GB.
**Expected behavior**
In restore point mode, commits that exist only in remote-tracking refs
are saved, and the workspace expires.
**Steps to reproduce**
1. In a nested checkout, fetch a remote branch, so
`refs/remotes/origin/<branch>` exists.
2. Delete that branch on the remote and garbage-collect it.
3. With restore points enabled, close the issue and run expiration: the
workspace is kept with `unpushed_managed_checkout`.
**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- **Strict mode is unchanged.** An earlier revision of this pull request
ignored remote-tracking refs. Review pointed to `does not trust a stale
remote-tracking ref`, which deliberately protects those commits, so this
pull request keeps that behavior and the test is unchanged.
- **Relaxed nested check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`):
- reads all reflogs in one bounded call (`--format=%H %gD`),
deduplicates them by commit, and splits them into local and
remote-tracking entries before applying the tip budget;
- local entries still have to be covered or upstream, as before;
- remote-tracking tips and remote reflog entries that the remote no
longer has are returned as `preserve` (new `tipsMissingUpstream`,
sharing `commitsMissingUpstream` with the existing check).
- **Capture:** the proof is shared as `provenManagedCheckoutHistory`.
`storeGitRestorePoint({ resolvePreserveCommits })` calls it after taking
the history digest and before capture, then compares the digest again
after capture. A fetch before the digest is included in the preserved
commits, and one after it changes the digest and aborts capture. The
commits flow into `captureRestorePoint`. They are bundled as
history-only commits, within the existing limits (50 preserved tips,
1,000 commits, 256 MB bundle). The checkout therefore gets a restore
point, and the final check uses the relaxed check for it.
- **Final check:** `restorePointHistoryDigest` now also covers
remote-tracking refs and their reflogs, so a fetch, ref change or
reflog-only change after capture rolls expiration back.
- **Restore:** the commits are imported as objects; the stale
remote-tracking refs themselves are not recreated.
- **Docs:** `doc/recoverable-disk-cleanup.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
src/__tests__/git-reproducible-managed-checkouts.test.ts -t "stale
remote-tracking|remote-tracking refs change"` (in `server/`): 5 passed.
- The existing strict test passes unchanged.
- A clean nested checkout whose remote-tracking ref points at a deleted,
garbage-collected branch is kept strictly, expires with a restore point,
and reopens on its commit with the stale commit present.
- A remote-tracking ref created after capture rolls expiration back, and
so does a reflog-only change to one.
- 6,000 repeated reflog entries for one commit do not hit the tip
budget.
- **Mutation checks:** each of these fails a case:
- dropping the preserved commits reproduces the instance's symptom
(`unpushed_managed_checkout`);
- dropping remote-tracking refs or their reflogs from the digest;
- removing deduplication (`managed_checkout_local_tip_budget`).
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 261 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled.** Strict expiration is unchanged.
- **Retention:** saved stale commits are kept until the restore point
expires (90 days after the tree closes by default), then deleted with
it.
- **Limits:** checkouts with more than 50 stale tips, or bundles over
the limits, skip capture and stay kept, as before.
- **Spurious rollbacks:** a fetch during the short window between
capture and deletion rolls back; the next sweep retries.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees after saving a
restore point of anything the remote cannot reproduce
> - Restore points left ignored files out, so any worktree with an
ignored file that was not a known regenerable directory was kept
> - On one instance that was the most common reason a closed worktree
was still kept, mostly build caches and agent evidence files
> - This pull request saves those ignored files in the restore point too
> - The benefit is that more closed worktrees are reclaimed, and nothing
an agent left behind is lost
## Linked Issues or Issue Description
Refs #17248, #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, one instance kept 15 of 43 inspected closed
worktrees with `unique_ignored_files`. Among 19 such worktrees on disk:
- 9 held `ui/tsconfig.tsbuildinfo`;
- others held build output outside the regenerable directory names
(`server/ui-dist/`, `test-results/`);
- 2 held materialized skill copies under `.claude/skills/`;
- about 5 held small files agents kept out of git: patches, evidence
archives, recovery receipts.
**Expected behavior**
Ignored files that are not regenerable are saved with the restore point
and restored on reopen, so the worktree can be removed.
**Steps to reproduce**
1. In a closed issue's worktree, leave an ignored file outside
`node_modules`, `dist` and the other regenerable directories (for
example `ui/tsconfig.tsbuildinfo`).
2. With restore points enabled, run expiration: the worktree is kept
with `unique_ignored_files`.
**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Capture** (`server/src/services/git-restore-point.ts`):
- `restorePointSavesIgnoredFile` defines which ignored files are saved.
It leaves out:
- `.env` and `.env.*` files;
- `.paperclip-repositories/` (nested checkouts have their own restore
points);
- `.claude/settings.local.json` (saved with the workspace contents);
- anything inside a directory named in
`REGENERABLE_WORKSPACE_DIR_NAMES`.
- Those files are scanned with the same limits and checks as untracked
files, and snapshotted into a separate `ignored` tree, committed on top
of `HEAD`. Its ref travels in the bundle.
- They are hashed with `hash-object --no-filters` (symlinks by their raw
target bytes), so `.gitattributes` conversions such as `text=auto` never
change their bytes. Every path given to `--stdin-paths` is C-quoted, so
git always reads the literal workspace file, even for names that start
with `"`.
- An ignored nested repository skips capture, which keeps the worktree.
- A worktree whose only extra files are ignored is no longer "nothing to
save".
**Format:** a payload with an ignored tree uses
`paperclip.git-restore-point-payload.v2`, and one without stays `v1`.
Decoding requires `ignored` exactly for `v2`, so older servers reject
the new payloads instead of restoring without the files.
**Restore:** `applyRestorePoint` writes the ignored files directly from
their blobs, read in one `cat-file --batch` call, so no filter is
applied.
- It checks every target before writing, refuses any path that would
leave the worktree, and never overwrites an existing file: a conflict
fails the restore closed.
- If a write fails partway, it removes the files it wrote, so a retry
starts clean. The post-apply verification, the before-deletion snapshot
check and `restorePointSnapshotMatches` all include the ignored tree.
The stored reference (`GitRestorePointRef.ignored`) is validated and
checked against the decoded manifest.
**Expiration** (`git-workspace-restoration.ts`): in restore point mode,
the reproducibility check accepts the ignored files a restore point
saves. Strict mode is unchanged.
**Docs:** `doc/recoverable-disk-cleanup.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-restore-point.test.ts` (in
`server/`): 27 passed. New cases:
- **Round trip:** ignored files round-trip as a `v2` payload with their
modes, while `dist/`, `node_modules/` and `.env` stay out.
- **Conflict:** an existing conflicting file fails the restore closed
and is left as it was.
- **Nested repository:** an ignored nested repository skips capture.
- **Forged payload:** a `v1` payload claiming an ignored tree is
rejected.
- **Line endings:** an ignored CRLF file in a `* text=auto` repository
restores byte for byte.
- **Partial failure:** a restore that fails partway leaves nothing
behind, and a retry succeeds.
- **Quoted names:** a crafted ignored path that looks like a quoted
absolute path saves and restores the workspace file, never the outside
file it names.
- **Symlinks:** a symlink target with bytes that are not valid UTF-8
restores exactly.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): new
cases.
- A closed worktree whose only extra files are an ignored build cache
and evidence file is kept without restore points, expires with one, and
reopens with those files and without `node_modules`.
- An ignored file changed after capture rolls back.
- **Mutation checks:** each of these fails at least one case:
- not relaxing the reproducibility check;
- not capturing ignored files (the end-to-end case then catches the
loss);
- leaving the ignored tree out of the comparison;
- not writing the files on apply;
- hashing with filters;
- dropping the partial-restore rollback;
- passing unquoted paths;
- decoding symlink targets as UTF-8.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration execution-workspaces
workspace-compaction heartbeat-git-restore-point-resume
src/services/heartbeat/environment-selection` (in `server/`): 296
passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled.** Strict expiration is unchanged.
- **Bigger restore points.** Ignored files count toward the existing
limits (20,000 files, 64 MB per file, 256 MB total). Larger caches
outside the regenerable names make capture skip, which keeps the
worktree as before.
- **Rollback:** older servers cannot restore `v2` restore points; they
fail closed. Disable restore points before downgrading.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 and #17257 let git cache expiration remove closed issues'
worktrees after saving encrypted restore points
> - Restore points are kept for 90 days after an issue tree closes, then
deleted
> - Reopening a workspace whose restore point had expired failed, so the
issue could not get a workspace back
> - This pull request starts a fresh workspace on the same branch
instead, and warns that the saved work is gone
> - The benefit is that old issues can always be resumed, with a clear
record of what was lost
## Linked Issues or Issue Description
Refs #17248, #17257. Bug report described in-PR (bug report template).
**What happened?**
After `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` passes, the retention
worker marks the restore point expired and deletes it. Reopening that
workspace then fails with the git cache restoration error, every time.
**Expected behavior**
Reopening succeeds with a fresh workspace on the issue's branch, and the
issue says that unsaved work from before the cleanup is no longer
available.
**Steps to reproduce**
1. Enable restore points and let a closed issue's worktree expire with
unsaved work.
2. Let its restore point expire.
3. Reopen the issue: the workspace rebuild fails.
**Paperclip version or commit:** `master` at 650581b2f0.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- **Detection:** `gitRestorePointExpired(recipe)` is true when the main
restore point or any nested checkout's restore point has `expiredAt`.
- **Reopen** (`execution-workspaces.ts`):
- `restoreGitCacheBase(..., { freshStart: true })` prepares the base
without the saved commit or the bundle.
- The saved recipe is left out of the metadata handed to the runtime,
and dropped from the stored metadata only when the reopen succeeds, so a
failed attempt can be retried and still warns. The runtime recreates the
worktree on its branch: the branch tip if the repository still has it,
otherwise a new branch from the base ref. When the expired recipe had
nested checkouts, the project's repositories are prepared with
`prepareProjectRepositoryWorkspaces` (without the expired saved state)
right after the worktree is recreated and before the provision command
runs, with the same copy identity and access rechecks as other reopen
copies.
- The result carries `warning`, the `execution_workspace.reopened`
activity records `restorePointExpired` and the warning, and a system
comment on the issue posts the warning in the same transaction.
- **Retention:** `markGitRestorePointExpired` now also marks nested
checkouts' restore points, matched by object key.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the fresh start.
No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 13
passed. New case: an expired restore point reopens fresh with the
warning and the system comment, on the original branch, even after the
base lost the branch and its unpushed commit; the recipe is gone. A
first attempt that fails keeps the expired recipe, and the retry still
warns and comments exactly once.
- New nested case in `git-workspace-expiration.test.ts`: an expired
nested checkout restore point is marked on the checkout entry and
reopens fresh with the warning, with the nested repository cloned again
at its pushed commit and without its expired uncommitted files.
- **Mutation checks:** each of these fails a case:
- removing the fresh base path (reopen fails);
- removing the system comment;
- handing the runtime the old recipe;
- dropping the stored recipe before the rebuild succeeds;
- marking only the main restore point expired;
- skipping nested repository preparation on a fresh start.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` plus the environment selection
and heartbeat resume suites (in `server/`): 257 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled:** recipes get `expiredAt` only
from the restore point retention worker, so nothing changes otherwise.
- **Lost work is intended:** the fresh workspace has no uncommitted
files and, when the repository no longer has the branch, no unpushed
commits. The warning says so.
- **Recipe dropped:** after a fresh start the workspace no longer
carries the expired recipe, so later expirations start from a clean
slate.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 and #17257 let git cache expiration remove a closed issue
tree's worktree after saving an encrypted restore point
> - Issues that stay open but go untouched for weeks still keep their
worktrees forever, so disks keep filling
> - This pull request expires those idle worktrees the same way, and has
the next run restore its workspace automatically
> - The benefit is bounded disk use for stalled open work, with nothing
lost when work resumes
## Linked Issues or Issue Description
Refs #17248, #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, expiration only considers worktrees whose
whole issue tree has been `done` or `cancelled` for a while. A worktree
whose issue stays `todo`, `in_progress`, `blocked` or `in_review` with
no activity for weeks is never removed.
**Expected behavior**
An open issue's worktree that has been idle long enough is saved to a
restore point and removed, and the next run on that issue gets it back
exactly.
**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Leave an issue open with an isolated worktree and no activity for
weeks.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at 650581b2f0.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Expiration** (`git-workspace-expiration.ts`):
- `isGitWorkspaceExpirationEligible(..., idleCutoff)`: in idle mode the
closed-tree conditions are replaced by idle ones. No issue in the tree,
and no other issue bound to the workspace, was updated after the cutoff,
and no relevant run was created, started or finished after it. Runs of
bound issues now count as relevant runs in both modes. Every other check
stays, including no live run, recovery action, unsettled finalization,
environment lease or running service.
- `gitWorkspaceLastActivityAt` returns the latest of those timestamps,
sharing the same tree and run scope.
- `expireReconstructableGitWorkspace` takes `idleCutoff` and passes it
to all three eligibility checks.
**Reaper** (`execution-workspaces.ts`):
- For an open tree's isolated git worktree, when restore points and git
cache expiration are on and `idleMs` is set, the reaper computes the
idle deadline (last activity + `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS`).
- Before the deadline, it records the deadline, so the next sweep wakes
then. Open trees cause no recurring checks before that.
- At the deadline, it runs expiration with a restore point. A kept
workspace is retried after an hour, like terminal ones.
- An expiration interrupted after archiving (`completed: false`) is
resumed by later sweeps even while the tree is open, with a retry
scheduled until cleanup finishes.
**Runs** (`heartbeat/environment-selection.ts`):
- When a run's issue is bound to a workspace that git cache expiration
removed (archived, `workspaceRemoved`), the run reopens it as the agent
through `reopenClosedIsolatedExecutionWorkspaceForIssue`, with the
restore point applied.
- It then clears the reopen fence, since the live run now holds the
workspace, and continues with the reopened row.
- A failed restore fails the run instead of silently starting a new
workspace.
- Comments and agent updates on the issue already reopened closed
workspaces through the issue routes.
**Config:** `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS` (default `7`, `0`
turns idle expiry off). Docs in `doc/recoverable-disk-cleanup.md` and
`doc/DEVELOPING.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts -t "idle open
issues"` (in `server/`): 7 passed, with real repositories and embedded
Postgres.
- **Round trip:** an open issue's worktree, idle for 8 days with unsaved
work, is skipped without idle mode, expires with a restore point in idle
mode, and reopens exactly.
- **Kept:** issue activity 3 days ago, or a run finished 2 days ago,
keeps the worktree.
- **Interrupted:** an idle expiration interrupted after archiving is
finished by the reaper while the issue stays open, and reopens exactly.
- **Shared workspace:** a recent update on another issue bound to the
workspace keeps it, and moves the last activity time.
- **Reaper:** before the deadline the sweep skips the worktree and
schedules exactly that deadline; at the deadline it expires it with an
active restore point. With idle expiry off, nothing is scheduled or
expired.
- `pnpm exec vitest run
src/services/heartbeat/environment-selection.test.ts` (in `server/`): 14
passed. New cases: a run reopens its expired workspace as the agent,
clears the fence with the reopen's generation and binds the reopened
row; a failed restore fails the run; a workspace archived for other
reasons is left alone.
- `pnpm exec vitest run
src/__tests__/heartbeat-git-restore-point-resume.test.ts` (in
`server/`): 1 passed. Assembled heartbeat journey with no mocked
restore: a real worktree with an unpushed commit and uncommitted notes
is idle-expired, a queued run executes, the agent runs in the original
worktree path with the original workspace id, the commit and notes are
back, the workspace is no longer archived with no reopen fence left, and
the issue stays bound. Disabling the run-side reopen fails it.
- `git-restore-point-expiration-config.test.ts`: the default is 7 and
`0` turns it off.
- **Mutation checks:** each of these fails a case:
- dropping the idle deadline;
- dropping the idle cutoff;
- dropping the run activity condition;
- not failing the run on a failed restore;
- not re-reading the reopened workspace;
- not resuming a pending expiration on an open tree;
- ignoring bound issues' activity.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents execution-workspaces
src/services/heartbeat/environment-selection
src/services/heartbeat/workspace-preparation` plus
`heartbeat-git-restore-point-resume` (in `server/`): 312 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Off unless enabled:** requires
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
`PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS=0` keeps restore points for
closed trees only.
- **First run after idle expiry is slower:** it waits for the restore,
about as long as a reopen.
- **Failed restore blocks runs:** a restore that fails closed (for
example a missing restore point) fails each run on that issue until an
operator fixes or rebinds the workspace. This is deliberate, so work is
never silently continued in an empty workspace.
- **Kept idle workspaces are rechecked hourly**, matching the existing
behavior for closed trees that cannot be expired.
- **Retention:** restore points of open trees are not deleted while the
tree stays open, as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 lets git cache expiration remove a closed issue's worktree
with unsaved work after saving an encrypted restore point
> - Workspaces with nested `.paperclip-repositories` checkouts were
still kept whenever a checkout had unpushed or uncommitted work
> - On one instance that was the second most common reason expiration
skipped a workspace
> - This pull request gives each nested checkout its own restore point
and restores it when the workspace comes back
> - The benefit is that multi-repository workspaces can be cleaned up as
safely as single-repository ones
## Linked Issues or Issue Description
Refs #17238, #17241, #17248. Bug report described in-PR (bug report
template).
**What happened?**
With restore points enabled, a workspace whose nested managed checkout
has unpushed commits or uncommitted files is still skipped with
`unpushed_managed_checkout` or `dirty_or_untracked_files`. On one
instance, `unpushed_managed_checkout` accounted for 32 of the last 111
expiration checks.
**Expected behavior**
A nested checkout's unsaved work is saved like the main worktree's, and
the workspace can expire and later be restored exactly.
**Steps to reproduce**
1. Use a project with an extra repository, so the task worktree contains
`.paperclip-repositories/<name>`.
2. Commit in that checkout without pushing, then close the issue.
3. With restore points enabled, the workspace is never expired.
**Paperclip version or commit:** `master` at 6d44b80a5e.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Relaxed checkout check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`, used only in restore point mode):
- **Allowed:** the checked-out branch's unpushed commits, plus its
uncommitted, untracked and history-only work.
- **Still blocks:**
- another local branch or tag whose tip the remote lacks and `HEAD` does
not contain (`unpushed_managed_checkout_branches`), because restore
points do not recreate other branches;
- commits in any reflog that are neither on the remote nor in the
bundled history (`HEAD`'s reflog and `ORIG_HEAD`), such as work left in
another branch's reflog;
- objects reachable from nothing;
- a `COMMIT_EDITMSG` that matches no commit in the checkout's history.
- **`COMMIT_EDITMSG`:** a nested checkout is a full clone, so its base
git storage check now accepts a `COMMIT_EDITMSG` that matches a commit
in its history, in strict and relaxed mode. It is read only if it is a
regular file of at most 64 KB, opened without following symlinks and
without blocking; anything else keeps the workspace
(`unsupported_commit_message`). The main worktree's existing
`COMMIT_EDITMSG` read uses the same bounded reader. A clean checkout
with only an unpushed commit therefore reports
`unpushed_managed_checkout`, which restore point mode can relax, instead
of `unknown_base_git_files`.
**Capture** (`git-workspace-expiration.ts`):
- **Expiration flow:** strict inspection still runs first.
`unpushed_managed_checkout` is now relaxable. The relaxed prover records
which checkouts need restore points.
- **Storage:** each checkout's restore point is stored under
`.../<generation>/checkouts/<name>/<id>.pcrp`, against a remote-verified
boundary from that checkout. It is recorded on its
`contents.checkouts[]` entry.
- **Activation:** all restore points of the generation are activated
together in the archive transaction. Any nested restore point makes the
recipe `paperclip.git-cache-restoration.v2`, so older servers reject it.
- **Schema:** migration `0374` adds `checkout_name` to
`execution_workspace_restore_points` and includes it in the
live-generation unique index.
- **Final check:** under the expiration locks, each checkout with a
restore point gets the relaxed check and must match its own snapshot.
Checkouts without one get the strict check and must still be clean.
Every restore point, the main worktree's included, records a digest of
its `HEAD` reflog and `ORIG_HEAD` at capture, and that history must be
unchanged, so a commit followed by a reset after capture rolls back.
Rollback drops every restore point reference, returns the recipe to v1,
and supersedes the stored restore points.
**Restore** (`git-workspace-contents.ts`):
- **Re-cloned checkouts:** re-cloning a checkout imports its bundle
before the saved commit is needed. After checking out the saved commit,
it applies and verifies the restore point.
- **Existing checkouts:** the same helper as the main worktree marks a
matching checkout applied, and fails closed instead of overwriting a
diverged one.
**Validation:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` validate nested restore point references
against their checkout names. Contents comparison ignores the reference.
**Docs:** `doc/recoverable-disk-cleanup.md` describes nested checkouts.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "restore points for nested checkouts"` (in `server/`): 10 passed,
with a real nested checkout and embedded Postgres. The cases:
- **Round trip:** an unpushed commit and an untracked file in the nested
checkout survive expiration and reopen. The recipe is v2, the restore
point lives on the checkout entry, and a ledger row carries the checkout
name.
- **Another branch:** unpushed work on another local branch keeps the
workspace.
- **Change after capture:** a nested change after capture rolls back,
keeps the files, returns the recipe to v1, and supersedes the restore
point.
- **Only an unpushed commit:** a clean nested checkout with only an
unpushed commit (and its `COMMIT_EDITMSG`) expires and restores.
- **Reflog-only history:** unpushed history held only in another
branch's reflog keeps the workspace.
- **History after capture:** a commit followed by a reset after capture
rolls back, in a nested checkout and in the main worktree.
- **Unsafe `COMMIT_EDITMSG`:** a named pipe, an oversized file, or a
symlink keeps the workspace without reading it unsafely.
- **Mutation checks:** each of these fails a case:
- removing the other-branch check, the per-checkout snapshot check, the
rollback cleanup, or the checkout apply;
- rejecting a saved `COMMIT_EDITMSG`;
- dropping reflog coverage;
- dropping the history digest check;
- removing the size cap.
Not covered by a dedicated case: a workspace with two nested checkouts
where only one has a restore point.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` (in `server/`): 232 passed.
- `pnpm run check:migrations` (in `packages/db`): pass.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.
## Risks
- **Off by default:** nothing changes unless
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- **Other local branches:** a nested checkout's other local branches are
not saved. Any of them with commits the remote lacks keeps the workspace
instead.
- **Migration:** adds a defaulted column and replaces one partial unique
index on a table only written when restore points are enabled.
- **Rollback:** disable the setting. Workspaces already expired with
nested restore points keep v2 recipes that require this code to restore.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes a closed issue's worktree only when the
remote can reproduce it exactly
> - Most closed worktrees hold unpushed commits or uncommitted files, so
they are kept forever and disks fill up
> - #17238 added restore points (an encrypted save of exactly what the
remote cannot reproduce) and #17241 added their lifecycle
> - This pull request lets expiration remove those worktrees after
saving a restore point, and restores them exactly when the work resumes
> - The benefit is bounded disk use for closed work without losing
anything a user could need, behind a setting that is off by default
## Linked Issues or Issue Description
Refs #17238, #17241. Bug report described in-PR (bug report template).
**What happened?**
With `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one
instance's last 111 expiration checks removed nothing:
| Skip reason | Checks |
|---|---|
| `dirty_or_untracked_files` | 39 |
| `unpushed_managed_checkout` | 32 |
| `live_or_unresolved_consumer` | 21 |
| `unique_ignored_files` | 9 |
| commit not on the remote | 10 |
Without expiration, the terminal workspace reaper archives only merged,
clean worktrees. Closed issues' worktrees therefore accumulate without
bound.
**Expected behavior**
A closed worktree whose only obstacle is unsaved work can be removed
after that work is saved, and restored exactly if the issue is reopened
or work resumes.
**Steps to reproduce**
1. Close an issue whose worktree has an unpushed commit or an
uncommitted file.
2. Enable git cache expiration with dry run off.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at 12bb7e1976.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Expiration** (`server/src/services/git-workspace-expiration.ts`):
- **When it applies:** with `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`
(default `false`), a worktree that fails the reproducibility check for a
relaxable reason is reconsidered once its issue tree has been closed for
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). The
relaxable reasons are uncommitted or untracked files, a commit not on
the remote, reflog or `ORIG_HEAD` history, and ignored `.env` files. A
leftover `COMMIT_EDITMSG` still blocks expiration unless it matches a
commit in the worktree's reflog, because restore points do not save
draft messages.
- **What still blocks:** a relaxed inspection
(`inspectGitCacheRecipeForRestorePoint`) keeps every other check. Nested
managed checkouts, submodules, special git state, local config and other
unique ignored files still prevent expiration.
- **Capture:** inside the existing source and lifecycle locks, a restore
point is captured with the #17238 library against a remote-verified
boundary (the `HEAD` itself, or its merge base with `origin/HEAD`), then
encrypted, uploaded, read back and hash-checked
(`git-restore-point-storage.ts`).
- **Commit before touching disk:** it is activated in the #17241 table
in the same transaction that archives the workspace with a
`paperclip.git-cache-restoration.v2` recipe referencing it, still before
anything on disk is touched.
- **Final check:** under the file locks, the worktree must still match
the captured snapshot (`HEAD`, index, working tree, modes). Otherwise
the existing rollback runs. The rollback now also drops the restore
point reference and returns the recipe to v1, so a later restore can
never apply a stale restore point over the live worktree; the stored
restore point is marked superseded.
- **Failure paths:** the `pending` row is inserted in a transaction that
first registers a `git-restore-points` notification. If the upload or
read-back fails, the row is marked `superseded` so retention deletes any
uploaded object. Nothing is deleted from disk.
- **Encryption key:** restore points are refused
(`restore_point_key_not_durable`) on `s3` storage unless the secrets
master key comes from `PAPERCLIP_SECRETS_MASTER_KEY`.
- **Dry run** reports `restorePoint: true` candidates without storing
anything.
**Restore** (`git-workspace-restoration.ts`, `workspace-runtime.ts`,
`execution-workspaces.ts`):
- **Recipe versions:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` accept v1 without a restore point, or v2
with a valid one. Older servers reject v2 instead of restoring the
commit without the saved work.
- **Before the saved commit is needed:** reopening
(`restoreGitCacheBase`) and the runtime path import the bundle when the
saved commit is missing locally.
- **After the worktree is re-added:** the runtime always imports the
bundle, so history-only commits are present again as objects even when
the base was rebuilt (they are not recreated as branches), then applies
the restore point, verifies the result, and records `appliedAt`. Reopen
persists that through a callback, as it already does for saved contents.
- **Never overwriting:** a worktree that already matches is only marked
applied. A worktree that diverged from the saved commit's clean state is
never overwritten; the restore fails closed and can be retried. Missing,
corrupt or expired restore points also fail closed.
**Retention:** the #17241 worker's expiry hook now marks the recipe's
restore point expired before its object is deleted.
**Config and docs:**
- `server/src/config.ts` adds the two settings;
`doc/recoverable-disk-cleanup.md` and `doc/DEVELOPING.md` describe them.
- No code comments were added.
**Not in this PR (follow-ups):**
- workspaces with nested `.paperclip-repositories` checkouts;
- removing idle worktrees of issues that are still open;
- starting a fresh workspace with a warning when a restore point has
expired.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 12
passed, with real repositories and embedded Postgres. The cases:
- **Round trip:** an unpushed commit, staged and unstaged edits of one
file, a deletion, and an untracked file are restored exactly on reopen,
`.env` is dropped, and the restore point is active. Restoring through
the runtime path works the same way, and a later run never re-applies
over newer work.
- **Kept:** a worktree that is not yet due, one with the feature
disabled (today's skip reason), and one whose encryption key is not
durable.
- **Change after capture:** rolls back, keeps the worktree, returns the
recipe to v1 and supersedes the restore point.
- **Missing restore point:** fails closed, keeps the row archived and
unapplied, and fails closed again on retry.
- **Dry run:** stores nothing.
- **Reproducible worktrees:** expire as before, with a v1 recipe.
- **History-only commits:** reopen imports them even after they are
pruned from the base.
- **Draft messages:** a draft commit message keeps the worktree.
- **Upload failure:** a failed upload marks the pending restore point
for cleanup and keeps the worktree.
- **Mutation checks:** each of these fails at least one case:
- removing the pre-deletion snapshot check;
- keeping the restore point reference on rollback;
- removing the runtime apply;
- removing the re-apply guard (the divergence check then fails closed
instead of overwriting);
- skipping the import for a matching worktree, relaxing the draft
message guard, or not marking a failed upload's row.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration-config.test.ts`: 5 passed.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point workspace-runtime execution-workspaces` (in
`server/`): 529 passed on the first revision. Two
`workspace-runtime.test.ts` cases about remote base refs fail, and they
also fail on unmodified `master` on this machine. After the review
fixes, the expiration, contents, restore point and execution workspace
suites passed again: 222 tests.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.
## Risks
- **Off by default:** nothing changes unless both
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` are set. Roll out with
`PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=true` first to see which
worktrees would receive restore points.
- **Data at rest:** restore points contain source and uncommitted files,
encrypted with keys wrapped by the instance secrets key. Losing that key
makes them unrecoverable, which is why `s3` storage requires an
environment-provided key. Untracked `.env` files are not stored.
- **Interrupted apply:** a crash in the middle of applying a restore
point leaves a worktree that neither matches the restore point nor is
clean. The restore then fails closed rather than overwriting it, and an
operator must reset that worktree to retry.
- **Rollback:** disable the setting. Already-expired worktrees keep
their v2 recipes and restore points, so reverting the code would leave
those recipes unreadable until restored by a server with this change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Closed issues' git worktrees pile up on disk because they can only
be removed once the remote reproduces them exactly
> - #17238 adds restore points: encrypted saves of the unpushed and
uncommitted parts of a worktree, so the worktree itself can be removed
> - Those saves live in file storage and must be deleted on a schedule,
and promptly when their issue or company is deleted
> - This pull request adds the table that tracks restore points and a
deadline-driven retention worker, with nothing creating restore points
yet
> - The benefit is a reviewable lifecycle foundation before capture is
wired into expiration
## Linked Issues or Issue Description
Refs #17238 (restore point library). Bug report described in-PR (bug
report template).
**What happened?**
Closed issues' worktrees accumulate without bound. For example, one
instance's last 111 git cache expiration checks removed nothing, because
each worktree held unpushed commits or uncommitted files. Restore points
(#17238) will let those worktrees be removed safely. That needs a
durable record of every stored restore point and a retention policy for
deleting them.
**Expected behavior**
- A restore point is kept while its work can still resume.
- It is deleted a fixed time after its issue tree closes.
- It is deleted promptly when its issue or company is deleted.
**Steps to reproduce**
Not applicable on its own: this adds the lifecycle that the capture
integration will use.
**Paperclip version or commit:** `master` at def4726bea.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**New table `execution_workspace_restore_points`** (migration `0372`,
schema `packages/db/src/schema/execution_workspace_restore_points.ts`):
- **Columns:** storage key, sha256, size, generation, source issue,
`status`, closing time, review/deletion time (`delete_after`), a
`review_generation` counter and last error.
- **Statuses:** `pending`, `active`, `expiring`, `superseded`,
`delete_requested`, `deleted`.
- **No foreign keys:** a row outlives the deletion of its workspace,
issue or company long enough to remove the stored object.
- **Indexes:**
- the storage key is unique;
- live generations are unique per workspace;
- an index on status and deadline serves the worker, and a partial index
on active rows' source issue serves the trigger.
**Issue tree trigger** (same migration):
`paperclip_restore_point_tree_changed` runs after an issue's `status` or
`parent_id` changes, and after an issue is deleted. It walks up from the
issue and from its old and new parents (at most 64 levels). For every
active restore point whose source issue is in that set, it makes the row
due now (keeping an earlier due time) and increments
`review_generation`. This happens in the same transaction, so a child's
deletion or move still reaches its former parent's tree.
**New `server/src/services/git-restore-point-ledger.ts`:**
- **`gitRestorePointRetentionService(db, { storage, retentionDays,
beforeExpire })`.** Each `tick` handles only due work, in batches of 50:
- **Review:** an active restore point is checked when its `delete_after`
time arrives. The due check compares against the later of the worker's
clock and the database clock, so a row the trigger just marked is never
skipped. At review:
- **Tree still open:** clear `delete_after`. The trigger marks it due
again on the next relevant issue change, so open trees cause no
recurring checks.
- **Tree `done` or `cancelled`:** expire
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` (default `90`, minimum `1`)
after the tree's latest close. Expiry moves the row to `expiring`, then
runs an optional `beforeExpire` hook, which the capture integration will
use to mark the workspace no longer restorable. Only then does it
request deletion. An `expiring` row's object is never deleted: after a
restart, or if the hook fails, the hook is retried (after 15 minutes
when it failed).
- **Source issue gone:** delete.
- **Concurrency:** a review result is written only if
`review_generation` is unchanged since the worker read the row. If the
trigger fired in between (for example, the tree closed or reopened), the
write is skipped and the row stays due for the next run. Expiry and
orphan deletion claim the row under the same check before the expiry
hook runs.
- **Abandoned captures:** a `pending` row older than one hour is
deleted.
- **Deletion:** `superseded` and `delete_requested` rows have their
object removed. They become `deleted` only once `headObject` reports the
object gone. Failures, including a local unlink error the provider
swallows, are recorded in `last_error` and retried after 15 minutes.
- **`nextDeadlineAt`** returns the earliest real deadline, or `null`
when nothing is due.
- **`requestRestorePointDeletion(tx, { sourceIssueId } | { companyId
})`** registers a delivery-work notification and marks the matching
`pending`/`active` rows `delete_requested` in the caller's transaction.
**Deletion hooks:**
- `issueService.remove` and `deleteCompany` call
`requestRestorePointDeletion` in their transactions, so a deleted
issue's or company's objects are removed promptly.
- Child issues cannot be deleted with their parent (the parent link
restricts deletion), so each issue removal goes through this hook.
**Scheduling** follows `rules/sleep-aware-background-work.mdc`:
- `registerGitRestorePointWork` registers a `git-restore-points` queue
with the shared delivery-work coordinator.
- The worker keeps no in-memory inputs. Due work is recorded in the
table, by the trigger or by deletion requests. Each run handles it and
schedules only `nextDeadlineAt`, so a failed run loses nothing and is
retried by the coordinator.
- It runs at startup, at real deadlines, and when terminal-workspace
notifications settle after issue changes, but only while restore points
exist. With nothing due, it schedules nothing. A full batch continues
immediately. There is no timer.
**Config and docs:**
- `server/src/config.ts` adds
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS`.
- `doc/recoverable-disk-cleanup.md` gains a "Git workspace restore
points" section; `doc/DEVELOPING.md` documents the variable.
- `drizzle-kit generate` pruned `meta/0367_snapshot.json`, following the
repository's snapshot-retention convention.
- No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-restore-point-ledger.test.ts`
(in `server/`): 26 passed against embedded Postgres, stable across
repeated runs. The cases:
- **Review outcomes:** an open tree clears its deadline and schedules
nothing; a closed tree is scheduled 90 days after its close; a
long-closed tree expires and is deleted; a missing source issue is
deleted.
- **Expiry durability:** a failing hook keeps the row `expiring` with
its object until a retry succeeds; an `expiring` row left by a restart
resumes the hook before deletion; an `expiring` row's object is never
deleted while its hook fails.
- **Trigger:** the last open child closing, a child being deleted, a
child moving to another parent, and the source issue's own status change
all mark the restore point due. Unrelated issues, inactive rows and
already-due rows are untouched. End to end, an open tree's restore point
waits without a deadline, then is scheduled 90 days after its tree
closes.
- **Races:** closing the tree, or reopening it, between the worker's
read and write leaves the row due, and the next run resolves it
correctly. A reopened tree's restore point is not deleted.
- **No rescanning:** 60 retained restore points with future review times
are not evaluated. A backlog of 60 due ones is processed as 50, then 10,
then nothing.
- **Captures and deletion:** abandoned pending rows are deleted while
recent ones are kept; superseded rows are deleted; an object still
present after deletion is not marked deleted; a failed deletion is
retried only after its 15-minute deadline.
- **Deadlines:** the next deadline is correct for each obligation, and
`null` when idle.
- **Deletion hooks:** `issueService.remove` marks only that issue's
restore points, and `deleteCompany` marks all of the company's; the
worker then deletes their objects.
- **Worker:** schedules nothing when idle, and only the next deadline
otherwise; wakes on issue changes only while restore points exist.
- **Mutation checks:** each of these fails at least one case:
- removing the issue hook or the company hook;
- making the trigger ignore former parents;
- removing the worker's generation check or the trigger's generation
bump;
- letting storage cleanup delete `expiring` rows, or not resuming the
hook;
- ignoring open trees;
- removing the deletion verification;
- evaluating rows that are not due;
- dropping the retry deadline.
- `pnpm exec vitest run company-deletion issue-delete issues-service
delivery-work issue-tree` (in `server/`): 190 passed.
- **CI note:** one earlier `Typecheck + Release Registry` run was
cancelled when the GitHub runner received a shutdown signal; it is not a
code failure.
- `pnpm exec vitest run` (in `packages/db`): 211 passed. One run had a
single failure in `issue-comment-derived-attribution-migration.test.ts`,
an older migration's test unrelated to this change. It passed alone and
on the full re-run.
- `pnpm exec vitest run
src/__tests__/git-restore-point-ledger-config.test.ts`: 5 passed.
- `pnpm run check:migrations` (in `packages/db`): numbering and safety
checks pass.
- `pnpm run typecheck` (in `server/`): pass.
## Risks
- **No behavior change today:** nothing creates restore points yet. The
worker runs once at startup, finds no rows and schedules nothing. The
capture integration follows in a separate PR.
- **Trigger cost:** every issue status change, parent change or deletion
now runs a short ancestor walk, plus an `UPDATE` on the new table that
matches only active restore points through a partial index. With no
restore points, the `UPDATE` matches nothing.
- **Missed wakes:** if a wake is ever missed, for example because the
process exits after commit, the due row stays in the table. The worker's
startup run or next deadline processes it.
- **Deletion paths:** issue and company deletion each gain one `UPDATE`
on the new table inside their existing transactions.
- **Migration:** `0372_mature_the_initiative.sql` creates one empty
table, its indexes, and the trigger function with two triggers on
`issues`; nothing is rewritten.
- **Rollback:** revert. The table can stay, since nothing else reads it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only measurements on live
instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex sessions in remote sandboxes copy the sandbox's Codex
home into a host-side failover backup after every turn
> - Ephemeral sandboxes restore that backup at the start of every turn,
so its size affects both disk use and turn latency
> - Most of each backup is remote plugin data that Codex itself treats
as a cache and re-downloads on startup
> - This pull request leaves that data out of Codex checkpoints, keeping
everything a resumed session needs
> - The benefit is backups roughly 80–90% smaller, less data moved per
turn, and much slower disk growth
## Linked Issues or Issue Description
Bug report described in-PR (bug report template). Related: #16866
(expires failover backups after their issues close).
**What happened?**
Failover backups grow quickly because each Codex checkpoint includes
`$CODEX_HOME` remote plugin data. On one instance, 204 backups held 22.1
GB:
- `codex-home/plugins/cache/openai-curated-remote`: 8.3 GB (about 29 MB
per backup);
- `codex-home/cache/remote_plugin_catalog`: 6.1 GB (one 38 MB catalog
file per backup);
- `codex-home/sessions` (the conversation history): 0.31 GB.
About 80 new backups (9.8 GB) were written per day. Ephemeral sandboxes
also copy all of this into a fresh sandbox before each turn.
**Expected behavior**
Checkpoints keep what a resumed session needs and omit data that Codex
re-downloads itself.
**Steps to reproduce**
1. Run a Codex agent with ChatGPT auth in a remote sandbox environment.
2. Inspect
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current/codex-home`.
3. Most bytes are under `plugins/cache/openai-curated-remote` and
`cache/remote_plugin_catalog`.
**Paperclip version or commit:** `master` at f89c37791c.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** Codex (native app-server and ACPX).
## What Changed
- **`server/src/services/native-runtime/native-session-executor.ts`:**
Codex checkpoints (direct and under ACPX) now also exclude:
- `plugins/cache/<marketplace>` for Codex's six remote marketplaces
(`openai-curated-remote`, `created-by-me-remote`, `workspace-directory`,
`workspace-shared-with-me`, `workspace-shared-with-me-private`,
`workspace-shared-with-me-unlisted`);
- `cache/remote_plugin_catalog`;
- `plugins/.remote-plugin-install-staging`, and
`.remote-plugin-install-staging` (the location older Codex versions
use).
- **Still kept:** local and bundled plugin caches, `plugins/data`, the
Apps tool cache (`cache/codex_apps_tools`, which avoids waiting up to 30
s for the Apps MCP server), session rollouts, and the state databases.
- **Tests:** a real-tar checkpoint test covers kept and omitted entries;
the profile expectations are updated.
- **`doc/recoverable-disk-cleanup.md`:** describes what Codex failover
copies omit and the first-turn effect.
- No code comments were added.
**Why the omitted data is safe to drop** (Codex source at commit
d4e03acb):
- **Server-side source of truth:** the installed remote plugin list
comes from the server (`/backend-api/ps/plugins/installed`) and is held
in memory; these directories are not the source of truth.
- **Self-healing startup:** at `app-server` startup, a background task
reconciles missing bundles and force-refreshes the catalog
(`core-plugins/src/manager.rs`, `remote_installed_plugin_sync.rs`).
- **No path dependencies:** rollouts and state store plugin ids, not
paths into these directories.
## Verification
- `pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts` (in
`server/`): 766 passed.
- **Mutation check:** dropping one remote marketplace from the exclusion
list fails both the new checkpoint test and the profile test.
- `pnpm run typecheck` (in `server/`): pass.
- **Manual resume with Codex CLI 0.162.1 and ChatGPT auth:**
1. Started a session in a fresh `CODEX_HOME`. This downloaded 28 MB of
remote plugins.
2. Copied the home without the excluded entries: 34 MB became 4.9 MB.
3. Ran `codex exec resume --last`. It recalled the conversation in 3 s,
and the remote plugins were re-downloaded in the background without
errors.
- **Compatibility:** restore compares only top-level directory names
(`compatibleNativeHarnessBackupManifests`), so existing backups that
still contain the plugins restore unchanged.
## Risks
- **First turn after a restore:** remote plugin skills and tools can be
missing until the background download finishes, typically seconds. Codex
leaves an unavailable remote plugin out of a turn; it does not fail the
turn. Later turns include it.
- **Network and auth:** re-downloading requires ChatGPT auth and network
access from the sandbox, which remote plugins require anyway, since
Codex fetches the installed list from the server on every start.
- **Per-turn download:** ephemeral sandboxes download the remote plugin
bundles each turn, about 28 MB in the observed case, in the background.
In exchange, about 90 MB less is copied from the host before each turn.
- **Rollback:** revert.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, Codex source reading, a local
Codex resume test, read-only measurements on a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent sessions that run in a remote sandbox keep a host-side
failover copy of the sandbox's harness state, so a replacement sandbox
can resume the conversation
> - Each session keeps that copy forever, and one copy can reach several
gigabytes
> - On a busy instance these copies fill the volume, and no existing
cleanup touches them
> - Once a session's work is closed, its copy matters only if someone
reopens that work later
> - This pull request expires the copy after the session's issues have
been closed for a configurable window, and makes a reopened issue start
a fresh session
> - The benefit is bounded disk use without breaking active,
recoverable, or leased sessions
## Linked Issues or Issue Description
Bug report described in-PR (bug report template).
**What happened?**
After every remote turn, `checkpointRemoteRunner` writes the sandbox's
runner state and provider session home (for example `codex-home`) to
`durable-sessions/<scope>/failover-backups/current`. Only the same
session's next checkpoint replaces it. No code path deletes a session's
copy when the session ends, its issue closes, or its agent goes away, so
the instance volume fills. On one instance:
- 246 sessions held 17 GB of copies, 84 of them older than a week;
- 24 new sessions added 5 GB in a single day, one copy alone 3.2 GB.
**Expected behavior**
Disk used by failover copies stays bounded. A copy is kept while its
session can still resume (open issue, live run, pending recovery, or
live lease), and expires once the session's issues have been closed for
a while. A reopened issue then starts a fresh native session, instead of
failing with `runner_harness_backup_unavailable` on a replacement
sandbox.
**Steps to reproduce**
1. Run agents with a native runtime in a remote sandbox environment.
2. Let them complete many issues.
3. Watch
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current`
grow without bound after the issues close.
**Paperclip version or commit:** `master` at 9179d8dcc3.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** not adapter-specific (core bug); observed
with Codex sessions.
**Breaking changes**
- Reopening an issue that has been closed for longer than the window
(default 7 days) starts a fresh agent session instead of resuming the
old conversation.
- Disable with
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or lengthen
the window with `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS`.
## What Changed
-
**`server/src/services/native-runtime/native-harness-backup-expiration.ts`
(new):** the sweep. It checks at most 200 sessions and expires at most
50 per batch, continuing from a cursor. A copy expires only when all of
these hold:
- every issue the session ran on is `done` or `cancelled`, closed before
the window;
- the copy was written before the window;
- no run of the session is queued, running, or unfinished;
- every native run finalization of the session is settled (`committed`
or `terminal_failure`), with no recovery state or lease owner;
- no `active`, `released`, `retained`, or `pending_cleanup` environment
lease references the copy (by backup stamp, session run, or recorded
lease);
- the session is not executing, initializing, warm, or closing in this
process, and its directory has no pending cleanup activation.
- **One transaction:** the database checks share-lock the issues and
delete the agent task session that points at the session, so a reopened
issue cannot resume the expired session.
- **Atomic, non-blocking removal:** the copy is renamed to
`failover-backups/.retiring-<UUID>`, and a final check looks for a run
of the session. If one appeared, or the check throws, the rename is
undone. Only a confirmed copy becomes `.expired-<UUID>` and is deleted
with asynchronous file operations, so a multi-gigabyte removal does not
block the event loop. A later sweep restores an unconfirmed
`.retiring-*` copy when `current` is missing, drops it when a newer
`current` exists, and finishes an interrupted deletion. The rest of the
session directory is kept.
- **`native-session-executor.ts`:** exports
`nativeSessionScopeDigestIsBusy`, which checks the in-process executing,
initializing, warm, and closing session maps.
- **`native-harness-backup-stamp.ts`:** exports the state base path the
sweep scans.
- **`config.ts` and `index.ts`:** three environment controls
(`_ENABLED`, `_DRY_RUN`, `_DAYS`) and the scheduled sweep. It starts two
minutes after boot, honors shutdown, standby, and idle drain, and logs
candidates, reclaimed bytes, cleared task sessions, and skip reasons.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the sweep;
`doc/DEVELOPING.md` lists the variables.
- No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/native-harness-backup-expiration.test.ts
src/__tests__/native-harness-backup-expiration-config.test.ts` (in
`server/`): 30 passed, 24 of them against embedded Postgres plus 6
config tests. Cases:
- expiry for done and cancelled issues, including clearing the task
session;
- each keep condition: open issue, recently closed, live run, pending
finalization, recovery in progress, live lease, recent copy, cleanup
activation, in-process session;
- settled finalizations and expired leases do not block;
- dry run changes nothing;
- bounded batches continue across ticks;
- an interrupted deletion is finished, and an unconfirmed retiring copy
is restored or dropped;
- the configured clock controls the closed window.
- **Mutation checks:** treating `in_progress` as closed, ignoring live
leases, ignoring the configured clock, or not restoring a retiring copy
each fail the corresponding tests.
- `pnpm exec vitest run native-harness-backup config` (in `server/`): 19
files, 293 tests passed.
- `pnpm run typecheck` (in `server/`): pass.
- **Real server:** after building and bundling,
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reached healthy.
- **Manual equivalent on a live instance:** applying these checks by
hand (with a 3-day window) to 242 copies freed 7.1 GB. Most of the
remainder belonged to open issues or live leases.
## Risks
- **Reopen after the window:** the agent starts a fresh session and
loses the old conversation's provider history. Issue comments,
documents, and work products are unaffected.
- **Race with a wake:** the checks share-lock the issues, so a reopen
waits for the transaction. A run created for the session between commit
and rename is caught by the post-rename check, which restores the copy.
Such a run would find no task session and start fresh.
- **Missing owner run:** if the owning run row is gone (for example,
after an agent was deleted), the copy is kept, because the sweep cannot
prove the session is closed.
- **Untouched:** quarantined session roots and `continuity-breaks`
archives are not touched by this change.
- **Rollback:** set
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or revert.
Expired copies cannot be restored.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, local builds, tests and boots,
read-only measurements on a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
