## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each isolated task workspace is a Git worktree on the server disk,
and the terminal workspace reaper and git cache expiration keep that
disk usage bounded
> - For some days, workspace preparation gave each isolated
`git_worktree` workspace its own full clone under
`isolated-repository-sources/`, with the worktree inside it. #16963
removed that creation path, but the existing clones stay, and restoring
an expired workspace still recreates its private clone
> - On one instance, 30 such clones use about 23 GB: 12.1 GB of `.git`
data, 5.2 GB of unused default-branch checkouts and 5.75 GB of real
worktrees. 28 of them back open workspaces, and the disk is nearly full
> - This pull request moves each such open workspace onto the project's
shared managed clone with its exact state, deletes the private clone,
and makes restore use the shared clone
> - The benefit is that disk use drops to one shared clone per project
repository, with no lost work and no change for workspaces that cannot
move safely
## Linked Issues or Issue Description
Refs #16963. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 30 per-issue clones under
`isolated-repository-sources/<companyId>/<issueId>/checkout` use about
23 GB. 28 of them back open workspaces. No process moves these
workspaces to the shared project clone that new workspaces use, and git
cache expiration does not remove a clone while its workspace is open.
Reopening an expired workspace whose recipe has `sourceKind:
"isolated_remote"` clones the full repository again. Separately, a
restore point with a non-default file mode cannot be restored: the
reference read back from jsonb has reordered `modes` keys, so
`loadGitRestorePoint` fails with `restore_point_manifest_mismatch`.
**Expected behavior**
An open workspace with a private clone moves to the shared project clone
with the same branch, HEAD, index, worktree, untracked and ignored
files, and file modes. The private clone is then deleted. Reopening an
expired workspace uses the shared clone when one exists. A restore point
with non-default file modes restores.
**Steps to reproduce**
1. Create an isolated `git_worktree` workspace with
`materializeIsolatedRemoteWorkspaceSource` and record
`isolatedRemoteSource` metadata, as preparation did before #16963.
2. Keep its issue open. The clone stays, because no sweep moves or
removes it.
3. Expire the workspace with a restore point that has a file with mode
`0600`, then reopen it. The private clone is created again, and the
restore fails with `restore_point_manifest_mismatch`.
**Paperclip version or commit:** `master` at ec2e0d6407.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- New `server/src/services/isolated-repository-rehome.ts` with
`rehomeIsolatedRepositoryWorkspace`. It holds the isolated source lock,
the `execution_workspace_lifecycle:<id>` advisory lock and
expiration-style Git lock files (`index.lock`, `HEAD.lock` and the
branch ref lock in the clone).
- Eligibility uses the safety checks of git cache expiration that matter
while a workspace stays open: no live or unfinished relevant runs, no
unsettled native finalizations, no unreleased environment leases, no
running runtime services, no pending reopen, and no other workspace on
the same path. It does not require a closed or idle issue tree. The
check runs again inside the update transaction.
- The target base is the project's existing shared managed clone for the
same `repoUrl` (the paths `ensureManagedProjectWorkspace` uses, verified
with the same origin and storage checks as restoration).
- The move captures a restore point of the old worktree. The upstream
boundary is commits that both repositories have, so the bundle is small.
It imports the bundle into the shared clone, creates
`refs/heads/<branch>` or fast-forwards it from an ancestor, and adds the
worktree at `<base>/.paperclip/worktrees/<branch>`. Then it applies the
restore point. It copies the `HEAD` reflog and `ORIG_HEAD`, copies the
branch's upstream config when it creates the branch, and moves
regenerable directories, untracked `.env`/`.env.*` files,
`.claude/settings.local.json` and nested `.paperclip-repositories/*`
checkouts by rename. Standalone nested clones move as is. Nested
checkouts that are worktrees of their own shared clone (#17382) move
too, followed by `git worktree repair` in that clone so the registration
follows the new path, and their `paperclip/<hash>/` branch is renamed to
the new path's prefix so ownership checks still pass. Entries inside a
moved nested checkout (for example its `node_modules`) move with it.
Rollback and interrupted-move recovery repair them again after moving
them back, and verification checks that each one still resolves to the
same shared clone.
- Verification: the restore point snapshot (HEAD, index, worktree,
ignored files, modes) of the new worktree must equal the old one before
and after the move. The reflog and `ORIG_HEAD` must match, and every
moved item must exist. Only then does one transaction set
`cwd`/`providerRef`, drop `isolatedRemoteSource` and
`gitCacheExpiration`, rewrite a saved active restoration recipe to
`managed_project`, and record `isolatedRepositoryRehome: { at, from, to,
cleanedAt }`. A failure before that removes the new worktree, moves
items back, restores the branch, and leaves the old worktree untouched.
- Safety of the checkout in the shared clone: the move skips with
`unsupported_base_git_config` when the shared clone's local config sets
filter drivers, includes, attribute files, worktree config or command
settings, or when it has `info/attributes`. It checks before planning
and again right before `git worktree add`.
- Crash safety before the update: the move records
`isolatedRepositoryRehomePending` (base, target, branch, head, previous
branch tip, planned moves) before the first change to the shared clone.
The next attempt undoes an interrupted move under the same locks: it
moves items back, removes the new worktree only when it is proven to be
this move's (a linked worktree of the shared clone on the issue branch
that no workspace row references), and restores the branch. A live
failure removes the target only after this attempt created it. When an
item exists in both places, rollback stops with
`rehome_rollback_conflict` and keeps the target and the record, so no
copy is lost.
- Cleanup after the update: remove the old worktree (rename to trash,
then delete, then `worktree prune`). Delete the clone (rename to trash,
then delete) only when no other workspace, project workspace or running
service uses it, it has no other worktree, its checkout has no changes
or non-regenerable files, its refs are only branches, tags and
remote-tracking refs, and every local commit is on a remote-tracking ref
or present in the shared clone. Cleanup reads at most 5,000 refs, reflog
entries and unique commits (`clone_history_budget`). `cleanedAt` records
completion. A crash after the update leaves a pending record that the
next pass finishes, also after the workspace is archived: the reaper
scan includes archived rows with unfinished re-home cleanup when the
feature is on.
- Reaper integration in `sweepTerminalWorkspaces`: behind
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED` (default `false`) and
`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN`, only when git cache
expiration is enabled. It logs `event:
"execution_workspace.isolated_source_rehome"` with outcome, reason,
candidate bytes and reclaimed bytes. A skipped workspace gets a retry
deadline one hour later in the reaper's existing deadline set. No new
timer.
- Restore: `reopenClosedIsolatedExecutionWorkspaceForIssue` calls
`rehomeExpiredIsolatedRestoration` first. For an expired workspace with
an `isolated_remote` recipe, it uses the shared clone when one exists
and the branch there is absent or at the saved commit. It keeps the old
behavior for recipes with worktree-backed nested checkouts on branches,
because those branch names are bound to the old path. It also requires
that the shared clone has every base commit (prerequisite) of the saved
restore point bundle. It rewrites the row and recipe to the shared clone
and the standard worktree path, then the existing restore runs
(`ensureGitRestorePointCommit`, `ensureSavedGitCommit`, apply). Without
a shared clone, the old behavior stays.
- Fix: restore point `modes` comparison in
`git-restore-point-storage.ts` compares by path and mode, so jsonb key
order does not matter.
- Small exports: `allocatedBytes`, `expirationScope`,
`managedSourcePaths`, `verifiedGitCacheSource`.
- Docs: new "Isolated repository sources" section in
`doc/recoverable-disk-cleanup.md`, new settings in `doc/DEVELOPING.md`.
Deviations from the requested design:
- The sweep does not create a missing shared clone. When none exists, it
skips with `no_shared_base`, because a new network clone in a cleanup
path can use more disk. Restore follows the same rule and keeps the old
behavior.
- History moves through the restore point bundle, not a separate fetch
from the clone. The bundle contains HEAD and every reflog/`ORIG_HEAD`
commit that the shared clone does not have. The move then checks that
all of them are present.
- The standard location is `<base>/.paperclip/worktrees/<branch>`. A
worktree outside the clone's standard location is skipped
(`nonstandard_worktree_location`).
- Lock files use a token derived from the workspace id, so a crashed
attempt's locks are recovered without a database marker.
- A clone that another open workspace still uses is kept and marked
`cloneRetained: "base_has_consumers"`. That workspace's own move removes
it later.
Skip reasons: `live_or_unresolved_consumer`,
`unsupported_base_git_config`, `rehome_rollback_conflict`,
`nested_branch_conflict`, `invalid_rehome_record`,
`interrupted_rehome_pending` (dry run only), `lifecycle_busy`,
`source_busy`, `workspace_changed`, `no_shared_base`,
`shared_base_repo_mismatch`, `missing_worktree`,
`nonstandard_worktree_location`, `source_identity_changed`,
`branch_identity_changed`, `unproven_worktree_admin`,
`unsupported_worktree_git_state`, `unique_worktree_refs`,
`unknown_worktree_git_logs`, `unsupported_local_git_state`,
`unique_git_refs`, `unsupported_index_flags`, `unsupported_branch_name`,
`invalid_restoration_recipe`, `unsupported_restoration_recipe`,
`target_worktree_exists`, `base_branch_checked_out`,
`base_branch_diverged`, `untransferable_ignored_files`,
`untransferable_managed_checkouts`, `worktree_git_busy`, `head_changed`,
`history_not_transferred`, `transfer_conflict`, `snapshot_mismatch`,
`history_mismatch`, `transfer_incomplete`, and restore point reasons
such as `untracked_nested_repository`, `submodules_unsupported`,
`filtered_paths`, `restore_point_too_large`. Clone kept reasons:
`base_has_consumers`, `base_has_live_services`, `base_has_worktrees`,
`clone_checkout_has_files`, `unsupported_clone_refs`,
`unique_clone_history`, `clone_history_budget`.
## Verification
- `pnpm exec vitest run
src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 18
passed. It uses real Git repositories and embedded Postgres. It covers:
- a move with an unpushed commit, staged and unstaged edits, a deletion,
an untracked file with mode `0600`, an ignored file, `node_modules`,
`.env`, `.claude/settings.local.json` and a nested
`.paperclip-repositories` checkout. Snapshots, branch, HEAD, `git
status` and reflog match, the row points to the new path without
`isolatedRemoteSource`, and the old worktree and clone are gone;
- skips with nothing changed on disk or in the database: live run,
branch in the shared clone at an unrelated commit, no shared clone, an
untracked nested repository, and a concurrent change between capture and
verify (hook). The new worktree is removed and the old one keeps its
moved items;
- the clone is kept while another workspace references it;
- a worktree-backed nested checkout moves with the task worktree, stays
registered in its shared clone at the new path (not prunable), and keeps
its changes, with its branch renamed to the new path's prefix. On
rollback it returns to the old path with its registration repaired and
its branch name restored. A standalone nested checkout moves together
with its own ignored `node_modules`;
- a file recreated in the old worktree before verification stops the
rollback and keeps both copies, then the move completes once the
conflict is gone;
- a hostile `filter.*.smudge` in the shared clone's config makes the
move skip without running it;
- a clone with more than 5,000 refs is kept, and is deleted once the
refs are gone;
- a move interrupted before the database update is reported (not undone)
by a dry run, then undone and completed on the next attempt;
- a target that another party creates right before `git worktree add` is
never deleted;
- a crash after the database update; after the row is archived, a later
reaper sweep removes the old worktree and clone;
- restore keeps the private-clone fallback when the shared clone lacks
the restore point's base commits;
- disabled by default (both flags needed), dry run changes nothing, and
the retry deadline is registered;
- reopening an expired `isolated_remote` workspace rebuilds it in the
shared clone with the restore point applied and no clone created.
- After the round 4 fixes, the related suites
(`git-workspace-expiration`, `git-restore-point-expiration`,
`execution-workspace-reopen`, `execution-workspaces-service`,
`heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`,
`isolated-remote-workspace-source`) with the new suite: 326 passed.
- After rebasing on #17382, the related suites (adding
`heartbeat-project-repositories`) with the new suite: 374 passed.
- After the first review fixes, the related suites
(`git-workspace-expiration`, `git-restore-point`,
`git-restore-point-expiration`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`) with the new suite: 301 passed.
- Related suites in `server/` (first commit): `pnpm exec vitest run` on
`git-workspace-expiration`, `git-workspace-expiration-eligibility`,
`git-restore-point`, `git-restore-point-expiration`,
`git-restore-point-expiration-config`, `git-restore-point-ledger`,
`git-restore-point-ledger-config`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`,
`isolated-remote-workspace-source-lock`, `git-workspace-base-storage`,
`git-reproducible-managed-checkouts`, `git-expiration-auth-fallback` and
`workspace-runtime`: 550 passed, 2 failed. The 2 failures are in
`workspace-runtime.test.ts` (base ref resolution) and also fail on
unmodified `master` on the same machine.
- `pnpm run typecheck` in `server/` passes (after `pnpm -r --filter
"./packages/**" build`).
- Mutation checks. Each reverted guard fails one test in the new suite:
- eligibility always true: "keeps a workspace with a live run" fails;
- snapshot verification removed: "removes the new worktree and keeps the
old one when the workspace changes during the move" fails;
- clone reference check removed: "keeps the clone while another
workspace still references it" fails;
- restore re-home removed from reopen: "restores an expired isolated
workspace into the shared project clone" fails;
- old `JSON.stringify` modes comparison: the same restore test fails;
- interrupted-move recovery removed: the interrupted-move test fails;
- target existence check before `git worktree add` replaced by assumed
ownership: the foreign-target test fails;
- bundle prerequisite check removed: the stale shared clone restore test
fails;
- archived-row cleanup scan removed: the crash test fails;
- dry run check before interrupted-move recovery removed: the
interrupted-move test fails;
- `git worktree repair` after the forward move removed: both nested
worktree tests fail. The repair after moving back removed: the rollback
test fails;
- nested branch rename forward or back removed: the matching nested test
fails;
- rollback conflict check removed: the reappearing-file test fails;
- covered-entry filter removed: the full re-home test fails;
- shared clone config check removed: the hostile filter test fails;
- ref budget removed: the many-ref test fails.
- Not run: the full server suite, and the move on a live instance.
## Risks
- Workspace paths change for re-homed workspaces. Agents that resume an
old session can remember the old path. The run's working directory comes
from the updated row, but text in a resumed conversation can still name
the old path.
- The feature is off by default. It runs only with
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` and git cache expiration
enabled. Dry run is available. The restore change is not behind the
flag: reopening an expired private-clone workspace now uses the shared
clone when one exists.
- A run can start between the last eligibility check and the database
update. The update transaction checks eligibility again, and the old
worktree is deleted only after another check. This is the same window
that git cache expiration accepts.
- Imported restore point objects stay in the shared clone as unreachable
objects after a failed attempt until Git garbage collection removes
them. Base expiration keeps a base with unreachable objects, so this
only delays base expiration.
- A workspace that cannot move (for example `no_shared_base`) is retried
each hour while the feature is on. Each retry runs a few Git commands
for that workspace.
- No migration. No schema change.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, Git, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration saves restore points and tracks them in
`execution_workspace_restore_points`
> - A unique index allows one row per workspace, generation and checkout
unless the row is deleted
> - A pending row left by an interrupted attempt therefore blocked every
retry for an hour, and superseded rows blocked retries until retention
deleted their objects
> - This pull request narrows the index to live rows and supersedes
stale pending rows before inserting
> - The benefit is that an interrupted expiration no longer stalls a
workspace's cleanup
## Linked Issues or Issue Description
Refs #17257. Bug report described in-PR (bug report template).
**What happened?**
On one instance, a closed workspace with a nested checkout failed
expiration with `inspection_failed` on every sweep. An earlier attempt
had left a `pending` row for the nested checkout at the workspace's next
generation. Each retry stored the main worktree's restore point, then
failed to insert the nested row because the unique index on
`(execution_workspace_id, generation, checkout_name) WHERE status <>
'deleted'` still held the stale row. Running the expiration by hand
reproduced the failing insert.
**Expected behavior**
A retry after an interrupted attempt replaces the stale row and expires
the workspace.
**Steps to reproduce**
1. Leave a `pending` restore point row for a workspace's next generation
and a nested checkout, as an interrupted attempt does.
2. Run expiration with restore points: it fails with `inspection_failed`
until retention supersedes the row after the one-hour pending grace
period.
**Paperclip version or commit:** `master` at 0fb152ee94.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- **Schema**
(`packages/db/src/schema/execution_workspace_restore_points.ts`,
migration `0375`): the unique index is renamed
`execution_workspace_restore_points_live_status_uniq` and covers only
`pending`, `active` and `expiring`. Superseded, delete-requested and
deleted rows no longer block a new row.
- **Storage** (`storeGitRestorePoint`): in the transaction that inserts
the new `pending` row, any `pending` row for the same workspace,
generation and checkout is first marked `superseded`. The expiration
holds the workspace lifecycle lock, so such a row can only come from an
attempt that already ended. Retention then deletes its object as usual.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "stale pending restore point"` (in `server/`): 1 passed. A workspace
with a stale `pending` nested row and a `superseded` main row from an
earlier attempt expires with a nested restore point, and the stale row
is superseded.
- **Mutation checks:** removing the supersede step, or keeping the old
index scope in the migration, each reproduces the instance's
`inspection_failed`.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 274 passed.
- `pnpm run check:migrations` (in `packages/db`): pass.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Migration:** drops and recreates one partial unique index on a small
table. It cannot fail on existing data, because the old index was
stricter. The index is briefly absent while the migration runs.
- **Superseding a pending row:** only rows for the same workspace,
generation and checkout, while the current attempt holds the workspace
lifecycle lock.
- **Not addressed:** why the earlier attempt left the row behind. #17341
logs the underlying error of `inspection_failed` so the next occurrence
can be diagnosed.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only and dry-run
inspection of a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees once their
remote, or a restore point, can reproduce them
> - A nested `.paperclip-repositories` checkout's remote-tracking refs
can point at commits the remote has since dropped, such as a branch
deleted after a squash merge
> - Strict expiration rightly keeps such a checkout, because it may hold
the only copy; but restore point mode also kept it forever, because
there was nothing else to save
> - This pull request saves those commits in the checkout's restore
point, so the workspace can be reclaimed without losing them
> - The benefit is that long-lived multi-repository workspaces are
reclaimed in restore point mode
## Linked Issues or Issue Description
Refs #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, four closed workspaces on one instance were
kept with `unpushed_managed_checkout`, after a dry run had counted them
as candidates. Each nested `paperclip` checkout was clean, on `main` at
a commit that is on GitHub `main`. But it held about 1,000
remote-tracking refs, and between 28 and 41 of them pointed at branches
the remote has since deleted or force-pushed (147 to 203 commits it no
longer has).
The relaxed check passed, so the checkout needed no restore point beyond
those commits. Capture saved only `HEAD` history, so it found nothing to
save and stored no restore point. The final check before deletion then
ran the strict check, which keeps stale remote-tracking commits, and
rolled back. The dry run never reaches that final check, so it
over-reported. Together these workspaces held about 2.4 GB.
**Expected behavior**
In restore point mode, commits that exist only in remote-tracking refs
are saved, and the workspace expires.
**Steps to reproduce**
1. In a nested checkout, fetch a remote branch, so
`refs/remotes/origin/<branch>` exists.
2. Delete that branch on the remote and garbage-collect it.
3. With restore points enabled, close the issue and run expiration: the
workspace is kept with `unpushed_managed_checkout`.
**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- **Strict mode is unchanged.** An earlier revision of this pull request
ignored remote-tracking refs. Review pointed to `does not trust a stale
remote-tracking ref`, which deliberately protects those commits, so this
pull request keeps that behavior and the test is unchanged.
- **Relaxed nested check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`):
- reads all reflogs in one bounded call (`--format=%H %gD`),
deduplicates them by commit, and splits them into local and
remote-tracking entries before applying the tip budget;
- local entries still have to be covered or upstream, as before;
- remote-tracking tips and remote reflog entries that the remote no
longer has are returned as `preserve` (new `tipsMissingUpstream`,
sharing `commitsMissingUpstream` with the existing check).
- **Capture:** the proof is shared as `provenManagedCheckoutHistory`.
`storeGitRestorePoint({ resolvePreserveCommits })` calls it after taking
the history digest and before capture, then compares the digest again
after capture. A fetch before the digest is included in the preserved
commits, and one after it changes the digest and aborts capture. The
commits flow into `captureRestorePoint`. They are bundled as
history-only commits, within the existing limits (50 preserved tips,
1,000 commits, 256 MB bundle). The checkout therefore gets a restore
point, and the final check uses the relaxed check for it.
- **Final check:** `restorePointHistoryDigest` now also covers
remote-tracking refs and their reflogs, so a fetch, ref change or
reflog-only change after capture rolls expiration back.
- **Restore:** the commits are imported as objects; the stale
remote-tracking refs themselves are not recreated.
- **Docs:** `doc/recoverable-disk-cleanup.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
src/__tests__/git-reproducible-managed-checkouts.test.ts -t "stale
remote-tracking|remote-tracking refs change"` (in `server/`): 5 passed.
- The existing strict test passes unchanged.
- A clean nested checkout whose remote-tracking ref points at a deleted,
garbage-collected branch is kept strictly, expires with a restore point,
and reopens on its commit with the stale commit present.
- A remote-tracking ref created after capture rolls expiration back, and
so does a reflog-only change to one.
- 6,000 repeated reflog entries for one commit do not hit the tip
budget.
- **Mutation checks:** each of these fails a case:
- dropping the preserved commits reproduces the instance's symptom
(`unpushed_managed_checkout`);
- dropping remote-tracking refs or their reflogs from the digest;
- removing deduplication (`managed_checkout_local_tip_budget`).
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 261 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled.** Strict expiration is unchanged.
- **Retention:** saved stale commits are kept until the restore point
expires (90 days after the tree closes by default), then deleted with
it.
- **Limits:** checkouts with more than 50 stale tips, or bundles over
the limits, skip capture and stay kept, as before.
- **Spurious rollbacks:** a fetch during the short window between
capture and deletion rolls back; the next sweep retries.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees after saving a
restore point of anything the remote cannot reproduce
> - Restore points left ignored files out, so any worktree with an
ignored file that was not a known regenerable directory was kept
> - On one instance that was the most common reason a closed worktree
was still kept, mostly build caches and agent evidence files
> - This pull request saves those ignored files in the restore point too
> - The benefit is that more closed worktrees are reclaimed, and nothing
an agent left behind is lost
## Linked Issues or Issue Description
Refs #17248, #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, one instance kept 15 of 43 inspected closed
worktrees with `unique_ignored_files`. Among 19 such worktrees on disk:
- 9 held `ui/tsconfig.tsbuildinfo`;
- others held build output outside the regenerable directory names
(`server/ui-dist/`, `test-results/`);
- 2 held materialized skill copies under `.claude/skills/`;
- about 5 held small files agents kept out of git: patches, evidence
archives, recovery receipts.
**Expected behavior**
Ignored files that are not regenerable are saved with the restore point
and restored on reopen, so the worktree can be removed.
**Steps to reproduce**
1. In a closed issue's worktree, leave an ignored file outside
`node_modules`, `dist` and the other regenerable directories (for
example `ui/tsconfig.tsbuildinfo`).
2. With restore points enabled, run expiration: the worktree is kept
with `unique_ignored_files`.
**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Capture** (`server/src/services/git-restore-point.ts`):
- `restorePointSavesIgnoredFile` defines which ignored files are saved.
It leaves out:
- `.env` and `.env.*` files;
- `.paperclip-repositories/` (nested checkouts have their own restore
points);
- `.claude/settings.local.json` (saved with the workspace contents);
- anything inside a directory named in
`REGENERABLE_WORKSPACE_DIR_NAMES`.
- Those files are scanned with the same limits and checks as untracked
files, and snapshotted into a separate `ignored` tree, committed on top
of `HEAD`. Its ref travels in the bundle.
- They are hashed with `hash-object --no-filters` (symlinks by their raw
target bytes), so `.gitattributes` conversions such as `text=auto` never
change their bytes. Every path given to `--stdin-paths` is C-quoted, so
git always reads the literal workspace file, even for names that start
with `"`.
- An ignored nested repository skips capture, which keeps the worktree.
- A worktree whose only extra files are ignored is no longer "nothing to
save".
**Format:** a payload with an ignored tree uses
`paperclip.git-restore-point-payload.v2`, and one without stays `v1`.
Decoding requires `ignored` exactly for `v2`, so older servers reject
the new payloads instead of restoring without the files.
**Restore:** `applyRestorePoint` writes the ignored files directly from
their blobs, read in one `cat-file --batch` call, so no filter is
applied.
- It checks every target before writing, refuses any path that would
leave the worktree, and never overwrites an existing file: a conflict
fails the restore closed.
- If a write fails partway, it removes the files it wrote, so a retry
starts clean. The post-apply verification, the before-deletion snapshot
check and `restorePointSnapshotMatches` all include the ignored tree.
The stored reference (`GitRestorePointRef.ignored`) is validated and
checked against the decoded manifest.
**Expiration** (`git-workspace-restoration.ts`): in restore point mode,
the reproducibility check accepts the ignored files a restore point
saves. Strict mode is unchanged.
**Docs:** `doc/recoverable-disk-cleanup.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-restore-point.test.ts` (in
`server/`): 27 passed. New cases:
- **Round trip:** ignored files round-trip as a `v2` payload with their
modes, while `dist/`, `node_modules/` and `.env` stay out.
- **Conflict:** an existing conflicting file fails the restore closed
and is left as it was.
- **Nested repository:** an ignored nested repository skips capture.
- **Forged payload:** a `v1` payload claiming an ignored tree is
rejected.
- **Line endings:** an ignored CRLF file in a `* text=auto` repository
restores byte for byte.
- **Partial failure:** a restore that fails partway leaves nothing
behind, and a retry succeeds.
- **Quoted names:** a crafted ignored path that looks like a quoted
absolute path saves and restores the workspace file, never the outside
file it names.
- **Symlinks:** a symlink target with bytes that are not valid UTF-8
restores exactly.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): new
cases.
- A closed worktree whose only extra files are an ignored build cache
and evidence file is kept without restore points, expires with one, and
reopens with those files and without `node_modules`.
- An ignored file changed after capture rolls back.
- **Mutation checks:** each of these fails at least one case:
- not relaxing the reproducibility check;
- not capturing ignored files (the end-to-end case then catches the
loss);
- leaving the ignored tree out of the comparison;
- not writing the files on apply;
- hashing with filters;
- dropping the partial-restore rollback;
- passing unquoted paths;
- decoding symlink targets as UTF-8.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration execution-workspaces
workspace-compaction heartbeat-git-restore-point-resume
src/services/heartbeat/environment-selection` (in `server/`): 296
passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled.** Strict expiration is unchanged.
- **Bigger restore points.** Ignored files count toward the existing
limits (20,000 files, 64 MB per file, 256 MB total). Larger caches
outside the regenerable names make capture skip, which keeps the
worktree as before.
- **Rollback:** older servers cannot restore `v2` restore points; they
fail closed. Disable restore points before downgrading.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 and #17257 let git cache expiration remove closed issues'
worktrees after saving encrypted restore points
> - Restore points are kept for 90 days after an issue tree closes, then
deleted
> - Reopening a workspace whose restore point had expired failed, so the
issue could not get a workspace back
> - This pull request starts a fresh workspace on the same branch
instead, and warns that the saved work is gone
> - The benefit is that old issues can always be resumed, with a clear
record of what was lost
## Linked Issues or Issue Description
Refs #17248, #17257. Bug report described in-PR (bug report template).
**What happened?**
After `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` passes, the retention
worker marks the restore point expired and deletes it. Reopening that
workspace then fails with the git cache restoration error, every time.
**Expected behavior**
Reopening succeeds with a fresh workspace on the issue's branch, and the
issue says that unsaved work from before the cleanup is no longer
available.
**Steps to reproduce**
1. Enable restore points and let a closed issue's worktree expire with
unsaved work.
2. Let its restore point expire.
3. Reopen the issue: the workspace rebuild fails.
**Paperclip version or commit:** `master` at 650581b2f0.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- **Detection:** `gitRestorePointExpired(recipe)` is true when the main
restore point or any nested checkout's restore point has `expiredAt`.
- **Reopen** (`execution-workspaces.ts`):
- `restoreGitCacheBase(..., { freshStart: true })` prepares the base
without the saved commit or the bundle.
- The saved recipe is left out of the metadata handed to the runtime,
and dropped from the stored metadata only when the reopen succeeds, so a
failed attempt can be retried and still warns. The runtime recreates the
worktree on its branch: the branch tip if the repository still has it,
otherwise a new branch from the base ref. When the expired recipe had
nested checkouts, the project's repositories are prepared with
`prepareProjectRepositoryWorkspaces` (without the expired saved state)
right after the worktree is recreated and before the provision command
runs, with the same copy identity and access rechecks as other reopen
copies.
- The result carries `warning`, the `execution_workspace.reopened`
activity records `restorePointExpired` and the warning, and a system
comment on the issue posts the warning in the same transaction.
- **Retention:** `markGitRestorePointExpired` now also marks nested
checkouts' restore points, matched by object key.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the fresh start.
No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 13
passed. New case: an expired restore point reopens fresh with the
warning and the system comment, on the original branch, even after the
base lost the branch and its unpushed commit; the recipe is gone. A
first attempt that fails keeps the expired recipe, and the retry still
warns and comments exactly once.
- New nested case in `git-workspace-expiration.test.ts`: an expired
nested checkout restore point is marked on the checkout entry and
reopens fresh with the warning, with the nested repository cloned again
at its pushed commit and without its expired uncommitted files.
- **Mutation checks:** each of these fails a case:
- removing the fresh base path (reopen fails);
- removing the system comment;
- handing the runtime the old recipe;
- dropping the stored recipe before the rebuild succeeds;
- marking only the main restore point expired;
- skipping nested repository preparation on a fresh start.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` plus the environment selection
and heartbeat resume suites (in `server/`): 257 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled:** recipes get `expiredAt` only
from the restore point retention worker, so nothing changes otherwise.
- **Lost work is intended:** the fresh workspace has no uncommitted
files and, when the repository no longer has the branch, no unpushed
commits. The warning says so.
- **Recipe dropped:** after a fresh start the workspace no longer
carries the expired recipe, so later expirations start from a clean
slate.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 lets git cache expiration remove a closed issue's worktree
with unsaved work after saving an encrypted restore point
> - Workspaces with nested `.paperclip-repositories` checkouts were
still kept whenever a checkout had unpushed or uncommitted work
> - On one instance that was the second most common reason expiration
skipped a workspace
> - This pull request gives each nested checkout its own restore point
and restores it when the workspace comes back
> - The benefit is that multi-repository workspaces can be cleaned up as
safely as single-repository ones
## Linked Issues or Issue Description
Refs #17238, #17241, #17248. Bug report described in-PR (bug report
template).
**What happened?**
With restore points enabled, a workspace whose nested managed checkout
has unpushed commits or uncommitted files is still skipped with
`unpushed_managed_checkout` or `dirty_or_untracked_files`. On one
instance, `unpushed_managed_checkout` accounted for 32 of the last 111
expiration checks.
**Expected behavior**
A nested checkout's unsaved work is saved like the main worktree's, and
the workspace can expire and later be restored exactly.
**Steps to reproduce**
1. Use a project with an extra repository, so the task worktree contains
`.paperclip-repositories/<name>`.
2. Commit in that checkout without pushing, then close the issue.
3. With restore points enabled, the workspace is never expired.
**Paperclip version or commit:** `master` at 6d44b80a5e.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Relaxed checkout check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`, used only in restore point mode):
- **Allowed:** the checked-out branch's unpushed commits, plus its
uncommitted, untracked and history-only work.
- **Still blocks:**
- another local branch or tag whose tip the remote lacks and `HEAD` does
not contain (`unpushed_managed_checkout_branches`), because restore
points do not recreate other branches;
- commits in any reflog that are neither on the remote nor in the
bundled history (`HEAD`'s reflog and `ORIG_HEAD`), such as work left in
another branch's reflog;
- objects reachable from nothing;
- a `COMMIT_EDITMSG` that matches no commit in the checkout's history.
- **`COMMIT_EDITMSG`:** a nested checkout is a full clone, so its base
git storage check now accepts a `COMMIT_EDITMSG` that matches a commit
in its history, in strict and relaxed mode. It is read only if it is a
regular file of at most 64 KB, opened without following symlinks and
without blocking; anything else keeps the workspace
(`unsupported_commit_message`). The main worktree's existing
`COMMIT_EDITMSG` read uses the same bounded reader. A clean checkout
with only an unpushed commit therefore reports
`unpushed_managed_checkout`, which restore point mode can relax, instead
of `unknown_base_git_files`.
**Capture** (`git-workspace-expiration.ts`):
- **Expiration flow:** strict inspection still runs first.
`unpushed_managed_checkout` is now relaxable. The relaxed prover records
which checkouts need restore points.
- **Storage:** each checkout's restore point is stored under
`.../<generation>/checkouts/<name>/<id>.pcrp`, against a remote-verified
boundary from that checkout. It is recorded on its
`contents.checkouts[]` entry.
- **Activation:** all restore points of the generation are activated
together in the archive transaction. Any nested restore point makes the
recipe `paperclip.git-cache-restoration.v2`, so older servers reject it.
- **Schema:** migration `0374` adds `checkout_name` to
`execution_workspace_restore_points` and includes it in the
live-generation unique index.
- **Final check:** under the expiration locks, each checkout with a
restore point gets the relaxed check and must match its own snapshot.
Checkouts without one get the strict check and must still be clean.
Every restore point, the main worktree's included, records a digest of
its `HEAD` reflog and `ORIG_HEAD` at capture, and that history must be
unchanged, so a commit followed by a reset after capture rolls back.
Rollback drops every restore point reference, returns the recipe to v1,
and supersedes the stored restore points.
**Restore** (`git-workspace-contents.ts`):
- **Re-cloned checkouts:** re-cloning a checkout imports its bundle
before the saved commit is needed. After checking out the saved commit,
it applies and verifies the restore point.
- **Existing checkouts:** the same helper as the main worktree marks a
matching checkout applied, and fails closed instead of overwriting a
diverged one.
**Validation:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` validate nested restore point references
against their checkout names. Contents comparison ignores the reference.
**Docs:** `doc/recoverable-disk-cleanup.md` describes nested checkouts.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "restore points for nested checkouts"` (in `server/`): 10 passed,
with a real nested checkout and embedded Postgres. The cases:
- **Round trip:** an unpushed commit and an untracked file in the nested
checkout survive expiration and reopen. The recipe is v2, the restore
point lives on the checkout entry, and a ledger row carries the checkout
name.
- **Another branch:** unpushed work on another local branch keeps the
workspace.
- **Change after capture:** a nested change after capture rolls back,
keeps the files, returns the recipe to v1, and supersedes the restore
point.
- **Only an unpushed commit:** a clean nested checkout with only an
unpushed commit (and its `COMMIT_EDITMSG`) expires and restores.
- **Reflog-only history:** unpushed history held only in another
branch's reflog keeps the workspace.
- **History after capture:** a commit followed by a reset after capture
rolls back, in a nested checkout and in the main worktree.
- **Unsafe `COMMIT_EDITMSG`:** a named pipe, an oversized file, or a
symlink keeps the workspace without reading it unsafely.
- **Mutation checks:** each of these fails a case:
- removing the other-branch check, the per-checkout snapshot check, the
rollback cleanup, or the checkout apply;
- rejecting a saved `COMMIT_EDITMSG`;
- dropping reflog coverage;
- dropping the history digest check;
- removing the size cap.
Not covered by a dedicated case: a workspace with two nested checkouts
where only one has a restore point.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` (in `server/`): 232 passed.
- `pnpm run check:migrations` (in `packages/db`): pass.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.
## Risks
- **Off by default:** nothing changes unless
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- **Other local branches:** a nested checkout's other local branches are
not saved. Any of them with commits the remote lacks keeps the workspace
instead.
- **Migration:** adds a defaulted column and replaces one partial unique
index on a table only written when restore points are enabled.
- **Rollback:** disable the setting. Workspaces already expired with
nested restore points keep v2 recipes that require this code to restore.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes a closed issue's worktree only when the
remote can reproduce it exactly
> - Most closed worktrees hold unpushed commits or uncommitted files, so
they are kept forever and disks fill up
> - #17238 added restore points (an encrypted save of exactly what the
remote cannot reproduce) and #17241 added their lifecycle
> - This pull request lets expiration remove those worktrees after
saving a restore point, and restores them exactly when the work resumes
> - The benefit is bounded disk use for closed work without losing
anything a user could need, behind a setting that is off by default
## Linked Issues or Issue Description
Refs #17238, #17241. Bug report described in-PR (bug report template).
**What happened?**
With `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one
instance's last 111 expiration checks removed nothing:
| Skip reason | Checks |
|---|---|
| `dirty_or_untracked_files` | 39 |
| `unpushed_managed_checkout` | 32 |
| `live_or_unresolved_consumer` | 21 |
| `unique_ignored_files` | 9 |
| commit not on the remote | 10 |
Without expiration, the terminal workspace reaper archives only merged,
clean worktrees. Closed issues' worktrees therefore accumulate without
bound.
**Expected behavior**
A closed worktree whose only obstacle is unsaved work can be removed
after that work is saved, and restored exactly if the issue is reopened
or work resumes.
**Steps to reproduce**
1. Close an issue whose worktree has an unpushed commit or an
uncommitted file.
2. Enable git cache expiration with dry run off.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at 12bb7e1976.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Expiration** (`server/src/services/git-workspace-expiration.ts`):
- **When it applies:** with `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`
(default `false`), a worktree that fails the reproducibility check for a
relaxable reason is reconsidered once its issue tree has been closed for
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). The
relaxable reasons are uncommitted or untracked files, a commit not on
the remote, reflog or `ORIG_HEAD` history, and ignored `.env` files. A
leftover `COMMIT_EDITMSG` still blocks expiration unless it matches a
commit in the worktree's reflog, because restore points do not save
draft messages.
- **What still blocks:** a relaxed inspection
(`inspectGitCacheRecipeForRestorePoint`) keeps every other check. Nested
managed checkouts, submodules, special git state, local config and other
unique ignored files still prevent expiration.
- **Capture:** inside the existing source and lifecycle locks, a restore
point is captured with the #17238 library against a remote-verified
boundary (the `HEAD` itself, or its merge base with `origin/HEAD`), then
encrypted, uploaded, read back and hash-checked
(`git-restore-point-storage.ts`).
- **Commit before touching disk:** it is activated in the #17241 table
in the same transaction that archives the workspace with a
`paperclip.git-cache-restoration.v2` recipe referencing it, still before
anything on disk is touched.
- **Final check:** under the file locks, the worktree must still match
the captured snapshot (`HEAD`, index, working tree, modes). Otherwise
the existing rollback runs. The rollback now also drops the restore
point reference and returns the recipe to v1, so a later restore can
never apply a stale restore point over the live worktree; the stored
restore point is marked superseded.
- **Failure paths:** the `pending` row is inserted in a transaction that
first registers a `git-restore-points` notification. If the upload or
read-back fails, the row is marked `superseded` so retention deletes any
uploaded object. Nothing is deleted from disk.
- **Encryption key:** restore points are refused
(`restore_point_key_not_durable`) on `s3` storage unless the secrets
master key comes from `PAPERCLIP_SECRETS_MASTER_KEY`.
- **Dry run** reports `restorePoint: true` candidates without storing
anything.
**Restore** (`git-workspace-restoration.ts`, `workspace-runtime.ts`,
`execution-workspaces.ts`):
- **Recipe versions:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` accept v1 without a restore point, or v2
with a valid one. Older servers reject v2 instead of restoring the
commit without the saved work.
- **Before the saved commit is needed:** reopening
(`restoreGitCacheBase`) and the runtime path import the bundle when the
saved commit is missing locally.
- **After the worktree is re-added:** the runtime always imports the
bundle, so history-only commits are present again as objects even when
the base was rebuilt (they are not recreated as branches), then applies
the restore point, verifies the result, and records `appliedAt`. Reopen
persists that through a callback, as it already does for saved contents.
- **Never overwriting:** a worktree that already matches is only marked
applied. A worktree that diverged from the saved commit's clean state is
never overwritten; the restore fails closed and can be retried. Missing,
corrupt or expired restore points also fail closed.
**Retention:** the #17241 worker's expiry hook now marks the recipe's
restore point expired before its object is deleted.
**Config and docs:**
- `server/src/config.ts` adds the two settings;
`doc/recoverable-disk-cleanup.md` and `doc/DEVELOPING.md` describe them.
- No code comments were added.
**Not in this PR (follow-ups):**
- workspaces with nested `.paperclip-repositories` checkouts;
- removing idle worktrees of issues that are still open;
- starting a fresh workspace with a warning when a restore point has
expired.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 12
passed, with real repositories and embedded Postgres. The cases:
- **Round trip:** an unpushed commit, staged and unstaged edits of one
file, a deletion, and an untracked file are restored exactly on reopen,
`.env` is dropped, and the restore point is active. Restoring through
the runtime path works the same way, and a later run never re-applies
over newer work.
- **Kept:** a worktree that is not yet due, one with the feature
disabled (today's skip reason), and one whose encryption key is not
durable.
- **Change after capture:** rolls back, keeps the worktree, returns the
recipe to v1 and supersedes the restore point.
- **Missing restore point:** fails closed, keeps the row archived and
unapplied, and fails closed again on retry.
- **Dry run:** stores nothing.
- **Reproducible worktrees:** expire as before, with a v1 recipe.
- **History-only commits:** reopen imports them even after they are
pruned from the base.
- **Draft messages:** a draft commit message keeps the worktree.
- **Upload failure:** a failed upload marks the pending restore point
for cleanup and keeps the worktree.
- **Mutation checks:** each of these fails at least one case:
- removing the pre-deletion snapshot check;
- keeping the restore point reference on rollback;
- removing the runtime apply;
- removing the re-apply guard (the divergence check then fails closed
instead of overwriting);
- skipping the import for a matching worktree, relaxing the draft
message guard, or not marking a failed upload's row.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration-config.test.ts`: 5 passed.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point workspace-runtime execution-workspaces` (in
`server/`): 529 passed on the first revision. Two
`workspace-runtime.test.ts` cases about remote base refs fail, and they
also fail on unmodified `master` on this machine. After the review
fixes, the expiration, contents, restore point and execution workspace
suites passed again: 222 tests.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.
## Risks
- **Off by default:** nothing changes unless both
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` are set. Roll out with
`PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=true` first to see which
worktrees would receive restore points.
- **Data at rest:** restore points contain source and uncommitted files,
encrypted with keys wrapped by the instance secrets key. Losing that key
makes them unrecoverable, which is why `s3` storage requires an
environment-provided key. Untracked `.env` files are not stored.
- **Interrupted apply:** a crash in the middle of applying a restore
point leaves a worktree that neither matches the restore point nor is
clean. The restore then fails closed rather than overwriting it, and an
operator must reset that worktree to retry.
- **Rollback:** disable the setting. Already-expired worktrees keep
their v2 recipes and restore points, so reverting the code would leave
those recipes unreadable until restored by a server with this change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
