## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Managed Git workspaces retain task and run obligations until cleanup
is safe.
> - Activity, expiration, retention and rehome share a run association
query.
> - That query repeatedly extracted three keys from wide historical JSON
contexts.
> - This pull request decodes those keys together and projects the six
fields its consumers use.
> - Native bindings, cleanup vetoes, lifecycle locks and fresh deletion
checks stay in place.
> - The benefit is less database work while preserving the same cleanup
decisions.
## Linked Issues or Issue Description
**What happened?**
Workspace activity and eligibility queries repeatedly decompress large
run contexts. An OR predicate extracts issue, task and workspace
compatibility keys separately. The run scope also materializes columns
its lifecycle consumers do not read.
**Expected behavior**
Read the same relevant runs and preserve all deletion and rehome vetoes
with less repeated JSON work. Keep existing native, historical and
pointer bindings independent.
**Steps to reproduce**
Run the synthetic PostgreSQL comparison below against baseline
`db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. Repeat the small, large-tree
and combined controls documented in the PR.
Related work: #17464 preserves workspace descendants. #17877
reconstructs native workspace obligations. This change preserves both
contracts and adds no scheduling change.
## What Changed
- Decode the three legacy context keys together with `jsonb_to_record`.
A strict root-object guard preserves scalar, array and null behavior.
Independent native and pointer bindings still match.
- Materialize the full textual tree-ID set once. This avoids inflated
recursive estimates triggering JIT compilation for the correlated
decoder.
- Project the six run lifecycle columns used by activity, expiration,
orphan retention and rehome.
- Add binding, lease, malformed-shape and fresh-read regression cases.
Add a reproducible four-control benchmark and detailed evidence
documentation.
## Verification
- Full repository typecheck passed on the integrated final source.
- Six relevant suites passed all 209 tests against real PostgreSQL,
including the 20 new binding/shape cases, 25 eligibility cases, full Git
expiration/restoration, task-directory retention and configuration, and
isolated repository rehome. These cover conflicting bindings, company
isolation, duplicate admission, pending cleanup, controller leases,
non-object contexts and newly active consumers.
- PostgreSQL 17.11 comparisons use two warmups and five alternating
samples per variant. Each sample runs eight sequential workspace
queries. Every complete caller result and six-column relevant-run
multiset matches.
- Stress fixture: 4,813 runs and about 332 MB of run storage, with
unique wide contexts. Last-activity median: 1,071.54 to 743.45 ms per
eight workspaces, 1.44 times faster. Closed eligibility: 814.29 to
713.86 ms, 1.14 times faster. Idle eligibility: 721.46 to 481.69 ms,
1.50 times faster.
- Small-history control improves 1.23–1.36 times. The 250-descendant
stress control improves 1.24–1.30 times. Combined
small-history/large-tree activity and idle checks improve 1.19–1.22
times. Combined closed eligibility is neutral: 61.11 to 62.82 ms per
eight, about 0.21 ms extra per workspace.
- Stress last-activity ranges do not overlap: 920.27–1,111.22 ms before
and 689.14–880.40 ms after. Closed-eligibility ranges overlap; its
smaller median gain is directional.
- Last-activity buffer accesses fall from 113,202 to 38,081. Eligibility
run width falls from about 1,414 to 57 planner bytes. All final plans
retain one history scan with no JIT.
- Reproduce: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-workspace-activity.ts
--baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2
--docker-postgres=17 --reject-new-jit
--output-dir=/tmp/workspace-activity-stress`.
- Repeat with `--small`, `--descendants=250`, and both flags, using
distinct output directories. Each run creates and removes its own
synthetic database.
- Full CI passed on `bc4edd700f983e22f093d0a01e2f4c795075b982`: 54
successful checks, one neutral review and two expected Storybook skips,
plus successful Snyk status. Greptile is 5/5 on this exact head. There
are zero unresolved threads and the merge state is clean. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/40530494291).
- Full repository typecheck and build passed on the integrated final
source. The local full test command was stopped after native PostgreSQL
initialization failed repeatedly, producing 16 failures and skipped
database suites. It is incomplete, not a pass. Focused real-database
suites used disposable Docker PostgreSQL. Current-head CI passed all
full gates separately.
## Risks
- Cleanup safety depends on complete run association. Tests compare
every retained run field and exercise downstream vetoes, not only row
counts.
- PostgreSQL plan estimates matter. Rejected per-key index probes were
up to 56 times slower on a small-history/large-tree control. An ANY
variant also regressed. The final shape passes all four controls under
default JIT settings.
- The final combined eligibility control is neutral. Synthetic SQL
timings do not establish deployed latency, provider behavior or actual
compute sleep.
- Review suggested a CASE/jsonb_typeof root guard as an optional
alternative. The strict JSONPath guard is retained because its scalar,
nested-array and independent-native behavior passes real PostgreSQL
tests and all four measured controls. No correctness defect was
identified. A replacement needs its own comparative measurements.
- Future consumers that need another run column must extend this
explicit projection.
- No migration, index, startup job, polling change or deployment is
included.
## Model Used
OpenAI Codex, configured model `gpt-6.1-sol` with xhigh reasoning, tool
use and code execution. Independent blast-radius review also used
`gpt-6-astra` and `gpt-6.1-sol`. The runtime context window was not
exposed.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration, isolated source re-home, nested clone
migration and workspace reopen fetch from the workspace's remotes to
prove history is on the remote or to rebuild workspaces
> - They built git credentials from the issue id alone, but the managed
GitHub identity is selected from the run's responsible user and agent
> - So the credential never resolved, they fell back to anonymous
access, and private repositories were unreadable
> - This pull request resolves credentials with the issue's responsible
user and assigned agent, the same identity agent runs use
> - The benefit is that workspaces with private repositories can expire,
re-home, migrate and reopen like public ones
## Linked Issues or Issue Description
Refs #17464, #17407, #17369. Bug report described in-PR (bug report
template).
**What happened?**
On one instance, 18 long-closed workspaces in one project were kept on
every expiration sweep with `managed_checkout_upstream_unavailable`.
Their nested checkout is a private GitHub repository. Expiration called
`createGitRemoteAuthProvider(db, companyId, { issueId })`. With that
context, `resolveManagedGitHubCredential` returned "No managed GitHub
identity is available for this run". `withAnonymousGitFallback` then
tried anonymous access, which cannot read the private repository. Called
with the issue's `responsibleUserId` and `agentId`, the same lookup
returned a credential.
**Expected behavior**
Background workspace jobs authenticate as the issue's identity, as agent
runs do, and can verify private repositories.
**Steps to reproduce**
1. Configure a project with a private GitHub repository and a managed
GitHub identity tied to the issue's responsible user.
2. Close an issue whose workspace includes that repository and let git
cache expiration inspect it.
3. It is kept with `managed_checkout_upstream_unavailable`.
**Paperclip version or commit:** `master` at fe85fd7ac4.
**Deployment mode:** any with managed GitHub identities.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- `createIssueGitRemoteAuthProvider(db, companyId, issueId)` in
`server/src/services/git-credentials.ts`:
- looks up the issue's `responsibleUserId` and `assigneeAgentId` once,
lazily, scoped to the company;
- builds `createGitRemoteAuthProvider` with `{ issueId,
responsibleUserId, agentId }`;
- falls back to the issue-only context when the issue is missing.
- Callers switched from the issue-only context:
- git cache expiration (`expirationGitAuth`);
- isolated source re-home;
- nested clone migration;
- both credential sites in
`reopenClosedIsolatedExecutionWorkspaceForIssue`.
- Anonymous fallback for public repositories is unchanged.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-credentials.test.ts
src/__tests__/git-expiration-auth-fallback.test.ts` (in `server/`): 32
passed. New cases:
- the helper passes the issue's responsible user and agent to the
credential lookup;
- it looks the issue up once across calls;
- it falls back to the issue-only context when the issue is missing.
- **Mutation check:** dropping the identity fields from the context
fails two cases.
- **Live instance:** for an affected workspace, the managed credential
resolved only with the responsible user and agent; with the issue id
alone, or with standing delegation, it did not.
- `pnpm exec vitest run git-credentials git-expiration-auth-fallback
git-workspace-expiration git-restore-point isolated-repository-rehome
nested-clone-migration` (in `server/`): 296 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Whose identity is used:** background jobs now use the issue's
responsible user's managed GitHub identity, which agent runs on that
issue already use. They only read: fetch, ls-remote and clone for
verification and restore.
- **Unchanged failure behavior:** if the identity is unavailable (for
example, the user is no longer a member), the existing anonymous
fallback applies, as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only diagnosis on a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents do task work in execution workspaces on the instance disk
> - A task without a project repository runs in a plain (non-Git) task
directory under `<instance>/isolated-workspaces/<company>/<issue>/`
> - Nothing removes these directories after the work is done, so the
disk fills up
> - The directories can hold the only copy of agent work, so plain
deletion is not safe
> - This pull request saves each finished task directory in an encrypted
restore point, removes it, and restores it when the work reopens. It
also removes orphaned directories after a grace period
> - The benefit is that finished task directories stop using disk, and
no agent work is lost silently
## Linked Issues or Issue Description
**What happened?**
On one instance, `instances/<id>/isolated-workspaces/` held 251 task
directories and 12 GB, and it grew every day. 61 directories belonged to
open workspace rows whose issues closed more than 7 days before (3.9
GB). 12 belonged to issues closed 3 to 7 days before (0.7 GB). 4.0 GB
were directories with no open workspace row (the row was archived or
missing).
The cause has two parts:
1. The terminal workspace reaper archives a workspace only when it
proves delivery through Git (a merged pull request or a merged branch).
A task directory is not a Git checkout, so the Git status inspection
fails and the reaper always skips the row as "undelivered". Git cache
expiration handles only `git_worktree` workspaces.
2. When a person archives such a workspace by hand, artifact cleanup
removes a `local_fs` directory only when `metadata.createdByRuntime` is
true. Task directory rows never set it, so the directory stays and the
row becomes `cleanup_failed`.
**Expected behavior**
Paperclip reclaims the task directories of finished work by itself. It
must not lose files that only exist in those directories.
**Steps to reproduce**
1. Run an agent on an issue without a project repository (for example a
low-trust sandbox task, or an issue with no project), so it works in
`isolated-workspaces/<company>/<issue>/`.
2. Let the agent write files there, then mark the issue `done`.
3. Wait any number of days. The directory and its workspace row stay.
The reaper logs the row as skipped (undelivered).
4. Archive the workspace by hand. The directory still stays.
**Paperclip version or commit**
`master` at fe85fd7ac4
**Deployment mode**
Self-hosted, local disk storage.
**Agent adapter(s) involved**
Any adapter that runs in a task directory (seen with local and sandbox
runs).
## What Changed
- Added `isolated-task-directory-snapshot.ts`: a deterministic snapshot
of a plain directory (files with modes, directories, symbolic links as
links). It never follows symbolic links, refuses special files, and
enforces the restore point limits (20,000 files, 64 MiB per file, 256
MiB in total). A `node_modules` directory next to a `package.json` is
left out. Restore writes to a staging directory, verifies the digest,
then renames it into place.
- Added `isolated-task-directory-retention.ts`:
- `expireIsolatedTaskDirectory` saves the directory in a restore point,
archives the workspace with the `isolatedTaskDirectoryRetention` state
in one transaction, checks again that the directory did not change,
renames it to trash and deletes it. A change after the capture rolls the
workspace back to active and drops the restore point. An interrupted
removal finishes on a later sweep.
- It uses the git cache expiration consumer checks
(`isGitWorkspaceExpirationEligible`): no live run, unsettled
finalization, live lease, live runtime service, unresolved recovery
action or reopen fence, and no other open workspace or project workspace
in the directory. It also requires the source issue to be bound to the
workspace, because reopen restores through that binding.
- `restoreIsolatedTaskDirectory` restores the directory on reopen. It
replaces an empty directory and fails closed on a non-empty directory
that does not match. When the restore point expired, it reopens an empty
directory and posts a system comment.
- `sweepIsolatedTaskDirectories` walks `isolated-workspaces/` in bounded
batches (8 per sweep, with a cursor). A closed workspace that its issue
is still bound to is saved and removed like an open one. Other folders
are orphans and are removed after a grace period when no live row,
project workspace, runtime service, run, finalization, lease or recovery
action refers to them.
- Reused the git restore point ledger, storage and encryption.
`storeSealedRestorePoint` and `readRestorePointObject` were extracted
from `storeGitRestorePoint`. `markGitRestorePointExpired` now also marks
a task directory restore point as expired.
- `isGitWorkspaceExpirationEligible` takes `closedSiblingsAllowed`, so
closed rows for the same issue directory do not block a task directory
forever. Git expiration keeps its old behavior.
- Terminal workspace reaper (`execution-workspaces.ts`): handles open
task directory rows after the terminal delay and registers that moment
as a deadline. It finishes pending removals, runs the folder walk at the
start of a rotation (once per server start, then only at real
deadlines), and saves at most 8 directories per sweep.
- Reopen accepts a closed `shared_workspace` row that holds task
directory state. The run path also reopens (and restores) a closed
workspace with that state before preparing the run.
- Settings: `PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED` (default
`false`), `PAPERCLIP_TASK_DIRECTORY_RETENTION_DRY_RUN` (default `true`),
`PAPERCLIP_TASK_DIRECTORY_ORPHAN_GRACE_DAYS` (default `7`, minimum `1`).
The delay is `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS`. Removal
needs `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- Logs: `execution_workspace.task_directory_retention`,
`execution_workspace.task_directory_cleanup` and
`execution_workspace.task_directory_restore`, with outcome, skip reason,
candidate bytes and reclaimed bytes.
- Documented the behavior in `doc/recoverable-disk-cleanup.md` and the
settings in `doc/DEVELOPING.md`.
**Preservation decision.** The code shows that these directories are not
scratch space. Nothing pushes or copies their contents, and the next run
of the same issue reuses the same path. So the files are saved before
removal, with the same ledger, encryption and retention as git restore
points, and restored on reopen. Orphans are not saved: no workspace row
can reopen them. The docs and the Risks section state this.
**Scheduling.** The work follows
`rules/sleep-aware-background-work.mdc`. It adds no timer and no
polling. Open rows use the existing reaper and its per-workspace
deadlines (the end of the terminal delay, or one hour after a retryable
skip). The folder walk runs once at startup. After that, workspace
archive and issue close notifications (the existing terminal workspace
hints) point the reaper at the affected folder, and a kept folder is
checked again only at its own deadline (end of a grace period or
terminal delay, or one hour after a busy lock or live consumer). When
the tracked state is full, the reaper schedules another walk instead of
dropping work. With nothing pending, it registers nothing.
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/isolated-task-directory-retention-config.test.ts
src/__tests__/heartbeat-task-directory-restore.test.ts` (embedded
Postgres): 48 tests pass. They cover:
- a closed issue's task directory is archived and removed after the
delay by the reaper, and reopen restores files, modes and symbolic links
exactly;
- the same through reopen for a `shared_workspace` row, and through a
real heartbeat run (the run restores the files and runs in them);
- the directory stays before the delay (and the deadline is the end of
the delay), while the issue is open, and when retention is off;
- the directory stays while each consumer check fails: live run, live
runtime service, reopen fence, unsettled finalization, live lease,
unresolved recovery action, issue bound elsewhere, live sibling
workspace in the directory;
- no removal without restore points, with a non-durable key, before
`notBefore`, or over the limits;
- dry run changes nothing; a change after the capture rolls back; an
interrupted removal finishes on the next sweep; an expired restore point
reopens empty with a warning; a diverged directory fails the reopen and
stays untouched;
- folder sweep: an orphan is removed after the grace period, kept when
recently modified, in dry run, when a live row points into it, and when
its issue is open; an archived unbound row's folder is removed; an
archived bound row's folder is saved, removed and restored on reopen;
the walk is bounded and continues from its cursor;
- review follow-ups: a file name with a backslash saves and restores;
after the startup walk, a workspace archived later is found through its
change hint; an orphan used by a live run is retried; skipped rows do
not starve later rows of the per-sweep budget; overflowing tracked
deadlines schedule another walk instead of being dropped; a closing
child sends the reaper back to its archived parent's folder; a pending
lease cleanup on a closed workspace keeps an orphan; a deep path at the
platform path limit restores; an expiry recorded during removal survives
and the archive and removal are recorded as activity; a task closed
beyond retention keeps its new restore point for the full period;
- symbolic links: a task directory that is a link, an orphan that is a
link, and a link inside a directory are never followed; the link targets
stay untouched.
- Mutation checks. Each change below made at least one of the tests
above fail, and the test passed again after revert:
- drop the run-path reopen condition; drop the skipped-row cache; drop
hint tracking; reject backslashes again; drop the live-consumer retry
for folders; evict overflowing folder deadlines; skip ancestor hints;
ignore workspace-attached leases for orphans; stage restores under the
old longer name; overwrite the whole metadata on the final write; retain
task directory restore points from the tree close only; drop the removal
activity; force `closedSiblingsAllowed` off; make live siblings not
block; skip the digest recheck before rename; skip the issue binding
check; remove both symbolic link refusals (task directory); remove the
orphan symbolic link refusal; ignore live rows in the folder walk; skip
the orphan modification grace; ignore dry run for orphans; skip the
restore on reopen; skip the terminal delay; skip the restore points
requirement; skip the orphan open-issue check.
- Removing only one of the two task directory symbolic link checks did
not fail a test, because the other check still refuses the link.
Removing both fails the test.
- Related suites: `pnpm exec vitest run execution-workspaces
isolated-task-directory workspace-runtime git-restore-point
git-workspace-expiration terminal-workspace workspace-reaper
heartbeat-git-restore heartbeat-task-directory
isolated-repository-rehome nested-clone-migration environment-selection`
in `server/`: 34 files, 749 tests pass, 4 skipped. Two
`workspace-runtime.test.ts` tests fail (`does not diagnose a missing
remote branch as a rejected credential`, `rejects explicit main on a
master-only remote ...`). They fail the same way on unchanged `master`
on the local machine, so they are not caused by this change.
- `pnpm run typecheck` in `server/`: passes.
- Not run: the full server suite, a live instance rollout, and a check
that the host disk usage drops after removal.
## Risks
- The feature is off by default, and it starts as a dry run when
enabled. It needs restore points to remove anything. Rollback: set
`PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED=false`. Saved restore points
and restore on reopen keep working.
- Orphan folders are removed without a restore point. This applies only
when no workspace row is bound to the issue at that path. If such an
issue is reopened later without another workspace, its next run starts
in an empty directory. The grace period (default 7 days of a closed tree
and no changes) and the dry run limit this risk.
- A directory over the restore point limits stays on disk (logged with
its reason). Special files keep a directory too.
- `node_modules` next to a `package.json` is not saved. A reopened task
must install its packages again.
- The restore point lives as long as git restore points (90 days by
default), counted from the later of the tree close and the removal. Only
task directory restore points use the removal time; git restore points
keep their retention. After that, reopen starts with an empty directory
and posts a warning comment.
- Reopen now accepts a closed `shared_workspace` row, but only when it
holds task directory state written by this feature.
- No schema migration. The ledger table and its columns are unchanged.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees and saves
restore points for unsaved work
> - It treated a workspace as closed only when the source issue and
every descendant were done or cancelled
> - A routine that closes a parent and leaves children open for days
therefore kept every parent worktree, although the children work in
their own workspaces
> - The idle path did not help, because the children are updated every
day
> - This pull request counts only the issues that use the workspace when
it decides that a closed workspace can expire
> - The benefit is that closed parent worktrees are removed while their
children work elsewhere, and a child that later binds to the workspace
gets it back from the restore point
## Linked Issues or Issue Description
Refs #17271. Bug report described in-PR (bug report template).
**What happened?**
On one instance, an hourly routine creates a parent issue, works it in
its own worktree, marks the parent done, and leaves child issues open
(`blocked`, `in_review` or `in_progress`) for days. The children work in
their own workspaces. They are not bound to the parent's workspace. Git
cache expiration kept every parent worktree (about 300 MB each), because
the closed-tree check and the workspace reaper required all descendants
to be closed. In one project, 18 of 19 long-closed parent worktrees were
kept only by open children that did not use them. The idle path never
ran, because the children were updated daily.
**Expected behavior**
A closed issue's worktree expires while descendants are open, as long as
no open descendant uses that workspace.
**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Close an issue that has an isolated git worktree, and leave a child
issue open that works in its own workspace (or has no workspace
binding).
3. Wait past the terminal delay and run the workspace reaper: the parent
worktree stays, and the reaper counts it as `skippedNonTerminalTree`.
**Paperclip version or commit:** `master` at 410e5c6eed.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- `isGitWorkspaceExpirationEligible` (closed path, not the idle path):
the "tree closed for one day" check now covers only the issues in the
tree that use the workspace. An issue uses the workspace when it is the
source issue, when its `execution_workspace_id` is the workspace, or
when its current `workspace_selection` or its
`workspace_pending_selection` selects it as the `existing` workspace.
Inheritance from a parent (`reuse_existing`) and `existing` selections
are both stored as `execution_workspace_id`, and heartbeat workspace
resolution reuses an existing workspace only through that binding or a
run's persisted native binding. An open descendant without one of these
references gets a new workspace on its next run, so it no longer keeps
the parent's worktree.
- All other checks stay the same: any open issue bound to the workspace,
live or unsettled runs of any issue in the tree or of the workspace
(including a run whose context names the workspace), finalizations,
leases, recovery actions, runtime services, shared cwd and the reopen
fence.
- Workspace reaper: when the source issue is closed, git cache
expiration applies, and no reopen is pending, the reaper routes the
workspace by the same rule. It takes the restore point due time from the
source issue and the descendants that use the workspace, also after
every descendant is closed. A recently closed child that used the
workspace still holds it for the terminal delay. The compaction delay
and the archive cooldown still use the whole tree when it is closed.
- Workspace reaper: compaction and the normal archive path keep their
behavior. While any descendant is open, the reaper does not compact the
worktree, and the expiration branch handles the workspace and continues.
The archive path still requires a closed subtree. With expiration off,
or a pending reopen, routing is unchanged.
- New exported helper `listIssuesUsingGitWorkspace` shares the SQL "uses
the workspace" predicate between eligibility and the reaper.
- Docs: `doc/recoverable-disk-cleanup.md` and the
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` entry in
`doc/DEVELOPING.md`.
- The existing `preserves live descendants and pending reopening` case
now binds its open child to the workspace. An unbound open child no
longer keeps the worktree by design.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/heartbeat-git-restore-point-resume.test.ts` in `server/`:
all pass.
- `pnpm exec vitest run src/__tests__/git-workspace-expiration
src/__tests__/git-restore-point src/__tests__/execution-workspace
src/__tests__/heartbeat-git-restore-point-resume
src/services/heartbeat/environment-selection
src/__tests__/isolated-repository` in `server/`: 15 of 17 files passed
in one parallel run. The other failures were the updated descendant case
(now fixed) and 45-second timeouts in restore point tests while 17
embedded Postgres files ran at the same time. Both files pass when run
on their own.
- `pnpm run typecheck` in `server/`: passes.
- New embedded Postgres cases with real git repositories:
- A closed parent with open `blocked`, `in_review`, `in_progress` and
`todo` children and a grandchild in their own workspaces (or with none):
the worktree expires with a restore point and the reopen restores it
exactly.
- The reaper archives it, timed from the issues that use the workspace.
A child closed one minute ago with its own workspace does not delay it.
- A recently closed child bound to the workspace holds it until its
terminal delay passes.
- When every child is closed and the most recent one used its own
workspace, the reaper does not restart the restore point delay.
- The worktree is kept when an open child is bound to it, when an open
grandchild under a child with its own workspace is bound to it, when a
child created later with `issueService.create` inherits it with
`reuse_existing`, when an open child selects it through
`workspace_selection` or `workspace_pending_selection`, when a child run
that references it is live, and when the parent was closed inside the
delay.
- The reaper never compacts the worktree while children are open, with
expiration on or off. A pending reopen keeps its current fence handling.
- After expiry, an open child that binds to the workspace reopens it
with the saved commits, index, working tree and untracked files.
- Heartbeat: a queued run for an open child that binds to the expired
workspace of a closed parent restores the workspace and runs the agent
in it with the saved state.
- Mutation checks (each run against the two restore point test files):
- Closed-tree check restored to the whole tree: 7 tests fail.
- `execution_workspace_id` removed from the rule: 3 fail.
- `workspace_selection` removed from the rule: 1 fails.
- `workspace_pending_selection` removed from the rule: 1 fails.
- Reaper `latestTerminalAt` from the whole tree: 3 fail.
- Reaper routing by the whole tree: 2 fail.
- Reaper restore point due time from the whole tree once it is closed: 1
fails.
- Pending-reopen gate removed: 1 fails.
- Compaction gate (`!treeOpen`) removed: no test fails. Compaction's own
guard under the lifecycle lock still rejects open trees, so the gate
only skips work that cannot succeed.
- Not run: the full server suite and the UI suites. CI covers them.
## Risks
- Closed parent worktrees now expire while their children are open. A
child that later binds to the workspace depends on the restore point and
on the run-side reopen from #17271. The new tests cover this path. When
restore points are off, only exactly reproducible worktrees expire.
- An open child with no workspace binding no longer keeps the parent's
worktree. Heartbeat resolution never reuses a workspace without a
binding, so this does not change where the child runs.
- The rule is conservative. A workspace selected only by an issue's
current or pending selection is kept, and live runs of any issue in the
tree still block.
- The reaper runs one extra indexed query for each closed workspace when
git cache expiration applies. No migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each isolated task workspace is a Git worktree on the server disk,
and the terminal workspace reaper and git cache expiration keep that
disk usage bounded
> - For some days, workspace preparation gave each isolated
`git_worktree` workspace its own full clone under
`isolated-repository-sources/`, with the worktree inside it. #16963
removed that creation path, but the existing clones stay, and restoring
an expired workspace still recreates its private clone
> - On one instance, 30 such clones use about 23 GB: 12.1 GB of `.git`
data, 5.2 GB of unused default-branch checkouts and 5.75 GB of real
worktrees. 28 of them back open workspaces, and the disk is nearly full
> - This pull request moves each such open workspace onto the project's
shared managed clone with its exact state, deletes the private clone,
and makes restore use the shared clone
> - The benefit is that disk use drops to one shared clone per project
repository, with no lost work and no change for workspaces that cannot
move safely
## Linked Issues or Issue Description
Refs #16963. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 30 per-issue clones under
`isolated-repository-sources/<companyId>/<issueId>/checkout` use about
23 GB. 28 of them back open workspaces. No process moves these
workspaces to the shared project clone that new workspaces use, and git
cache expiration does not remove a clone while its workspace is open.
Reopening an expired workspace whose recipe has `sourceKind:
"isolated_remote"` clones the full repository again. Separately, a
restore point with a non-default file mode cannot be restored: the
reference read back from jsonb has reordered `modes` keys, so
`loadGitRestorePoint` fails with `restore_point_manifest_mismatch`.
**Expected behavior**
An open workspace with a private clone moves to the shared project clone
with the same branch, HEAD, index, worktree, untracked and ignored
files, and file modes. The private clone is then deleted. Reopening an
expired workspace uses the shared clone when one exists. A restore point
with non-default file modes restores.
**Steps to reproduce**
1. Create an isolated `git_worktree` workspace with
`materializeIsolatedRemoteWorkspaceSource` and record
`isolatedRemoteSource` metadata, as preparation did before #16963.
2. Keep its issue open. The clone stays, because no sweep moves or
removes it.
3. Expire the workspace with a restore point that has a file with mode
`0600`, then reopen it. The private clone is created again, and the
restore fails with `restore_point_manifest_mismatch`.
**Paperclip version or commit:** `master` at ec2e0d6407.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- New `server/src/services/isolated-repository-rehome.ts` with
`rehomeIsolatedRepositoryWorkspace`. It holds the isolated source lock,
the `execution_workspace_lifecycle:<id>` advisory lock and
expiration-style Git lock files (`index.lock`, `HEAD.lock` and the
branch ref lock in the clone).
- Eligibility uses the safety checks of git cache expiration that matter
while a workspace stays open: no live or unfinished relevant runs, no
unsettled native finalizations, no unreleased environment leases, no
running runtime services, no pending reopen, and no other workspace on
the same path. It does not require a closed or idle issue tree. The
check runs again inside the update transaction.
- The target base is the project's existing shared managed clone for the
same `repoUrl` (the paths `ensureManagedProjectWorkspace` uses, verified
with the same origin and storage checks as restoration).
- The move captures a restore point of the old worktree. The upstream
boundary is commits that both repositories have, so the bundle is small.
It imports the bundle into the shared clone, creates
`refs/heads/<branch>` or fast-forwards it from an ancestor, and adds the
worktree at `<base>/.paperclip/worktrees/<branch>`. Then it applies the
restore point. It copies the `HEAD` reflog and `ORIG_HEAD`, copies the
branch's upstream config when it creates the branch, and moves
regenerable directories, untracked `.env`/`.env.*` files,
`.claude/settings.local.json` and nested `.paperclip-repositories/*`
checkouts by rename. Standalone nested clones move as is. Nested
checkouts that are worktrees of their own shared clone (#17382) move
too, followed by `git worktree repair` in that clone so the registration
follows the new path, and their `paperclip/<hash>/` branch is renamed to
the new path's prefix so ownership checks still pass. Entries inside a
moved nested checkout (for example its `node_modules`) move with it.
Rollback and interrupted-move recovery repair them again after moving
them back, and verification checks that each one still resolves to the
same shared clone.
- Verification: the restore point snapshot (HEAD, index, worktree,
ignored files, modes) of the new worktree must equal the old one before
and after the move. The reflog and `ORIG_HEAD` must match, and every
moved item must exist. Only then does one transaction set
`cwd`/`providerRef`, drop `isolatedRemoteSource` and
`gitCacheExpiration`, rewrite a saved active restoration recipe to
`managed_project`, and record `isolatedRepositoryRehome: { at, from, to,
cleanedAt }`. A failure before that removes the new worktree, moves
items back, restores the branch, and leaves the old worktree untouched.
- Safety of the checkout in the shared clone: the move skips with
`unsupported_base_git_config` when the shared clone's local config sets
filter drivers, includes, attribute files, worktree config or command
settings, or when it has `info/attributes`. It checks before planning
and again right before `git worktree add`.
- Crash safety before the update: the move records
`isolatedRepositoryRehomePending` (base, target, branch, head, previous
branch tip, planned moves) before the first change to the shared clone.
The next attempt undoes an interrupted move under the same locks: it
moves items back, removes the new worktree only when it is proven to be
this move's (a linked worktree of the shared clone on the issue branch
that no workspace row references), and restores the branch. A live
failure removes the target only after this attempt created it. When an
item exists in both places, rollback stops with
`rehome_rollback_conflict` and keeps the target and the record, so no
copy is lost.
- Cleanup after the update: remove the old worktree (rename to trash,
then delete, then `worktree prune`). Delete the clone (rename to trash,
then delete) only when no other workspace, project workspace or running
service uses it, it has no other worktree, its checkout has no changes
or non-regenerable files, its refs are only branches, tags and
remote-tracking refs, and every local commit is on a remote-tracking ref
or present in the shared clone. Cleanup reads at most 5,000 refs, reflog
entries and unique commits (`clone_history_budget`). `cleanedAt` records
completion. A crash after the update leaves a pending record that the
next pass finishes, also after the workspace is archived: the reaper
scan includes archived rows with unfinished re-home cleanup when the
feature is on.
- Reaper integration in `sweepTerminalWorkspaces`: behind
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED` (default `false`) and
`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN`, only when git cache
expiration is enabled. It logs `event:
"execution_workspace.isolated_source_rehome"` with outcome, reason,
candidate bytes and reclaimed bytes. A skipped workspace gets a retry
deadline one hour later in the reaper's existing deadline set. No new
timer.
- Restore: `reopenClosedIsolatedExecutionWorkspaceForIssue` calls
`rehomeExpiredIsolatedRestoration` first. For an expired workspace with
an `isolated_remote` recipe, it uses the shared clone when one exists
and the branch there is absent or at the saved commit. It keeps the old
behavior for recipes with worktree-backed nested checkouts on branches,
because those branch names are bound to the old path. It also requires
that the shared clone has every base commit (prerequisite) of the saved
restore point bundle. It rewrites the row and recipe to the shared clone
and the standard worktree path, then the existing restore runs
(`ensureGitRestorePointCommit`, `ensureSavedGitCommit`, apply). Without
a shared clone, the old behavior stays.
- Fix: restore point `modes` comparison in
`git-restore-point-storage.ts` compares by path and mode, so jsonb key
order does not matter.
- Small exports: `allocatedBytes`, `expirationScope`,
`managedSourcePaths`, `verifiedGitCacheSource`.
- Docs: new "Isolated repository sources" section in
`doc/recoverable-disk-cleanup.md`, new settings in `doc/DEVELOPING.md`.
Deviations from the requested design:
- The sweep does not create a missing shared clone. When none exists, it
skips with `no_shared_base`, because a new network clone in a cleanup
path can use more disk. Restore follows the same rule and keeps the old
behavior.
- History moves through the restore point bundle, not a separate fetch
from the clone. The bundle contains HEAD and every reflog/`ORIG_HEAD`
commit that the shared clone does not have. The move then checks that
all of them are present.
- The standard location is `<base>/.paperclip/worktrees/<branch>`. A
worktree outside the clone's standard location is skipped
(`nonstandard_worktree_location`).
- Lock files use a token derived from the workspace id, so a crashed
attempt's locks are recovered without a database marker.
- A clone that another open workspace still uses is kept and marked
`cloneRetained: "base_has_consumers"`. That workspace's own move removes
it later.
Skip reasons: `live_or_unresolved_consumer`,
`unsupported_base_git_config`, `rehome_rollback_conflict`,
`nested_branch_conflict`, `invalid_rehome_record`,
`interrupted_rehome_pending` (dry run only), `lifecycle_busy`,
`source_busy`, `workspace_changed`, `no_shared_base`,
`shared_base_repo_mismatch`, `missing_worktree`,
`nonstandard_worktree_location`, `source_identity_changed`,
`branch_identity_changed`, `unproven_worktree_admin`,
`unsupported_worktree_git_state`, `unique_worktree_refs`,
`unknown_worktree_git_logs`, `unsupported_local_git_state`,
`unique_git_refs`, `unsupported_index_flags`, `unsupported_branch_name`,
`invalid_restoration_recipe`, `unsupported_restoration_recipe`,
`target_worktree_exists`, `base_branch_checked_out`,
`base_branch_diverged`, `untransferable_ignored_files`,
`untransferable_managed_checkouts`, `worktree_git_busy`, `head_changed`,
`history_not_transferred`, `transfer_conflict`, `snapshot_mismatch`,
`history_mismatch`, `transfer_incomplete`, and restore point reasons
such as `untracked_nested_repository`, `submodules_unsupported`,
`filtered_paths`, `restore_point_too_large`. Clone kept reasons:
`base_has_consumers`, `base_has_live_services`, `base_has_worktrees`,
`clone_checkout_has_files`, `unsupported_clone_refs`,
`unique_clone_history`, `clone_history_budget`.
## Verification
- `pnpm exec vitest run
src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 18
passed. It uses real Git repositories and embedded Postgres. It covers:
- a move with an unpushed commit, staged and unstaged edits, a deletion,
an untracked file with mode `0600`, an ignored file, `node_modules`,
`.env`, `.claude/settings.local.json` and a nested
`.paperclip-repositories` checkout. Snapshots, branch, HEAD, `git
status` and reflog match, the row points to the new path without
`isolatedRemoteSource`, and the old worktree and clone are gone;
- skips with nothing changed on disk or in the database: live run,
branch in the shared clone at an unrelated commit, no shared clone, an
untracked nested repository, and a concurrent change between capture and
verify (hook). The new worktree is removed and the old one keeps its
moved items;
- the clone is kept while another workspace references it;
- a worktree-backed nested checkout moves with the task worktree, stays
registered in its shared clone at the new path (not prunable), and keeps
its changes, with its branch renamed to the new path's prefix. On
rollback it returns to the old path with its registration repaired and
its branch name restored. A standalone nested checkout moves together
with its own ignored `node_modules`;
- a file recreated in the old worktree before verification stops the
rollback and keeps both copies, then the move completes once the
conflict is gone;
- a hostile `filter.*.smudge` in the shared clone's config makes the
move skip without running it;
- a clone with more than 5,000 refs is kept, and is deleted once the
refs are gone;
- a move interrupted before the database update is reported (not undone)
by a dry run, then undone and completed on the next attempt;
- a target that another party creates right before `git worktree add` is
never deleted;
- a crash after the database update; after the row is archived, a later
reaper sweep removes the old worktree and clone;
- restore keeps the private-clone fallback when the shared clone lacks
the restore point's base commits;
- disabled by default (both flags needed), dry run changes nothing, and
the retry deadline is registered;
- reopening an expired `isolated_remote` workspace rebuilds it in the
shared clone with the restore point applied and no clone created.
- After the round 4 fixes, the related suites
(`git-workspace-expiration`, `git-restore-point-expiration`,
`execution-workspace-reopen`, `execution-workspaces-service`,
`heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`,
`isolated-remote-workspace-source`) with the new suite: 326 passed.
- After rebasing on #17382, the related suites (adding
`heartbeat-project-repositories`) with the new suite: 374 passed.
- After the first review fixes, the related suites
(`git-workspace-expiration`, `git-restore-point`,
`git-restore-point-expiration`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`) with the new suite: 301 passed.
- Related suites in `server/` (first commit): `pnpm exec vitest run` on
`git-workspace-expiration`, `git-workspace-expiration-eligibility`,
`git-restore-point`, `git-restore-point-expiration`,
`git-restore-point-expiration-config`, `git-restore-point-ledger`,
`git-restore-point-ledger-config`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`,
`isolated-remote-workspace-source-lock`, `git-workspace-base-storage`,
`git-reproducible-managed-checkouts`, `git-expiration-auth-fallback` and
`workspace-runtime`: 550 passed, 2 failed. The 2 failures are in
`workspace-runtime.test.ts` (base ref resolution) and also fail on
unmodified `master` on the same machine.
- `pnpm run typecheck` in `server/` passes (after `pnpm -r --filter
"./packages/**" build`).
- Mutation checks. Each reverted guard fails one test in the new suite:
- eligibility always true: "keeps a workspace with a live run" fails;
- snapshot verification removed: "removes the new worktree and keeps the
old one when the workspace changes during the move" fails;
- clone reference check removed: "keeps the clone while another
workspace still references it" fails;
- restore re-home removed from reopen: "restores an expired isolated
workspace into the shared project clone" fails;
- old `JSON.stringify` modes comparison: the same restore test fails;
- interrupted-move recovery removed: the interrupted-move test fails;
- target existence check before `git worktree add` replaced by assumed
ownership: the foreign-target test fails;
- bundle prerequisite check removed: the stale shared clone restore test
fails;
- archived-row cleanup scan removed: the crash test fails;
- dry run check before interrupted-move recovery removed: the
interrupted-move test fails;
- `git worktree repair` after the forward move removed: both nested
worktree tests fail. The repair after moving back removed: the rollback
test fails;
- nested branch rename forward or back removed: the matching nested test
fails;
- rollback conflict check removed: the reappearing-file test fails;
- covered-entry filter removed: the full re-home test fails;
- shared clone config check removed: the hostile filter test fails;
- ref budget removed: the many-ref test fails.
- Not run: the full server suite, and the move on a live instance.
## Risks
- Workspace paths change for re-homed workspaces. Agents that resume an
old session can remember the old path. The run's working directory comes
from the updated row, but text in a resumed conversation can still name
the old path.
- The feature is off by default. It runs only with
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` and git cache expiration
enabled. Dry run is available. The restore change is not behind the
flag: reopening an expired private-clone workspace now uses the shared
clone when one exists.
- A run can start between the last eligibility check and the database
update. The update transaction checks eligibility again, and the old
worktree is deleted only after another check. This is the same window
that git cache expiration accepts.
- Imported restore point objects stay in the shared clone as unreachable
objects after a failed attempt until Git garbage collection removes
them. Base expiration keeps a base with unreachable objects, so this
only delays base expiration.
- A workspace that cannot move (for example `no_shared_base`) is retried
each hour while the feature is on. Each retry runs a few Git commands
for that workspace.
- No migration. No schema change.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, Git, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can attach extra repositories, and every task worktree
gets a managed checkout of each one under
`.paperclip-repositories/<name>`
> - Each of these nested checkouts is a full clone. The clone uses
`--dissociate`, so no objects are shared between tasks
> - On one instance, 44 task worktrees held nested checkouts of the same
repository: 24 GB of 27 GB of worktrees. Each new task added about 0.5
GB, and the disk filled twice in one day
> - Main task worktrees already avoid this, because they are git
worktrees of one managed project clone
> - This pull request makes nested checkouts git worktrees of one shared
clone per project and repository, and adapts expiration, restore points,
restore and cleanup to them without weaker safety checks
> - The benefit is that repository objects are stored once, and a new
task adds only its working files
## Linked Issues or Issue Description
No public issue exists. This is the in-PR bug report.
**What happened?**
Every task in a project with extra repositories got a full, independent
clone of each extra repository under `.paperclip-repositories/`. On one
instance this used 24 GB of 27 GB of task worktrees (16 GB of it `.git`
data) and filled the disk twice in one day.
**Expected behavior**
Nested checkouts of the same repository share one object store, like
main task worktrees share the managed project clone. Disk use per new
task is only its working files.
**Steps to reproduce**
1. Create a project with a primary repository and a second repository
that has a repository URL and no local folder.
2. Run several tasks in isolated git worktrees in that project.
3. Look at `<task worktree>/.paperclip-repositories/<name>/.git`: each
one is a full directory with its own copy of every object.
**Paperclip version or commit:** `master` at ff835a3fc6.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
Related: #15662 (SSH restores skip nested worktrees) touches a different
path; this PR does not change adapter sync.
## What Changed
- New `server/src/services/nested-repository-worktrees.ts`. It finds the
shared clone (the managed project checkout for the repository URL, from
`ensureManagedProjectWorkspace`), adds nested worktrees, restores them,
removes their registrations, and serializes all of these operations and
fetches per shared clone with an in-process lock.
- `prepareProjectRepositoryWorkspaces` takes a `sharedBase` (company and
project). For a repository without a configured local folder, it fetches
the shared clone and adds a worktree at the same
`.paperclip-repositories/<name>-<key>` path. Each worktree gets its own
branch `paperclip/<id>/<branch>` that tracks `origin/<branch>` (`<id>`
derives from the worktree path), or a detached HEAD for a tag or commit
ref. An existing worktree must belong to the project's shared clone.
Configured local-folder copies and existing standalone clones are
unchanged. Moving a worktree to `detached-repositories` repairs its
registration. Heartbeat preparation and fresh-start reopening pass
`sharedBase`.
- `assertReproducibleManagedCheckout` and `provenManagedCheckoutHistory`
accept worktree-backed checkouts. They verify the worktree link, its
admin directory, the shared clone identity and origin, and that HEAD is
detached or on a branch under the worktree's own prefix. History checks
scope to HEAD, the per-worktree HEAD reflog, ORIG_HEAD and the owned
branches with their reflogs. Other tasks' branches are not inspected.
Stashes, unsupported refs and custom config in the shared clone still
block. The upstream proof reads objects from the common git directory.
- Restoration contents record `worktree: { base, admin }` for a
worktree-backed checkout. Validation checks its shape and the owned
branch name, and every use checks the base against the project's
expected clone paths.
- Expiration locks for a worktree-backed checkout sit in the shared
clone (`.git/worktrees/<admin>/index.lock`, `HEAD.lock`, and the owned
branch ref lock). `gitCacheExpirationLockLocations` is now async.
- `finishGitCacheTrash` removes the expired worktree's registration and
deletes its owned branch only at the recorded commit.
- `restoreGitWorkspaceContents` recreates a worktree-backed checkout
from the shared clone, and clones the shared clone again if it is gone.
It imports the restore point into the shared clone, adds the worktree on
the saved branch and commit, writes reflog-only commits back into the
new HEAD reflog, and applies the restore point.
- `cleanupExecutionWorkspaceArtifacts` (terminal reaper archival, close
and delete) collects the task's nested worktrees first. After the task
directory is gone, it removes their registrations and deletes owned
branches only when every commit is on a remote-tracking ref. An unpushed
branch stays in the shared clone.
- Review follow-ups. Owned branches always track origin, including names
with dots, and the config check accepts their tracking keys. A ref held
by a retained detached checkout gets `paperclip/<id>/r<n>/<branch>`.
Restore derives the upstream from the saved source ref. Each new
worktree fetches with `--prune`, refreshes `origin/HEAD` when no ref is
configured, and strips `refs/heads/` from branch refs. Git commands on
the shared clone run with hooks disabled. The worktree `gitdir` pointer
is read through a bounded, non-following, non-blocking handle, and more
than 64 owned branches stop the proof before reflog reads.
- Docs: new "Nested project repository worktrees" section in
`doc/recoverable-disk-cleanup.md`, and updates in `doc/DEVELOPING.md`.
## Verification
- `pnpm -r --filter "./packages/**" build`, then `pnpm run typecheck` in
`server/`: passes.
- `pnpm exec vitest run` in `server/` for `git-workspace-expiration`,
`git-reproducible-managed-checkouts`, `git-workspace-contents`,
`git-restore-point*`, `heartbeat-project-repositories`,
`execution-workspaces-service`, `execution-workspaces-routes`,
`execution-workspace-compaction`, `workspace-compaction`,
`heartbeat-git-restore-point-resume`, `managed-repository-checkout`,
`task-workspace-repositories*`, `native-workspace-sync*`,
`heartbeat-managed-clone-credentials`,
`services/heartbeat/workspace-preparation`,
`project-repositories-persistence`, `git-workspace-base-storage`,
`agent-directory-probe`, `workspace-runtime`,
`workspace-preparation-history` and `native-completion-feedback`: 866
passed, 2 failed. The 2 failures are in `workspace-runtime.test.ts`
(`rejects explicit main on a master-only remote...` and `does not
diagnose a missing remote branch...`). They fail the same way on
unmodified `master` on this machine.
- New tests (embedded Postgres, real git repositories) in
`git-workspace-expiration.test.ts`, section "nested checkouts as
worktrees of a shared base":
- Two tasks get worktrees of one shared clone. `--git-common-dir` points
to it, each task has a distinct `paperclip/<id>/main` branch, upstream
is `origin/main` and the remote is the repository URL.
- Concurrent preparation of three tasks succeeds, and preparation waits
while the shared clone lock is held.
- A legacy standalone clone is still adopted, and no shared clone is
created for it.
- Expiring a clean nested worktree removes only its registration and
branch. The sibling worktree and branch stay. Reopening recreates the
worktree from the shared clone on the same branch and commit, with its
local settings.
- Reopening clones a deleted shared clone again.
- Unpushed commits and uncommitted files block plain expiration, are
saved in a restore point and are restored exactly.
- Reflog-only history is saved and written back into the restored HEAD
reflog, and it survives `gc --prune=now` of the shared clone.
- Stashes, a checked-out branch outside the worktree's prefix, and
unpushed owned branches or owned reflog entries outside HEAD history
block expiration.
- Another task's unpushed branch does not block, and it stays after
expiration.
- A worktree of a foreign clone at the nested path is rejected by
preparation and by the prover.
- Expiration locks in the shared clone are recovered after a process
exit.
- Deleting a task removes only its nested registrations, keeps its
unpushed branch and deletes the sibling's pushed branch when the sibling
is deleted.
- Follow-up tests: dotted branch tracking, a ref switched away and back
with a retained checkout, a changed remote default branch, a renumbered
branch restored next to a same-named remote branch, a fully qualified
branch ref after the remote advances, a branch deleted upstream, planted
`post-checkout` and `reference-transaction` hooks, more than 64 owned
branches, and a FIFO, oversized or symlinked `gitdir` pointer. Each fix
was mutation-checked.
- `heartbeat-project-repositories.test.ts`: a repository-URL-only
sibling is now a worktree of the shared clone on an owned branch, and
its detached copy is still a valid checkout after its repository leaves
the project. In `workspace-preparation.test.ts`, the publication guard
now runs before any worktree exists.
- Mutation checks. I reverted one guard at a time and confirmed that a
test fails: the owned-HEAD check, the owned-branch check in restore
point mode, the owned-reflog check, the shared-clone identity check, the
pushed-only branch deletion on cleanup, the cleanup release hook, the
registration removal on expiration, the reflog-only history write-back
on restore, and the shared clone lock. The owned-branch-tip check alone
is also covered by the owned-reflog check, so removing only one of the
two keeps the tests green.
- Not run: the full server test suite, the UI and e2e suites, and a live
instance migration.
## Risks
- Behavior change for agents: a nested checkout of a URL-only repository
is now on `paperclip/<id>/<branch>`, not `<branch>`. `git pull` and `git
status` work as before. A plain `git push` now fails because the branch
names differ. Agents must push with an explicit destination, for example
`git push origin HEAD:<branch>`. Branches in the shared clone are
visible to all tasks of the project, like main task worktree branches.
- Agent-created branches outside the worktree's prefix, stashes, and
local config live in the shared clone. Stashes and custom config block
expiration of every worktree of that clone (fail-closed). Other branches
stay in the shared clone after a task is deleted.
- The shared clone is the managed project checkout for that repository
URL. Fetches into it update its remote-tracking refs. A concurrent
restore point capture of a sibling can then roll back and retry on the
next sweep.
- The base lock is per process. Across processes, git's own ref and
index locks apply.
- No migration in this PR. Existing standalone nested clones keep their
disk use until their tasks are removed or expired. A safe migration
helper is a follow-up. Task repositories (`task-repo-*`) and configured
local-folder copies stay standalone clones.
- Restoration recipes now can carry `contents.checkouts[].worktree`. An
older server rejects such a recipe as invalid (fail-closed) and does not
restore without it.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a finalization row, and tasks can keep recovery
actions while work is stuck
> - Git cache expiration and native harness backup expiration read these
rows to decide if closed work is still live
> - Commit paths never cleared
`native_run_finalizations.recovery_state`, so committed rows kept
`resuming_session` forever
> - Recovery actions on done or cancelled tasks also stayed active when
the close did not go through the issue routes
> - So workspaces and backups of tasks that closed weeks ago stayed live
and used disk space
> - This pull request clears the stale state at commit, backfills old
rows, resolves stale recovery actions on closed tasks, and lets the two
expiration jobs ignore settled state on closed trees
> - The benefit is that closed work expires on schedule, while live
leases, unsettled phases, unverified owners, and in-flight system
recovery still keep it
## Linked Issues or Issue Description
Refs #17248
**What happened?**
On one instance, Git cache expiration and native harness backup
expiration kept the workspaces and backups of tasks that were done for
more than 7 days. Three kinds of stale state caused this:
1. 51 `native_run_finalizations` rows had `phase = 'committed'` and
`recovery_state = 'resuming_session'`. The commit writes (status
decision commit, attention replay, result persistence, and the finalizer
claim) never cleared `recovery_state`. No reader uses `recovery_state`
on a row that has a result, except the expiration predicates.
2. Rows with `phase = 'terminal_failure'` and `recovery_state =
'blocked'` (or NULL) are permanent states. Nothing clears them, and
continuations are refused for closed tasks.
3. `issue_recovery_actions` with status `active` or `escalated` stayed
open after the source task became done or cancelled. Only the issue
routes cancel them at close. The periodic sweep
`reconcileActiveRecoveryActions` skipped every action unless
`wakePolicy.type` was a bounded owner policy or the owner was the board.
Agent-owned `resume_native_run` actions (which use `wakePolicy.kind`)
and system-owned actions were never resolved.
**Expected behavior**
When a task tree is closed for the configured time, its Git cache and
native backup expire. Only live state keeps them: a held finalization
lease, an unsettled finalization phase, an unverified native workspace
owner, or system recovery that is scheduled, running, or awaiting
evidence.
**Steps to reproduce**
1. Run a native task. Let the runner restart so that restart recovery
sets `recovery_state = 'resuming_session'` on the finalization row.
2. Let the resumed run commit its result and close the task as done.
3. Wait past the expiration window (or run the sweep with a later
clock).
4. See that the finalization row is still `committed` +
`resuming_session`, and that Git cache expiration and native backup
expiration skip the closed workspace.
**Paperclip version or commit**
`master` at 837add0caa
**Deployment mode**
Self-hosted server with native runtime enabled
**Agent adapter(s) involved**
Not adapter-specific (core bug)
## What Changed
- Clear `recovery_state` and `recovery_request_id` in every write that
persists or commits a native result: `recordCoordinatorDecision` in the
status decision committer, the attention replay in
`routePersistedNativeResultAttention`, `completeRun` in both
`NativeRunCoordinatorStore` and `PaperclipControlPlanePort`, and
`claimCoordinator` in the finalizer.
- Add data migration `0376_clear_committed_native_recovery_state.sql`
with its unchanged schema snapshot (the oldest snapshot is pruned, as
the generator does). It clears `recovery_state` and
`recovery_request_id` on rows that are `committed` with no lease owner.
`recovery_history` keeps the audit trail.
- In `reconcileActiveRecoveryActions`, resolve active and escalated
actions on done or cancelled tasks (`source_terminal`) before the wake
policy and owner filter. The existing owner-unverified guard still runs
first. Actions with system recovery in `scheduled`, `running`, or
`awaiting_evidence` are skipped.
- In `isGitWorkspaceExpirationEligible` (closed-tree mode only), a
recovery action blocks only if its cause is
`native_workspace_finalization_owner_unverified` or its system recovery
is in flight. A finalization blocks only if it has a lease owner or its
phase is not `committed` or `terminal_failure`. Idle mode keeps the old
strict checks.
- In native harness backup expiration, a settled finalization
(`committed` or `terminal_failure`, no lease owner) no longer blocks
once the session's tasks are closed for the configured days.
## Verification
- `pnpm exec vitest run` in `server/` on the touched and related files:
`git-workspace-expiration`, `git-workspace-expiration-eligibility`,
`native-harness-backup-expiration`, `native-finalization-migration`,
`native-interaction-bridge`, `native-run-finalizer-telemetry`,
`paperclip-control-plane-port`, `issue-recovery-actions`,
`native-status-arbiter-corpus`: 9 files, 321 tests passed. After the
tests moved to `native-finalization-recovery-state.test.ts`, that file,
`git-workspace-expiration` and `native-run-finalizer-telemetry` passed
again (95 tests).
- `pnpm exec vitest run src/migration-snapshot-drift.test.ts` in
`packages/db`: passed.
- `pnpm exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts`: 408 tests passed.
- `pnpm run typecheck` in `server/`: passed.
- `pnpm run check:migrations` in `packages/db`: passed.
- New tests:
- Result persistence (port and coordinator store), finalizer claim and
status decision commit (new focused file
`native-finalization-recovery-state.test.ts`), and attention replay each
clear `recovery_state` and `recovery_request_id`.
- The migration clears committed, unleased rows. It keeps leased
committed rows, `observed` rows, and `terminal_failure` rows unchanged,
and keeps `recovery_history`.
- The sweep resolves an agent-owned `resume_native_run` action and an
escalated system action on a done task, and keeps an action with
scheduled system recovery. The existing test "retains the exact native
owner hold without changing a terminal task's disposition" still covers
the owner-unverified hold.
- Git expiration eligibility: a closed tree expires with `committed` +
`resuming_session`, `terminal_failure` + `blocked`, an active agent
recovery action, or an escalated system action. It stays with a leased
finalization, an `observed` finalization, an owner-unverified hold,
in-flight system recovery, an open tree, and in idle mode.
- A full Git expiration of a closed tree with stale native state, then a
reopen to the recorded commit. The existing "preserves scheduled runs,
unresolved recovery, and live previews" test now uses an `observed` and
a leased finalization as the unresolved recovery.
- Native backup expiration expires `committed` + `resuming_session` and
`terminal_failure` + `blocked`, and keeps an `observed` or leased
finalization.
- Mutation checks. I reverted each fix on its own, saw the matching test
fail, and restored it:
- Status decision commit, finalizer claim, port `completeRun`, store
`completeRun`, attention replay: each new clearing test failed.
- Migration (`WHERE false`): the backfill test failed.
- Sweep order (wake policy filter moved back in front) and the in-flight
system recovery guard: the sweep test failed.
- Git eligibility: the closed-tree action relaxation, the finalization
relaxation, the owner-unverified guard, and idle strictness each made
the matching eligibility tests fail.
- Backup expiration (`recovery_state IS NOT NULL` restored): both new
expiry cases failed.
- Not run: the full server suite and the e2e suites. CI runs them.
## Risks
- The migration is a data-only `UPDATE` on `native_run_finalizations`.
It touches only committed rows with no lease owner and a non-null
`recovery_state`. The table is not on the known-large list, and the
migration safety check passes. `recovery_history` is not changed.
- Clearing `recovery_state` at commit and claim: the readers of this
column need `phase = 'observed'` (continuation contract), `result_id IS
NULL` (recovery service, failed-run retry), or running runs (heartbeat
recovery). A row that has a result does not match any of them. The
restart recovery summary counts will drop because they no longer count
committed rows.
- The sweep now resolves more actions on closed tasks. A done or
cancelled task cannot continue, so these actions cannot act. Unverified
owner holds and in-flight system recovery are kept.
- Expiration now removes more closed workspaces and backups. Restore
points and Git cache restoration still allow a reopen to rebuild the
workspace.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration keeps a workspace and reports
`inspection_failed` when an unexpected error interrupts its checks
> - It dropped that error, so the log never said what went wrong
> - This pull request adds a sanitized description of the error to the
expiration result, which the workspace reaper already logs
> - The benefit is that unexpected expiration failures can be diagnosed
without exposing commands or credentials
## Linked Issues or Issue Description
Refs #17248. Bug report described in-PR (bug report template).
**What happened?**
On one instance, a closed workspace was kept with `inspection_failed`.
The log entry carried only that reason, so the cause could not be
determined from the logs.
**Expected behavior**
An unexpected failure is logged with enough detail to diagnose it,
without leaking the command line or credentials.
**Steps to reproduce**
1. Break a closed issue's worktree so a git command fails, for example
by pointing its `.git` file at a missing directory.
2. Run expiration: the log says `inspection_failed` and nothing else.
**Paperclip version or commit:** `master` at 0fb152ee94.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- `GitExpirationResult.failure` is set when the skip reason is
`inspection_failed`, by all three places that record a skip (the outer
catch and the two base-inspection catches, through one `recordSkip`
helper). It contains:
- `name`;
- `code` (for example git's exit code);
- `at` (the first stack frame);
- `stderr` (up to 500 characters).
- `describeGitExpirationFailure` builds it. It never includes the error
message, because a failed git command's message contains the full
command line, which can include `http.extraHeader` authentication. In
stderr, credentials in URLs and `authorization`, `token`, `password` or
`secret` values are redacted to the end of the line.
- The workspace reaper already spreads the result into its `Git
workspace cache expiration inspected` log entry, so no logging code
changed.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "unexpected inspection failure|redacts credentials|while inspecting
the base"` (in `server/`): 4 passed.
- **Broken worktree:** a worktree whose `.git` link points at a missing
directory reports `inspection_failed` with a code, a stack frame and
git's "not a git repository" stderr, and is kept.
- **Failing command:** a real failing `git ls-remote` with credentials
in its arguments produces a description without them or the command.
- **Echoed credentials:** a stderr that echoes a credentialed URL and an
`Authorization: Bearer` header is redacted.
- **Base inspection:** an unreadable directory in the base, which fails
base inspection after the worktree is removed, reports
`inspection_failed` with code `EACCES` and a stack frame. Recording only
the reason in that catch fails this case.
- **Mutation check:** removing the redaction fails the
echoed-credentials case. That case also caught an earlier version that
redacted only the first word of a header value.
- `pnpm exec vitest run git-workspace-expiration execution-workspaces
git-restore-point` (in `server/`): 258 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Log volume:** one bounded object (about 1 KB at most) per failed
inspection.
- **Redaction is pattern-based:** the message and command line, where
git places configuration values, are never logged; stderr is redacted by
pattern.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees once their
remote, or a restore point, can reproduce them
> - A nested `.paperclip-repositories` checkout's remote-tracking refs
can point at commits the remote has since dropped, such as a branch
deleted after a squash merge
> - Strict expiration rightly keeps such a checkout, because it may hold
the only copy; but restore point mode also kept it forever, because
there was nothing else to save
> - This pull request saves those commits in the checkout's restore
point, so the workspace can be reclaimed without losing them
> - The benefit is that long-lived multi-repository workspaces are
reclaimed in restore point mode
## Linked Issues or Issue Description
Refs #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, four closed workspaces on one instance were
kept with `unpushed_managed_checkout`, after a dry run had counted them
as candidates. Each nested `paperclip` checkout was clean, on `main` at
a commit that is on GitHub `main`. But it held about 1,000
remote-tracking refs, and between 28 and 41 of them pointed at branches
the remote has since deleted or force-pushed (147 to 203 commits it no
longer has).
The relaxed check passed, so the checkout needed no restore point beyond
those commits. Capture saved only `HEAD` history, so it found nothing to
save and stored no restore point. The final check before deletion then
ran the strict check, which keeps stale remote-tracking commits, and
rolled back. The dry run never reaches that final check, so it
over-reported. Together these workspaces held about 2.4 GB.
**Expected behavior**
In restore point mode, commits that exist only in remote-tracking refs
are saved, and the workspace expires.
**Steps to reproduce**
1. In a nested checkout, fetch a remote branch, so
`refs/remotes/origin/<branch>` exists.
2. Delete that branch on the remote and garbage-collect it.
3. With restore points enabled, close the issue and run expiration: the
workspace is kept with `unpushed_managed_checkout`.
**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- **Strict mode is unchanged.** An earlier revision of this pull request
ignored remote-tracking refs. Review pointed to `does not trust a stale
remote-tracking ref`, which deliberately protects those commits, so this
pull request keeps that behavior and the test is unchanged.
- **Relaxed nested check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`):
- reads all reflogs in one bounded call (`--format=%H %gD`),
deduplicates them by commit, and splits them into local and
remote-tracking entries before applying the tip budget;
- local entries still have to be covered or upstream, as before;
- remote-tracking tips and remote reflog entries that the remote no
longer has are returned as `preserve` (new `tipsMissingUpstream`,
sharing `commitsMissingUpstream` with the existing check).
- **Capture:** the proof is shared as `provenManagedCheckoutHistory`.
`storeGitRestorePoint({ resolvePreserveCommits })` calls it after taking
the history digest and before capture, then compares the digest again
after capture. A fetch before the digest is included in the preserved
commits, and one after it changes the digest and aborts capture. The
commits flow into `captureRestorePoint`. They are bundled as
history-only commits, within the existing limits (50 preserved tips,
1,000 commits, 256 MB bundle). The checkout therefore gets a restore
point, and the final check uses the relaxed check for it.
- **Final check:** `restorePointHistoryDigest` now also covers
remote-tracking refs and their reflogs, so a fetch, ref change or
reflog-only change after capture rolls expiration back.
- **Restore:** the commits are imported as objects; the stale
remote-tracking refs themselves are not recreated.
- **Docs:** `doc/recoverable-disk-cleanup.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
src/__tests__/git-reproducible-managed-checkouts.test.ts -t "stale
remote-tracking|remote-tracking refs change"` (in `server/`): 5 passed.
- The existing strict test passes unchanged.
- A clean nested checkout whose remote-tracking ref points at a deleted,
garbage-collected branch is kept strictly, expires with a restore point,
and reopens on its commit with the stale commit present.
- A remote-tracking ref created after capture rolls expiration back, and
so does a reflog-only change to one.
- 6,000 repeated reflog entries for one commit do not hit the tip
budget.
- **Mutation checks:** each of these fails a case:
- dropping the preserved commits reproduces the instance's symptom
(`unpushed_managed_checkout`);
- dropping remote-tracking refs or their reflogs from the digest;
- removing deduplication (`managed_checkout_local_tip_budget`).
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 261 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled.** Strict expiration is unchanged.
- **Retention:** saved stale commits are kept until the restore point
expires (90 days after the tree closes by default), then deleted with
it.
- **Limits:** checkouts with more than 50 stale tips, or bundles over
the limits, skip capture and stay kept, as before.
- **Spurious rollbacks:** a fetch during the short window between
capture and deletion rolls back; the next sweep retries.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 and #17257 let git cache expiration remove a closed issue
tree's worktree after saving an encrypted restore point
> - Issues that stay open but go untouched for weeks still keep their
worktrees forever, so disks keep filling
> - This pull request expires those idle worktrees the same way, and has
the next run restore its workspace automatically
> - The benefit is bounded disk use for stalled open work, with nothing
lost when work resumes
## Linked Issues or Issue Description
Refs #17248, #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, expiration only considers worktrees whose
whole issue tree has been `done` or `cancelled` for a while. A worktree
whose issue stays `todo`, `in_progress`, `blocked` or `in_review` with
no activity for weeks is never removed.
**Expected behavior**
An open issue's worktree that has been idle long enough is saved to a
restore point and removed, and the next run on that issue gets it back
exactly.
**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Leave an issue open with an isolated worktree and no activity for
weeks.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at 650581b2f0.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Expiration** (`git-workspace-expiration.ts`):
- `isGitWorkspaceExpirationEligible(..., idleCutoff)`: in idle mode the
closed-tree conditions are replaced by idle ones. No issue in the tree,
and no other issue bound to the workspace, was updated after the cutoff,
and no relevant run was created, started or finished after it. Runs of
bound issues now count as relevant runs in both modes. Every other check
stays, including no live run, recovery action, unsettled finalization,
environment lease or running service.
- `gitWorkspaceLastActivityAt` returns the latest of those timestamps,
sharing the same tree and run scope.
- `expireReconstructableGitWorkspace` takes `idleCutoff` and passes it
to all three eligibility checks.
**Reaper** (`execution-workspaces.ts`):
- For an open tree's isolated git worktree, when restore points and git
cache expiration are on and `idleMs` is set, the reaper computes the
idle deadline (last activity + `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS`).
- Before the deadline, it records the deadline, so the next sweep wakes
then. Open trees cause no recurring checks before that.
- At the deadline, it runs expiration with a restore point. A kept
workspace is retried after an hour, like terminal ones.
- An expiration interrupted after archiving (`completed: false`) is
resumed by later sweeps even while the tree is open, with a retry
scheduled until cleanup finishes.
**Runs** (`heartbeat/environment-selection.ts`):
- When a run's issue is bound to a workspace that git cache expiration
removed (archived, `workspaceRemoved`), the run reopens it as the agent
through `reopenClosedIsolatedExecutionWorkspaceForIssue`, with the
restore point applied.
- It then clears the reopen fence, since the live run now holds the
workspace, and continues with the reopened row.
- A failed restore fails the run instead of silently starting a new
workspace.
- Comments and agent updates on the issue already reopened closed
workspaces through the issue routes.
**Config:** `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS` (default `7`, `0`
turns idle expiry off). Docs in `doc/recoverable-disk-cleanup.md` and
`doc/DEVELOPING.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts -t "idle open
issues"` (in `server/`): 7 passed, with real repositories and embedded
Postgres.
- **Round trip:** an open issue's worktree, idle for 8 days with unsaved
work, is skipped without idle mode, expires with a restore point in idle
mode, and reopens exactly.
- **Kept:** issue activity 3 days ago, or a run finished 2 days ago,
keeps the worktree.
- **Interrupted:** an idle expiration interrupted after archiving is
finished by the reaper while the issue stays open, and reopens exactly.
- **Shared workspace:** a recent update on another issue bound to the
workspace keeps it, and moves the last activity time.
- **Reaper:** before the deadline the sweep skips the worktree and
schedules exactly that deadline; at the deadline it expires it with an
active restore point. With idle expiry off, nothing is scheduled or
expired.
- `pnpm exec vitest run
src/services/heartbeat/environment-selection.test.ts` (in `server/`): 14
passed. New cases: a run reopens its expired workspace as the agent,
clears the fence with the reopen's generation and binds the reopened
row; a failed restore fails the run; a workspace archived for other
reasons is left alone.
- `pnpm exec vitest run
src/__tests__/heartbeat-git-restore-point-resume.test.ts` (in
`server/`): 1 passed. Assembled heartbeat journey with no mocked
restore: a real worktree with an unpushed commit and uncommitted notes
is idle-expired, a queued run executes, the agent runs in the original
worktree path with the original workspace id, the commit and notes are
back, the workspace is no longer archived with no reopen fence left, and
the issue stays bound. Disabling the run-side reopen fails it.
- `git-restore-point-expiration-config.test.ts`: the default is 7 and
`0` turns it off.
- **Mutation checks:** each of these fails a case:
- dropping the idle deadline;
- dropping the idle cutoff;
- dropping the run activity condition;
- not failing the run on a failed restore;
- not re-reading the reopened workspace;
- not resuming a pending expiration on an open tree;
- ignoring bound issues' activity.
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents execution-workspaces
src/services/heartbeat/environment-selection
src/services/heartbeat/workspace-preparation` plus
`heartbeat-git-restore-point-resume` (in `server/`): 312 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Off unless enabled:** requires
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
`PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS=0` keeps restore points for
closed trees only.
- **First run after idle expiry is slower:** it waits for the restore,
about as long as a reopen.
- **Failed restore blocks runs:** a restore that fails closed (for
example a missing restore point) fails each run on that issue until an
operator fixes or rebinds the workspace. This is deliberate, so work is
never silently continued in an empty workspace.
- **Kept idle workspaces are rechecked hourly**, matching the existing
behavior for closed trees that cannot be expired.
- **Retention:** restore points of open trees are not deleted while the
tree stays open, as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 lets git cache expiration remove a closed issue's worktree
with unsaved work after saving an encrypted restore point
> - Workspaces with nested `.paperclip-repositories` checkouts were
still kept whenever a checkout had unpushed or uncommitted work
> - On one instance that was the second most common reason expiration
skipped a workspace
> - This pull request gives each nested checkout its own restore point
and restores it when the workspace comes back
> - The benefit is that multi-repository workspaces can be cleaned up as
safely as single-repository ones
## Linked Issues or Issue Description
Refs #17238, #17241, #17248. Bug report described in-PR (bug report
template).
**What happened?**
With restore points enabled, a workspace whose nested managed checkout
has unpushed commits or uncommitted files is still skipped with
`unpushed_managed_checkout` or `dirty_or_untracked_files`. On one
instance, `unpushed_managed_checkout` accounted for 32 of the last 111
expiration checks.
**Expected behavior**
A nested checkout's unsaved work is saved like the main worktree's, and
the workspace can expire and later be restored exactly.
**Steps to reproduce**
1. Use a project with an extra repository, so the task worktree contains
`.paperclip-repositories/<name>`.
2. Commit in that checkout without pushing, then close the issue.
3. With restore points enabled, the workspace is never expired.
**Paperclip version or commit:** `master` at 6d44b80a5e.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Relaxed checkout check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`, used only in restore point mode):
- **Allowed:** the checked-out branch's unpushed commits, plus its
uncommitted, untracked and history-only work.
- **Still blocks:**
- another local branch or tag whose tip the remote lacks and `HEAD` does
not contain (`unpushed_managed_checkout_branches`), because restore
points do not recreate other branches;
- commits in any reflog that are neither on the remote nor in the
bundled history (`HEAD`'s reflog and `ORIG_HEAD`), such as work left in
another branch's reflog;
- objects reachable from nothing;
- a `COMMIT_EDITMSG` that matches no commit in the checkout's history.
- **`COMMIT_EDITMSG`:** a nested checkout is a full clone, so its base
git storage check now accepts a `COMMIT_EDITMSG` that matches a commit
in its history, in strict and relaxed mode. It is read only if it is a
regular file of at most 64 KB, opened without following symlinks and
without blocking; anything else keeps the workspace
(`unsupported_commit_message`). The main worktree's existing
`COMMIT_EDITMSG` read uses the same bounded reader. A clean checkout
with only an unpushed commit therefore reports
`unpushed_managed_checkout`, which restore point mode can relax, instead
of `unknown_base_git_files`.
**Capture** (`git-workspace-expiration.ts`):
- **Expiration flow:** strict inspection still runs first.
`unpushed_managed_checkout` is now relaxable. The relaxed prover records
which checkouts need restore points.
- **Storage:** each checkout's restore point is stored under
`.../<generation>/checkouts/<name>/<id>.pcrp`, against a remote-verified
boundary from that checkout. It is recorded on its
`contents.checkouts[]` entry.
- **Activation:** all restore points of the generation are activated
together in the archive transaction. Any nested restore point makes the
recipe `paperclip.git-cache-restoration.v2`, so older servers reject it.
- **Schema:** migration `0374` adds `checkout_name` to
`execution_workspace_restore_points` and includes it in the
live-generation unique index.
- **Final check:** under the expiration locks, each checkout with a
restore point gets the relaxed check and must match its own snapshot.
Checkouts without one get the strict check and must still be clean.
Every restore point, the main worktree's included, records a digest of
its `HEAD` reflog and `ORIG_HEAD` at capture, and that history must be
unchanged, so a commit followed by a reset after capture rolls back.
Rollback drops every restore point reference, returns the recipe to v1,
and supersedes the stored restore points.
**Restore** (`git-workspace-contents.ts`):
- **Re-cloned checkouts:** re-cloning a checkout imports its bundle
before the saved commit is needed. After checking out the saved commit,
it applies and verifies the restore point.
- **Existing checkouts:** the same helper as the main worktree marks a
matching checkout applied, and fails closed instead of overwriting a
diverged one.
**Validation:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` validate nested restore point references
against their checkout names. Contents comparison ignores the reference.
**Docs:** `doc/recoverable-disk-cleanup.md` describes nested checkouts.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "restore points for nested checkouts"` (in `server/`): 10 passed,
with a real nested checkout and embedded Postgres. The cases:
- **Round trip:** an unpushed commit and an untracked file in the nested
checkout survive expiration and reopen. The recipe is v2, the restore
point lives on the checkout entry, and a ledger row carries the checkout
name.
- **Another branch:** unpushed work on another local branch keeps the
workspace.
- **Change after capture:** a nested change after capture rolls back,
keeps the files, returns the recipe to v1, and supersedes the restore
point.
- **Only an unpushed commit:** a clean nested checkout with only an
unpushed commit (and its `COMMIT_EDITMSG`) expires and restores.
- **Reflog-only history:** unpushed history held only in another
branch's reflog keeps the workspace.
- **History after capture:** a commit followed by a reset after capture
rolls back, in a nested checkout and in the main worktree.
- **Unsafe `COMMIT_EDITMSG`:** a named pipe, an oversized file, or a
symlink keeps the workspace without reading it unsafely.
- **Mutation checks:** each of these fails a case:
- removing the other-branch check, the per-checkout snapshot check, the
rollback cleanup, or the checkout apply;
- rejecting a saved `COMMIT_EDITMSG`;
- dropping reflog coverage;
- dropping the history digest check;
- removing the size cap.
Not covered by a dedicated case: a workspace with two nested checkouts
where only one has a restore point.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` (in `server/`): 232 passed.
- `pnpm run check:migrations` (in `packages/db`): pass.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.
## Risks
- **Off by default:** nothing changes unless
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- **Other local branches:** a nested checkout's other local branches are
not saved. Any of them with commits the remote lacks keeps the workspace
instead.
- **Migration:** adds a defaulted column and replaces one partial unique
index on a table only written when restore points are enabled.
- **Rollback:** disable the setting. Workspaces already expired with
nested restore points keep v2 recipes that require this code to restore.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes a closed issue's worktree only when the
remote can reproduce it exactly
> - Most closed worktrees hold unpushed commits or uncommitted files, so
they are kept forever and disks fill up
> - #17238 added restore points (an encrypted save of exactly what the
remote cannot reproduce) and #17241 added their lifecycle
> - This pull request lets expiration remove those worktrees after
saving a restore point, and restores them exactly when the work resumes
> - The benefit is bounded disk use for closed work without losing
anything a user could need, behind a setting that is off by default
## Linked Issues or Issue Description
Refs #17238, #17241. Bug report described in-PR (bug report template).
**What happened?**
With `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one
instance's last 111 expiration checks removed nothing:
| Skip reason | Checks |
|---|---|
| `dirty_or_untracked_files` | 39 |
| `unpushed_managed_checkout` | 32 |
| `live_or_unresolved_consumer` | 21 |
| `unique_ignored_files` | 9 |
| commit not on the remote | 10 |
Without expiration, the terminal workspace reaper archives only merged,
clean worktrees. Closed issues' worktrees therefore accumulate without
bound.
**Expected behavior**
A closed worktree whose only obstacle is unsaved work can be removed
after that work is saved, and restored exactly if the issue is reopened
or work resumes.
**Steps to reproduce**
1. Close an issue whose worktree has an unpushed commit or an
uncommitted file.
2. Enable git cache expiration with dry run off.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at 12bb7e1976.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Expiration** (`server/src/services/git-workspace-expiration.ts`):
- **When it applies:** with `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`
(default `false`), a worktree that fails the reproducibility check for a
relaxable reason is reconsidered once its issue tree has been closed for
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). The
relaxable reasons are uncommitted or untracked files, a commit not on
the remote, reflog or `ORIG_HEAD` history, and ignored `.env` files. A
leftover `COMMIT_EDITMSG` still blocks expiration unless it matches a
commit in the worktree's reflog, because restore points do not save
draft messages.
- **What still blocks:** a relaxed inspection
(`inspectGitCacheRecipeForRestorePoint`) keeps every other check. Nested
managed checkouts, submodules, special git state, local config and other
unique ignored files still prevent expiration.
- **Capture:** inside the existing source and lifecycle locks, a restore
point is captured with the #17238 library against a remote-verified
boundary (the `HEAD` itself, or its merge base with `origin/HEAD`), then
encrypted, uploaded, read back and hash-checked
(`git-restore-point-storage.ts`).
- **Commit before touching disk:** it is activated in the #17241 table
in the same transaction that archives the workspace with a
`paperclip.git-cache-restoration.v2` recipe referencing it, still before
anything on disk is touched.
- **Final check:** under the file locks, the worktree must still match
the captured snapshot (`HEAD`, index, working tree, modes). Otherwise
the existing rollback runs. The rollback now also drops the restore
point reference and returns the recipe to v1, so a later restore can
never apply a stale restore point over the live worktree; the stored
restore point is marked superseded.
- **Failure paths:** the `pending` row is inserted in a transaction that
first registers a `git-restore-points` notification. If the upload or
read-back fails, the row is marked `superseded` so retention deletes any
uploaded object. Nothing is deleted from disk.
- **Encryption key:** restore points are refused
(`restore_point_key_not_durable`) on `s3` storage unless the secrets
master key comes from `PAPERCLIP_SECRETS_MASTER_KEY`.
- **Dry run** reports `restorePoint: true` candidates without storing
anything.
**Restore** (`git-workspace-restoration.ts`, `workspace-runtime.ts`,
`execution-workspaces.ts`):
- **Recipe versions:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` accept v1 without a restore point, or v2
with a valid one. Older servers reject v2 instead of restoring the
commit without the saved work.
- **Before the saved commit is needed:** reopening
(`restoreGitCacheBase`) and the runtime path import the bundle when the
saved commit is missing locally.
- **After the worktree is re-added:** the runtime always imports the
bundle, so history-only commits are present again as objects even when
the base was rebuilt (they are not recreated as branches), then applies
the restore point, verifies the result, and records `appliedAt`. Reopen
persists that through a callback, as it already does for saved contents.
- **Never overwriting:** a worktree that already matches is only marked
applied. A worktree that diverged from the saved commit's clean state is
never overwritten; the restore fails closed and can be retried. Missing,
corrupt or expired restore points also fail closed.
**Retention:** the #17241 worker's expiry hook now marks the recipe's
restore point expired before its object is deleted.
**Config and docs:**
- `server/src/config.ts` adds the two settings;
`doc/recoverable-disk-cleanup.md` and `doc/DEVELOPING.md` describe them.
- No code comments were added.
**Not in this PR (follow-ups):**
- workspaces with nested `.paperclip-repositories` checkouts;
- removing idle worktrees of issues that are still open;
- starting a fresh workspace with a warning when a restore point has
expired.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 12
passed, with real repositories and embedded Postgres. The cases:
- **Round trip:** an unpushed commit, staged and unstaged edits of one
file, a deletion, and an untracked file are restored exactly on reopen,
`.env` is dropped, and the restore point is active. Restoring through
the runtime path works the same way, and a later run never re-applies
over newer work.
- **Kept:** a worktree that is not yet due, one with the feature
disabled (today's skip reason), and one whose encryption key is not
durable.
- **Change after capture:** rolls back, keeps the worktree, returns the
recipe to v1 and supersedes the restore point.
- **Missing restore point:** fails closed, keeps the row archived and
unapplied, and fails closed again on retry.
- **Dry run:** stores nothing.
- **Reproducible worktrees:** expire as before, with a v1 recipe.
- **History-only commits:** reopen imports them even after they are
pruned from the base.
- **Draft messages:** a draft commit message keeps the worktree.
- **Upload failure:** a failed upload marks the pending restore point
for cleanup and keeps the worktree.
- **Mutation checks:** each of these fails at least one case:
- removing the pre-deletion snapshot check;
- keeping the restore point reference on rollback;
- removing the runtime apply;
- removing the re-apply guard (the divergence check then fails closed
instead of overwriting);
- skipping the import for a matching worktree, relaxing the draft
message guard, or not marking a failed upload's row.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration-config.test.ts`: 5 passed.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point workspace-runtime execution-workspaces` (in
`server/`): 529 passed on the first revision. Two
`workspace-runtime.test.ts` cases about remote base refs fail, and they
also fail on unmodified `master` on this machine. After the review
fixes, the expiration, contents, restore point and execution workspace
suites passed again: 222 tests.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.
## Risks
- **Off by default:** nothing changes unless both
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` are set. Roll out with
`PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=true` first to see which
worktrees would receive restore points.
- **Data at rest:** restore points contain source and uncommitted files,
encrypted with keys wrapped by the instance secrets key. Losing that key
makes them unrecoverable, which is why `s3` storage requires an
environment-provided key. Untracked `.env` files are not stored.
- **Interrupted apply:** a crash in the middle of applying a restore
point leaves a worktree that neither matches the restore point nor is
clean. The restore then fails closed rather than overwriting it, and an
operator must reset that worktree to retry.
- **Rollback:** disable the setting. Already-expired worktrees keep
their v2 recipes and restore points, so reverting the code would leave
those recipes unreadable until restored by a server with this change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use isolated Git workspaces with managed project repository
copies.
> - Finished workspaces can expire when Git can reproduce their
contents.
> - The saved recipe records the main commit but omits nested commits
and local settings.
> - Reopening can therefore use newer nested code and lose merged user
settings.
> - This pull request saves and restores those contents before workspace
provisioning.
> - It also permits verified FETCH_HEAD metadata so ordinary fetches do
not block safe expiration.
## Linked Issues or Issue Description
Refs #16457, #16586, #16621. Related performance work: #16579.
**What happened?**
After a finished workspace expires, its nested project checkout can
reopen at the current upstream commit instead of its previous commit.
Cleanup also treats local Claude settings as disposable, although that
file can contain user settings. Ordinary linked-worktree FETCH_HEAD
metadata prevents otherwise safe cleanup.
**Expected behavior**
Restore the main and nested repositories at their saved commits.
Preserve local settings bytes and permissions. Skip files and Git state
that cannot be reproduced. Permit standard fetched metadata only when
its commits remain recoverable.
**Steps to reproduce**
1. Prepare a managed nested project repository at commit A and add
ignored local settings.
2. Advance upstream to B, finish the task tree, and expire its workspace
and base.
3. Reopen the workspace through the normal service path. Before this
fix, nested preparation selects B.
## What Changed
- Extend the existing restoration recipe with exact nested commits,
branches, source identity, and encrypted local settings. Keep the schema
compatible with older recipes.
- Restore nested repositories and settings before provisioning. Recheck
company/project source access and preserve conflicting files on retry.
- Recheck recorded contents under cleanup locks before deletion. Persist
lock ownership before atomic publication so process exits recover safely
and foreign locks stay intact. Preserve changed or unsupported contents,
including tracked settings.
- Keep saved project repository selections through normal preparation.
Checkpoint restored contents before provisioning so failed setup can
retry without overwriting later edits.
- Accept bounded linked-worktree FETCH_HEAD records only after local
ancestry or fresh upstream proof. Preserve local-only fetched commits.
- Add real Git/Postgres lifecycle cases and focused content, access,
publication-retry, and fetched-state controls. Document the behavior.
## Verification
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- The original real Git/Postgres experiment failed with nested B instead
of A. The same path passes with this change.
- The integrated lifecycle suite passed 42 cases in 91.66s. Related
preparation, credential, content, and Git metadata controls passed 46
cases in 24.41s. The final heartbeat checkpoint persistence case passed
separately in 8.17s.
- Coverage includes exact main/nested commits, encrypted settings and
modes, upstream failure and retry, provisioning order, concurrent
reopening, changed unique files, tracked settings, malformed metadata,
and company boundaries.
- Exact-head CI passed the implementation, typecheck, build, security,
and test matrices. One browser shard passed on a single rerun after an
unrelated fixture teardown raced an activity-log write; its feature
assertions passed in the initial attempt.
- The broad local test run was stopped after about 12 minutes of
unrelated serial tests. Full parallel CI is required before merge; no
local full-suite pass is claimed.
- Single-workspace deployment, real-backlog reclamation, and ordinary
product reopening remain required acceptance work after merge.
## Risks
- Expired Git repositories still depend on upstream access. A failed
restoration returns the existing retryable error and never substitutes
the default branch.
- Saved settings use the existing local encryption provider. Recovery
requires that encryption key to remain available.
- Unsupported files, tracked settings, unknown sources, and unverified
Git state remain on disk. This can reduce reclaimed space.
- No database migration, recovery-retention change, public API, or
fleet-default change is included.
## Model Used
OpenAI Codex, GPT-6 family, with tool use, code execution, and an
independent agent review. The runtime does not expose a more specific
model identifier.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
CONTRIBUTING.md says PRs add no code comments, but #16485, #16586, #16621 and #16672 added some. This removes the 215 comment-only lines git blame on master attributes to those merge commits, across 12 code files, plus one blank line the removal left doubled. Rewritten (not added) comments in scripts/assert-cloud-image-sentry.mjs and comments in .github workflows are left alone. No code changes.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Git expiration created its auth provider with only the issue ID; with a managed GitHub connection that provider throws outside a run ("No managed GitHub identity is available for this run"), so every upstream commit proof failed and was logged as inspection_failed. Expiration now falls back to anonymous access when the provider throws: public repositories verify, private ones stay kept. Upstream verification failures are reported as upstream_commit_unverified, and a managed checkout whose upstream cannot be reached or authenticated as managed_checkout_upstream_unavailable; local proof-store failures keep their own error.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Check the existing global execution/project shared-path veto before scanning retained run contexts. Preserve the full unshared-path query, lifecycle locks and all deletion-phase rechecks.
Verify with 35 focused real PostgreSQL/Git cases, full repository typecheck/build, exact-head full CI/E2E and Greptile Base 5/5. A read-only same-snapshot query pair retained the false verdict while reducing round trip from1089ms to9.4ms; this is not a full-agent latency result. The first security scan was inconclusive after an infrastructure checkout failure; one actual rescan passed.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reconstructable Git expiration rejected nearly every terminal worktree with unique_ignored_files, because Paperclip writes two ignored paths into every worktree itself: .claude/settings.local.json and the managed project checkouts under .paperclip-repositories/. Expiration now accepts the settings file, and accepts a managed checkout only through a prover the expiration worker supplies. The prover maps the checkout back to its project workspace row and proves it reproducible against that row's repository URL: worktree and stock-storage checks, no linked worktrees, no unreachable objects, and every ref, reflog and HEAD commit reachable from upstream refs fetched fresh into a proof store. Every inspection Git command now runs with core.fsmonitor disabled.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip manages AI agents and their task workspaces.
> - Finished tasks can release reproducible Git files and restore them
later.
> - Unique work must stay on disk, while regenerable dependencies can be
removed.
> - Linux Git templates include an empty legacy directory that blocked
base cleanup.
> - The new expiration branch also skipped the existing guarded
dependency cleanup.
> - This change accepts the standard empty directory and retains that
cleanup order.
> - Private bases can expire, and protected workspaces still shed
regenerable files.
## Linked Issues or Issue Description
Refs: #16457
Live acceptance of #16457 removed a worktree but retained its private
base because Linux Git created an empty `.git/branches` directory. A
pending-base dry run then tried to resolve the already removed worktree
and hid the preservation reason.
A local integration comparison also showed that enabling Git expiration
skipped existing dependency compaction. It deferred pending startup
compaction by an hour. Both failures were reproduced before the fix.
## What Changed
- Accept `.git/branches` only when it is an empty real directory.
Preserve populated entries, files, and symlinks.
- Inspect pending base cleanup without resolving an absent worktree.
- Run existing guarded compaction before Git expiration. Exclude already
closed rows from compaction and retain its pending deadline.
- Keep failed expiration checks from reaching legacy whole-workspace
deletion. Existing compaction remains independent of Git expiration
dry-run mode.
- Extend real Git/PostgreSQL tests and retain the acceptance failures in
the delivery state note.
## Verification
- Linux-template baseline: private-base expiration failed before the
fix. The first sandboxed attempt could not bind PostgreSQL; the
authorized retry reproduced the behavior.
- Compaction baseline: two regression cases failed in 7.67 seconds. They
reproduced skipped dependency cleanup and the lost startup deadline.
- After the fixes, `pnpm exec vitest run
server/src/__tests__/execution-workspace-compaction.test.ts
server/src/__tests__/git-workspace-expiration.test.ts
server/src/__tests__/terminal-workspace-work.test.ts` passed all 49
cases in 43.79 seconds.
- Controls cover unique work, active runs, unfinished descendants,
previews, concurrent reopening, interrupted removal, exact-commit
restoration, and retryable upstream failures.
- Full `pnpm -r typecheck` and `pnpm build` passed for the final commit.
All exact-head CI and security checks passed. Greptile is 5/5 with all
findings resolved.
- Final staging acceptance will let the restarted worker resume the
retained base cleanup, then reopen normally at the recorded commit and
persist subsequent work. The base has not been manually removed. The
superseded preview build was cancelled before dispatching a replacement.
## Risks
- Legacy branch definitions can hold unique data. Only an empty real
directory qualifies for deletion.
- Compaction preserves its existing lifecycle guard and
regenerable-directory allowlist. It cannot fall through to legacy
deletion after an expiration failure.
- Already closed workspaces must only resume their recorded expiration;
they do not enter compaction.
## Model Used
OpenAI Codex, GPT-6.1-sol, with reasoning, code execution, Git,
database, and browser tools. The exact context-window size is not
exposed by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip manages AI agents and their task workspaces.
> - Isolated Git worktrees and private repository copies remain on the
host after tasks finish.
> - These files can fill the disk even when every commit is available
upstream.
> - Cleanup must retain unique files, local Git state, live work, and
recovery data.
> - This pull request records and verifies an exact-commit restoration
recipe before expiration.
> - Normal reopening restores that commit even after upstream advances.
> - The benefit is lower host disk use while finished tasks remain
available for later work.
## Linked Issues or Issue Description
Refs #10285 and #2022. Follows #16433 for skill-cache expiration.
Related #11936 provides separate host-local cleanup tooling; this change
extends the existing company-scoped lifecycle worker and normal
workspace reopening. ROADMAP.md does not specify a competing
implementation.
## What Changed
- Add opt-in expiration for verified Paperclip-owned Git worktrees after
their whole task tree has been done or cancelled for 24 hours.
- Preserve active and scheduled runs, previews, pending reopening,
unresolved recovery, dirty files, unknown ignored files, and unsupported
or unique Git state.
- Verify an authenticated fetch in a fresh object store. Save repository
identity, exact commit, branch, and ownership before any deletion.
- Rename eligible files into recorded managed trash. Resume interrupted
cleanup under source and lifecycle locks.
- Expire private base clones only after consumer, file, reference,
reflog, and object checks pass. Preserve shared project base clones.
- Rebuild missing bases and worktrees through normal reopening at the
recorded commit. Return an explicit retryable error when repository
access fails.
- Add independent workspace/base switches, dry-run reports,
reclaimed-byte counts, and operational documentation. No table, public
API, UI, or recovery-retention change.
## Verification
- Real local bare remote plus Postgres: create at A, advance upstream to
B, expire the worktree and private base, reopen at A, then make and
retain a subsequent commit.
- Final lifecycle suite: all 20 tests passed in 46.41 seconds at
`d72947a2e45be5d0c912debb03c46b15863b4798`. Review fixes cover
second-cycle expiration, newer-branch recovery, normal preparation after
restoration, branch adoption, and Git administration path controls.
Eleven existing branch reconciliation controls also passed. Covers
unique files and Git state, live descendants, scheduled runs, recovery,
previews, unauthorized access, upstream failure, concurrent reopening,
interrupted deletion, shared bases, and nested private worktrees.
- Final guard regressions: 3 affected tests passed. Existing
workspace/runtime/preparation/source controls: 328 tests passed in
238.69 seconds.
- Full repository typecheck and build passed locally. The full local
test attempt did not pass: initial native-session cases hit macOS
Postgres shared-memory exhaustion, source changes invalidated two
discovery cases, and the checkout disappeared near the end. The
final-code retry of the remaining skill and unrelated controls passed
all 103 tests. All checks passed on this exact head, including
typecheck, general and serialized tests, Runner checks, build, canary
dry-run, browser E2E, and security scans. Greptile reviewed the head at
5/5 with all findings resolved.
- Fixed a blocking CI policy false positive for
`docker-compose-secrets.test.mjs`. The test still rejects GitHub secret
expressions, including dot, bracket, and multiline forms. Its full
15-test suite passes.
- Map validation and `git diff --check` passed.
- Staging dry-run inspection, measured reclamation, and ordinary product
acceptance remain pending. No live acceptance is claimed.
## Risks
- Git restoration depends on continued upstream availability and
credentials. Cleanup skips unverified commits; restoration never
substitutes the default branch.
- File deletion requires multiple conservative proofs. Unknown or
unsupported state is retained and can limit reclamation.
- Source and lifecycle locks serialize preparation, reopening, and
expiration. Remote verification has a deadline but can delay work on the
same source.
- Start with both expiration switches disabled. When enabled, dry-run
remains the default. Rollback disables further deletion while retaining
restoration support.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository inspection, tool
use, and code execution. The exact backend model ID and context-window
size are not exposed in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused tests; full local
limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
