## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents do task work in execution workspaces on the instance disk
> - A task without a project repository runs in a plain (non-Git) task
directory under `<instance>/isolated-workspaces/<company>/<issue>/`
> - Nothing removes these directories after the work is done, so the
disk fills up
> - The directories can hold the only copy of agent work, so plain
deletion is not safe
> - This pull request saves each finished task directory in an encrypted
restore point, removes it, and restores it when the work reopens. It
also removes orphaned directories after a grace period
> - The benefit is that finished task directories stop using disk, and
no agent work is lost silently
## Linked Issues or Issue Description
**What happened?**
On one instance, `instances/<id>/isolated-workspaces/` held 251 task
directories and 12 GB, and it grew every day. 61 directories belonged to
open workspace rows whose issues closed more than 7 days before (3.9
GB). 12 belonged to issues closed 3 to 7 days before (0.7 GB). 4.0 GB
were directories with no open workspace row (the row was archived or
missing).
The cause has two parts:
1. The terminal workspace reaper archives a workspace only when it
proves delivery through Git (a merged pull request or a merged branch).
A task directory is not a Git checkout, so the Git status inspection
fails and the reaper always skips the row as "undelivered". Git cache
expiration handles only `git_worktree` workspaces.
2. When a person archives such a workspace by hand, artifact cleanup
removes a `local_fs` directory only when `metadata.createdByRuntime` is
true. Task directory rows never set it, so the directory stays and the
row becomes `cleanup_failed`.
**Expected behavior**
Paperclip reclaims the task directories of finished work by itself. It
must not lose files that only exist in those directories.
**Steps to reproduce**
1. Run an agent on an issue without a project repository (for example a
low-trust sandbox task, or an issue with no project), so it works in
`isolated-workspaces/<company>/<issue>/`.
2. Let the agent write files there, then mark the issue `done`.
3. Wait any number of days. The directory and its workspace row stay.
The reaper logs the row as skipped (undelivered).
4. Archive the workspace by hand. The directory still stays.
**Paperclip version or commit**
`master` at fe85fd7ac4
**Deployment mode**
Self-hosted, local disk storage.
**Agent adapter(s) involved**
Any adapter that runs in a task directory (seen with local and sandbox
runs).
## What Changed
- Added `isolated-task-directory-snapshot.ts`: a deterministic snapshot
of a plain directory (files with modes, directories, symbolic links as
links). It never follows symbolic links, refuses special files, and
enforces the restore point limits (20,000 files, 64 MiB per file, 256
MiB in total). A `node_modules` directory next to a `package.json` is
left out. Restore writes to a staging directory, verifies the digest,
then renames it into place.
- Added `isolated-task-directory-retention.ts`:
- `expireIsolatedTaskDirectory` saves the directory in a restore point,
archives the workspace with the `isolatedTaskDirectoryRetention` state
in one transaction, checks again that the directory did not change,
renames it to trash and deletes it. A change after the capture rolls the
workspace back to active and drops the restore point. An interrupted
removal finishes on a later sweep.
- It uses the git cache expiration consumer checks
(`isGitWorkspaceExpirationEligible`): no live run, unsettled
finalization, live lease, live runtime service, unresolved recovery
action or reopen fence, and no other open workspace or project workspace
in the directory. It also requires the source issue to be bound to the
workspace, because reopen restores through that binding.
- `restoreIsolatedTaskDirectory` restores the directory on reopen. It
replaces an empty directory and fails closed on a non-empty directory
that does not match. When the restore point expired, it reopens an empty
directory and posts a system comment.
- `sweepIsolatedTaskDirectories` walks `isolated-workspaces/` in bounded
batches (8 per sweep, with a cursor). A closed workspace that its issue
is still bound to is saved and removed like an open one. Other folders
are orphans and are removed after a grace period when no live row,
project workspace, runtime service, run, finalization, lease or recovery
action refers to them.
- Reused the git restore point ledger, storage and encryption.
`storeSealedRestorePoint` and `readRestorePointObject` were extracted
from `storeGitRestorePoint`. `markGitRestorePointExpired` now also marks
a task directory restore point as expired.
- `isGitWorkspaceExpirationEligible` takes `closedSiblingsAllowed`, so
closed rows for the same issue directory do not block a task directory
forever. Git expiration keeps its old behavior.
- Terminal workspace reaper (`execution-workspaces.ts`): handles open
task directory rows after the terminal delay and registers that moment
as a deadline. It finishes pending removals, runs the folder walk at the
start of a rotation (once per server start, then only at real
deadlines), and saves at most 8 directories per sweep.
- Reopen accepts a closed `shared_workspace` row that holds task
directory state. The run path also reopens (and restores) a closed
workspace with that state before preparing the run.
- Settings: `PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED` (default
`false`), `PAPERCLIP_TASK_DIRECTORY_RETENTION_DRY_RUN` (default `true`),
`PAPERCLIP_TASK_DIRECTORY_ORPHAN_GRACE_DAYS` (default `7`, minimum `1`).
The delay is `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS`. Removal
needs `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- Logs: `execution_workspace.task_directory_retention`,
`execution_workspace.task_directory_cleanup` and
`execution_workspace.task_directory_restore`, with outcome, skip reason,
candidate bytes and reclaimed bytes.
- Documented the behavior in `doc/recoverable-disk-cleanup.md` and the
settings in `doc/DEVELOPING.md`.
**Preservation decision.** The code shows that these directories are not
scratch space. Nothing pushes or copies their contents, and the next run
of the same issue reuses the same path. So the files are saved before
removal, with the same ledger, encryption and retention as git restore
points, and restored on reopen. Orphans are not saved: no workspace row
can reopen them. The docs and the Risks section state this.
**Scheduling.** The work follows
`rules/sleep-aware-background-work.mdc`. It adds no timer and no
polling. Open rows use the existing reaper and its per-workspace
deadlines (the end of the terminal delay, or one hour after a retryable
skip). The folder walk runs once at startup. After that, workspace
archive and issue close notifications (the existing terminal workspace
hints) point the reaper at the affected folder, and a kept folder is
checked again only at its own deadline (end of a grace period or
terminal delay, or one hour after a busy lock or live consumer). When
the tracked state is full, the reaper schedules another walk instead of
dropping work. With nothing pending, it registers nothing.
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/isolated-task-directory-retention-config.test.ts
src/__tests__/heartbeat-task-directory-restore.test.ts` (embedded
Postgres): 48 tests pass. They cover:
- a closed issue's task directory is archived and removed after the
delay by the reaper, and reopen restores files, modes and symbolic links
exactly;
- the same through reopen for a `shared_workspace` row, and through a
real heartbeat run (the run restores the files and runs in them);
- the directory stays before the delay (and the deadline is the end of
the delay), while the issue is open, and when retention is off;
- the directory stays while each consumer check fails: live run, live
runtime service, reopen fence, unsettled finalization, live lease,
unresolved recovery action, issue bound elsewhere, live sibling
workspace in the directory;
- no removal without restore points, with a non-durable key, before
`notBefore`, or over the limits;
- dry run changes nothing; a change after the capture rolls back; an
interrupted removal finishes on the next sweep; an expired restore point
reopens empty with a warning; a diverged directory fails the reopen and
stays untouched;
- folder sweep: an orphan is removed after the grace period, kept when
recently modified, in dry run, when a live row points into it, and when
its issue is open; an archived unbound row's folder is removed; an
archived bound row's folder is saved, removed and restored on reopen;
the walk is bounded and continues from its cursor;
- review follow-ups: a file name with a backslash saves and restores;
after the startup walk, a workspace archived later is found through its
change hint; an orphan used by a live run is retried; skipped rows do
not starve later rows of the per-sweep budget; overflowing tracked
deadlines schedule another walk instead of being dropped; a closing
child sends the reaper back to its archived parent's folder; a pending
lease cleanup on a closed workspace keeps an orphan; a deep path at the
platform path limit restores; an expiry recorded during removal survives
and the archive and removal are recorded as activity; a task closed
beyond retention keeps its new restore point for the full period;
- symbolic links: a task directory that is a link, an orphan that is a
link, and a link inside a directory are never followed; the link targets
stay untouched.
- Mutation checks. Each change below made at least one of the tests
above fail, and the test passed again after revert:
- drop the run-path reopen condition; drop the skipped-row cache; drop
hint tracking; reject backslashes again; drop the live-consumer retry
for folders; evict overflowing folder deadlines; skip ancestor hints;
ignore workspace-attached leases for orphans; stage restores under the
old longer name; overwrite the whole metadata on the final write; retain
task directory restore points from the tree close only; drop the removal
activity; force `closedSiblingsAllowed` off; make live siblings not
block; skip the digest recheck before rename; skip the issue binding
check; remove both symbolic link refusals (task directory); remove the
orphan symbolic link refusal; ignore live rows in the folder walk; skip
the orphan modification grace; ignore dry run for orphans; skip the
restore on reopen; skip the terminal delay; skip the restore points
requirement; skip the orphan open-issue check.
- Removing only one of the two task directory symbolic link checks did
not fail a test, because the other check still refuses the link.
Removing both fails the test.
- Related suites: `pnpm exec vitest run execution-workspaces
isolated-task-directory workspace-runtime git-restore-point
git-workspace-expiration terminal-workspace workspace-reaper
heartbeat-git-restore heartbeat-task-directory
isolated-repository-rehome nested-clone-migration environment-selection`
in `server/`: 34 files, 749 tests pass, 4 skipped. Two
`workspace-runtime.test.ts` tests fail (`does not diagnose a missing
remote branch as a rejected credential`, `rejects explicit main on a
master-only remote ...`). They fail the same way on unchanged `master`
on the local machine, so they are not caused by this change.
- `pnpm run typecheck` in `server/`: passes.
- Not run: the full server suite, a live instance rollout, and a check
that the host disk usage drops after removal.
## Risks
- The feature is off by default, and it starts as a dry run when
enabled. It needs restore points to remove anything. Rollback: set
`PAPERCLIP_TASK_DIRECTORY_RETENTION_ENABLED=false`. Saved restore points
and restore on reopen keep working.
- Orphan folders are removed without a restore point. This applies only
when no workspace row is bound to the issue at that path. If such an
issue is reopened later without another workspace, its next run starts
in an empty directory. The grace period (default 7 days of a closed tree
and no changes) and the dry run limit this risk.
- A directory over the restore point limits stays on disk (logged with
its reason). Special files keep a directory too.
- `node_modules` next to a `package.json` is not saved. A reopened task
must install its packages again.
- The restore point lives as long as git restore points (90 days by
default), counted from the later of the tree close and the removal. Only
task directory restore points use the removal time; git restore points
keep their retention. After that, reopen starts with an empty directory
and posts a warning comment.
- Reopen now accepts a closed `shared_workspace` row, but only when it
holds task directory state written by this feature.
- No schema migration. The ledger table and its columns are unchanged.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Closed issues' git worktrees pile up on disk because they can only
be removed once the remote reproduces them exactly
> - #17238 adds restore points: encrypted saves of the unpushed and
uncommitted parts of a worktree, so the worktree itself can be removed
> - Those saves live in file storage and must be deleted on a schedule,
and promptly when their issue or company is deleted
> - This pull request adds the table that tracks restore points and a
deadline-driven retention worker, with nothing creating restore points
yet
> - The benefit is a reviewable lifecycle foundation before capture is
wired into expiration
## Linked Issues or Issue Description
Refs #17238 (restore point library). Bug report described in-PR (bug
report template).
**What happened?**
Closed issues' worktrees accumulate without bound. For example, one
instance's last 111 git cache expiration checks removed nothing, because
each worktree held unpushed commits or uncommitted files. Restore points
(#17238) will let those worktrees be removed safely. That needs a
durable record of every stored restore point and a retention policy for
deleting them.
**Expected behavior**
- A restore point is kept while its work can still resume.
- It is deleted a fixed time after its issue tree closes.
- It is deleted promptly when its issue or company is deleted.
**Steps to reproduce**
Not applicable on its own: this adds the lifecycle that the capture
integration will use.
**Paperclip version or commit:** `master` at def4726bea.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**New table `execution_workspace_restore_points`** (migration `0372`,
schema `packages/db/src/schema/execution_workspace_restore_points.ts`):
- **Columns:** storage key, sha256, size, generation, source issue,
`status`, closing time, review/deletion time (`delete_after`), a
`review_generation` counter and last error.
- **Statuses:** `pending`, `active`, `expiring`, `superseded`,
`delete_requested`, `deleted`.
- **No foreign keys:** a row outlives the deletion of its workspace,
issue or company long enough to remove the stored object.
- **Indexes:**
- the storage key is unique;
- live generations are unique per workspace;
- an index on status and deadline serves the worker, and a partial index
on active rows' source issue serves the trigger.
**Issue tree trigger** (same migration):
`paperclip_restore_point_tree_changed` runs after an issue's `status` or
`parent_id` changes, and after an issue is deleted. It walks up from the
issue and from its old and new parents (at most 64 levels). For every
active restore point whose source issue is in that set, it makes the row
due now (keeping an earlier due time) and increments
`review_generation`. This happens in the same transaction, so a child's
deletion or move still reaches its former parent's tree.
**New `server/src/services/git-restore-point-ledger.ts`:**
- **`gitRestorePointRetentionService(db, { storage, retentionDays,
beforeExpire })`.** Each `tick` handles only due work, in batches of 50:
- **Review:** an active restore point is checked when its `delete_after`
time arrives. The due check compares against the later of the worker's
clock and the database clock, so a row the trigger just marked is never
skipped. At review:
- **Tree still open:** clear `delete_after`. The trigger marks it due
again on the next relevant issue change, so open trees cause no
recurring checks.
- **Tree `done` or `cancelled`:** expire
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` (default `90`, minimum `1`)
after the tree's latest close. Expiry moves the row to `expiring`, then
runs an optional `beforeExpire` hook, which the capture integration will
use to mark the workspace no longer restorable. Only then does it
request deletion. An `expiring` row's object is never deleted: after a
restart, or if the hook fails, the hook is retried (after 15 minutes
when it failed).
- **Source issue gone:** delete.
- **Concurrency:** a review result is written only if
`review_generation` is unchanged since the worker read the row. If the
trigger fired in between (for example, the tree closed or reopened), the
write is skipped and the row stays due for the next run. Expiry and
orphan deletion claim the row under the same check before the expiry
hook runs.
- **Abandoned captures:** a `pending` row older than one hour is
deleted.
- **Deletion:** `superseded` and `delete_requested` rows have their
object removed. They become `deleted` only once `headObject` reports the
object gone. Failures, including a local unlink error the provider
swallows, are recorded in `last_error` and retried after 15 minutes.
- **`nextDeadlineAt`** returns the earliest real deadline, or `null`
when nothing is due.
- **`requestRestorePointDeletion(tx, { sourceIssueId } | { companyId
})`** registers a delivery-work notification and marks the matching
`pending`/`active` rows `delete_requested` in the caller's transaction.
**Deletion hooks:**
- `issueService.remove` and `deleteCompany` call
`requestRestorePointDeletion` in their transactions, so a deleted
issue's or company's objects are removed promptly.
- Child issues cannot be deleted with their parent (the parent link
restricts deletion), so each issue removal goes through this hook.
**Scheduling** follows `rules/sleep-aware-background-work.mdc`:
- `registerGitRestorePointWork` registers a `git-restore-points` queue
with the shared delivery-work coordinator.
- The worker keeps no in-memory inputs. Due work is recorded in the
table, by the trigger or by deletion requests. Each run handles it and
schedules only `nextDeadlineAt`, so a failed run loses nothing and is
retried by the coordinator.
- It runs at startup, at real deadlines, and when terminal-workspace
notifications settle after issue changes, but only while restore points
exist. With nothing due, it schedules nothing. A full batch continues
immediately. There is no timer.
**Config and docs:**
- `server/src/config.ts` adds
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS`.
- `doc/recoverable-disk-cleanup.md` gains a "Git workspace restore
points" section; `doc/DEVELOPING.md` documents the variable.
- `drizzle-kit generate` pruned `meta/0367_snapshot.json`, following the
repository's snapshot-retention convention.
- No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-restore-point-ledger.test.ts`
(in `server/`): 26 passed against embedded Postgres, stable across
repeated runs. The cases:
- **Review outcomes:** an open tree clears its deadline and schedules
nothing; a closed tree is scheduled 90 days after its close; a
long-closed tree expires and is deleted; a missing source issue is
deleted.
- **Expiry durability:** a failing hook keeps the row `expiring` with
its object until a retry succeeds; an `expiring` row left by a restart
resumes the hook before deletion; an `expiring` row's object is never
deleted while its hook fails.
- **Trigger:** the last open child closing, a child being deleted, a
child moving to another parent, and the source issue's own status change
all mark the restore point due. Unrelated issues, inactive rows and
already-due rows are untouched. End to end, an open tree's restore point
waits without a deadline, then is scheduled 90 days after its tree
closes.
- **Races:** closing the tree, or reopening it, between the worker's
read and write leaves the row due, and the next run resolves it
correctly. A reopened tree's restore point is not deleted.
- **No rescanning:** 60 retained restore points with future review times
are not evaluated. A backlog of 60 due ones is processed as 50, then 10,
then nothing.
- **Captures and deletion:** abandoned pending rows are deleted while
recent ones are kept; superseded rows are deleted; an object still
present after deletion is not marked deleted; a failed deletion is
retried only after its 15-minute deadline.
- **Deadlines:** the next deadline is correct for each obligation, and
`null` when idle.
- **Deletion hooks:** `issueService.remove` marks only that issue's
restore points, and `deleteCompany` marks all of the company's; the
worker then deletes their objects.
- **Worker:** schedules nothing when idle, and only the next deadline
otherwise; wakes on issue changes only while restore points exist.
- **Mutation checks:** each of these fails at least one case:
- removing the issue hook or the company hook;
- making the trigger ignore former parents;
- removing the worker's generation check or the trigger's generation
bump;
- letting storage cleanup delete `expiring` rows, or not resuming the
hook;
- ignoring open trees;
- removing the deletion verification;
- evaluating rows that are not due;
- dropping the retry deadline.
- `pnpm exec vitest run company-deletion issue-delete issues-service
delivery-work issue-tree` (in `server/`): 190 passed.
- **CI note:** one earlier `Typecheck + Release Registry` run was
cancelled when the GitHub runner received a shutdown signal; it is not a
code failure.
- `pnpm exec vitest run` (in `packages/db`): 211 passed. One run had a
single failure in `issue-comment-derived-attribution-migration.test.ts`,
an older migration's test unrelated to this change. It passed alone and
on the full re-run.
- `pnpm exec vitest run
src/__tests__/git-restore-point-ledger-config.test.ts`: 5 passed.
- `pnpm run check:migrations` (in `packages/db`): numbering and safety
checks pass.
- `pnpm run typecheck` (in `server/`): pass.
## Risks
- **No behavior change today:** nothing creates restore points yet. The
worker runs once at startup, finds no rows and schedules nothing. The
capture integration follows in a separate PR.
- **Trigger cost:** every issue status change, parent change or deletion
now runs a short ancestor walk, plus an `UPDATE` on the new table that
matches only active restore points through a partial index. With no
restore points, the `UPDATE` matches nothing.
- **Missed wakes:** if a wake is ever missed, for example because the
process exits after commit, the due row stays in the table. The worker's
startup run or next deadline processes it.
- **Deletion paths:** issue and company deletion each gain one `UPDATE`
on the new table inside their existing transactions.
- **Migration:** `0372_mature_the_initiative.sql` creates one empty
table, its indexes, and the trigger function with two triggers on
`issues`; nothing is rewritten.
- **Rollback:** revert. The table can stay, since nothing else reads it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only measurements on live
instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
