mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
Devin FoleyandPaperclip
12bb7e1976
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Closed issues' git worktrees pile up on disk because they can only
be removed once the remote reproduces them exactly
> - #17238 adds restore points: encrypted saves of the unpushed and
uncommitted parts of a worktree, so the worktree itself can be removed
> - Those saves live in file storage and must be deleted on a schedule,
and promptly when their issue or company is deleted
> - This pull request adds the table that tracks restore points and a
deadline-driven retention worker, with nothing creating restore points
yet
> - The benefit is a reviewable lifecycle foundation before capture is
wired into expiration
## Linked Issues or Issue Description
Refs #17238 (restore point library). Bug report described in-PR (bug
report template).
**What happened?**
Closed issues' worktrees accumulate without bound. For example, one
instance's last 111 git cache expiration checks removed nothing, because
each worktree held unpushed commits or uncommitted files. Restore points
(#17238) will let those worktrees be removed safely. That needs a
durable record of every stored restore point and a retention policy for
deleting them.
**Expected behavior**
- A restore point is kept while its work can still resume.
- It is deleted a fixed time after its issue tree closes.
- It is deleted promptly when its issue or company is deleted.
**Steps to reproduce**
Not applicable on its own: this adds the lifecycle that the capture
integration will use.
**Paperclip version or commit:** `master` at def4726bea.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**New table `execution_workspace_restore_points`** (migration `0372`,
schema `packages/db/src/schema/execution_workspace_restore_points.ts`):
- **Columns:** storage key, sha256, size, generation, source issue,
`status`, closing time, review/deletion time (`delete_after`), a
`review_generation` counter and last error.
- **Statuses:** `pending`, `active`, `expiring`, `superseded`,
`delete_requested`, `deleted`.
- **No foreign keys:** a row outlives the deletion of its workspace,
issue or company long enough to remove the stored object.
- **Indexes:**
- the storage key is unique;
- live generations are unique per workspace;
- an index on status and deadline serves the worker, and a partial index
on active rows' source issue serves the trigger.
**Issue tree trigger** (same migration):
`paperclip_restore_point_tree_changed` runs after an issue's `status` or
`parent_id` changes, and after an issue is deleted. It walks up from the
issue and from its old and new parents (at most 64 levels). For every
active restore point whose source issue is in that set, it makes the row
due now (keeping an earlier due time) and increments
`review_generation`. This happens in the same transaction, so a child's
deletion or move still reaches its former parent's tree.
**New `server/src/services/git-restore-point-ledger.ts`:**
- **`gitRestorePointRetentionService(db, { storage, retentionDays,
beforeExpire })`.** Each `tick` handles only due work, in batches of 50:
- **Review:** an active restore point is checked when its `delete_after`
time arrives. The due check compares against the later of the worker's
clock and the database clock, so a row the trigger just marked is never
skipped. At review:
- **Tree still open:** clear `delete_after`. The trigger marks it due
again on the next relevant issue change, so open trees cause no
recurring checks.
- **Tree `done` or `cancelled`:** expire
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` (default `90`, minimum `1`)
after the tree's latest close. Expiry moves the row to `expiring`, then
runs an optional `beforeExpire` hook, which the capture integration will
use to mark the workspace no longer restorable. Only then does it
request deletion. An `expiring` row's object is never deleted: after a
restart, or if the hook fails, the hook is retried (after 15 minutes
when it failed).
- **Source issue gone:** delete.
- **Concurrency:** a review result is written only if
`review_generation` is unchanged since the worker read the row. If the
trigger fired in between (for example, the tree closed or reopened), the
write is skipped and the row stays due for the next run. Expiry and
orphan deletion claim the row under the same check before the expiry
hook runs.
- **Abandoned captures:** a `pending` row older than one hour is
deleted.
- **Deletion:** `superseded` and `delete_requested` rows have their
object removed. They become `deleted` only once `headObject` reports the
object gone. Failures, including a local unlink error the provider
swallows, are recorded in `last_error` and retried after 15 minutes.
- **`nextDeadlineAt`** returns the earliest real deadline, or `null`
when nothing is due.
- **`requestRestorePointDeletion(tx, { sourceIssueId } | { companyId
})`** registers a delivery-work notification and marks the matching
`pending`/`active` rows `delete_requested` in the caller's transaction.
**Deletion hooks:**
- `issueService.remove` and `deleteCompany` call
`requestRestorePointDeletion` in their transactions, so a deleted
issue's or company's objects are removed promptly.
- Child issues cannot be deleted with their parent (the parent link
restricts deletion), so each issue removal goes through this hook.
**Scheduling** follows `rules/sleep-aware-background-work.mdc`:
- `registerGitRestorePointWork` registers a `git-restore-points` queue
with the shared delivery-work coordinator.
- The worker keeps no in-memory inputs. Due work is recorded in the
table, by the trigger or by deletion requests. Each run handles it and
schedules only `nextDeadlineAt`, so a failed run loses nothing and is
retried by the coordinator.
- It runs at startup, at real deadlines, and when terminal-workspace
notifications settle after issue changes, but only while restore points
exist. With nothing due, it schedules nothing. A full batch continues
immediately. There is no timer.
**Config and docs:**
- `server/src/config.ts` adds
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS`.
- `doc/recoverable-disk-cleanup.md` gains a "Git workspace restore
points" section; `doc/DEVELOPING.md` documents the variable.
- `drizzle-kit generate` pruned `meta/0367_snapshot.json`, following the
repository's snapshot-retention convention.
- No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-restore-point-ledger.test.ts`
(in `server/`): 26 passed against embedded Postgres, stable across
repeated runs. The cases:
- **Review outcomes:** an open tree clears its deadline and schedules
nothing; a closed tree is scheduled 90 days after its close; a
long-closed tree expires and is deleted; a missing source issue is
deleted.
- **Expiry durability:** a failing hook keeps the row `expiring` with
its object until a retry succeeds; an `expiring` row left by a restart
resumes the hook before deletion; an `expiring` row's object is never
deleted while its hook fails.
- **Trigger:** the last open child closing, a child being deleted, a
child moving to another parent, and the source issue's own status change
all mark the restore point due. Unrelated issues, inactive rows and
already-due rows are untouched. End to end, an open tree's restore point
waits without a deadline, then is scheduled 90 days after its tree
closes.
- **Races:** closing the tree, or reopening it, between the worker's
read and write leaves the row due, and the next run resolves it
correctly. A reopened tree's restore point is not deleted.
- **No rescanning:** 60 retained restore points with future review times
are not evaluated. A backlog of 60 due ones is processed as 50, then 10,
then nothing.
- **Captures and deletion:** abandoned pending rows are deleted while
recent ones are kept; superseded rows are deleted; an object still
present after deletion is not marked deleted; a failed deletion is
retried only after its 15-minute deadline.
- **Deadlines:** the next deadline is correct for each obligation, and
`null` when idle.
- **Deletion hooks:** `issueService.remove` marks only that issue's
restore points, and `deleteCompany` marks all of the company's; the
worker then deletes their objects.
- **Worker:** schedules nothing when idle, and only the next deadline
otherwise; wakes on issue changes only while restore points exist.
- **Mutation checks:** each of these fails at least one case:
- removing the issue hook or the company hook;
- making the trigger ignore former parents;
- removing the worker's generation check or the trigger's generation
bump;
- letting storage cleanup delete `expiring` rows, or not resuming the
hook;
- ignoring open trees;
- removing the deletion verification;
- evaluating rows that are not due;
- dropping the retry deadline.
- `pnpm exec vitest run company-deletion issue-delete issues-service
delivery-work issue-tree` (in `server/`): 190 passed.
- **CI note:** one earlier `Typecheck + Release Registry` run was
cancelled when the GitHub runner received a shutdown signal; it is not a
code failure.
- `pnpm exec vitest run` (in `packages/db`): 211 passed. One run had a
single failure in `issue-comment-derived-attribution-migration.test.ts`,
an older migration's test unrelated to this change. It passed alone and
on the full re-run.
- `pnpm exec vitest run
src/__tests__/git-restore-point-ledger-config.test.ts`: 5 passed.
- `pnpm run check:migrations` (in `packages/db`): numbering and safety
checks pass.
- `pnpm run typecheck` (in `server/`): pass.
## Risks
- **No behavior change today:** nothing creates restore points yet. The
worker runs once at startup, finds no rows and schedules nothing. The
capture integration follows in a separate PR.
- **Trigger cost:** every issue status change, parent change or deletion
now runs a short ancestor walk, plus an `UPDATE` on the new table that
matches only active restore points through a partial index. With no
restore points, the `UPDATE` matches nothing.
- **Missed wakes:** if a wake is ever missed, for example because the
process exits after commit, the due row stays in the table. The worker's
startup run or next deadline processes it.
- **Deletion paths:** issue and company deletion each gain one `UPDATE`
on the new table inside their existing transactions.
- **Migration:** `0372_mature_the_initiative.sql` creates one empty
table, its indexes, and the trigger function with two triggers on
`issues`; nothing is rewritten.
- **Rollback:** revert. The table can stay, since nothing else reads it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only measurements on live
instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>