mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 15:09:57 +02:00
Devin FoleyandPaperclip
5b29b9062e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each task can run in its own isolated Git worktree, and each worktree installs its own dependencies > - The terminal-workspace reaper removes a worktree only when the task tree is finished, the delivery is merged, the files are clean, and a 7-day cooldown has passed > - A recurring task that runs every hour keeps a week of worktrees, and each worktree holds gigabytes of `node_modules`. A finished task with no merged pull request, or with leftover agent files, keeps its worktree forever > - On a busy instance this fills the disk in days, and a larger volume only delays the next failure > - This pull request adds compaction: one hour after a task tree finishes, the reaper removes regenerable directories from the worktree and keeps the workspace, the branch, and all source changes > - The benefit is that finished tasks stop holding gigabytes of disk, and no work is lost because a resumed task regenerates the files when its workspace is provisioned again ## Linked Issues or Issue Description Refs #2022 (retention for workspace cleanup). Related: #11936 and #14608 add external reclaim scripts (a scheduled tool and a disk-guard step). This pull request does the reclaim inside the server's terminal-workspace reaper instead. That way it uses the reaper's lifecycle lock, issue-tree checks, and live-run checks, and needs no extra scheduler. **What existing behavior does this improve?** Disk retention of execution workspaces after their tasks finish. **Current behavior** A finished task keeps its full worktree, including installed dependencies and build output, for at least 7 days. If the delivery is not merged, or the worktree has uncommitted or untracked files, the reaper never removes it. On one instance, an hourly recurring task created about 22 worktrees per day at about 3 GB each (about 2.8 GB of that was `node_modules`). Together with finished tasks that had no merged pull request, they filled a 100 GB volume in about two days. **Proposed behavior** One hour after a task tree becomes terminal, the reaper removes regenerable directories (installed dependencies, build output, tool caches) from the worktree. It keeps the worktree, the branch, the commits, and every tracked, untracked, or other ignored file. When the task resumes, the existing reuse path runs the provision step, which reinstalls the dependencies. Clean, merged worktrees are archived after 1 day instead of 7. **Reason and benefit** Most of a finished worktree's size is regenerable. Compaction removes that part without the risk of deleting work, so it can apply to workspaces that the reaper must keep (no merged pull request, leftover files). The shorter cooldown limits the cost of the source checkouts that stay. **Breaking changes** None. The default archive cooldown changes from 7 days to 1 day. `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS` still overrides it. `PAPERCLIP_WORKSPACE_COMPACTION_ENABLED=false` turns compaction off. ## What Changed - New `server/src/services/workspace-compaction.ts`: - Finds directories with a regenerable name (`node_modules`, `dist`, `build`, `coverage`, `storybook-static`, `target`, `.cache`, `.next`, `.nuxt`, `.parcel-cache`, `.svelte-kit`, `.turbo`, `.vite`). - Keeps only the directories that the repository's own `.gitignore` files ignore and that hold no tracked file. A negated pattern (`!build/`), a global excludes file, and `.git/info/exclude` do not count. Each candidate is checked per path, so dependencies below an ignored parent (`vendor/tool/node_modules`) are found. - Searches inside a regenerable-named directory that does not qualify (for example a tracked `build/`), so its ignored dependencies still go. - Never enters `.git` or `.paperclip` (the worktree instance data), and never follows a symlink. - Each Git read has a 60-second deadline and stops when the sweep is aborted. - Moves the directories into `.paperclip-compaction-trash` beside the worktree with a same-filesystem rename, and can restore them. - `sweepTerminalWorkspaces` compacts each terminal workspace once per terminal cycle, after the compaction delay and before the archive cooldown check: - Only isolated linked Git worktrees qualify. A shared workspace, a project's main checkout, or a row whose `.git` is a directory is never compacted. - The move runs inside a transaction that holds the workspace lifecycle lock. The guard checks a fresh row and the same lifecycle generation. It also checks that the issue tree is terminal, that no queued or running run works on the tree or the workspace (by checkout or execution run, or by run context), and that no runtime service runs from the workspace. - The guard runs before the move and again after it. If the second check fails (for example, the task reopened or a run started), the directories move back. Otherwise the trash is deleted. - The worker wakes at the compaction deadline, not only at the archive deadline. - Stale trash from an interrupted compaction or a failed delete is purged by later sweeps. The sweep checks each worktree parent at most every 30 minutes, whether or not anything is left to compact. - New counters `compacted` and `compactionFailed` in the sweep result. The activity log gets an `execution_workspace.regenerable_files_removed` entry with the removed paths. - Config: new `PAPERCLIP_WORKSPACE_COMPACTION_DELAY_HOURS` (default `1`) and `PAPERCLIP_WORKSPACE_COMPACTION_ENABLED`. The default for `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS` changes from `7` to `1`. Both new keys are added to the run-failure diagnostics allowlists. - Docs: `doc/DEVELOPING.md` documents the new variables and has a new "Terminal workspace compaction" section. `doc/SPEC-implementation.md` describes compaction. - `execution-workspaces.ts` was already above 1,000 lines. The file-system logic is in the new module. Only the guarded orchestration, which uses the existing lifecycle lock and run predicates, is in the service. ## Verification - `pnpm exec vitest run src/__tests__/workspace-compaction.test.ts src/__tests__/execution-workspace-compaction.test.ts` (in `server/`): 27 passed. `workspace-reaper-cooldown-config.test.ts`: passed. - Unit tests: directory selection (ignored, untracked, regenerable name only, `.env` and local data kept, tracked `coverage` kept, `.paperclip` skipped, symlinks not followed), nested candidates under a source `build/` and under an ignored `vendor/`, a negated `!build/` kept, global and `info/exclude` rules ignored, glob characters in names, abort, main checkout vs linked worktree, move and restore, restore that keeps a recreated directory, and stale-trash purge. - Embedded Postgres tests: - A no-PR, unmerged, dirty workspace is compacted. HEAD, the branch ref, `git status --porcelain --untracked-files=all`, untracked notes, `.env`, local data, and `.paperclip` data stay the same, and the row stays active. - A delivered workspace inside the cooldown is compacted. - The delay is respected, and the worker wakes at the delay. - An active checkout run, a queued run found through its context, an unfinished child issue, and a running runtime service all block compaction. - The task reopens during the move: the files are restored and unchanged. - A run starts during the move: the files are restored. - A shared workspace and a main checkout are never compacted. - Compaction runs once per terminal cycle and again after a later cycle. - Disabled compaction does nothing. - End to end: `realizeExecutionWorkspace` creates a worktree, the sweep compacts it, and a second `realizeExecutionWorkspace` reuses the same worktree. The provision command regenerates `node_modules`, and the untracked notes and the unmerged commit are still present. - Leftover trash is purged when the workspace has nothing left to compact. - Mutation checks: when the post-move guard is disabled, both race tests fail. Without the negation fix or the repository-only source fix, their tests fail. - `pnpm exec vitest run` for `execution-workspaces-service.test.ts`, `terminal-workspace-work.test.ts`, `execution-workspace-reopen.test.ts`, `server-startup-feedback-export.test.ts`, and `workspace-runtime.test.ts`: 335 passed. Two `workspace-runtime.test.ts` cases fail locally with and without this change (`does not diagnose a missing remote branch as a rejected credential`, `rejects explicit main on a master-only remote ...`). They also fail on unmodified `master` on the same machine, so the cause is the local Git environment. - `pnpm typecheck` in `server/`: passed. `pnpm check:module-boundaries`: passed. ## Risks - A repository can keep something that is not regenerable in an ignored `dist`, `build`, `target`, or `.cache` directory. Compaction removes it. The fixed name list, the ignore check, and the tracked-file check limit this. `PAPERCLIP_WORKSPACE_COMPACTION_ENABLED=false` turns compaction off. - A repository without a provision command does not get dependencies back automatically. This is the same state as a new worktree for that repository, where the agent already installs them. - A resumed task pays for a dependency install again. This happens only for tasks that finished more than an hour earlier. - The shorter default cooldown removes clean, merged worktrees after 1 day instead of 7. An operator who wants the old hold can set `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS=7`. - Self-hosted instances get the same behavior. There is no schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, test runs). A separate Claude agent read the reaper code during the diagnosis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>