mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep stores one deadline per workspace: cooldown end, compaction, cleanup retry, Git idle. > - When any deadline was due, the worker ran a full rotation over every open candidate workspace. That costs a task tree read and more for each one. > - A workspace held by a `running` run kept a past deadline, so the 5-minute deadline rotation came back for it forever. A run that never finishes would keep the instance awake indefinitely. > - This pull request makes a due deadline inspect only the due workspaces. A held workspace waits for its run instead of for a timer. > - The benefit is that an idle instance does work only for workspaces whose deadline really arrived. The cost no longer grows with the number of open workspaces. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** How the terminal workspace worker (`server/src/services/terminal-workspace-work.ts`) acts on due deadlines and on workspaces held by an active run. **Current behavior** `hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest deadline is past, the next run starts a full keyset rotation, 50 workspaces per page with 1 second between pages, over every candidate. A workspace whose issue still has a `queued` or `running` run is skipped, but the archive deadline set earlier in the same pass stays. It is already past, so the rotation repeats every 5 minutes. **Proposed behavior** - When the run fires for a due deadline, the worker takes the due workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and runs a targeted inspection of only those workspaces. The service first clears the deadlines of the targeted workspaces; the inspection sets fresh ones. - If more than 50 workspaces are due, or a due deadline belongs to the isolated source, nested clone or task directory work, the worker still runs a full rotation. Startup still runs a full rotation. - A workspace skipped because a run holds it drops its deadline. - `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`) publishes `issue:<runIssueId>` to the terminal workspace queue when a run reaches a terminal status. That targeted inspection reaches the workspace the run held. - **Backstop.** About 200 code paths write `heartbeat_runs` directly (reapers, recovery terminalization, bulk cancels), and they do not pass through `writeRunStatus`. So the worker keeps a full rotation as a backstop, armed only by activity, like the recovery backstop from #21552: - Any terminal-workspace hint arms it. - Once armed, a full rotation runs at most once per hour (`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation. - An idle instance with no hint schedules nothing. - A missed release hint therefore delays archiving by at most one hour after activity, without a restart. - The hour is a compromise. On master today, the 5-minute rotation re-reads every candidate. On the hosted instance that is about 1,000 finished remote workspaces, roughly 3,000 statements per lap. The hourly backstop re-reads them at most once per hour while there is activity, about 1 statement per second averaged. **Reason and benefit** Background work must be driven by deadlines and change notifications, and its cost must not grow with the backlog. Every guarantee keeps a named trigger: - cooldown end, compaction, retry backoff and Git idle each have a stored deadline, and a due deadline inspects exactly that workspace; - the release of a holding run publishes a hint; - task status changes and workspace writes still publish hints; - startup still reconciles everything. ## What Changed - `server/src/services/execution-workspaces.ts`: - New `takeDueTerminalWorkspaceTargets()`. - A targeted inspection clears the deadlines of its targets before it inspects them. - A workspace skipped for an active run drops its deadline. - `server/src/services/terminal-workspace-work.ts`: a due run uses the due targets instead of a full rotation, unless they overflow or a non-workspace deadline is due. A due run that finds nothing due does nothing. Any hint arms the hourly backstop rotation. - `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run status hints the run's task to the terminal workspace queue. - `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new tests on real PostgreSQL. They use the real coordinator and worker on a fake clock. - `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616): its idle window now bounds the number of sweeps instead of requiring zero targeted sweeps, because a due deadline is now a targeted inspection. ## Verification - `inspects only the workspace whose cooldown ends, however many trees stay open`: 60 workspaces with open task trees and 1 done workspace whose 1-day cooldown ends 1 hour later. - Before the change: after startup, every due deadline starts a full rotation (27 full sweeps in 2 hours). - After the change: no full rotation; the done workspace is inspected alone (`checked: 1`) once its cooldown ends; fewer than 200 statements in 2 hours. - `waits for the release of the run that holds a workspace instead of rechecking it`: a done workspace held by a `running` run. - Before the change: a full rotation every 5 minutes (22 sweeps in 2 hours). - After the change: 0 statements in 2 idle hours. When the run is set to `succeeded` through the lifecycle, the workspace is inspected (`checked: 1`, `skippedActiveRun: 0`). - `leaves a thousand finished remote workspaces unread at idle`. This is the hosted state: 1,000 `active` remote workspaces with a location, a committed snapshot and a task done 5 days ago, and zero runs. - After startup and 2 idle hours: no full rotation. No remote workspace is read again; the only inspections are of the one due local workspace. Statements stay within 10 per inspection. - On master, every due deadline re-reads all 1,000. - `archives a workspace whose run ended outside the run lifecycle within the activity-armed backstop`: - A workspace held by a `running` run. The run is set to `failed` by a direct `UPDATE`, so no release hint is published, and one unrelated hint arrives. - The workspace is archived within one hour and one minute, without a restart. - With the backstop scheduling disabled, it stays `active`. - `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts src/__tests__/terminal-workspace-work.test.ts src/__tests__/execution-workspaces-service.test.ts src/__tests__/isolated-task-directory-retention.test.ts src/__tests__/git-workspace-expiration.test.ts src/__tests__/workspace-delivery-cleanup.integration.test.ts src/__tests__/nested-clone-migration.test.ts src/__tests__/execution-workspace-compaction.test.ts src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333 tests pass. - `pnpm check:module-boundaries` passes. Local fixture with #21616 applied as the base, 30 idle minutes on a fake clock, real PostgreSQL. Each fixture holds open trees, a stranded `running` holder, cooldown, done local workspaces with missing files, closed remote workspaces with and without a committed snapshot: | Workspaces | Before: sweeps | Before: statements | After: sweeps | After: statements | | --- | --- | --- | --- | --- | | 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 | | 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 | After the change, no sweep reads the task tree of a workspace whose tree is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420 workspaces). What remains comes only from workspaces with a due retry or inspection backoff, so it grows with the number of stuck workspaces, not open ones. Per inspected workspace a pass issues the same 3–6 statements as before; batching them per page is a separate pull request. ## Risks - **A run that leaves `running` without `writeRunStatus`** does not hint. The activity-armed backstop catches it within one hour of activity, and startup reconciles. This pull request does not decide that a stranded run is gone; it waits for a status change. - **A workspace whose deadline is reset to "now"** (a reopen-pending flag within its grace period, task directory budget overflow) is inspected alone on each due run. The inspection backoff in `nextTerminalWorkspaceSweepAt` still spaces these runs. - **Workspaces without a stored deadline** (for example an undelivered workspace whose deadline was deleted) are inspected again only when a hint or the startup rotation reaches them. Before this change they were also reached by the rotation that a different workspace's deadline started. ## Open decision **Should a terminal workspace whose local files are gone be archived, or should it wait?** This pull request does not change the behaviour; it leaves the decision to the owner. Today such a workspace fails its close-readiness inspection and is reported as `skippedUndelivered`. A non-remote workspace keeps its past deadline, so the 5-minute inspection backoff re-inspects it again and again. That costs about 3–4 statements per stuck workspace every 5 minutes (from the local fixture: 255 inspections, 866 statements), and it never ends while the files stay missing. The cause is typically a replaced container. The two options: - **Archive it.** Nothing remains on disk to keep. - **Store a "files missing" state.** It is then inspected again only when a write announces a change. Either option removes this cost. Archiving also removes the row from later rotations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>