mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep walks every candidate workspace in a full pass. On staging that is about 4,000 finished remote workspaces, and each one costs a persistence probe and a work product read. > - #21617 limited full passes to startup and an hourly backstop that only activity arms. But three other paths in the worker still started a full pass at once: > - a wake whose hints do not parse to workspace or task targets; > - more than 50 pending targets; > - any run that finds no targets. > - Across one staging deploy the probe count rose by about 2.8 full passes in 26 minutes. At 1,000 concurrent agents, more than 50 run-finish hints between passes would be normal, so the overflow path alone would bring back passes proportional to history. > - This pull request works through targets in batches of 50 and turns hintless wakes into the backstop. It gives the one caller that needs a broad look, a workspace provider plugin becoming ready, its own hint that inspects only plugin workspaces. > - The benefit is that a full pass happens only at startup and at most once per backstop hour, whatever the activity. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On staging, the hourly steady state holds (one full pass per hour, no probes between). But across a deploy the probe count rose by 11,351 in under 26 minutes, about 2.8 full passes over the roughly 4,000 finished remote workspaces. **Steps to reproduce** The new test file `server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs the real coordinator and worker against real PostgreSQL on a fake clock, with finished remote workspaces that cannot be archived (the staging state). These are the counts on master. The numbers are workspaces checked by full passes; a pass over N workspaces checks N: | Scenario | Full-pass checks on master | Expected | | --- | --- | --- | | A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2 extra passes) | 0 | | One hintless wake and one unparseable hint, 120 workspaces, 30 minutes | 1,490 (about 12 passes) | 0, then 1 at the backstop | | A workspace provider plugin becomes ready (hintless notify from `plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin workspaces | | Restart: old instance drains while hints arrive, new instance starts and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30 minutes | 2,424 (12 passes) | 202 (the startup pass) | **Cause of the passes across a deploy:** - The new instance runs its startup pass. - Each workspace provider plugin that turns `ready` sends a hintless notification (`plugin-registry.ts`), which started a full pass. - Run recovery at startup finishes many runs, and their `issue:` hints overflow 50, which started another full pass. - Any later wake without targets also started a full pass, including the coordinator's 5-minute retry timer. In the fixture this produced two back-to-back passes every 5 minutes. - The draining old instance starts nothing, because the worker's `canRun` is false during drain. The test confirms this. **Expected behavior** A full pass runs only at startup and when the activity-armed backstop is due, at most once per hour. Targets are worked through in targeted batches. ## What Changed - `server/src/services/terminal-workspace-work.ts`: - **Pending targets.** Workspace and task targets go into pending sets with no cap. Each run inspects at most 50 of them, and the rest stay queued for the next run, which is scheduled at once. A failed targeted run puts its targets back. - **Hintless or unparseable wakes** only arm the hourly backstop. - **No targets, nothing due:** the run does nothing. - **Due deadlines:** the due workspaces are queued as targets, including isolated re-home and nested clone deadlines keyed by a workspace. A due non-workspace deadline (the isolated source orphan sweep, task directory work) runs a targeted sweep for its prelude and arms the backstop. - **A failed plugin workspace listing** restores the plugin refresh flag, so the coordinator's retry lists again. - **Full passes** happen only at startup, when the backstop is due, and on continuation pages of a pass already under way. - **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`): it queues the ids of plugin-provider candidate workspaces from `listPluginTerminalWorkspaceIds`. - The worker now always tells the service `targeted: true`, so a hint during a pass no longer requests a rescan lap. - `server/src/services/execution-workspaces.ts`: - New `listPluginTerminalWorkspaceIds` reads the plugin-provider candidates. - A targeted sweep reads all matches of its at most 50 targets, with no page limit and no rescan request. - A failed targeted sweep no longer requests a rescan lap, because the worker re-queues its targets. - `server/src/services/plugin-registry.ts`: the two workspace-provider notifications send `plugin-workspaces` instead of no hint. - `server/src/__tests__/terminal-workspace-work.test.ts`: a new test, `retries the plugin workspace listing after it fails instead of waiting for the backstop`, fails without the retry fix. Three existing tests now describe the new behaviour: - unscoped commits go to the backstop; - more than 50 hinted workspaces are inspected in batches of 50 and 1; - a due deadline on a service without targeting still runs a full pass. - `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`: four new tests, the scenarios above. The startup pass stays. The 1,000-workspace test in `terminal-workspace-deadlines.test.ts` shows that durable deadlines keep an idle instance quiet after startup. But only the startup pass reconciles workspaces whose hints were lost while no instance was running, so removing it would need a stored "last full pass" marker first. ## Verification - The four new tests fail on master with the counts in the table and pass after the change: - 0 full-pass checks for the burst, worked as targeted batches of at most 50 (120 checks in total); - 0 full-pass checks for 30 minutes after hintless and unparseable wakes, then 1 pass at the backstop; - only the 3 plugin workspaces are inspected after the plugin hint; - exactly 1 full pass across the restart: none on the draining instance, only the startup pass on the new one. - Prompt archiving and the missed-hint backstop keep their existing tests in `terminal-workspace-deadlines.test.ts`: - the release of a holding run inspects its workspace at once; - a run made terminal outside `writeRunStatus` is archived within 1 hour 1 minute; - 1,000 finished remote workspaces stay unread at idle. - `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`, `terminal-workspace-work`, `terminal-workspace-deadlines`, `terminal-workspace-idle`, `terminal-workspace-page-batch`, `execution-workspaces-service`, `plugin-workspaces.integration`, `workspace-operations-reconciliation`, `execution-workspace-reopen`, `isolated-task-directory-retention`, `git-workspace-expiration` and `workspace-delivery-cleanup.integration`. - `pnpm check:module-boundaries` passes. ## Risks - **A notification that relied on an immediate broad pass now waits up to an hour.** The only caller I found that sent such a notification on purpose is the workspace provider readiness in `plugin-registry.ts`, and it now has its own targeted hint. Other hintless notifications arm the backstop. - **The isolated source orphan sweep** runs only at the start of a full pass. Its due deadline now arms the backstop instead of starting a pass at once. This only matters when the isolated source re-home feature is enabled. - **A targeted sweep has no page limit.** Its at most 50 targets bound the matched workspaces to those tasks, their ancestors and their execution workspaces. - **A burst of N hints costs N/50 targeted runs, back to back.** The cost scales with activity, not with history. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>