Commit Graph
5 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 a18479bbba perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep walks every candidate workspace in a
full pass. On staging that is about 4,000 finished remote workspaces,
and each one costs a persistence probe and a work product read.
> - #21617 limited full passes to startup and an hourly backstop that
only activity arms. But three other paths in the worker still started a
full pass at once:
>   - a wake whose hints do not parse to workspace or task targets;
>   - more than 50 pending targets;
>   - any run that finds no targets.
> - Across one staging deploy the probe count rose by about 2.8 full
passes in 26 minutes. At 1,000 concurrent agents, more than 50
run-finish hints between passes would be normal, so the overflow path
alone would bring back passes proportional to history.
> - This pull request works through targets in batches of 50 and turns
hintless wakes into the backstop. It gives the one caller that needs a
broad look, a workspace provider plugin becoming ready, its own hint
that inspects only plugin workspaces.
> - The benefit is that a full pass happens only at startup and at most
once per backstop hour, whatever the activity.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On staging, the hourly steady state holds (one full pass per hour, no
probes between). But across a deploy the probe count rose by 11,351 in
under 26 minutes, about 2.8 full passes over the roughly 4,000 finished
remote workspaces.

**Steps to reproduce**

The new test file
`server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs
the real coordinator and worker against real PostgreSQL on a fake clock,
with finished remote workspaces that cannot be archived (the staging
state). These are the counts on master. The numbers are workspaces
checked by full passes; a pass over N workspaces checks N:

| Scenario | Full-pass checks on master | Expected |
| --- | --- | --- |
| A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2
extra passes) | 0 |
| One hintless wake and one unparseable hint, 120 workspaces, 30 minutes
| 1,490 (about 12 passes) | 0, then 1 at the backstop |
| A workspace provider plugin becomes ready (hintless notify from
`plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin
workspaces |
| Restart: old instance drains while hints arrive, new instance starts
and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30
minutes | 2,424 (12 passes) | 202 (the startup pass) |

**Cause of the passes across a deploy:**
- The new instance runs its startup pass.
- Each workspace provider plugin that turns `ready` sends a hintless
notification (`plugin-registry.ts`), which started a full pass.
- Run recovery at startup finishes many runs, and their `issue:` hints
overflow 50, which started another full pass.
- Any later wake without targets also started a full pass, including the
coordinator's 5-minute retry timer. In the fixture this produced two
back-to-back passes every 5 minutes.
- The draining old instance starts nothing, because the worker's
`canRun` is false during drain. The test confirms this.

**Expected behavior**

A full pass runs only at startup and when the activity-armed backstop is
due, at most once per hour. Targets are worked through in targeted
batches.

## What Changed

- `server/src/services/terminal-workspace-work.ts`:
- **Pending targets.** Workspace and task targets go into pending sets
with no cap. Each run inspects at most 50 of them, and the rest stay
queued for the next run, which is scheduled at once. A failed targeted
run puts its targets back.
  - **Hintless or unparseable wakes** only arm the hourly backstop.
  - **No targets, nothing due:** the run does nothing.
- **Due deadlines:** the due workspaces are queued as targets, including
isolated re-home and nested clone deadlines keyed by a workspace. A due
non-workspace deadline (the isolated source orphan sweep, task directory
work) runs a targeted sweep for its prelude and arms the backstop.
- **A failed plugin workspace listing** restores the plugin refresh
flag, so the coordinator's retry lists again.
- **Full passes** happen only at startup, when the backstop is due, and
on continuation pages of a pass already under way.
- **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`):
it queues the ids of plugin-provider candidate workspaces from
`listPluginTerminalWorkspaceIds`.
- The worker now always tells the service `targeted: true`, so a hint
during a pass no longer requests a rescan lap.
- `server/src/services/execution-workspaces.ts`:
- New `listPluginTerminalWorkspaceIds` reads the plugin-provider
candidates.
- A targeted sweep reads all matches of its at most 50 targets, with no
page limit and no rescan request.
- A failed targeted sweep no longer requests a rescan lap, because the
worker re-queues its targets.
- `server/src/services/plugin-registry.ts`: the two workspace-provider
notifications send `plugin-workspaces` instead of no hint.
- `server/src/__tests__/terminal-workspace-work.test.ts`: a new test,
`retries the plugin workspace listing after it fails instead of waiting
for the backstop`, fails without the retry fix. Three existing tests now
describe the new behaviour:
  - unscoped commits go to the backstop;
  - more than 50 hinted workspaces are inspected in batches of 50 and 1;
- a due deadline on a service without targeting still runs a full pass.
- `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`:
four new tests, the scenarios above.

The startup pass stays. The 1,000-workspace test in
`terminal-workspace-deadlines.test.ts` shows that durable deadlines keep
an idle instance quiet after startup. But only the startup pass
reconciles workspaces whose hints were lost while no instance was
running, so removing it would need a stored "last full pass" marker
first.

## Verification

- The four new tests fail on master with the counts in the table and
pass after the change:
- 0 full-pass checks for the burst, worked as targeted batches of at
most 50 (120 checks in total);
- 0 full-pass checks for 30 minutes after hintless and unparseable
wakes, then 1 pass at the backstop;
  - only the 3 plugin workspaces are inspected after the plugin hint;
- exactly 1 full pass across the restart: none on the draining instance,
only the startup pass on the new one.
- Prompt archiving and the missed-hint backstop keep their existing
tests in `terminal-workspace-deadlines.test.ts`:
  - the release of a holding run inspects its workspace at once;
- a run made terminal outside `writeRunStatus` is archived within 1 hour
1 minute;
  - 1,000 finished remote workspaces stay unread at idle.
- `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`,
`terminal-workspace-work`, `terminal-workspace-deadlines`,
`terminal-workspace-idle`, `terminal-workspace-page-batch`,
`execution-workspaces-service`, `plugin-workspaces.integration`,
`workspace-operations-reconciliation`, `execution-workspace-reopen`,
`isolated-task-directory-retention`, `git-workspace-expiration` and
`workspace-delivery-cleanup.integration`.
- `pnpm check:module-boundaries` passes.

## Risks

- **A notification that relied on an immediate broad pass now waits up
to an hour.** The only caller I found that sent such a notification on
purpose is the workspace provider readiness in `plugin-registry.ts`, and
it now has its own targeted hint. Other hintless notifications arm the
backstop.
- **The isolated source orphan sweep** runs only at the start of a full
pass. Its due deadline now arms the backstop instead of starting a pass
at once. This only matters when the isolated source re-home feature is
enabled.
- **A targeted sweep has no page limit.** Its at most 50 targets bound
the matched workspaces to those tasks, their ancestors and their
execution workspaces.
- **A burst of N hints costs N/50 targeted runs, back to back.** The
cost scales with activity, not with history.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:19:43 -06:00
DottaandClaude Opus 5.5 cefef8f11d perf(workspaces): inspect only hinted workspaces after a scoped commit (#21519)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up execution workspaces whose task tree is done. A
background sweep (`terminal-workspaces`) finds these workspaces.
> - Every workspace write and every task status change sends a hint to
this sweep. Each hint starts a full rotation over every open candidate
workspace, and each candidate costs a recursive read of its task tree.
> - Runs write workspaces all the time, so with live runs the sweep
never stops, and its cost grows with the number of open workspaces.
> - On a hosted instance with 5 live runs, the task tree read ran 11.6
times per second. On a local copy with run churn it ran 81 to 101 times
per second.
> - This pull request inspects only the workspaces that a hint names or
reaches. Full rotations stay for startup, saved deadlines, failures and
hints without a scope.
> - The benefit is that a workspace change costs a few statements, not
one tree read per open workspace.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The terminal workspace sweep (`registerTerminalWorkspaceWork` in
`server/src/services/terminal-workspace-work.ts` and
`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).

**Current behavior**

Producers send hints such as `workspace:<id>` and `issue:<id>`. The
worker ignores the scope: each settled hint wakes the queue, and the run
starts or continues a full keyset rotation over all candidate
workspaces, 50 per page with 1 second between pages. A hint that arrives
during a rotation also requests a rescan, so a new rotation starts after
the current one. Every candidate with an open task tree costs one
recursive tree read and is skipped.

**Proposed behavior**

The worker keeps the workspace ids and task ids from scoped hints. When
the queue runs only because of such hints, it calls
`sweepTerminalWorkspaces` with `targets`. The sweep then reads only
these candidates:

- the workspaces named by `workspace:` hints;
- the workspaces whose source task is the hinted task or one of its
ancestors (the tree of a workspace is its source task and its
descendants);
- the execution workspace of each hinted task.

A targeted inspection does not move the rotation cursor or its boundary
and does not clear saved deadlines. The per-workspace checks are the
same code as in a rotation.

A full rotation still runs at startup, when a saved deadline is due,
when a hint has no scope or an unknown scope, while a rotation is in
progress, and after a failed targeted inspection.

**Reason and benefit**

Background work must not grow with the backlog of open workspaces. With
4 candidate workspaces, a hint for one task tree now reads 1 tree
instead of 4. A hint during a rotation no longer starts another full
rotation.

## What Changed

- `server/src/services/terminal-workspace-work.ts`: parse scoped hints
into targets. Run a targeted inspection when only scoped hints are
pending. Keep the full rotation for startup, due deadlines, unscoped
hints, continuation pages and failures.
- `server/src/services/terminal-workspace-work.ts` also falls back to a
full rotation when more than 50 workspaces wait for inspection. It keeps
the 1-second pause between rotation pages and runs targeted inspections
between them.
- `server/src/services/issues.ts`: moving a task also hints its former
parent. Removing a task also hints its parent and its execution
workspace. Without this, a former parent's workspace could never be
reached.
- `server/src/services/execution-workspaces.ts`:
`sweepTerminalWorkspaces` accepts `targets`. A targeted page that fills
its limit requests a full rescan. and filters the candidates to the
targeted workspaces. A targeted inspection leaves the rotation state
alone. `terminalWorkspaceWorkSettled` takes `{ targeted: true }` from
the worker, and then a scoped hint does not request a full rescan.
Direct callers keep the old rescan behavior.
- `server/src/index.ts`: pass the new option through the existing
wrapper.
- Tests in `server/src/__tests__/terminal-workspace-work.test.ts` and
`server/src/__tests__/execution-workspaces-service.test.ts`.

## Verification

- New test `inspects only the workspaces a hint reaches, however many
trees stay open`: 3 workspaces with open trees and 1 done tree whose
child task is hinted. Before the change the targeted call checks 4
workspaces. After the change it checks 1, reads 1 tree, and archives the
done workspace.
- New test `inspects only hinted workspaces after a commit, and keeps
unscoped commits and deadlines on the full rotation`: before the change
the worker never passes targets. After the change a scoped commit runs a
targeted inspection, and an unscoped commit and a due deadline still run
a full rotation.
- `npx vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/git-workspace-expiration.test.ts` (in `server/`) pass.
- New tests `falls back to the full rotation when more workspaces are
hinted than one inspection reads`, `keeps the pause between rotation
pages while hinted workspaces wait` and `hints the former parent and
workspace when a task moves or is removed`. Each fails on the first
commit and passes on the head.
- `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all`. The fixture had 1 company, 60 agents, and open
tasks scaled from 150 to 5,000. Recovery re-dispatched runs to agents
that fail at once, and every run creates or updates an execution
workspace, so the hints keep coming. Each window is 180 seconds and
starts 150 seconds after the server is ready. The machine was shared.

The table gives the rate of the task tree read (`select id, status,
completed_at, cancelled_at, updated_at from issues ... WITH RECURSIVE
issue_tree`):

| Open tasks | Before (per s) | After (per s) |
| --- | --- | --- |
| 150 | 17.56 | 0.93 |
| 1,500 | 109.83 | 5.16 |
| 5,000 | 113.82 | 4.40 |

Total statements and transactions per second depend mostly on how fast
the failing runs cycle in the fixture, so they are noisy. Before →
after: 343 → 287 at 150, 1,647 → 1,562 at 1,500, 1,530 → 1,678 at 5,000
statements/s. In that window the 5,000-task run after the change also
had 10% more run events. With no activity, the server issued 0
statements before and after the change.

## Risks

- A workspace that waits on something without a hint is no longer
inspected as a side effect of unrelated activity. The known cases keep
their path: a workspace held by an active run keeps its past-due
deadline, so the 5-minute deadline rotation inspects it again; cooldown,
compaction, Git idle and retry deadlines still start full rotations. A
delivery that changes outside Paperclip without a work product change
was already only found by a later rotation.
- A failed targeted inspection requests a full rotation, so a lost
target is recovered by the rotation.
- Low migration risk. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:04:35 -06:00
Nicky LeachandPaperclip aab87ce707 Prove workspace delivery before inspecting cleanup candidates (#18493)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Completed tasks leave workspaces that need safe cleanup.
> - Providers without ancestry inspection currently need a recorded
merged PR before cleanup can start.
> - A squash merge changes commit identity, so direct ancestry cannot
prove that delivery.
> - This change proves delivery from the finalized HEAD and captured
destination before storage inspection.
> - Fresh storage checks and a second delivery check still protect
removal.
> - This reuses the existing worker and receipts. It adds no table or
timer.

## Linked Issues or Issue Description

Refs #18410. Related: #13400 addresses workspace identity during
inspection; this change adds destination-aware delivery discovery.

## What Changed

- Resolve direct ancestry or discover an associated merged GitHub PR
without a recorded work product. Require exact HEAD, destination
repository and branch, and a reachable integration commit.
- Use the latest settled finalization receipt as a candidate. Reject
missing, failed, pending, conflicting, or changed evidence.
- Inspect storage only after positive remote proof for providers that
disable ancestry inspection. Recheck HEAD, branch, source identity and
delivery before removal intent.
- Add an optional provider capability for a specific PR recovery hint
and an optional prepared destination receipt. Older providers keep their
existing protocol.
- Wake cleanup when finalization writes settle or gain a workspace
binding. Keep remote-only retries for concrete unresolved delivery. Park
missing or unsupported candidate identity.
- Reuse the source project's explicit GitHub consent for delivery checks
through the existing authorization service. Revalidate the bound
repository and consent before removal; missing or revoked consent
retains storage without switching identities. Public sources remain
anonymous.
- Update the contracts, documentation, access map, and targeted
regressions.

## Verification

- Initial-head GitHub CI passed at
`5acd4c818be6f1764f8319f42891f9956c357c26`, including full build and
typecheck. Standard Greptile is 5/5 with no open threads. The first Apex
review scored 3/5 and found startup latency from remote proof lookups
and removal retry identity changes across provider capability changes.
Both corrections are pushed in `9d1a38df3d`, with the qualified
destination-ref length fix. Red regressions reproduced each issue.
Latest targeted checks passed: cleanup/worker 26, provider integration
41, SDK contract 8. Head `9d1a38df3d` passed standard and Apex reviews
at 5/5 with no open threads, plus all CI including full build/typecheck.
Canary Dry Run initially failed on an npm 404 fetching
`rollup-4.67.3.tgz`; the exact tarball subsequently returned HTTP 200,
and one targeted retry passed without source changes. Aggregate
verification also passed. After #18403 merged, master introduced
conflicts in shared workspace cleanup and metadata handling. Resolved
them in `7700832536`, preserving remote snapshot cleanup and Git
delivery-proof cleanup. Targeted merge verification: 93 tests passed in
the first run (59 opt-in skips), then all 16 remote lifecycle tests
passed with explicit database opt-in; map checks passed. The isolated
database is stopped. The merge head received standard review 5/5. Its CI
found two failures in unchanged test cleanup paths: native runtime
temporary-directory ENOTEMPTY and agent-chat cancellation lock timeout.
Those failures remain recorded and require successful final-head
verification. The subsequent project-consent change has targeted
coverage: authorization36, resolver33 and cleanup19 tests passed,
including revoked consent, anonymous public access and source-binding
races. The positive consent test failed before the fix. Current head
`c772679e05` has green GitHub CI, including full build/typecheck and
both previously failing test jobs, plus standard Greptile 5/5 with no
open threads. Neither cleanup failure required changes to its test path.
Apex also completed at 5/5 on this exact head (check `124651548532`,
summary updated 2026-10-31 20:45 UTC), with no open review threads. The
PR remains draft. Earlier failures remain recorded above.

- Final focused run on the rebased branch: **182 tests passed across six
files**, covering the resolver, delivery cleanup, provider bridge,
finalization notifications, existing workspace service and SDK
contracts. The disposable test database was stopped afterward.
- Targeted resolver tests: 33 passed. These cover ancestry, discovered
squash PRs, wrong HEAD/base/repository, removed integration commits,
bounded responses, cancellation and credential isolation.
- Targeted cleanup integration tests: 17 passed. Provider bridge
integration: 36 passed. SDK contract tests: 7 passed. Finalization
notification tests: 9 passed.
- A negative control that removed the final delivery check failed the
race regression as expected. The check was restored.
- Read-only live GitHub probe: the new resolver discovered PR #18410
from HEAD `3f5ade4cd42b707040e9a82fdaed343e3d736a09`, without PR hints.
It proved integration commit `6484f96c28bfb11a793ce4be6cc5ac15643a5a02`
reachable from observed master
`5bc029bf6c1db6647e96acea323f3723fd1d3eba`. GitHub also advertised the
exact original HEAD at `refs/pull/18410/head`.
- `node scripts/check-maps.mjs` passed. Structural checks do not prove
deployment behavior.
- Local tests are targeted by user request. Full build, typecheck and
test suite run in GitHub CI. A standalone server typecheck could not
complete because local Runner build declarations were absent; no
changed-file diagnostics were reported.
- Initial local execution hit a stalled pnpm wrapper, sandbox socket
denial and stale generated SDK declarations. Direct installed binaries,
the authorized isolated test database and rebuilt SDK resolved those
setup failures. No deployment was performed.

## Risks

- Destructive cleanup still requires fresh storage safety and lifecycle
checks. Remote proof alone never authorizes removal. A persisted removal
intent preserves existing recovery semantics.
- Legacy workspaces without an unambiguous destination or completion
HEAD remain retained. This does not guess the original default branch.
- Private repositories use the explicit standing GitHub consent on the
bound source project, with current membership, grant, repository access
and trust checks. No separate company token is required when that
consent is valid. Missing or revoked consent and unavailable or
rate-limited access retain storage; the worker does not borrow a task or
agent identity.
- A known undelivered commit keeps the existing five-minute retry
because external GitHub changes have no guaranteed ingress to this lane.
This can keep cleanup scheduled. Missing evidence parks instead.
- Providers need the new recovery capability to clean a squash merge
whose original HEAD is reachable only through its PR ref. Providers
without it remain conservative.
- The resolver currently supports GitHub.com. Existing local and
other-provider ancestry paths stay in place.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools and
parallel code agents. The exact runtime model ID and context-window size
were not exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 07:59:17 -08:00
Devin FoleyandPaperclip aec80cbc8c fix(server): keep workspace compaction out of the startup sweep (#16328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When the server starts, `/api/health` answers "starting" until
startup recovery finishes, and deploy tooling waits for "ok"
> - Startup recovery waits for the first sweep of the terminal-workspace
worker
> - #16304 added workspace compaction to that sweep. Compaction walks
worktrees, runs Git, and deletes dependency trees, and it re-checks
every eligible workspace after a restart
> - Compaction is never needed to serve requests, so it should not delay
readiness
> - This pull request skips compaction in the startup sweep only; later
sweeps compact as before
> - The benefit is that a compaction backlog cannot make a restart
slower

## Linked Issues or Issue Description

Follow-up to #16304.

**What existing behavior does this improve?**
Server startup time after #16304.

**Current behavior**
`await terminalWorkspaceWorker.ready` blocks the readiness flip until
the first terminal-workspace sweep ends. That sweep compacts every
eligible workspace in its page. The per-cycle compaction memo is in
memory, so after each restart every eligible workspace is walked and
checked with Git again, and any directories found are deleted before the
sweep ends.

**Proposed behavior**
The first sweep skips compaction. Pending compactions keep their
deadlines, so the worker wakes and a later sweep compacts them.

**Reason and benefit**
Readiness depends only on work that is needed to serve requests. On one
measured instance the first sweep took 5–8 s with compaction; with a
larger backlog, compaction could add seconds per workspace.

**Breaking changes**
None. Compaction happens one sweep later after a restart.

## What Changed

- `sweepTerminalWorkspaces(limit, signal, { compact })`: `compact:
false` skips compaction for that sweep and keeps each pending compaction
deadline.
- `registerTerminalWorkspaceWork` passes `compact: false` to its first
sweep (the one readiness waits for) and `compact: true` after that.
- `server/src/index.ts` forwards the option through the logging wrapper.
- `doc/DEVELOPING.md`: the compaction section says the startup sweep
never compacts.

## Verification

- `pnpm exec vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/workspace-compaction.test.ts` (in `server/`): 35 passed.
- New: the worker passes `compact: false` only to the startup sweep and
`compact: true` to the next one.
- New: a sweep with `compact: false` leaves the files in place, the
compaction deadline stays due, and the next sweep compacts.
- `pnpm typecheck` in `server/`: passed.

## Risks

- Low risk. After a restart, compaction runs a few seconds later than
before. Archive behavior does not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-14 15:27:28 -07:00
DottaandPaperclip 7c016c049b perf(workspaces): wake terminal cleanup from committed changes (#16282)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use execution workspaces to retain their files and Git
branches.
> - A reaper closes eligible workspaces after their task tree is
terminal.
> - The reaper scanned every heartbeat tick and read Git before cheap
eligibility checks.
> - Permanent scans also query idle databases when no cleanup work
exists.
> - This pull request wakes cleanup after transaction settlement and
restores bounded work at startup.
> - The benefit is fewer scans while fresh cleanup guards continue to
protect retained work.

## Linked Issues or Issue Description

Related work: Refs #13619, Refs #11753. Those PRs bound scans or back
off unchanged candidates. This PR uses the transaction work coordinator
to remove the permanent reaper tick. It overlaps their early
terminal-state filtering but does not change their runtime restart
behavior or introduce a second Git scheduler cache.

**What existing behavior does this improve?**

Terminal workspace cleanup and Git inspection for workspace detail
reads.

**Subsystem affected**

Server orchestration, execution workspaces, and idle sleep admission.

**Current behavior**

The heartbeat tick scans workspace candidates every thirty seconds. It
runs Git before rejecting nonterminal trees, cooldowns, and active runs.
Display inspections expire after five seconds.

**Proposed behavior**

Recover workspace rows in bounded startup pages. Wake after relevant
issue, workspace, and pull-request work-product transactions settle.
Schedule the terminal tree cooldown or a five-minute eligibility retry.
Back off unchanged candidates across unrelated wakes, and invalidate
their backoff after relevant committed changes. Park empty and
nonterminal-only lanes. Reuse display inspections for thirty seconds.
Read fresh Git before destructive cleanup.

**Reason and benefit**

Avoid recurring database and Git work when no cleanup is possible.
Retain transaction admission holds, restart recovery, and fresh deletion
checks. Retention eligibility can wait for the next host wake; active
and failed recovery work still blocks sleep.

**Breaking changes**

No schema or API shape changes. Display Git state can lag by thirty
seconds. Cleanup eligibility retries can take five minutes.

## What Changed

- Add a terminal-workspace worker to the existing transaction work
coordinator and scheduler.
- Signal relevant writes before mutation on the owning transaction.
Outer transaction settlement wakes cleanup.
- Derive cooldown deadlines from the full company-scoped terminal task
tree. Continue bounded recovery pages without a permanent sweep.
- Bound unchanged candidate inspections to five minutes, retain at most
256 entries, and invalidate affected entries with scoped transaction
hints. Repeated wakes preserve the earliest inspection deadline.
- Collect deadlines from task trees already read by bounded pages.
Restore the cursor after a failed page and rescan commits received
during a rotation.
- Check terminal state, cooldown, and active runs before Git. Preserve
lifecycle locks, generation fences, and fresh final checks.
- Keep failed or incomplete recovery visible to idle admission. Preserve
the passive retention exception.
- Extend the bounded display cache to thirty seconds and document the
scheduling and wake limits.

## Verification

- Focused workspace, coordinator, cache, and local sleep suites: 111
tests passed with real local PostgreSQL and Git worktrees where
applicable.
- Final worker, display cache, and local sleep suites: 28 tests passed.
- The regressions prove three ineligible workspaces invoke zero Git
scans, cached clean display state cannot remove new untracked files, and
real producer notifications wait for outer transaction settlement.
- `pnpm -r typecheck` and `pnpm build` passed locally. Map and
module-boundary checks passed.
- Work-product and sleep-safety suites: 221 tests passed. Reopen and
issue-tree control suites: 46 tests passed. Final backoff, workspace,
worker, and work-product suites: 87 tests passed.
- Final workspace and worker suites: 81 tests passed. Deadline selection
performs zero database reads. A failed full page retries its unread
terminal workspace before recovery can park.
- Full CI coverage passed on head `2c397970f7`: general tests,
serialized server suites, runner verification, E2E shards, typecheck,
build, and the final verification gate. The duplicate serial local `pnpm
test:run` was stopped after full CI coverage passed; it is not reported
as a completed local pass.
- Greptile: 5/5 with both findings resolved. The security scan also
passed. No failing or pending PR checks remain.

## Risks

- Display state can be stale for thirty seconds. Destructive guards
bypass the display cache.
- Undelivered or otherwise ineligible terminal work retries after five
minutes instead of each heartbeat tick.
- Transaction signals cover the owning `createDb` scope. Direct SQL and
independent server processes require startup recovery.
- Local cooldown timers are maintenance hints. They do not establish an
external wake alarm or whole-instance sleep safety. Existing blockers
remain in force; failed bounded recovery keeps the host awake.

## Model Used

OpenAI GPT-6, model ID `gpt-6`, through Codex. Reasoning, repository
inspection, shell execution, and browser tools assisted the change. The
runtime does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-14 08:28:45 -05:00