mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
a18479bbba222e8e95d0cdf1fbd4043328c0d28d
8030
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a18479bbba |
perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep walks every candidate workspace in a full pass. On staging that is about 4,000 finished remote workspaces, and each one costs a persistence probe and a work product read. > - #21617 limited full passes to startup and an hourly backstop that only activity arms. But three other paths in the worker still started a full pass at once: > - a wake whose hints do not parse to workspace or task targets; > - more than 50 pending targets; > - any run that finds no targets. > - Across one staging deploy the probe count rose by about 2.8 full passes in 26 minutes. At 1,000 concurrent agents, more than 50 run-finish hints between passes would be normal, so the overflow path alone would bring back passes proportional to history. > - This pull request works through targets in batches of 50 and turns hintless wakes into the backstop. It gives the one caller that needs a broad look, a workspace provider plugin becoming ready, its own hint that inspects only plugin workspaces. > - The benefit is that a full pass happens only at startup and at most once per backstop hour, whatever the activity. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On staging, the hourly steady state holds (one full pass per hour, no probes between). But across a deploy the probe count rose by 11,351 in under 26 minutes, about 2.8 full passes over the roughly 4,000 finished remote workspaces. **Steps to reproduce** The new test file `server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs the real coordinator and worker against real PostgreSQL on a fake clock, with finished remote workspaces that cannot be archived (the staging state). These are the counts on master. The numbers are workspaces checked by full passes; a pass over N workspaces checks N: | Scenario | Full-pass checks on master | Expected | | --- | --- | --- | | A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2 extra passes) | 0 | | One hintless wake and one unparseable hint, 120 workspaces, 30 minutes | 1,490 (about 12 passes) | 0, then 1 at the backstop | | A workspace provider plugin becomes ready (hintless notify from `plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin workspaces | | Restart: old instance drains while hints arrive, new instance starts and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30 minutes | 2,424 (12 passes) | 202 (the startup pass) | **Cause of the passes across a deploy:** - The new instance runs its startup pass. - Each workspace provider plugin that turns `ready` sends a hintless notification (`plugin-registry.ts`), which started a full pass. - Run recovery at startup finishes many runs, and their `issue:` hints overflow 50, which started another full pass. - Any later wake without targets also started a full pass, including the coordinator's 5-minute retry timer. In the fixture this produced two back-to-back passes every 5 minutes. - The draining old instance starts nothing, because the worker's `canRun` is false during drain. The test confirms this. **Expected behavior** A full pass runs only at startup and when the activity-armed backstop is due, at most once per hour. Targets are worked through in targeted batches. ## What Changed - `server/src/services/terminal-workspace-work.ts`: - **Pending targets.** Workspace and task targets go into pending sets with no cap. Each run inspects at most 50 of them, and the rest stay queued for the next run, which is scheduled at once. A failed targeted run puts its targets back. - **Hintless or unparseable wakes** only arm the hourly backstop. - **No targets, nothing due:** the run does nothing. - **Due deadlines:** the due workspaces are queued as targets, including isolated re-home and nested clone deadlines keyed by a workspace. A due non-workspace deadline (the isolated source orphan sweep, task directory work) runs a targeted sweep for its prelude and arms the backstop. - **A failed plugin workspace listing** restores the plugin refresh flag, so the coordinator's retry lists again. - **Full passes** happen only at startup, when the backstop is due, and on continuation pages of a pass already under way. - **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`): it queues the ids of plugin-provider candidate workspaces from `listPluginTerminalWorkspaceIds`. - The worker now always tells the service `targeted: true`, so a hint during a pass no longer requests a rescan lap. - `server/src/services/execution-workspaces.ts`: - New `listPluginTerminalWorkspaceIds` reads the plugin-provider candidates. - A targeted sweep reads all matches of its at most 50 targets, with no page limit and no rescan request. - A failed targeted sweep no longer requests a rescan lap, because the worker re-queues its targets. - `server/src/services/plugin-registry.ts`: the two workspace-provider notifications send `plugin-workspaces` instead of no hint. - `server/src/__tests__/terminal-workspace-work.test.ts`: a new test, `retries the plugin workspace listing after it fails instead of waiting for the backstop`, fails without the retry fix. Three existing tests now describe the new behaviour: - unscoped commits go to the backstop; - more than 50 hinted workspaces are inspected in batches of 50 and 1; - a due deadline on a service without targeting still runs a full pass. - `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`: four new tests, the scenarios above. The startup pass stays. The 1,000-workspace test in `terminal-workspace-deadlines.test.ts` shows that durable deadlines keep an idle instance quiet after startup. But only the startup pass reconciles workspaces whose hints were lost while no instance was running, so removing it would need a stored "last full pass" marker first. ## Verification - The four new tests fail on master with the counts in the table and pass after the change: - 0 full-pass checks for the burst, worked as targeted batches of at most 50 (120 checks in total); - 0 full-pass checks for 30 minutes after hintless and unparseable wakes, then 1 pass at the backstop; - only the 3 plugin workspaces are inspected after the plugin hint; - exactly 1 full pass across the restart: none on the draining instance, only the startup pass on the new one. - Prompt archiving and the missed-hint backstop keep their existing tests in `terminal-workspace-deadlines.test.ts`: - the release of a holding run inspects its workspace at once; - a run made terminal outside `writeRunStatus` is archived within 1 hour 1 minute; - 1,000 finished remote workspaces stay unread at idle. - `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`, `terminal-workspace-work`, `terminal-workspace-deadlines`, `terminal-workspace-idle`, `terminal-workspace-page-batch`, `execution-workspaces-service`, `plugin-workspaces.integration`, `workspace-operations-reconciliation`, `execution-workspace-reopen`, `isolated-task-directory-retention`, `git-workspace-expiration` and `workspace-delivery-cleanup.integration`. - `pnpm check:module-boundaries` passes. ## Risks - **A notification that relied on an immediate broad pass now waits up to an hour.** The only caller I found that sent such a notification on purpose is the workspace provider readiness in `plugin-registry.ts`, and it now has its own targeted hint. Other hintless notifications arm the backstop. - **The isolated source orphan sweep** runs only at the start of a full pass. Its due deadline now arms the backstop instead of starting a pass at once. This only matters when the isolated source re-home feature is enabled. - **A targeted sweep has no page limit.** Its at most 50 targets bound the matched workspaces to those tasks, their ancestors and their execution workspaces. - **A burst of N hints costs N/50 targeted runs, back to back.** The cost scales with activity, not with history. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.14 |
||
|
|
890dcbdd66 |
fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap: the outgoing server still serves while the incoming server starts new runs > - A warm agent session stays open in the server process that ran the last turn, after that run finished > - The incoming server took over a warm session while the outgoing server was alive and still held it, because the finished run's finalization lease ends at commit > - The takeover then failed, the task went blocked with a Retry, and in one case a retry loop ran 27 times on the same sandbox > - This pull request makes the incoming server wait while the server that holds the warm session is alive, makes the outgoing server close idle warm sessions as soon as it leaves new runs to the replacement, and stops the retry loop when the promised fresh sandbox does not come > - The benefit is that a follow-up during a deploy finishes without a person, and no Runner ever gets two controllers ## Linked Issues or Issue Description **What happened?** A zero-downtime release-swap test (Gate B, wave B1, mastercanary/v2026.1114.0-canary.13 |
||
|
|
45531b719e |
perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Stranded-work recovery checks tasks that are assigned to an agent
but have no live run. It runs as a scoped-deadline queue that a change
notification or a stored deadline wakes.
> - For a failed legacy run, the check calls
`terminalizeLegacyExecution` with the run's current status. When the
run's recovery decision is already recorded (for example "continue in a
fresh run from saved work"), that call changes nothing. But it still
updated the run row and notified about 18 delivery queues, including the
run's own recovery graph.
> - The graph notification re-checked the same task, which re-asserted
the same run again. The loop ran about once a second per waiting task,
forever, with zero runs active. On a hosted instance this shows up as
constant idle transactions.
> - This pull request makes a re-assertion that changes nothing return
before it writes or notifies.
> - The benefit is that an idle instance does no stranded-recovery work
for tasks that wait on a recorded decision. A real change (another run
fails, a comment, a status change) still notifies and recovers at once.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
With zero runs active, the server runs many transactions per second. A
CPU profile groups most samples under `reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`). A local fixture on real
PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent,
whose latest legacy run failed and whose recovery decision is recorded.
Each check reaches the branch that terminalizes a failed legacy run that
needs reconciliation (`service.ts`, the
`legacyExecutionNeedsReconciliationWithEvidence` branch, and the same
call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution`
then:
- updates the run with the same status (only `updated_at` changes);
- calls `notifyExecutionRunWork`, `notifyInstructionCollection` and
`notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the
task and about 18 queues;
- finds the recorded decision and returns.
The graph hint re-checks the same task after the queue's 1-second pass
interval, so the loop never ends.
**Expected behavior**
A check that finds nothing to change stores nothing and wakes nothing.
Only a real change re-checks the task.
**Steps to reproduce**
1. Create a task in `in_progress` assigned to an agent, with a failed
legacy run (`adapter_failed`).
2. Start the issue-state recovery work. The boot pass records a
fresh-continuation decision for the run.
3. Leave the instance idle. On master: 591,045 statements in 30 minutes
for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update
"heartbeat_runs"` per task per second.
**Paperclip version or commit**
master at
|
||
|
|
68d5c9c120 |
feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner can already keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started (#21423, #21316). Core can already wait for that turn (#21453, #21454, #21455). No run does this yet, because no controller asks the Runner for it. > - The request goes through the transport of the Runner package. This pull request gives the transport an option for it and the behavior that the option needs. > - No caller sets the option in this pull request. It changes no production run. > - The benefit is a small base for the two pull requests after it: the recovery after a controller restart, and the switch that turns the behavior on. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. **This pull request:** the transport can ask the Runner for provider-started turns. No caller asks. 2. #21612: the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It adds the support that parts 2 and 3 use. **Subsystem affected** Paperclip Runner package, the controller side: `src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a caller that sets the new option. **Current behavior** - The transport has no way to ask the Runner for provider-started turns. **Proposed behavior** - The transport options have a new field, `providerStartedTurns`. The transport uses it only for ACPX with the agent `claude`. In each other case the field has no effect. - With the option, the transport sends `providerStartedTurns: true` in `turn.start`. It does not send it for a compaction turn. - With the option, a tool call in a provider-started turn waits until the transport gave each earlier event to the driver. Without this, a `paperclip_finish` of that turn could reach the driver before the `turn.started` of that turn, and the driver would refuse it. - With the option, the transport gives a `run.terminal` of the Runner to the driver as a marked internal notification (`paperclip/runTerminal`). The driver uses it only during a wait, so a wait ends at once when the Runner says that the run is over. - With no option, the transport does the same as before this pull request. Live sessions, evals, the CLI runner and the probes do not set the option. **Reason and benefit** - The opt-in is a choice of the caller, not a property of the agent. Only a consumer that can wait for a provider-started turn must ask for one. `CapabilityLiveSession` completes a turn at the first `turn/completed`, so it must not get the behavior because its agent is Claude. ## What Changed - `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the option, the field in `turn.start`, the order of the tool call, and the `paperclip/runTerminal` notification. - `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the source sequence of a tool call that belongs to a provider-started turn. - `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7 cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the real consumer of Core. - `src/live/live-session.test.ts` (12 lines added): a Claude live session gives the transport no such option. - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only): the mode `provider-result-resolved`. In this mode the provider-started turn calls `paperclip_finish` and waits for the answer of Core through the Runner (`tool.resolve`), as the real sidecar does. The mode `provider-result` is not changed. ## State saved by the previous release - No persisted state changes in this part. - **A Runner that does not know the field** ignores it. - **Rollback of the Runner package:** the previous transport does not send the field, so the Runner does not wait. ## Verification At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and 3: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node --test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed in each. The cases: - background work, then the turn of Claude calls `paperclip_finish`; Core answers; the Runner gives the answer to the sidecar with exactly one `tool.resolve`; the turn completes; the run finalizes with one `run.result.accepted`; - a Claude run with no option from the caller takes the path of today: no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`, no wait, and no `session.provider_turn` to the sidecar; - the limit ends the wait, the run takes the path of today, and the provider processes had exited before the cleanup of the test; - the provider process is lost during the wait, and the run settles in seconds; - the sidecar exits during the wait, and the run settles in seconds; - a turn with no background work does not wait; - an ACPX agent that is not Claude is never opted in, also when the caller sets the option. - Each case asserts the result of the close of the session: clean in 6 cases, and a named failure in the case where the sidecar is killed. Each clean case also asserts that there is no `native_session_settlement_incomplete` diagnostic and no "conflicting duplicate tool result" in the durable state. - `src/live/live-session.test.ts`, `src/live/runnerd-codex-transport.test.ts`, `src/live/runnerd-acpx-provider-turn-wait.test.ts`, `src/native-session-runtime.provider-turn.test.ts`: 434 passed. - Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102 passed). - Mutation checks, run on the full change before the split: with the option ignored, 6 of the 7 assertions of the case with no option fail; with the old mode of the fake, the case with the result fails at the close of the session. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No caller sets the option. Core sets it in part 3. - **The transport is shared by each provider.** The three new behaviors are behind one condition: the option, ACPX, and the agent `claude`. With the condition false, the code path is the path of today. - **The tool-call ordering uses a 10 ms poll** while earlier events are not delivered, only in a provider-started turn. - **A controller restart during a wait or during a provider-started turn is not handled in this part.** Part 2 handles it. Part 3 must not merge before part 2. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.12 |
||
|
|
7db92ea319 |
perf(workspaces): read a sweep page's task trees and active runs in two statements (#21618)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep inspects candidate workspaces 50 per
page. For each workspace it read the task tree with its own recursive
query. For many of them it also ran one or two queries to check for an
active run.
> - The cost of a page therefore grew with the page: about 50 to 110
statements. On the hosted instance, building the SQL text of these
statements costs about as much main-thread time as parsing their
results.
> - The archive statement re-checks the terminal tree and the absence of
a live run in the same statement, and the remote and plugin cleanups
re-check active runs under the lifecycle lock. So these early reads are
only pre-filters.
> - This pull request reads the task trees and the active-run workspaces
of a whole page in two fixed-text statements.
> - The benefit is that a page costs a constant number of these reads,
whatever its size.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The per-page reads of the terminal workspace sweep
(`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).
**Current behavior**
Each candidate calls `listWorkspaceIssueTree`, one recursive query. A
candidate that reaches the active-run checks calls
`workspaceHasActiveRun`: one query for the linked tasks' run ids, then
one for live runs.
**Proposed behavior**
- `readPageIssueTrees` reads the tree rows of every root on the page in
one statement. Its SQL text is fixed: the roots are one `jsonb`
parameter. The trees are then built in memory and keep the same row
shape.
- `readPageActiveRunWorkspaces` returns, in one fixed-text statement,
the page workspaces that have a `queued` or `running` run on a linked
task.
- The two pre-filter checks in the loop use these results. A workspace
whose source task changed earlier in the same pass (rehome) still reads
its own tree.
- The authoritative checks are unchanged: the archive `UPDATE` with its
terminal-tree and live-run predicates, the active-run check inside the
remote and plugin cleanup under the lifecycle lock, and the
per-workspace check before a stale reopen flag is cleared.
**Reason and benefit**
One rotation pass should issue a bounded number of statements per page.
For a page of 50 workspaces (40 with open trees, 10 held by a run), the
tree reads fall from 50 to 1 and the active-run reads from 20 to 1.
## What Changed
- `server/src/services/execution-workspaces.ts`: add
`readPageIssueTrees` and `readPageActiveRunWorkspaces`, and use them for
the per-page pre-filter.
- `server/src/__tests__/terminal-workspace-page-batch.test.ts`: two new
tests on real PostgreSQL.
## Verification
- `reads the task trees and active runs of a page in a fixed number of
statements`: 2 companies, 50 workspaces, each with a child task, 10 held
by a `running` run.
- Before the change it fails: 50 tree reads.
- After the change: one tree read, one active-run read, fewer than 10
statements for the page. The results are the same (`checked: 50`,
`skippedNonTerminalTree: 40`, `skippedActiveRun: 10`).
- `does not archive a workspace whose run starts after the page was
read`:
- The page is read first. Then, during that workspace's close-readiness
check, a `running` run is inserted and set as its task's checkout run.
- The sweep counts the workspace `eligible: 1`, `archived: 0`,
`skippedRace: 1`, and it stays `active`.
- After the run succeeds, the next sweep archives it (`archived: 1`).
- `npx vitest run src/__tests__/terminal-workspace-page-batch.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/plugin-workspaces.integration.test.ts` (in `server/`).
- `pnpm check:module-boundaries` passes.
## Risks
- **The pre-filter reads happen once, at the start of the page.** A page
can take long, because cleanups do remote and Git I/O. A run that starts
mid-page on a workspace the pre-filter saw as idle is still caught by a
check made at the destructive step itself, in the same statement or
under a lock. Line numbers are on this head:
- **Archive** of an active workspace: the archive `UPDATE` in
`sweepTerminalWorkspaces` carries `NOT EXISTS (… live_run.status IN
('queued', 'running'))`
(`server/src/services/execution-workspaces.ts:4291`). It changes 0 rows
and the workspace counts as `skippedRace` (`:4340`).
- **Remote cleanup** (destroying the sandbox):
`runTerminalWorkspaceCleanup` re-runs `workspaceHasActiveRun` under the
lifecycle lock before `cleanupRemoteWorkspaceAllocation` (`:2702`).
- **Plugin workspace removal**: the same check under the lifecycle lock
before `removePluginWorkspace` (`:2706`).
- **Git cache expiration** and **isolated task directory expiry**:
`isGitWorkspaceExpirationEligible` re-reads the tree and refuses any run
that is not finished, inside the locked transaction
(`server/src/services/git-workspace-expiration.ts:311` and `:416`,
`server/src/services/isolated-task-directory-retention.ts:244` and
`:290`).
- A run that ends mid-page leaves its workspace for the run-release hint
or the next inspection.
- **Tree rows are now deduplicated** (`UNION` instead of `UNION ALL`).
Parent cycles therefore cannot repeat rows.
- Low risk otherwise. No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.11
|
||
|
|
081dacba1d |
perf(workspaces): inspect only due workspaces and wait for a holding run's release (#21617)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep stores one deadline per workspace: cooldown end, compaction, cleanup retry, Git idle. > - When any deadline was due, the worker ran a full rotation over every open candidate workspace. That costs a task tree read and more for each one. > - A workspace held by a `running` run kept a past deadline, so the 5-minute deadline rotation came back for it forever. A run that never finishes would keep the instance awake indefinitely. > - This pull request makes a due deadline inspect only the due workspaces. A held workspace waits for its run instead of for a timer. > - The benefit is that an idle instance does work only for workspaces whose deadline really arrived. The cost no longer grows with the number of open workspaces. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** How the terminal workspace worker (`server/src/services/terminal-workspace-work.ts`) acts on due deadlines and on workspaces held by an active run. **Current behavior** `hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest deadline is past, the next run starts a full keyset rotation, 50 workspaces per page with 1 second between pages, over every candidate. A workspace whose issue still has a `queued` or `running` run is skipped, but the archive deadline set earlier in the same pass stays. It is already past, so the rotation repeats every 5 minutes. **Proposed behavior** - When the run fires for a due deadline, the worker takes the due workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and runs a targeted inspection of only those workspaces. The service first clears the deadlines of the targeted workspaces; the inspection sets fresh ones. - If more than 50 workspaces are due, or a due deadline belongs to the isolated source, nested clone or task directory work, the worker still runs a full rotation. Startup still runs a full rotation. - A workspace skipped because a run holds it drops its deadline. - `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`) publishes `issue:<runIssueId>` to the terminal workspace queue when a run reaches a terminal status. That targeted inspection reaches the workspace the run held. - **Backstop.** About 200 code paths write `heartbeat_runs` directly (reapers, recovery terminalization, bulk cancels), and they do not pass through `writeRunStatus`. So the worker keeps a full rotation as a backstop, armed only by activity, like the recovery backstop from #21552: - Any terminal-workspace hint arms it. - Once armed, a full rotation runs at most once per hour (`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation. - An idle instance with no hint schedules nothing. - A missed release hint therefore delays archiving by at most one hour after activity, without a restart. - The hour is a compromise. On master today, the 5-minute rotation re-reads every candidate. On the hosted instance that is about 1,000 finished remote workspaces, roughly 3,000 statements per lap. The hourly backstop re-reads them at most once per hour while there is activity, about 1 statement per second averaged. **Reason and benefit** Background work must be driven by deadlines and change notifications, and its cost must not grow with the backlog. Every guarantee keeps a named trigger: - cooldown end, compaction, retry backoff and Git idle each have a stored deadline, and a due deadline inspects exactly that workspace; - the release of a holding run publishes a hint; - task status changes and workspace writes still publish hints; - startup still reconciles everything. ## What Changed - `server/src/services/execution-workspaces.ts`: - New `takeDueTerminalWorkspaceTargets()`. - A targeted inspection clears the deadlines of its targets before it inspects them. - A workspace skipped for an active run drops its deadline. - `server/src/services/terminal-workspace-work.ts`: a due run uses the due targets instead of a full rotation, unless they overflow or a non-workspace deadline is due. A due run that finds nothing due does nothing. Any hint arms the hourly backstop rotation. - `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run status hints the run's task to the terminal workspace queue. - `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new tests on real PostgreSQL. They use the real coordinator and worker on a fake clock. - `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616): its idle window now bounds the number of sweeps instead of requiring zero targeted sweeps, because a due deadline is now a targeted inspection. ## Verification - `inspects only the workspace whose cooldown ends, however many trees stay open`: 60 workspaces with open task trees and 1 done workspace whose 1-day cooldown ends 1 hour later. - Before the change: after startup, every due deadline starts a full rotation (27 full sweeps in 2 hours). - After the change: no full rotation; the done workspace is inspected alone (`checked: 1`) once its cooldown ends; fewer than 200 statements in 2 hours. - `waits for the release of the run that holds a workspace instead of rechecking it`: a done workspace held by a `running` run. - Before the change: a full rotation every 5 minutes (22 sweeps in 2 hours). - After the change: 0 statements in 2 idle hours. When the run is set to `succeeded` through the lifecycle, the workspace is inspected (`checked: 1`, `skippedActiveRun: 0`). - `leaves a thousand finished remote workspaces unread at idle`. This is the hosted state: 1,000 `active` remote workspaces with a location, a committed snapshot and a task done 5 days ago, and zero runs. - After startup and 2 idle hours: no full rotation. No remote workspace is read again; the only inspections are of the one due local workspace. Statements stay within 10 per inspection. - On master, every due deadline re-reads all 1,000. - `archives a workspace whose run ended outside the run lifecycle within the activity-armed backstop`: - A workspace held by a `running` run. The run is set to `failed` by a direct `UPDATE`, so no release hint is published, and one unrelated hint arrives. - The workspace is archived within one hour and one minute, without a restart. - With the backstop scheduling disabled, it stays `active`. - `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts src/__tests__/terminal-workspace-work.test.ts src/__tests__/execution-workspaces-service.test.ts src/__tests__/isolated-task-directory-retention.test.ts src/__tests__/git-workspace-expiration.test.ts src/__tests__/workspace-delivery-cleanup.integration.test.ts src/__tests__/nested-clone-migration.test.ts src/__tests__/execution-workspace-compaction.test.ts src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333 tests pass. - `pnpm check:module-boundaries` passes. Local fixture with #21616 applied as the base, 30 idle minutes on a fake clock, real PostgreSQL. Each fixture holds open trees, a stranded `running` holder, cooldown, done local workspaces with missing files, closed remote workspaces with and without a committed snapshot: | Workspaces | Before: sweeps | Before: statements | After: sweeps | After: statements | | --- | --- | --- | --- | --- | | 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 | | 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 | After the change, no sweep reads the task tree of a workspace whose tree is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420 workspaces). What remains comes only from workspaces with a due retry or inspection backoff, so it grows with the number of stuck workspaces, not open ones. Per inspected workspace a pass issues the same 3–6 statements as before; batching them per page is a separate pull request. ## Risks - **A run that leaves `running` without `writeRunStatus`** does not hint. The activity-armed backstop catches it within one hour of activity, and startup reconciles. This pull request does not decide that a stranded run is gone; it waits for a status change. - **A workspace whose deadline is reset to "now"** (a reopen-pending flag within its grace period, task directory budget overflow) is inspected alone on each due run. The inspection backoff in `nextTerminalWorkspaceSweepAt` still spaces these runs. - **Workspaces without a stored deadline** (for example an undelivered workspace whose deadline was deleted) are inspected again only when a hint or the startup rotation reaches them. Before this change they were also reached by the rotation that a different workspace's deadline started. ## Open decision **Should a terminal workspace whose local files are gone be archived, or should it wait?** This pull request does not change the behaviour; it leaves the decision to the owner. Today such a workspace fails its close-readiness inspection and is reported as `skippedUndelivered`. A non-remote workspace keeps its past deadline, so the 5-minute inspection backoff re-inspects it again and again. That costs about 3–4 statements per stuck workspace every 5 minutes (from the local fixture: 255 inspections, 866 statements), and it never ends while the files stay missing. The cause is typically a replaced container. The two options: - **Archive it.** Nothing remains on disk to keep. - **Store a "files missing" state.** It is then inspected again only when a write announces a change. Either option removes this cost. Archiving also removes the row from later rotations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.10 |
||
|
|
20ccf03676 |
fix(workspaces): settle closed remote workspaces after one confirmed destroy (#21616)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server cleans up closed execution workspaces in a background sweep (`terminal-workspaces`). For a remote workspace, cleanup destroys the sandbox lease after it confirms that the saved snapshot is committed. > - A closed remote workspace stays a sweep candidate while `metadata.remoteWorkspace.location` is an object. A confirmed destroy did not clear that location. > - The cleanup write goes through the lifecycle gateway, and the gateway published a `workspace:<id>` hint for the same workspace. That hint started a new inspection, which cleaned the workspace again and published again. > - On a hosted instance with no runs, this loop kept three statements at about 15–24 per second each and a dedicated lock connection per inspection. Every pass also asked the sandbox provider to destroy a sandbox that was already destroyed. > - This pull request clears the location after a confirmed destroy, skips the provider call when the stored lease already has a destroy receipt, and stops the sweep from hinting itself for its own writes. > - The benefit is that an idle instance with closed remote workspaces settles: each workspace is cleaned once, and the provider is called at most once per lease. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** With no runs, the terminal workspace sweep never became idle. On the hosted instance these statements ran all the time: `workspaceHasActiveRun` (22.8/s), the task tree read (22.6/s) and the snapshot persistence read in `inspectRemoteWorkspacePersistence` (14.8/s). `SET application_name` for the `paperclip-workspace-finalization-lock` connections ran at 9.6/s. **Steps to reproduce** 1. Create closed (`archived`) remote workspaces that have a committed snapshot, a clean authority and a `location`. 2. Start the terminal workspace worker with no other activity. 3. Watch the sweeps. Each cleanup reports `archived`, the gateway publishes `workspace:<id>`, and the next targeted inspection cleans the same workspace again. Locally, a fixture with 140 workspaces (40 of them in this state) ran 568 sweeps, 136,888 statements and 23,880 lease destroy calls in 30 idle minutes. The destroys always hit the same 40 workspaces. **Expected behavior** A closed remote workspace is cleaned once. After the destroy is confirmed it leaves the candidate set, and an idle instance issues no statements between real deadlines. ## What Changed - `server/src/services/remote-workspace-lifecycle.ts` (`cleanupRemoteWorkspaceAllocation`): - Read the stored lease first. When it already carries a destroy receipt for the same environment, lease and provider lease, do not call `destroyLease` again. - After a confirmed destroy, set `metadata.remoteWorkspace.location` to `null`. One fixed-text `jsonb_set` update does this, and only when the stored location is still the location that was destroyed. - `server/src/services/execution-workspaces.ts`: the gateway takes `sweepOwned`. The sweep's own cleanup, cleanup-failure and reopen-flag writes set it, and then the gateway does not publish a terminal workspace hint for that workspace. All other gateway callers still publish. - `server/src/__tests__/terminal-workspace-idle.test.ts`: two new tests on real PostgreSQL. ## Verification - `settles after one cleanup of each closed remote workspace and inspects nothing on its own writes`: - Fixture: 40 workspaces (open trees, a stranded `running` holder, cooldown, done, closed remote with a committed snapshot, closed remote without a snapshot). - Runs the real coordinator and worker on a fake clock for 2 minutes, then 30 idle minutes. - Asserts: each closed remote workspace is destroyed exactly once; no targeted inspection in the idle window; fewer than 5,000 statements in 30 minutes; every cleaned workspace has `location: null`. - Before the change it fails: 850 destroy calls instead of 10 in the warm-up alone. After the change it passes. - `clears the location of an already destroyed lease without calling the provider again`: the stored lease has a destroy receipt. Before the change, the sweep calls `destroyLease`. After the change the provider mock is never called, the location is cleared, and the next inspection finds no candidate. - `npx vitest run src/__tests__/terminal-workspace-idle.test.ts src/services/remote-workspace-lifecycle.test.ts src/__tests__/terminal-workspace-work.test.ts src/__tests__/execution-workspaces-service.test.ts src/__tests__/workspace-delivery-cleanup.integration.test.ts` (in `server/`). Local fixture, 30 idle minutes, 140 workspaces (40 closed remote with a committed snapshot): | | Before | After | | --- | --- | --- | | Sweeps | 568 (all targeted, continuous) | 20 (one deadline rotation every 5 minutes) | | Statements | 136,888 | 1,371 | | Lease destroy calls | 23,880 | 40 (once per workspace) | | `workspaceHasActiveRun` reads | 45,440 | 260 | | Snapshot persistence reads | 22,720 | 40 | The remaining rotation every 5 minutes comes from deadlines of workspaces that cannot make progress (a stranded active-run holder, cleanup backoff). A separate pull request handles it. ## Risks - **Who reads `remoteWorkspace.location`.** For a closed, clean workspace with a committed snapshot and a destroyed lease, none of these readers needs the location: - the live run binding checks (`remote-workspace-lifecycle`, `fresh-continuation-fallback`, `workspace-recovery-fallback`, `workspace-recovery-source`) need a current writer or a dirty run reference; - the abandonment and hold paths (`remote-workspace-epoch-abandonment`, `remote-workspace-abandoned-files-hold`, `remote-workspace-unchanged-proof`) need a dirty or held allocation; - `reusable-sandbox-resume-replacement` treats `null` as no conflicting sandbox; - `remote-workspace-files` reads the saved snapshot when the location is `null`. With a destroyed location it would fail as unavailable. - The lease-release path (`clearRemoteWorkspaceLocationForReleasedLease`) already writes the same `null` for the same state. This change makes the sweep stop depending on that side effect. - **Provider calls.** Before this change every loop pass re-claimed the lease and called the driver's `destroyRunLease` again, through `destroyClaimedSandboxLease` in `server/src/services/environment-runtime.ts`. After it, a lease with a stored destroy receipt causes no provider call. - **Existing backlog after deploy.** The startup rotation and the next deadline rotation visit each closed remote workspace that still has a location once, 50 per page with 1 second between pages. A lease that already has a destroy receipt costs no provider call, and its location is cleared. Only leases without a receipt get one destroy call. The workspaces then leave the candidate set. - **No self-hint.** The sweep already handles the workspace it writes, and an inspection's deadline bookkeeping is unchanged. Other writers still publish hints. The git restore point and task directory paths depend on task or explicit hints, not on these writes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.9 |
||
|
|
c1451814c1 |
perf(chat): scope GitHub review check run hints to the issue and throttle company-wide rebuilds (#21610)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel keeps each unresolved "Paperclip Review" check in step with its task and run. A background queue (`chat-github-review-checks`) rebuilds its checks when a hint arrives > - Each run milestone (`notifyChatRunMilestones`) sends this queue a company-wide hint, also when the milestone knows the issue. Each company-wide hint rebuilds every unfinished check of the company and reads its full state again. Passes are limited to one a second > - So in a company with a steady stream of run milestones, each unresolved check reads its whole state about once a second, although nothing about it changed > - On a hosted test database the five state statements of a review check ran 10.3 times a second each. #21608 removes the endless 5-second head poll. This is the second cause > - This pull request sends the issue with the hint, rebuilds only the checks of that issue, and rebuilds for a company-wide hint at most once every 30 seconds > - The benefit is that run activity no longer makes every unresolved check of the company read its state each second. A run change of the check's own task still publishes at once ## Linked Issues or Issue Description Refs #21608 (backs off the head poll of an unresolved check; the other cause of the same hosted load). Refs #21580 (the same 30-second rule for the GitHub response sweeps). Refs #18384 (the same rule for external objects). **What happened?** On a hosted test database with zero active runs, the five statements of `githubReviewCheckState` (`chat_endpoints`, `chat_github_reviews`, `chat_message_links`, the `heartbeat_runs` `wakeCommentId` lookup and `chat_deliveries`) ran 10.3 times a second each. One source is the rebuild of the review check queue for company-wide hints. **Current behavior** `notifyChatRunMilestones` sends `{"companyId": ...}` to `chat-github-review-checks` for every milestone, also when its scope has an issue. The queue scope has no issue field. So each milestone rebuilds every received and retryable check of the company, and calls `githubReviewCheckState` for each received check. **Proposed behavior** A milestone with an issue sends `{"companyId", "issueId"}`. The queue matches a check to an issue through the issue of its latest review for the pull request head. This is the same review that `githubReviewCheckState` reads. A company-wide hint (company ID only) rebuilds at most once every 30 seconds. All narrower hints, the startup scope (`*`) and saved deadlines are not changed. **Steps to reproduce** 1. Admit a GitHub review check that stays unresolved (a review with no run that the check can find). 2. Send a run milestone for the company once a second (for example, status changes of other runs). 3. Count statements. Each hint rebuilds the check and reads its full state. ## What Changed - `review-check-work.ts`: the scope has an optional `issueId`. `page()` reads the issue of the latest review for the check head (`reviewIssueId`, through the existing `chat_github_reviews_pull_idx` index) and filters on it for an issue scope. `matches()` uses the same value. - `review-check-work.ts`: set `reconstructionIntervalMs` to `githubReviewCheckReconstructionIntervalMs`: 30 seconds for a company-only scope, 0 for all other scopes. - `chat-run-milestone-notifications.ts`: send the issue with the review check hint when the milestone has one. - `chat-channels.integration.test.ts`: two regression tests on the real queue, the real `dispatch` and embedded PostgreSQL with a controlled clock. The test helper now counts only the state read of `chat_github_reviews`, not the new subquery in `page()`. ## Verification Red/green regression tests: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "company-wide hints at most once per interval|at once from its issue milestone"` - Red (without the fix): `expected [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ] to deeply equal [ 1, +0, +0, +0, +0, +0, +0, +0, …(2) ]`. Each of 10 company-wide hints read the check state. The milestone hint was `{"companyId"}` only. - Green: 2 of 2 pass. The first company-wide hint reads the state once. The next 9 read nothing. After 30 seconds the waiting hint reads it once. A hint for the check's issue reads it at once each time. A hint for another issue or another company reads nothing. A real `notifyChatRunMilestones` with the check's issue publishes `in_progress` on the next pass and resets the head poll interval. Statement counts for one unresolved check with one company-wide hint each second (120 simulated seconds, real queue and `dispatch`, embedded PostgreSQL): | | Statements a second | Check state reads a second | | --- | --- | --- | | Before | 6.0 | 1.0 | | After | 0.2 | 0.033 | In the real-coordinator reproduction of #21608, one hint each second raised one stuck check from 0.70 to 1.67 state reads a second and from 8.4 to 15.2 statements a second. Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: passed, including all review check queue tests. - `cd server && npx vitest run src/__tests__/native-status-reassessment-work.test.ts src/__tests__/chat-run-milestone-work.test.ts src/services/native-runtime/external-chat-wait.integration.test.ts src/services/chat-github-review-repair.test.ts src/__tests__/idle-completed-history.test.ts src/__tests__/chat-admission-condition.test.ts src/__tests__/app-private-hostname-gate.test.ts`: 518 passed. - `cd server && node ../scripts/run-tsc.mjs` and `node scripts/typecheck-layers.mjs` (with test layers): passed. - `pnpm verify --explain` retains 2,718 test files conservatively, so I did not run it locally. Required CI supplies that coverage. ## Risks - A change that only a company-wide hint reports can take up to 30 seconds to reach a check. Company-wide hints now come from milestones without an issue (a run with no task, a comment merged into an active run) and from company authority changes. A run with no task cannot be the run of a review check. For a merged comment, the run's next status change sends an issue hint. The saved head poll deadline still applies. Each dispatch reads the full state again before it publishes. - An issue hint finds a check through the issue of its latest review for the head. If a run of another issue were the check's run, its change would wait for the next company-wide rebuild (30 seconds at most) or the next head poll. Run status and run identity milestones use the run's task issue, which is the review's issue for a GitHub review task. - The deadline reconciliation snapshot reports the queue as not reconciled while a company-wide hint waits (up to 30 seconds). This is the same as the GitHub response sweeps and the external-object refresh sweep. - `page()` reads one more indexed subquery for each row. It uses `chat_github_reviews_pull_idx`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local embedded PostgreSQL reproductions). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.8 |
||
|
|
a7453eba60 |
perf(chat): back off GitHub review check head polls while the check cannot change (#21608)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel publishes a "Paperclip Review" check for each admitted pull request head. A background queue (`chat-github-review-checks`) keeps each unresolved check in step with its task and run > - While a check is queued or running, the queue polls GitHub for the pull request head every 5 seconds. This finds a push whose webhook was lost > - The poll never stops while the check is unresolved. A check whose run never comes, or whose run the check cannot find, stays queued forever. It re-reads its whole state and calls GitHub every 5 seconds, also when nothing changes > - On a hosted test database with zero active runs, the five state statements of this check ran 10.3 times a second each > - This pull request doubles the head poll interval for each poll that finds the same state, up to 5 minutes. A state change resets it to 5 seconds > - The benefit is that an idle unresolved check costs about one poll each 5 minutes, not one each 5 seconds. Local changes still publish at once ## Linked Issues or Issue Description Refs #17662 (added the 5-second head reconciliation for unresolved checks). Refs #21580 (the same kind of fix for the GitHub response sweeps). **What happened?** On a hosted test database (Neon, 488-second window, zero running or queued runs, 2 runs created in 30 minutes), these five statements ran at exactly 10.3 a second each: the `chat_endpoints` source read, the `chat_github_reviews` read, the `chat_message_links` comment read, the `heartbeat_runs` `wakeCommentId` lookup and the `chat_deliveries` state read. All five come from `githubReviewCheckState` (`server/src/services/chat-channels/github/review-check-state.ts`). **Expected behavior** A check whose state cannot change makes no recurring database reads or GitHub calls until something that can change it happens: a local hint, a lease expiry, or a saved deadline. The missed-webhook head check stays, at a bounded cost. **Steps to reproduce** 1. Admit a GitHub review check. Bind a task review to it, but give it no run that the check can find (for example, the run was never created, or its wake comment is not the delivery's comment). 2. Start the review check queue and leave the instance idle. 3. Count statements. The check is due again 5 seconds after each poll, for ever. Each poll evaluates the check state about 3 to 4 times (before dispatch, in dispatch, after dispatch, and in the rebuild that the publication lease release starts). Each poll also asks GitHub for an installation token and the pull request. **Cause** `dispatch` (`server/src/services/chat-github-checks.ts`) saved `retryAt = now + 5s` after each poll. `entry()` in `review-check-work.ts` uses that deadline when the state is unchanged. So an unresolved check was due every 5 seconds without end. ## What Changed - `chat-github-checks.ts`: save `unchangedPolls` in the check result. A poll that finds the same state as the saved state adds one (up to 6). A changed state, a first publication and a recovered failure set it to 0. The next head poll is due after `min(5 minutes, 5 seconds × 2^unchangedPolls)`. - Export `GITHUB_REVIEW_CHECK_HEAD_POLL_MS` (5 seconds) and `GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS` (5 minutes). - `chat-channels.integration.test.ts`: two regression tests. They use the real review check queue, the real `dispatch`, embedded PostgreSQL, a controlled clock and one pass each second. The fixture has the hosted state: a review with no run, a delivery with a linked comment, and no run for that comment. ## Verification Reproduction with the real delivery coordinator, the real queue and embedded PostgreSQL. One queued check with no findable run, 30 seconds of real time, before this change: | Company-wide hints | Statements a second | Check state reads a second | GitHub pull reads a second | | --- | --- | --- | --- | | None | 8.4 | 0.70 | 0.17 | | One each second (`notifyChatRunMilestones`) | 15.2 | 1.67 | 0.17 | So one stuck check costs about 0.7 state reads a second from the head poll alone. The hosted rate (10.3 a second) is consistent with about 15 such checks, or fewer checks with frequent company-wide hints. Company-wide hints are a separate cause and a separate pull request. Red/green regression test, 600 simulated seconds with one pass each second: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "unresolved GitHub review check cannot change|publishes a local change at once"` - Red (without the fix): `expected 120 to be less than or equal to 7`. Over 600 seconds: 5,160 statements, 360 check state reads, 120 GitHub pull reads. - Green: 258 statements, 18 check state reads, 6 GitHub pull reads (8.6 → 0.43 statements a second, a 95% drop). The saved interval stops at 5 minutes. - Correctness (green): after the back-off, a restarted queue makes no early GitHub call. A run bound to the review publishes `in_progress` at the next pass and resets the interval. A pushed head is still found within 5 minutes, the check is cancelled and the review is superseded. Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed, including all 12 review check queue tests. - `cd server && npx vitest run src/services/chat-github-review-repair.test.ts src/__tests__/idle-completed-history.test.ts src/__tests__/chat-admission-condition.test.ts src/__tests__/app-private-hostname-gate.test.ts`: 109 passed. - `cd server && node ../scripts/run-tsc.mjs` and `node scripts/typecheck-layers.mjs` (with test layers): passed. - `pnpm verify --explain` retains 2,718 test files conservatively, so I did not run it locally. Required CI supplies that coverage. ## Risks - A push whose webhook is lost is found later: after up to 5 minutes, not 5 seconds, when the check has not changed for a while. The new head's own webhook still creates its new check at once. Only the old head's check stays queued or running on GitHub a little longer before it is cancelled. - Local changes are not delayed. `entry()` still makes a check due at once when its state differs from the saved state, and run, review, enqueue and lease hints still rebuild it. - Saved actions from an older release have no `unchangedPolls`. They start at 0. An invalid saved value also starts at 0. No migration. - Restart recovery is unchanged: the deadline is saved on the action, and startup rebuilds all unfinished checks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local embedded PostgreSQL reproductions). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7c0c3c4f31 |
fix(mcp): preserve Tasks sleep authority and re-enable recovery (#21603)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected apps can retain unfinished provider work through MCP Tasks. > - Managed defaults can enable Tasks without a stored experimental flag. > - The sleep check used the stored flag and could miss an active provider task. > - Enabling Tasks also failed to notify an owner that had become idle. > - This pull request uses the effective setting and wakes the existing owner after the setting commits. > - Pending work keeps its recovery path. Completed history can become quiet. ## Linked Issues or Issue Description Refs #21263, which added durable provider Tasks. The current roadmap covers the governed MCP Tool Gateway. This change repairs two existing paths. **What happened?** The gateway could enable MCP Tasks through managed defaults while the idle check reported no pending work. After an operator disabled and enabled Tasks, an existing task could stay idle until a separate task write woke its owner. **Expected behavior** The gateway and sleep check must use the same effective setting. Enabling Tasks must notify the existing owner. It must retain the task's original deadline and backoff. **Steps to reproduce** 1. Retain a task with `next_poll_at`. Use managed Tasks defaults without a stored true flag. Read the normal sleep projection. 2. Disable Tasks, retain the same task, then enable Tasks through the settings service without writing the task again. ## What Changed - Read the effective Tasks flag inside the same read-only sleep transaction. Bind that value to the existing pending-task query. - Register the existing Tasks queue notification before the settings transaction writes. The normal commit protocol owns the wake. - Add two real-PostgreSQL regressions to the existing nine-case suite. Extend eight mock transactions with the read projection they now need. Keep their assertions unchanged. - Publish the Tailscale enrollment fixture state by rename so its CLI watcher cannot parse a truncated JSON file. Existing enrollment assertions and time limits remain unchanged. ## Verification The two regressions fail against the preceding current-master products: the pending-work projection reports `none`, and re-enable does not rearm the owner within the original two-second oracle. After the fix, all nine PostgreSQL cases pass at the recorded source, including completed history, disabled Tasks, credential debt and fail-closed query errors. At current `72bc32f`, the normal DB build and migrations, all nine PostgreSQL cases, 155 affected mock cases, and the complete normal Server typecheck/planner pass. The mock run includes three selected safety families and six complete files; it explicitly excludes 196 existing durable-PG cases. All original case deadlines and assertions remain. The startup file's earlier 133-case pass is limited to its unchanged mock contract; a new hot-restart implementation is not relabeled as tested. Root independently verified all current outputs and process closure. The MCP product and regression files remain byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale test fixture repair below. Exact current-head CI passes: 55 successful checks, four explicit skips and one neutral result; Snyk succeeds. Fresh Greptile is 5/5 for this exact head. The complete latest edited summary and all paginated threads were inspected, with no open findings. An earlier additional whole-eight-file run reached its 180-second aggregate watchdog while the existing mixed PostgreSQL suite was still running. It produced no individual failing assertion. Its startup file passed 133 cases. The original failure remains recorded. The follow-up selects the three affected mock families and runs the other affected mock files whole; it does not claim that the interrupted PostgreSQL file passed. No test assertion or case deadline changed. Full required CI remains a merge gate. The first remote run at `72bc32f` passed 51 checks but failed one Tailscale enrollment case and an interrupted browser shard; two aggregate gates then failed. The browser log records a runner shutdown signal; its underlying cause is unknown, and one specific-job recovery was requested. The Tailscale case failed its status CLI on both original CI attempts. A local control then held publication until the original file watcher observed an empty state: the CLI threw `Unexpected end of JSON input`, and the existing enrollment assertion failed. Publishing the state atomically passed the same control. All nine original enrollment cases now pass in 4.393 seconds with the fixture repair. The recorded remote interleaving remains unknown; the original logs and the initial local non-reproduction are retained. No retry count, assertion, timeout or product behavior was weakened. The later exact-head run passes all required checks. A subsequent server shard was interrupted by a runner shutdown and later recovered at the same source; the interruption and unknown underlying cause remain recorded. Use Node24/pnpm9, one worker, an owned external PostgreSQL test database and the existing fixture/admin URL options. The two new regressions are in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`. These checks do not establish deployed suspension or a live provider's availability. ## Risks - The read adds the normal effective-settings projection to the existing bounded snapshot. Query errors must still block sleep. - A settings update now notifies the existing task owner. Transaction rollback must not publish that notification. - This adds no schema, provider protocol, timer or replacement task framework. Existing active task polling remains required until its work settles. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix. GPT-6-based Codex also reviewed and integrated it. The host does not expose every supervising model ID or context-window size. The agents used source editing, Git, code execution and regression tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (the focused scope above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the existing pending-task contract remains accurate) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
aaec67b891 |
fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure
## Linked Issues or Issue Description
**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.
I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.
**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.
**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.
**Paperclip version or commit**
`master` at
canary/v2026.1114.0-canary.7
|
||
|
|
12ac7cfd9f |
perf(recovery): read only the run keys that active execution recovery checks use (#21602)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Live recovery checks each running run when recovery work wakes it. The check is `recoverActiveExecutions` in `server/src/services/heartbeat/recovery.ts`. > - Each check read the full `heartbeat_runs` row and the agent's `adapter_config`. That includes `context_snapshot` and `runner_profile_json`, which are large JSON columns. > - Most checks only defer the run to its lease deadline. They use a few JSON keys and never use the rest of the row. > - Recovery wakes come from runner checkpoints, run events and company work signals. Each running run is checked about 4.5 times per minute under that load, so this read grows with the number of running runs. > - This pull request reads only the JSON keys that the check gates use. A run that recovery settles as lost reads its full context, runner profile and adapter config before the settle. > - The benefit is about 97% fewer bytes per check, with the same recovery decisions. ## Linked Issues or Issue Description **What existing behavior does this improve?** The read that live recovery does for each running run (`recoverActiveExecutions`). **Subsystem affected** Server: heartbeat run recovery (`server/src/services/heartbeat/recovery.ts`). **Current behavior** Each check selects the whole `heartbeat_runs` row plus `agents.adapter_config`. With a realistic row (`context_snapshot` 115 KB of text, 23 KB stored; `runner_profile_json` 48 KB of text, 10 KB stored), one check receives 175 KB from Postgres. A server with 20 running runs did 90 checks per minute, which is 15.7 MB per minute from this one statement. That was 99% of all bytes in the measured window. **Proposed behavior** The check selects all scalar run columns, a small JSON object with only the keys its gates read, and no output excerpts or adapter config. The process-lost settle path reads the full `context_snapshot`, `runner_profile_json` and `adapter_config` for that one run before it decides on a retry. **Reason and benefit** The staging server spends 21-24% of main-thread CPU on postgres.js result parsing at about 26 KB per statement. Whole-row reads in recurring loops cause that overfetch. This change removes about 171 KB from each check. **Breaking changes** None. No schema, API or contract change. **Additional context** I examined the `+1 s` re-arm (when revocation finds a child process that the inventory snapshot did not have). I kept it. It only fires when the revocation sees a changed row. The revocation also writes a new 60-second lease for a random boot ID, so the next pass defers on that live lease. It does not loop on unchanged state. In the 60-second measurement it did not fire. ## What Changed - `recoverActiveExecutions` selects `activeRecoveryRunColumns`. This is all `heartbeat_runs` columns, except that `context_snapshot` holds only `wakeReason`, `issueId` and `nativeQuestionCancellation`, `runner_profile_json` holds only `nativeStartupContext`, `nativeWorkspaceSync` and `remoteWorkspace`, and the stdout/stderr excerpts are null. It does not select `agents.adapter_config`. - New `readActiveRecoverySettlement` reads the full `context_snapshot`, `runner_profile_json` and `adapter_config` of one run. The process-lost settle uses that data for the conversation-adapter check, the retry budget, the process-loss diagnostic, the stop metadata and the first-heartbeat flag. - New tests in `server/src/services/heartbeat/recovery.test.ts`: - A regression test counts the bytes Postgres sends during one deferred check through a TCP proxy. It must be below 32 KB. - A settle test (two cases) gives a lost run its retry budget, first-heartbeat flag and conversation adapter only in the JSON keys outside the narrow read. The test checks that the settle still uses them. - A test checks that a run with `nativeQuestionCancellation` in its context is still skipped. ## Verification - Red/green: the byte test fails on `origin/master` (`expected 454426 to be less than 32768`) and passes with this change. The other three new tests pass both before and after. - Focused local runs pass: `recovery.test.ts`, `live-recovery-work.test.ts`, `native-startup-cleanup-recovery.test.ts`, `ownership-reconciliation-report.test.ts` (4 files, 255 tests). - `pnpm verify` selected 2,715 of 3,541 files (conservative retention). On this machine 1,689 files passed and 32 failed. The failures were `afterAll` hook timeouts (30 s) and unrelated tool-access, email and runner artifact suites under local load. The recovery suites above pass when run alone. Required CI gives the full coverage. - Before/after measurement (local embedded Postgres, a wire proxy that assigns each server response to its statement, the real live-recovery scheduler, 60 s window, 20 running runs: 10 native runs held by a live controller and 10 legacy runs with a live peer lease; a run recovery signal every 5 s for each run, a company work signal every 10 s, legacy lease renewal every 20 s): | | Before | After | | --- | --- | --- | | Checks per minute | 90 | 90 | | Bytes per check (inventory statement) | 174,954 | 4,470 | | Inventory bytes per minute | 15,745,819 | 402,298 | | All bytes per minute in the window | 15,848,137 | 504,640 | ## Risks - Low risk. The narrow read keeps the keys that the gates read. The settle path reads the full row data before it uses it. - The settle path now reads `context_snapshot`, `runner_profile_json` and `adapter_config` after the lease revocation, not at the start of the check. The values can be some milliseconds newer. The revocation already uses the committed row state, so this agrees with the rest of the settle. - The settle path has one more query. It only runs when a run is settled as lost. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as an agent in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
30fc64e212 |
fix(instructions): collect a waiting working copy as soon as its stop proof is written (#21596)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. A queue collects the copy after the provider is proven to
have stopped.
> - #21586 stopped this queue from re-reading copies forever: a copy
that waits for proof of the stop is now checked again after half of its
wait, capped at 6 hours.
> - With only that change, a copy whose stop is recorded T after its run
finished is collected up to T/2 later (at most 6 h). Before #21586 this
took at most 30 seconds. A sandbox that takes minutes to stop therefore
delays the save of an agent's instruction changes, and copies of
stranded runs wait up to 6 h after their stop is finally recorded.
> - This pull request makes the write that records the proof bring the
run's waiting copy forward, so the queue checks it on its next pass.
> - The benefit: collection is prompt again when the proof arrives,
without polling copies that are still waiting.
## Linked Issues or Issue Description
Follow-up to #21586. No public issue exists. Bug description:
**What happened?**
After #21586, a working copy waiting for proof of the provider stop
keeps its saved backoff deadline when the proof arrives. Proof is a
`native.local_process_stopped` event or a remote lease termination
receipt. A copy whose run finished 30 days ago is collected up to 6
hours after its stop is recorded.
**Expected behavior**
When a stop proof for a run is written, that run's waiting copy is
checked on the next queue pass. A write for a run without a waiting copy
costs one indexed no-op UPDATE and sends no work signal.
**Steps to reproduce**
1. Create a working copy for a native run that finished 30 days ago and
has no stop event, then let the collection queue check it once. Its next
check is about 6 h later.
2. Record the local stop (`recordNativeLocalProcessStopInTransaction`),
or release the run's remote lease with a termination receipt.
3. On master the copy stays in `pending_collection` until the old
deadline.
**Paperclip version or commit**
`origin/master` at
|
||
|
|
0ab63cf4d9 |
fix(restart-chaos): ignore a done the server rejected as incomplete (#21598)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs staging-mix waves and grades each task for duplicated side effects and extra turns > - The grader counts a run as a completer when its last accepted result reports `done` > - The server can reject that `done` as incomplete. The status arbiter then queues a same-agent continuation wake with key `native-completion-incomplete` > - The grader ignores the rejection. It reports a false duplicate completion and anchors completion on the rejected run, so the continuation run that finished the work shows as an extra turn > - This pull request marks a `done` as rejected when a `native-completion-incomplete` wake for the issue follows it, and ignores that `done` unless the same run posted the DONE comment > - The benefit is that wave results stop reporting false duplicates and false extra turns, so real regressions stay visible ## Linked Issues or Issue Description **What happened?** In wave `w-20261114T081656-s7105`, task t005, run d80d3ab3 reported `done` with no DONE comment. The server rejected it as incomplete and queued a `native-completion-incomplete` continuation wake. Run a777038c then posted the only DONE comment and completed the issue. The grader reported "duplicate completion: runs d80d3ab3, a777038c" and listed a777038c as an extra turn. **Expected behavior** The grader reports one completer (a777038c), anchors completion on a777038c, and lists no extra turn. **Steps to reproduce** 1. Build wake evidence where run A has an accepted `done` result and no DONE comment. 2. Add a continuation wake with payload `continuationIdempotencyKey: "native-completion-incomplete"` created after A's result. 3. Add run B from that wake. B posts the DONE comment and reports `done`. 4. Grade the task with `gradeStagingMix`, `completedAtAnchor` and `runsAfterDone`. **Additional context** This is a test-harness change only. It changes no product code. ## What Changed - `tests/restart-chaos/follow-up-delivery.ts`: add `rejectedDoneRuns`. It attributes each `native-completion-incomplete` wake to the latest run whose accepted `done` came at or before the wake. - `tests/restart-chaos/wake-evidence.ts` (`completionEvidence`) and `tests/restart-chaos/staging-mix-retry.ts` (`evidenceCompletion`): set a new optional `doneRejected` flag on each completion run. - `tests/restart-chaos/staging-mix.ts`: `completedAtAnchor` and `completionRunIds` ignore a rejected `done`, unless that run posted the DONE comment. - `tests/restart-chaos/staging-mix-retry.test.ts`: add the t005 red/green test, a test for a rejected run that posted DONE itself, and a G5c t004 guard. - `tests/restart-chaos/README.md`: document the rule. ## Verification - Red on master, green with the fix: `a done the server rejected as incomplete > anchors on the run that finished the work, with one completer and no extra turn (w-20261114T081656-s7105 t005)`. On master the anchor is the rejected run's result time (1794646168026) instead of the finisher's (1794646218506). - Guard, green on master and with the fix: `a done the server rejected as incomplete > keeps a run woken by recovery after a real completion as an extra turn and a duplicate completion (G5c t004)`. - `npx vitest run --config tests/restart-chaos/vitest.config.ts --maxWorkers=3`: 71 files, 816 tests pass. - `tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - The real t005 evidence from the wave now grades as one completer (a777038c) with no extra turns. - `pnpm verify` exits 2 because `follow-up-delivery.ts` is a global CI input. Required CI supplies the broad coverage. ## Risks - Low risk. The change affects only the restart-chaos grader. - The grader trusts the wake key. If the server queued `native-completion-incomplete` for another reason, the grader would skip the latest `done` before it. The server queues this key only from the incomplete branch of `arbitrateNativeStatus`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0b2b7aa737 |
chore(lockfile): refresh pnpm lockfiles (#21604)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>canary/v2026.1114.0-canary.6 |
||
|
|
d25667f370 |
perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly
## Linked Issues or Issue Description
**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.
The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.
**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.
**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.
**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.
## What Changed
- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:
```diff
remove: (x) => {
- const index = xs.indexOf(x)
- return index === -1
+ const at = xs.indexOf(x, index)
+ return at === -1
? null
- : (xs.splice(index, 1), x)
+ : (xs.splice(at, 1), x)
},
...
if (index === xs.length) {
index = 0
xs = []
+ } else if (index > 1024 && index * 2 > xs.length) {
+ xs = xs.slice(index)
+ index = 0
} else {
xs[index - 1] = undefined
}
```
The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:
| Measurement | Before | After |
| --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |
## Risks
- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
009c8884a4 |
perf(attention): read blocker retention state once per page (#21595)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items. One
kind is `blocker_attention`: a blocked task whose terminal blocker needs
a person.
> - When the queue loads a page of blocked tasks, it reads `keep` and
`archived_at` from `decision_retention` for each of them, one full-row
`select` per task.
> - A company hint walks every page. On a hosted instance with 2,112
blocker attention items and one active run, these per-task reads ran at
about 16 per second.
> - This pull request reads the retention state for the whole page in
one narrow statement. It also scopes three remaining run-path attention
hints to the task graph.
> - The benefit is one retention read per page instead of one per
blocked task, with the same decisions.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The `blocker_attention` eligibility read in
`server/src/services/attention-retention.ts` (`eligible()`), and three
run-path producers of company-wide attention hints.
**Subsystem affected**
Server background retention (`attention-retention.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-review-participant.ts`,
`native-runtime/native-workspace-reconstruction.ts`).
**Current behavior**
- For each blocked task whose blocker attention is terminal (and for the
owner variant), `eligible()` ran `select <all columns> from
decision_retention where company_id = $1 and source_kind = $2 and
source_id = $3`. It did this during `load`, once per task, on every
company hint.
- The hosted statement is identified by that column prefix: `id,
company_id, source_kind, source_id, source_activity_at, …`.
- I checked every other full-row reader of `decision_retention`. None
fires per run event:
- `syncItems`: attention summary and list requests, which a board calls
once per load; retention `check` and the sweep, which run for due
entries only; Keep and Archive actions.
- `getState`: user actions only.
- `archiveReviewedManifest`: proposal acceptance only.
- `decisions.list`: only when archive proposals exist.
- `native-provider-capacity-retry`, `native-review-participant` and
`native-workspace-reconstruction` sent `{ companyId }` attention hints,
which make the queue walk every page.
**Proposed behavior**
- `pageEvidence` reads `source_id`, `keep` and `archived_at` for every
blocker root and every terminal blocker on the page in one statement
(ids passed as one JSON parameter). `eligible()` uses that map when it
has page evidence.
- A missing row means "not kept, not archived", exactly as the per-task
read returning no row did.
- The per-entry `check` path still reads one row, now only `keep` and
`archived_at`.
- The three producers send `notifyIssueAttentionRetentionWork` with
their task (and run). That is the task, its parents and the tasks it
blocks, with a company fallback past 100.
- A capacity retry changes the failed-run items of its task.
- A review participant claim changes the review of its task.
- A workspace reconstruction resolves the recovery action of its task,
which can also change the blocker attention of the tasks it blocks.
- The configuration, lifecycle, import and fallback producers keep their
company hints.
**Reason and benefit**
Background work must not grow with the backlog. A page now costs one
retention read instead of up to 100.
## What Changed
- `server/src/services/attention-retention.ts`: batched and narrowed
blocker retention read; `PageEvidence.blockerRetention`.
-
`server/src/services/native-runtime/native-provider-capacity-retry.ts`,
`native-review-participant.ts`, `native-workspace-reconstruction.ts`:
scoped attention hints.
- Tests in `server/src/__tests__/attention-retention-work.test.ts` and
`server/src/services/native-runtime/native-workspace-reconstruction.test.ts`.
## Verification
- `reads blocker retention state once per page, not once per blocked
task` (I saw it fail first). 160 blocked tasks, each behind its own
unassigned blocker; the blockers' retention rows are kept (40), archived
(40), plain (40) or missing (40). The full-size response capture and the
statement log measure the load of both pages:
- before: one `decision_retention` read per task (160), 26,240 result
bytes;
- after: 2 reads (one per page), 5,480 bytes, none of them selecting
`source_activity_at`.
- The same tasks are listed: only plain and missing.
- `check` still drops the kept and archived ones.
- `scopes the attention retention hint of a reconstruction to the task
graph` (I saw it fail first): before, the reconstruction sent `{
companyId }`; after, it sends a hint that names the task.
- `batches 100 blocker graphs with exact terminal timestamps and fresh
terminal visibility` pinned 200 retention reads per page (one per root
and variant). It now expects 1; the rest of that test is unchanged and
passes.
- Not separately tested: the capacity-retry and review-participant hint
changes. Each is one line that passes the task the function already
holds, the same pattern as the reconstruction test.
- `attention-retention-work` (58), `attention-retention-candidates`,
`native-workspace-reconstruction`, `native-review-participant` and
`heartbeat-run-dispatch-work` pass, each file run alone (in `server/`,
against dedicated local `_test` databases).
- `pnpm check:module-boundaries` passes.
## Risks
- The page read happens once, before the per-entry decisions of that
page. Before, each decision read its row just in time. A Keep or Archive
that lands between the page read and an entry's decision is seen by the
next load or by `check`, which re-reads under a row lock, as before.
- A run-path hint that changes more than its task graph would now miss
the wider effect. The 90-day archive delay makes a missed reload cheap.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
cf09024292 |
fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once
## Linked Issues or Issue Description
**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.
**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.
**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.
**Paperclip version or commit**
`master` at
|
||
|
|
633dec7e43 |
fix(instructions): back off working copies that cannot make progress (#21586)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. Background queues collect the copy after the provider stops
and clean up its scratch directory.
> - When the queue cannot prove that the provider stopped, or cannot
finish a cleanup, it re-arms the same copy every 30 seconds with no
limit. The attempt count does not increase on the "stop not proven"
branch.
> - Every re-arm is a transaction that sends work signals to its own
queue, to the stranded-agent-status queue and to the skill-retention
queue. A self-signal also reloads the copy's scope. On a hosted instance
with only a few live runs, these copies were one of the largest sources
of SQL calls and kept the main thread busy.
> - This pull request makes a copy that cannot progress wait on a
growing, capped backoff. It also removes the work signals from a write
that only moves a deadline later, and reads the copy once per check.
> - The benefit: a copy that waits for proof of the provider stop costs
about 1/350 of the old steady-state load, and a copy that becomes
collectable is still collected automatically.
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
the instruction working-copy queues re-reading the same unresolved
copies forever, at about 25–30 statements/s. A copy whose run is
terminal but whose provider stop is not proven was re-armed every 30 s.
A completed agent-files copy whose remote scratch cleanup cannot finish
(its lease is released but not destroyed, or the lease row is gone) was
also re-armed every 30 s. Each re-arm was a transaction with work
signals that woke three queues.
**Expected behavior**
A copy that cannot progress is not re-read until a growing, capped
backoff elapses. A write that only moves a deadline later does not wake
other queues. A copy that becomes collectable is still collected
automatically, and no data is lost.
**Steps to reproduce**
1. Create working copies of terminal native runs that have no
`native.local_process_stopped` event, and completed agent-files copies
with `cleanupPending: true` whose remote lease does not exist.
2. Register the instruction-collection and directory-cleanup queues.
3. Count statements while time advances. With 300 + 300 such copies,
master sends 252 statements/s, 38.7 transactions/s and 38.7 work
signals/s.
**Paperclip version or commit**
`origin/master` at
canary/v2026.1114.0-canary.5
|
||
|
|
c3f9a0c040 |
fix(retention): evaluate native retention blockers for many sessions in one statement (#21588)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep runner checkpoints and harness backups so that a
session can resume. Retention expires them after the session can no
longer resume.
> - A retention blocker decides whether a session can still resume: a
live or recent run, an open or recently closed task, an unsettled
finalization, or a live lease.
> - The startup checkpoint pass evaluated that blocker in its own
transaction, once for each retained checkpoint, with three reads. A
session that stays blocked (for example, a long-lived open task) was
evaluated again for each of its runs on every server start. On a hosted
instance with frequent deploys and a 40-minute overlap of two servers,
this pass ran almost continuously at about 7 checkpoints/s, about 21
statements/s.
> - Backup reconciliation also used four reads and a transaction for
each run hint that reached a backup.
> - This pull request evaluates the blocker for many sessions in one
fixed-text statement. Only candidates that are not blocked enter the
existing locked transaction.
> - The benefit: a restart costs 2 statements for each page of 200
blocked checkpoints instead of 5 statements and 1 transaction for each
checkpoint, and a backup hint costs 2 statements with no transaction.
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
three retention-blocker reads (`company_id, agent_id from
heartbeat_runs`; the session's runs; the session's issues) at exactly
7.0/s each, each in its own transaction. The startup checkpoint pass
sent them once for each retained checkpoint older than 7 days whose
session was still blocked, and it repeated the full pass after every
start.
**Expected behavior**
A checkpoint or backup that is blocked from expiring costs no
transaction and no statements of its own, because one statement
evaluates its session together with the rest of its page. It expires as
soon as its blocker clears: checkpoints on the next startup pass (the
current design), and backups at their exact deadline.
**Steps to reproduce**
1. Create 100 native sessions with 3 finished runs each (finished 10
days ago), settled finalizations, a checkpoint header for each run, and
an open task.
2. Register the startup checkpoint expiration and count statements. Then
register it again, as a restart.
3. On master each start costs 1,502 statements and 300 transactions, and
nothing expires.
**Paperclip version or commit**
`origin/master` at
|
||
|
|
f452b449fd |
fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.
**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.
**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.
`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.
**Paperclip version or commit**
`master` at
|
||
|
|
e8c79235ca |
perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `attention-retention` queue archives old attention items (failed runs, reviews, questions, recovery actions, blocked tasks). A company hint makes it reload every candidate of the company, 100 per page, with eligibility reads for each page. > - Several run lifecycle producers sent that company hint: every agent status change after a run, the execution-control writes and legacy run terminalization when they name only a run, scheduled retries, and the Runner's task monitor tool. > - Each page also read whole rows it did not use: full `heartbeat_runs` rows (recent ones carry tens of KB of `context_snapshot` and `runner_profile_json`), full `issues` rows (descriptions), and the whole agent roster with its configuration. > - This pull request names the task graph or the source kind in those hints, and reads only the columns each eligibility decision uses. > - The benefit is fewer company reloads per run event and far fewer result bytes per reconstruction, with the same decisions. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds on that pull request's change to `execution-control-notifications.ts`); review that one first. **What existing behavior does this improve?** The attention retention queue (`server/src/services/attention-retention.ts`) and the producers that wake it on run lifecycle paths. **Subsystem affected** Server background retention (`attention-retention.ts`, `retention-work-notifications.ts`, `execution-control-notifications.ts`, `heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `native-runtime/paperclip-runner-tool-authority.ts`). **Current behavior** - `heartbeat/agent-status.ts` sends `{ companyId }` on every agent status change after a run (two places). `heartbeat/retries.ts` sends it for each scheduled retry. `writeExecutionControl` and `notifyExecutionRunWork` send it when the scope names only a run. The Runner's `set_task_monitor` tool sends it for one task. - On a local fixture with 1,500 open tasks and run churn, the queue loaded 47 company candidate rows per second from these hints. - `pageEvidence` read `select *` from `heartbeat_runs`, `issue_recovery_actions` and (through `getTableColumns`) `issues`; `reviewEligibility` read `select *` from `issues`; `interactionEligibility` read the full agent roster; the per-entry `check` repeated full-row reads. **Proposed behavior** - Agent status changes send `{ companyId, sourceKind: "agent_error_alert" }`. The status moves between `running`, `idle` and `error`. All three are invokable for blocker attention (`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions (`agent-eligibility.ts`). Failed-run attention only excludes terminated agents, and this path never sets that. So only agent error alerts can change. - Run-scoped execution-control writes, legacy run terminalization and scheduled retries use `notifyIssueAttentionRetentionWork` with the run id (the run's task, its parents and the tasks it blocks; company fallback past 100). This is the same helper as #21521. The task monitor tool scopes to its task. - `notifyIssueAttentionRetentionWork` matched the run's saved task id with `issue.id::text = …`. It now validates and casts the id, so the primary key is used, as #21552 does for recovery. - Narrow reads, same decisions: - runs: `id`, `agent_id`, `created_at` and the saved task id computed in SQL with the same rules as `readRunIssueId` (`issueId` unless it is absent or JSON null, then `taskId`; only non-empty strings). - recovery actions: `id`, `source_issue_id`. - issues for evidence and `visibleIssue`: the blocker-attention input columns plus `unblock_descriptor` and `blocked_transition_at`. - review issues: `id`, `company_id`, `status`, `execution_state`, `assignee_user_id`. - agent roster for question addressees: `id`, `company_id`, `name`, `reports_to`, `status` (the `AgentOrgRow` that `evaluateAgentInvokability` takes). **Reason and benefit** A run event no longer reloads every attention candidate of the company. A reconstruction no longer transfers whole run, task and agent rows. ## What Changed - `server/src/services/attention-retention.ts`: column lists for runs, recovery actions, issues and the roster; the run probe type replaces whole rows. - `server/src/services/heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `execution-control-notifications.ts`, `native-runtime/paperclip-runner-tool-authority.ts`: scoped attention hints. - `server/src/services/retention-work-notifications.ts`: indexed lookup of a run's saved task. - `server/src/__tests__/attention-retention-work.test.ts`: two tests. ## Verification - `reads only the fields its decisions use from large runs, reviews and agents` (I saw it fail first): 50 failed runs, 50 tasks in review with large descriptions, 10 questions to a paused agent, about 40 KB of payload in each run's snapshot, profile and stdout and in the agents' configuration. It loads the three kinds with the full-size response capture (`full-database-response-capture.ts`). Before: `heartbeat_runs` 6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After: under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions are the same before and after: 50 failed runs, 40 reviews and 10 questions. - `scopes run lifecycle attention hints to the run's task graph and agent status to agent alerts` (I saw it fail first): `writeExecutionControl` and `notifyExecutionRunWork` with a run id send the hint for the task and its parent. `finalizeAgentStatus` sends the agent-alert hint. Before, all three sent `{ companyId }`. - `attention-retention-work` (57 tests), `attention-retention-candidates`, `attention-service`, `execution-control-work` and `heartbeat-agent-status-recovery` pass, each file run alone (in `server/`, against a dedicated local `_test` database). - `pnpm check:module-boundaries` passes. Not done in this pull request, and why: - I did not reproduce `syncItems` re-syncing unchanged rows from the retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old recovery-action and blocker retention rows, run churn), `syncItems` ran 0 times in a 180-second window and `decision_retention` reads were 0 per second. `check` calls `syncItems` only for entries that are already due (older than the 90-day archive delay), and a due entry is archived and leaves the candidates. So I did not batch `syncItems` or skip checks of unchanged rows. - `readIssues` still selects the `issues` columns that blocker attention needs. I did not narrow `issueService.listBlockerAttentionBatch`. ## Risks - A producer that changes more than the run's task graph would now miss the wider effect. Company-level changes (agent configuration, company status, budgets) keep their company hints. The 90-day archive delay makes a missed reload cheap: the item is reloaded on the next hint that covers it. - The saved task id now comes from SQL. The SQL mirrors `readRunIssueId` for strings, absent keys and JSON null. A non-string `issueId` value now yields no task, as before. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a1c37b5cc4 |
perf(chat): rebuild GitHub response sweeps for company-wide hints at most every 30 seconds (#21580)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel has two background sweeps for request response comments: acknowledgements, and terminal responses. Each sweep rebuilds its candidate set when a hint arrives, and it reads the latest run of each candidate request (`readGitHubResponseRun`) on each rebuild > - A run milestone for a run with no task sends a company-wide hint to both sweeps. Examples: each status change of a run without a task (`setRunStatus`), and a comment merged into an active run in the wake queue. In a busy company these hints arrive more than once a second > - So both sweeps rebuild the whole company about once a second (their minimum pass interval). Each rebuild reads every parked request again, although nothing about those requests changed. On a hosted test database, `readGitHubResponseRun` ran 6.7 times a second > - This pull request rebuilds a GitHub response sweep for a company-wide hint at most once every 30 seconds. Issue, request, conversation, endpoint and lease hints still rebuild immediately. `external-objects.ts` uses the same rule for its refresh sweep > - The benefit is that a steady stream of company-wide hints no longer makes each sweep read every parked request every second ## Linked Issues or Issue Description Refs #21577 (makes each `readGitHubResponseRun` call cheap; this pull request reduces how often the sweeps call it). Refs #18384 (the same reconstruction interval for company-wide external-object hints). **What existing behavior does this improve?** The candidate rebuild of `registerGitHubAcknowledgementWork` (`acknowledgement-work.ts`) and `registerGitHubTerminalResponseWork` (`terminal-response-work.ts`) for company-wide hints. **Subsystem affected** GitHub chat channel: request acknowledgement and terminal-response recovery sweeps. **Current behavior** `notifyChatRunMilestones` calls `notifyChatResponseAvailability` with its scope. When the scope has no issue (only a company, or a company and a run), the hint for both GitHub response queues is company-wide (`{"companyId": ...}`), because the response scope has no run field. Each company-wide hint makes each sweep load every candidate of the company and call `readGitHubResponseRun` for each one. Passes are limited to one a second, so with a steady hint stream the cost is about (parked candidates) reads a second. **Proposed behavior** Both sweeps set `reconstructionIntervalMs`. A company-wide scope (company ID only) waits 30 seconds after its last rebuild before it rebuilds again. All narrower scopes, the startup scope (`*`), and saved deadlines are not changed. Entries that are already loaded stay available for dispatch while a company-wide hint waits, and each check still reads the request again from the database. **Reason and benefit** On the hosted test database, `readGitHubResponseRun` ran 6.7 times a second for no change in state. In a local reproduction, the call rate drops from about one call for each parked request each second to one call for each parked request each 30 seconds. **Breaking changes** None. ## What Changed - Add `response-sweep-scope.ts` with the shared scope type of the two GitHub response sweeps and `githubResponseReconstructionIntervalMs`: 30 seconds for a company-wide scope, 0 for all other scopes. - Use it as `reconstructionIntervalMs` in the acknowledgement sweep and the terminal-response sweep. Both files now import the shared scope type instead of declaring it. - Add `response-sweep-scope.test.ts`. It registers both real sweeps on a real database with 6 parked requests (3 for each sweep) and a controlled clock. It sends company-wide hints and issue hints, and counts `readGitHubResponseRun` statements on each pass. ## Verification Cause, from a local reproduction with the real delivery coordinator and both real sweeps on embedded PostgreSQL 18.1, with 8 parked requests (4 for each sweep), each with a succeeded native run that waits for its presentation: | Hints | Hints a second | `readGitHubResponseRun` calls a second | | --- | --- | --- | | None | 0 | 0 | | Company-wide run milestone (`{ companyId, runId }`) | 9.7 | 8.75 | | Company-wide run milestone | 1.0 | 7.98 | | Run milestone for an unrelated issue | 9.7 | 0 | So the call rate is about one call for each parked request each second, from company-wide hints only. The hosted rate (6.7 a second) is consistent with about 7 parked GitHub requests in one company. Red/green regression test: - `cd server && npx vitest run src/services/chat-channels/github/response-sweep-scope.test.ts` - Red (without the two `reconstructionIntervalMs` lines): `expected [ 6, 6, 6, 6, 6 ] to deeply equal [ 6, +0, +0, +0, +0 ]`. Each of 5 company-wide hints rebuilt all 6 parked requests. - Green: 2 of 2 pass. The first company-wide hint rebuilds once (6 reads). The next 4 hints read nothing. After 30 seconds, the waiting hint rebuilds (6 reads). Issue hints still read exactly the one request of that issue at once, and a hint for another company reads nothing. No acknowledgement is sent for a parked request. Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed (all GitHub chat-channel integration tests, including the acknowledgement and terminal-response sweeps). - `cd server && npx tsc --noEmit`: passed. - After the rebase onto #21577: both new test files pass (4 of 4), and the server typecheck passes. - `pnpm verify` selected most of the server suite, because the sweep modules are imported widely. I stopped that broad local run. Required CI supplies that coverage. ## Risks - A change that only a company-wide hint reports can take up to 30 seconds to rebuild a GitHub response sweep. I checked the callers. Changes to one request use issue, delivery, conversation or lease hints: run status changes with a task, native presentation commits (`status-decision-committer.ts`, `native-run-finalizer.ts`, `native-chat-review-presentation.ts`), publications, fast responses, and the publication lease. Those stay immediate. Company-wide hints come from runs with no task, from a comment merged into an active run (the wake queue sends only the run ID), and from company authority changes. A run with no task cannot change the latest run of a GitHub request's task. For a merged comment, the active run's next status change sends an issue hint, and staging the request's response sends a delivery hint; both rebuild at once. Only a mid-run ownership update can wait up to 30 seconds. For an authority change, each check reads the request and its authority again before it acts. - The deadline reconciliation snapshot reports these two sweeps as not reconciled while a company-wide hint waits (up to 30 seconds). This is the same as the external-object refresh sweep. - The startup rebuild (`*`) and failure retries do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local PostgreSQL reproductions). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
340157d1d6 |
perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.
**Subsystem affected**
Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).
**Current behavior**
`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.
**Proposed behavior**
Design. For each task queue, a run event of run R on task T can change:
| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |
The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.
- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.
Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:
- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.
Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:
1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.
Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.
Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.
- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.
**Reason and benefit**
Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.
## What Changed
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.
## Verification
New tests. I saw each red test fail before the change.
- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).
Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.
Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).
| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |
New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.
With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).
## Risks
- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
nightly/v2026.1114.0-nightly.0
canary/v2026.1114.0-canary.4
|
||
|
|
c1d3f90b37 |
fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `resolved-dependencies` recovery queue loads the blocked tasks whose blockers are all done (#21517). It runs this load on recovery hints. > - The load has two correlated subqueries on `issue_relations`. PostgreSQL turns them into a semi join and an anti join, and lets the statistics choose how. > - After a bulk change in a company that the statistics have not seen (a large import, a restore, a big cleanup), the planner estimates one row for the company. It then picks nested loops whose inner side scans every relation of the company and filters by task id. On a local copy with 5,000 open tasks the query ran for more than 20 seconds (once 19 minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms. > - This pull request keeps both subqueries as per-task subplans, so each one is an index probe by (company, task id) whatever the statistics say. > - The benefit is that the load stays proportional to the blocked tasks it reads, also right after bulk changes. This is prevention: the hosted instance does not show this query as slow today. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a local database that had been analyzed with a smaller dataset and then loaded with 5,000 open tasks in a new company, the `resolved-dependencies` load query did not finish within 20 seconds; in a benchmark run it stayed active for 19 minutes after its server stopped. **Steps to reproduce** 1. Load a company with about 1,500 tasks and run `ANALYZE issues, issue_relations`. 2. With autovacuum off for these tables, load another company with 5,000 tasks, 1,900 of them blocked with `blocks` relations. 3. Run the load query of `resolved-dependencies` for the new company. Plan with stale statistics (abridged): ``` Limit -> Nested Loop Semi Join (rows=1) Join Filter: (issues.id = r.related_issue_id) -> Nested Loop Anti Join (rows=1) Join Filter: (r_1.related_issue_id = issues.id) -> Index Scan using issues_company_id_uq on issues Index Cond: (company_id = ...) -> Nested Loop -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1 Index Cond: (company_id = ...) -> Index Scan using issues_company_id_uq on issues blocker Index Cond: (company_id = ...) -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r Index Cond: (company_id = ...) ``` The index condition holds only `company_id`. The task id is a join filter, so each blocked task rescans every relation (and every task, for the blocker) of the company. **Expected behavior** Each blocked task probes its own relations through `issue_relations_company_related_issue_idx` (company_id, related_issue_id) and its blockers through `issues_company_id_uq` (company_id, id). ## What Changed - `server/src/services/recovery/work.ts`: the candidate predicate is a named export, `resolvedDependencyCandidate`, and both subqueries end with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up into a join, so it stays a correlated subplan. Its index condition then includes the outer task id. The predicate and its results are otherwise unchanged. - `server/src/__tests__/resolved-dependency-candidates.test.ts`: two tests. Indexes that the subplans use (all existing): `issue_relations_company_related_issue_idx` (company_id, related_issue_id) for both relation probes, and `issues_company_id_uq` (company_id, id) for the blocker status. `issue_relations_company_issue_idx` (company_id, issue_id) is not needed here. ## Verification - `selects the same tasks as the earlier predicate`: 40 blocked tasks with a done blocker, an open blocker, a done and a cancelled blocker, or no blocker. The new predicate returns the same 10 tasks as the earlier one. - `probes the relations of each task, whatever the planner statistics say`: analyzes a small company, then loads 3,000 blocked tasks in a new company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi or anti join, and the load returns 100 rows within a 5-second statement timeout. I saw it fail first with the earlier predicate: the plan had a semi join and an anti join. - Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1` and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND (related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms. - `issue-recovery-deadlines.test.ts` and `issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm check:module-boundaries` passes. ## Risks - With fresh statistics the planner could use a merge or hash join for a large page; the subplan form gives up that choice. Each blocked task now costs two index probes, which is the same order as the fresh-statistics plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once each on a loaded machine). - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6b6c94a4ea |
perf(chat): read the GitHub response run by issue run indexes (#21577)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel keeps one response comment for each GitHub request. Two background sweeps (acknowledgements and terminal responses) and the acknowledgement step read the latest run of the assigned agent for the request's issue with `readGitHubResponseRun` > - That function sends two statements. Both match runs with `coalesce(native_issue_id::text, context_snapshot->>'issueId') = <issue id>`. No index has that expression, so each statement reads every run of the agent and decompresses the large `context_snapshot` of each legacy run > - On a hosted test database, these two statements were the two most expensive statements by database time. In a 701 s window, each ran 6.7 times a second, for 203 s and 137 s of database time (about 30 ms and 20 ms for each call). Together they kept about half of one database session busy > - This pull request reads the latest run with two index lookups (native binding, and legacy context without a native binding), the same way #20671 changed the terminal-response sweep. It decides request ownership on that one run in the same statement > - The benefit is that each call reads a few index entries in one statement instead of the agent's whole run history in two statements. The results do not change, and no migration is necessary ## Linked Issues or Issue Description Refs #20671 (the same rewrite for the terminal-response discovery page; its Risks section names this function as the remaining case). **What existing behavior does this improve?** `readGitHubResponseRun` in `server/src/services/chat-channels/github/response-run-state.ts`. The GitHub acknowledgement sweep calls it once for each candidate request. The terminal-response sweep and the acknowledgement step call it through `githubResponseExecution`. **Subsystem affected** GitHub chat channel: request acknowledgement and terminal-response recovery. **Current behavior** The function sends two statements: 1. The latest run of the agent for the issue that woke for the request: `heartbeat_runs` joined to the inbound `chat_message_links` row of the delivery on `wakeCommentId` or `wakeCommentIds`, ordered by `created_at desc`, limit 1. 2. The latest run of the agent for the issue, with the `responseRunColumns` projection, ordered by `created_at desc, id desc`, limit 1. Both statements filter on `coalesce(native_issue_id::text, context_snapshot->>'issueId') = $issueId`. The planner can only use `heartbeat_runs_company_agent_started_idx` or `heartbeat_runs_company_created_at_desc_idx`. It reads all runs of the agent and evaluates the `coalesce` on each row, which decompresses the context of each legacy run. On the hosted test database, `heartbeat_runs` has about 31,000 rows and 1.96 GB, because `context_snapshot` is large. **Proposed behavior** One statement: - The latest run ID comes from two disjoint branches, each `order by created_at desc, id desc limit 1`: runs with `native_issue_id = $issueId` (through `heartbeat_runs_company_native_issue_contract_id_uq`), and runs with `native_issue_id is null and context_snapshot->>'issueId' = $issueId` (through `heartbeat_runs_company_ctx_issue_created_idx`). Together the branches are exactly the rows where the old `coalesce` equals the issue ID. - The statement reads that one run by primary key with the same `responseRunColumns` projection. - An `exists` subquery on the inbound `chat_message_links` row of the delivery decides whether that run woke for the request. It uses the same `wakeCommentId` / `wakeCommentIds` match as before, but on one run only. - When the issue ID is null, the function returns no run without a query. The old statements could not match a null issue ID either. **Reason and benefit** These were the two most expensive statements on the hosted test database. On a local fixture with the hosted shape, the median call goes from 111.6 ms to 1.03 ms (about 108 times faster), and the buffers for one call go from about 73,000 to 98. **Breaking changes** None. The function signature, the returned run, and the ownership result do not change. ## What Changed - `readGitHubResponseRun` reads the latest run of the agent for the issue through the two index-backed branches, and decides request ownership on that run in the same statement. Two statements become one. - Add `response-run-state.test.ts`. It runs the old two-statement function (kept in the test as the oracle) and the new function on the same inputs, and it requires equal results. It also requires a plan bound: one call must read fewer than 150 `heartbeat_runs` rows while the agent has 1,500 unrelated runs. - Add `scripts/benchmark-github-response-run-state.ts`. It loads the baseline module from a given commit, compares the complete output of both versions for every input, measures both, and prints the plans. ## Verification Why the old statements were slow, and the thesis: each statement filters the agent's whole run history with an expression that no index has, and detoasts the context of every legacy run. If each precedence branch uses its own issue index and ownership is checked on the one latest run, the cost no longer depends on the agent's run count, and the results stay identical. Red/green regression test: - `cd server && npx vitest run src/services/chat-channels/github/response-run-state.test.ts` - Red (test file on the old `response-run-state.ts`): the plan bound fails with `expected 3597 to be less than 150`. The equivalence test passes, because the oracle is the old code. - Green (this change): 2 of 2 pass. The equivalence fixture has two companies, two agents each, 60 issues each, and one or two GitHub requests for each issue. Runs cover these cases: native binding, legacy context without a native binding, a native binding to another issue while the legacy context names this issue, another agent, another company, `wakeCommentId`, `wakeCommentIds` as an array, `wakeCommentIds` as a JSON string, a link to the outbound message of the delivery, a link to another company's comment, a link to the sibling request of the same issue, a request with no inbound comment, ties on `created_at`, and a null issue ID. It also calls with another company's endpoint. The test requires more than 20 results in each outcome (owned, latest run not owned, no run) and both binding kinds. Local benchmark with the hosted shape: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-response-run-state.ts --baseline-ref=3c3e6c2d75 ``` Fixture (synthetic): one company, 4 agents with about 7,750 runs each (31,000 runs), 2,000 issues, 300 GitHub requests. Each run has about 60 KiB of incompressible context, and `heartbeat_runs` is 1,932 MB. Runs are native or legacy at random, and about half link to the request comment. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted database is Neon, so absolute times are local measurements only. The benchmark compares the complete output for all 600 inputs (262 owned, 329 latest run not owned, 9 no run): equal. It then times 20 inputs with runs (two agents), with 2 warmups and 7 alternating samples. | | Statements per call | Median ms per call (range) | Buffers for one call | | --- | --- | --- | --- | | Before | 2 | 111.6 (100.9–240.5) | 18,358 hit + 26,498 read, then 14,997 hit + 13,489 read | | After | 1 | 1.03 (0.81–1.39) | 80 hit + 18 read | Plans for one call. Before: statement 1 is a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that removes 7,674 rows by filter (75.7 ms). Statement 2 is an `Index Scan` on `heartbeat_runs_company_created_at_desc_idx` that removes 19,801 rows by filter (60.2 ms). After: one `Index Scan` on `heartbeat_runs_pkey`. Its `InitPlan` is a `Merge Append` of two `Limit` index scans on `heartbeat_runs_company_native_issue_contract_id_uq` and `heartbeat_runs_company_ctx_issue_created_idx`, and its `SubPlan` is one `Index Scan` on `chat_message_links_delivery_idx` (0.128 ms). Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed (all GitHub chat-channel integration tests, including the acknowledgement and terminal-response sweeps). - `cd server && npx tsc --noEmit`: passed. - `pnpm verify` exited 2 because a file under `scripts/` changed, which requires the broad CI coverage. Required CI covers the rest. Why the function runs 6.7 times a second. The acknowledgement sweep and the terminal-response sweep rebuild their candidate set when a hint arrives, and they call this function once for each candidate on each rebuild. A run milestone for a run with no issue (for example, every status change of a run without a task in `setRunStatus`, or a comment merged into an active run in the wake queue) sends a company-wide hint to both sweeps. So in a busy company, both sweeps rebuild about once a second (their minimum pass interval), and each rebuild reads every parked candidate again, although nothing about those candidates changed. In a local reproduction with 8 parked candidates (4 for each sweep) and the real coordinator and workers, 10 company-wide milestone hints a second gave 8.75 calls a second; 1 hint a second gave 7.98 calls a second; 10 hints a second for an unrelated issue gave 0 calls; and no hints gave 0 calls. So the hosted rate is consistent with about 7 parked GitHub requests in one company and a steady stream of company-wide run milestones. This pull request makes each call cheap. A separate pull request will address the repeated rebuilds. ## Risks - Low risk. The change is one function. The run match is the same set of runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id = <uuid>` for the canonical UUID text that callers pass (the issue IDs come from UUID columns), and a null `native_issue_id` makes the old `coalesce` use the legacy context. - Ownership: the old code compared the latest run that woke for the request (ordered by `created_at` only) with the latest run (ordered by `created_at, id`). That is true exactly when the latest run woke for the request, except when two runs that both woke for the request tie on `created_at` with the latest run. In that case the old result depended on the physical row order. The new code always returns the deterministic answer (the latest run woke for the request). - The number of calls does not change in this pull request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e977c3b922 |
feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Part 2 gave the consumer a wait with a limit and two inputs: a hook that ends the wait early, and a start time. > - This pull request gives both from the server for a native ACPX Claude run. A new comment from a person ends the wait. The start time comes from an event that is already saved, so a restart of Core does not make the wait longer. > - The benefit is that a person who comments does not wait up to 10 minutes for the agent to see it. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 3. Its base branch is `feat/core-provider-turn-consumer`. Refs #19912 (design). **What existing behavior does this improve?** Nothing visible yet. What Core gives to the consumer of a native ACPX Claude run. **Subsystem affected** Server: `server/src/services/native-runtime`. **Current behavior** - A comment from a person during a run is queued as a deferred wake. The next run delivers it. This does not change. - Core has no wait for a provider-started turn. **Proposed behavior** - For a native ACPX Claude run, Core gives `executeNativeSession` the option `providerTurnWait`. - One delivery worker does the person-comment check for all runs that wait. The server registers it at startup with the shared delivery work coordinator, as it registers the cost accounting worker. The worker listens to the existing notification for deferred wakes. On each notice it runs one query for each run that waits: a comment from a person on this issue, made after the start of the run, that a deferred wake of this agent names. A comment that the query finds ends the wait of that run. The run then takes the path of today, and the next wake delivers the comment as today. - If a database read fails, that run stays owed a check, and the coordinator runs the worker again after 5 seconds, as for each other delivery worker. A read that fails for one run does not stop the check of the other runs. When no run is owed a check, the worker has no deadline and does no query. - The start time of the wait is the time of the first saved `turn.completed` event of the run that has `awaitingProviderTurn`. With no such event it is empty. - Each query has the company id in its condition. **Reason and benefit** - The comment window starts at the start of the run, not at the start of the wait. A comment made during the prompt turn must not wait for the limit. ## What Changed - `server/src/services/native-runtime/native-provider-turn-wait.ts` (new, 109 lines) and 4 lines in `native-session-executor.ts`. - `server/src/index.ts` (4 lines): the registration of the worker, its stop at server close, and its place in the list of delivery workers that must be ready. `delivery-work-notifications.ts` (1 line): the queue key of the worker. The key needs no database change, because each topic uses the one shared channel. - `server/src/vendor/paperclip-runner/index.ts`: one type export. - Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one case in `native-usage-accounting.test.ts` (25 lines). ## State saved by the previous release - No new persisted state and no schema change. The start time is read from `heartbeat_run_events`. - **A run that the previous release started:** it has no waiting terminal, so the start time is empty. - **Core restarts during a wait:** the new process reads the start time, so the limit is the original one. The consumer then continues the wait (#21454), and the worker checks the comments again from the start of the run, so a comment that was made before the restart still ends the wait. - **Rollback of Core during a wait:** the previous Core stops at the waiting terminal and finalizes the run as `native_finalization_missing`, as today. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres with the real coordinator: a comment from a person in a deferred wake ends the wait; a comment from an agent, a comment before the run, and a comment of a different company do not; the start time comes from the first waiting terminal, and a run of the previous release has none. - The 6 new tests: a read that fails one time is tried again after 5 seconds and finds a comment that is already saved, with no new notice; a notice during a read that fails gives a new check; a read that fails for one run does not hold back a second run, and each run ends only for its own comment; a run that starts to wait during a check is checked; a wait that is aborted is removed, nothing checks it again and no deadline stays; with no run owed a check there is no deadline and no query in one hour of test time. - The test helper now registers the wake intent before it writes the deferred wake, in the same transaction. This is the order of each production writer (`modules/wake-queue/adapters/postgres.ts` and `services/heartbeat/queue.ts`). Production code did not change for this. - Mutation checks: a read that fails clears the owed check (3 tests fail); an aborted wait is not removed (the abort test fails). - Also run, unchanged and green: `delivery-work-coordinator.test.ts`, `budget-enforcement-worker.test.ts`, `server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`. The 7 files together: 1,699 passed, 7 skipped. - `native-usage-accounting.test.ts`: a prompt turn and the provider-started turn after it are each counted one time, with the events as the executor receives them. The cost of the provider-started turn is the session total in the cumulative channel, and it is not added two times. The accounting code is not changed. Not run: a run through the executor with a database; the full suite. CI supplies the broad coverage. ## Risks - **One more query at each start of a native ACPX Claude run,** on `heartbeat_run_events` by company and run. - **The worker is registered in each server process, and it is idle with no wait.** A notice with no run that waits does no query. No wait starts before part 4. - **After an abort during a retry, the 5 second retry of the coordinator runs one more time.** It finds nothing owed and does no query. The coordinator has no call to cancel the retry of one worker. - **A process that does not register the worker does no comment check.** The server registers it with no condition at startup, and native runs execute only in that process. Only a unit test can be in this state. - **A comment ends the wait but does not interrupt a turn that is running.** That is the rule of part 2. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2102492806 |
fix(discord): refresh due command registration after task drains (#21573)
## Thinking Path > - Paperclip runs agents and connects their work to chat services. > - Discord command registration uses a shared worker and saved deadlines. > - A task drain can hold that worker while registration is already due. > - The coordinator can keep a 30-second fallback after the drain ends. > - This change refreshes the command owner on drain release or observed expiry. > - Normal admission, earliest-time and provider retry checks still apply. ## Linked Issues or Issue Description Refs: #17557, #21562. Related: #15532 checks credential reads when registration is not due. This change handles resumption of due work in the separate command worker. **What happened?** A due command registration can wait for the coordinator's fallback after a task drain ends. The worker retains the request but has no drain-release subscription. This delays command work and its return to an idle state. **Expected behavior** Resume due command work when admission returns. Preserve future earliest times and provider failure backoffs. Leave an empty or stopped worker idle. **Steps to reproduce** 1. Register the normal command worker while an idle task drain is active. 2. Retain a due registration and capture the coordinator's fallback. 3. Release the drain, or observe its expiry, before that fallback. 4. Check that the due endpoint is checked before the fallback. Both regression cases fail before this change. **Paperclip version or commit** Main base `3c15acd3fc1b200ef0e36ac904a03cd5fcba00c0`. **Deployment mode** Built from source. The focused tests use the real coordinator and command owner with controlled registration operations. They do not contact Discord. ## What Changed - Subscribe the command owner to normal drain changes and observed expiry. - Wake it only when current admission allows work and its existing next time is due. - Remove both subscriptions through the normal worker stop callback. - Add eight controls for release, expiry, another closed gate, future earliest time, provider backoff, empty state and both stop paths. ## Verification Local checks used the integrated branch that exposed this contract gap. This main port contains the same two tested file afterimages. Its admission, scheduler and fairness modules match those local inputs. Main's coordinator and DB modules have other changes. Fresh CI on `aec6ecc65717aa6372bad526e044ec4aa3888bec` now qualifies this main context. - Both release and expiry cases fail at the intended missing endpoint-check assertion before the product change. No setup or timeout failure is used as causal proof. - All eight focused cases pass after the change. No case is excluded. - The complete normal Server typecheck and affected-file program checks pass on the integrated branch. - Normal committed Source SDK verification passes on this main branch. - Exact-head main CI passes, including typecheck, build, the general and serialized suites, and all eight new controls (142 ms) in server shard 2/12. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43565579653). A full local repository build and suite were not run in this checkout. - Greptile reviewed the exact head at 5/5 with no actionable findings. The complete edited summary and all paginated review threads were inspected; no unresolved threads remain. Run the focused controls with: ```sh pnpm --filter @paperclipai/server exec vitest run src/services/chat-channels/discord/command-work.test.ts --maxWorkers=1 --no-file-parallelism --retry=0 ``` The existing admission-drain contract remains accurate. This fix adds no configuration or command, so no documentation change is required. Full transport sleep acceptance and deployed suspension are separate claims. ## Risks A release event can request an earlier pass for already due work. The existing `canRun`, earliest-time, batching and per-item backoff checks remain active. Both worker stop and coordinator stop must remove the subscriptions; the controls cover each path. No scheduler policy, claim or provider deadline changes. ## Model Used OpenAI `gpt-6.1-sol` assisted with implementation and independent review. The Codex managing agent performed technical review and verification. The host does not expose its exact runtime model ID or context-window size. Work used source inspection, code edits and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (not required: the existing contract remains accurate) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3c3e6c2d75 |
fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").
**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.
**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.
**Paperclip version or commit**
`master` at
|
||
|
|
b1f608296e |
feat(runner): keep a run open for a limited time while the Runner waits for the provider's own turn (#21454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - `consumeTurn` reads the events of one turn and returns at the terminal. It already continues past a terminal for a Codex goal. > - This pull request makes it continue past a terminal that waits for a turn of the provider, and it limits that wait. > - The benefit is that a run continues in the turn that Claude starts, and a wait always ends. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 2. Its base branch is `feat/core-provider-turn-driver`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the consumer of a native session does after a turn that ended with no result while the Runner waits. **Subsystem affected** Paperclip Runner package, the controller side: `src/native-session-runtime.ts` and a new module `src/native-provider-turn-wait.ts`. **Current behavior** - `consumeTurn` returns at the first terminal. With no result, the run is finalized as `native_finalization_missing`. **Proposed behavior** - At a terminal with `awaitingProviderTurn` and no result, `consumeTurn` continues to read. It consumes the provider-started turn as the next turn of the same run. A turn that reports a result finalizes as a normal run. - The wait has a limit of 10 minutes from the first terminal that waits. A later terminal that waits does not start the clock again. The limit is never later than 5 seconds before the timeout of the run. - The limit and an early end stop the wait only. They never interrupt a provider-started turn that is running; that turn stays under the timeout of the run. If that turn ends and the Runner waits again, the run takes the path of today at once. - When the wait ends with no turn active, the consumer returns the last terminal, and the existing code for a run with no result decides (`resolveMissingResult`, then `native_finalization_missing`). No new failure code. - The wait ends at once when the driver gives the acknowledgement of part 1. This covers Stop, the limit, and a Runner that says the wait is over. - `executeNativeSession` gets one new option, `providerTurnWait`: the start time of a wait that an earlier controller process saw, and a hook that ends the wait early. - When the wait ends a turn, the consumer first waits for the receipts of the text deltas that are committed in batches (#21460), as each other return path of `consumeTurn` does. Without this, a delta that arrived during the wait was not in the event count of the turn, and a delta that failed to commit was not reported. - When the interrupt that ends the wait is not available or fails, the limit forces the end of the wait. If that forced end starts a new turn (the continuation that a restart owes), the consumer continues to read that turn. The forced-end signal is used one time only, and the read that is open at that moment is kept until it gets its event. Before this fix, the consumer returned the terminal of the old turn and lost the first event of the new turn, and the run could end as `native_finalization_missing`. - If the controller restarts during the wait, the run continues to wait. When the saved checkpoint still shows the first turn as active, the recovered session gives the waiting terminal again, and the wait starts again with the start time that was recorded. When the checkpoint was saved after the waiting terminal, the consumer finds that terminal in the journal, starts the wait again, and does not send the disposition-only recovery prompt. The consumer does this only if two conditions are true: the recovered session has the waiting state (`awaitingProviderTurn`, saved by the driver since #21453), and the terminal in the journal has the marker. If one condition is false, the run uses the recovery path of today. When a wait that started again ends with no turn, the existing code for a run with no result decides, as for each other wait. **Reason and benefit** - The 10 minutes are the maximum time of a foreground command in the pinned Claude Code. - A limit on the wait, not on the work, means that a long test run in the turn of Claude is not cut. ## What Changed - `src/native-provider-turn-wait.ts` (new, 129 lines) and `src/native-session-runtime.ts` (58 lines added, 4 removed), `src/index.ts` (1 export). - `src/native-session-runtime.provider-turn.test.ts`: written again for the new behavior (557 lines added, 61 removed). ## State saved by the previous release - This part adds no persisted state. The wait is in memory. Part 3 reads its start time from an event row that is already saved. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - `src/native-session-runtime.provider-turn.test.ts`: 30 tests. They cover: a controller restart with the first turn still active in the checkpoint (the wait continues; a provider-started turn settles its result; with no turn the limit ends the wait on the recorded clock); a controller restart with the checkpoint saved after the waiting terminal (the same two results, no recovery prompt, no event of the journal appended again); a limit that passed before the restart ends the wait at once, with and without an interrupt; the recovery prompt of today when the terminal in the journal has no marker, when the journal has no terminal, and when the recovered session has no waiting state; a second restart during the wait; a forced end that starts a new turn settles the result of the new turn, with each event of that turn appended one time and in order; an event that reaches the open read after the forced end is appended exactly one time; a delta that arrives during the wait is counted when the limit ends the turn (the test fails when the new wait for receipts is removed); continue and settle the result of the provider-started turn; a second wait does not start the clock again; the limit with no turn active; the limit and a comment during a running turn (not interrupted); a running turn that ends and waits again after the limit; Stop; a `run.terminal` that the Runner held; the Runner stream ends during the wait; the limit under the timeout of the run; a start time from an earlier controller; a plan that a provider-started turn proposes settles as the plan review wait. - `src/native-session-runtime.test.ts`, not changed by this PR: 297 passed. - Mutation checks, each made tests fail: no continue at a waiting terminal; no limit; no early end; no clock from an earlier controller; the clock starts again at each wait; a running turn is interrupted at the limit. - Mutation checks for the forced end, each made the two new tests fail: the forced-end signal used on each loop (the run ends as `native_finalization_missing`); a new read started after the forced end (the first event of the new turn is lost). - Mutation checks for the restart, each made named tests fail: the recovery prompt not held back; the wait not started again in `consumeTurn`; the wait started again on the session state alone (the test with no marker in the journal fails). Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No terminal has the marker before part 4. - **`consumeTurn` runs for each native turn of each agent.** It makes one small object for each turn and reads one more payload field at a terminal. - **If the interrupt that ends the wait is not available or fails,** the limit forces the end: the consumer returns the last terminal, or continues to read when the forced end starts a new turn. This is a fallback path. On a forced return, one read of the event stream stays open until the session closes, as before this PR. - **A wait that starts again after a restart holds back the recovery prompt.** This is the one change to the recovery flow. It needs the two conditions above, and both exist only for a run that asked for provider-started turns. - **Known limit.** The driver does not clear its waiting state when the consumer ends a wait with an interrupt. A restart in the short time after that can start the wait again; the recorded clock and the comment check then end it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3c15acd3fc |
perf(recovery): look up held runs by key so the fresh-continuation deadline stops scanning run history (#21568)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off, the fresh-continuation fallback continues the
task in a new run. The execution-disposition worker schedules its next
pass with `nextFreshContinuationCheckAt` and lists due work with
`heldExecutionsAwaitingFreshContinuation`
> - Both queries share two predicates (`heldExecutionWhere` and
`recentInterruptionSourceSql`). These predicates matched the run as
`source.id::text = <evidence text>` and the task's runs as
`native_issue_id::text = <issue id>::text`. A cast on an indexed column
prevents an index lookup
> - So for each held record, PostgreSQL read the company's whole run
history (two times), and it decoded each run's wide context JSON. On
staging, the deadline query took about 13.5 s of database time in each
27 s worker cycle. It was the largest statement by time
> - This pull request compares the uuid columns directly. It turns the
recorded run id into a uuid only when the text is the canonical
lowercase form, which is exactly when `id::text = text` can be true
> - The benefit is that each held record costs a few index lookups. The
same records are selected, and the worker cycle no longer keeps a
database session busy
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The fresh-continuation scheduler queries: `nextFreshContinuationCheckAt`
(the disposition worker's deadline),
`heldExecutionsAwaitingFreshContinuation` (the backlog listing, also
probed by the schedule), and the "left alone" count. They are in
`server/src/services/fresh-continuation-fallback.ts`.
**Subsystem affected**
Server: recovery and the execution-disposition worker
(`executionControlSchedule().disposition` in
`execution-control-work.ts`).
**Current behavior**
Neon statement statistics on staging (window of 160 s on 2026-11-14):
the `select
min((issue_recovery_actions.evidence#>>'{freshContinuationCheck,nextCheckAt}')::bigint)
…` statement ran 6 times at about 13.5 s each. That is 81 s of database
time in 160 s, about half of one busy session. All other statements in
the window used less than 35 s together. Staging has 31,399 runs (1.96
GB, wide `context_snapshot`), 75,285 recovery records, and 282 records
with a check deadline.
The local plan shows the cause. For each candidate record:
- `source.id::text = coalesce(evidence->>'runId',
evidence->>'sourceRunId')` cannot use the primary key. PostgreSQL
filters the company's failed runs (6,767 rows removed in each loop, and
6,798 again in the outer semi-join).
- In the recent-interruption check, `max(held.finished_at)` matches runs
of the task with `native_issue_id::text = source_issue_id::text or
issue_id::text = … or context_snapshot->>'issueId' = …`. The casts
cannot use the task-link indexes, so PostgreSQL filters the company's
run history and decodes each run's context JSON (29,937 rows removed in
each loop).
**Proposed behavior**
The queries look up the source run by its primary key and the task's
runs by the existing `(company_id, native_issue_id, …)`, `(company_id,
issue_id, …)` and `(company_id, context_snapshot->>'issueId', …)`
indexes. The results do not change.
**Reason and benefit**
On a staging-shaped local fixture, the deadline query goes from 2,330 ms
to 27 ms (87x), and the backlog listing and count go from 3.7-5.7 s to
19-27 ms. Buffer accesses go from 2.15 M to 28 k.
**Breaking changes**
None. No schema change and no migration.
## What Changed
- `server/src/services/fresh-continuation-fallback.ts`:
- Add `sourceRunKey`. It compares the recorded run id with
`heartbeat_runs.id` through the existing `canonicalExecutionUuid` guard
(cast only when the text matches `^[0-9a-f]{8}-…-[0-9a-f]{12}$`).
`uuid::text` always gives that lowercase form, so this matches exactly
the rows `id::text = text` matched. Uppercase, braced, padded, numeric
and invalid ids still match nothing, and no cast error is possible.
- Use it in `heldExecutionWhere`, `interruptionEligibility` and the
listing's order expression.
- `runOfTask` compares `native_issue_id` and `issue_id` with the issue
id as a uuid. Both callers pass a uuid: the `source_issue_id` column, or
the id of a task row read in the same transaction.
- Export `heldExecutionWhere` and `recentInterruptionSourceSql` so the
regression test can compare them with the previous predicates row by
row.
- Add `server/src/services/fresh-continuation-schedule-queries.test.ts`
(regression test, see Verification).
- Add `scripts/benchmark-fresh-continuation-deadline.ts`. It seeds a
staging-shaped fixture, captures the SQL that the baseline revision and
the working tree send, compares complete positional rows, and reports
timings and plans.
## Verification
- Red/green: `fresh-continuation-schedule-queries.test.ts` "finds the
deadline by run-key and task-link lookups, not a scan of run history per
record" counts the `heartbeat_runs` rows that the deadline query's plan
reads (loops x (rows + rows removed)). With the previous predicates it
fails: 549,166 rows read against a bound of 9,442 (two times the run
count). With this change it passes.
- Equivalence: the same file seeds 900 held records with a seeded random
generator. The records cover these cases: `runId`, `sourceRunId`, both
keys, uppercase, braced, padded, numeric, invalid and unknown ids; a
source run in another company; source statuses (failed, timed_out,
interrupted, cancelled, succeeded, running) with and without
`finished_at`; the assignee and other agents; newer runs linked by
native id, legacy issue id, or context only; unrelated runs;
cancelled-at-dispatch runs; live (queued, running, scheduled_retry)
runs; earlier and later creation times; retries of the source; person
Retry wakes; old and recent finish times; restart, person-owned and
sandbox-resume error codes; and issue statuses and assignees. For every
record it compares `heldExecutionWhere()`, `heldExecutionWhere({
includeActive: true })` and `recentInterruptionSourceSql(now)` with
verbatim copies of the previous predicates. It also compares
`nextFreshContinuationCheckAt` and the full ordered
`heldExecutionsAwaitingFreshContinuation` rows with the previous
queries. The test checks that each predicate is true and false for more
than 5 records, so the comparison is meaningful.
- Focused suites pass locally: `fresh-continuation-fallback.test.ts` and
`fresh-continuation-schedule-queries.test.ts` (82 tests). Related
recovery suites pass: `fresh-continuation-retry-delivery`, `task-retry`,
`fresh-continuation-move-forward-offer`,
`execution-recovery-dead-controller`,
`automatic-run-brake-recovery-passes`, `execution-control-work`,
`recovery-wake-herd` (with the new test: 8 files, 252 tests). `node
server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a global CI input),
so CI supplies the broad coverage.
- Benchmark: local embedded PostgreSQL 18.1, `jit=on`,
`shared_buffers=128MB`, macOS arm64, a shared machine with other work.
Baseline `67652580ee`. Fixture: 5,669 issues, 31,351 runs with 60 KiB
incompressible context JSON (1,962 MB), 75,285 recovery records (282
with a check deadline, a third overdue). One warm-up and 5 alternating
timed pairs. Both versions return identical positional rows for every
query.
| Query | Baseline median (samples, ms) | This PR median (samples, ms) |
Speedup | Shared buffers |
| --- | --- | --- | --- | --- |
| `nextFreshContinuationCheckAt` | 2,330 (1,962-2,421) | 27 (21-28) |
87x | 2,149,802 -> 28,227 |
| `heldExecutionsAwaitingFreshContinuation` limit 5 | 3,692
(3,296-4,014) | 19 (18-24) | 194x | 2,831,539 -> 23,970 |
| same, limit 1 (schedule probe) | 3,554 (3,210-4,861) | 20 (18-24) |
179x | 2,831,539 -> 23,970 |
| left-alone count | 5,746 (5,090-5,992) | 27 (25-28) | 213x | 5,435,614
-> 40,953 |
With 1 KiB contexts (rows stored inline) the deadline query goes from
4,096 ms to 19 ms. Staging measured 13.5 s for the deadline query. The
local fixture reproduces the same plan shape, but the staging data
distribution and Neon storage are not reproduced exactly, so the staging
gain is not verified yet.
```sh
pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-fresh-continuation-deadline.ts \
--baseline-ref=67652580ee --context-kib=60 --samples=5 --warmups=1
--print-plans
```
- Plan after the change: the source run is an `Index Scan using
heartbeat_runs_pkey` (0 rows removed), and each task-run probe is a
`BitmapOr` over the native-id, issue-id and context-issue indexes. The
newer-run `not exists` already used that `BitmapOr` before. It does not
change.
## Risks
- Low risk. The SQL change is limited to how two keys are compared. The
regression test compares the old and new predicates for every record of
a varied fixture.
- `runOfTask` now casts its issue id to uuid. A caller that passes a
non-uuid text would get an error instead of no match. The two callers
pass a uuid column and a task row's id.
- Follow-up, not changed here: the disposition schedule calls
`deadline()`, which returns "retry in 15 s" when the earliest check is
already due. An active hold that the disposition pass does not pick up
(for example, one that records its run only as `sourceRunId`, which
`unresolvedExecutionDispositions` does not read) keeps the deadline
overdue, so the worker runs again every 15 s with nothing to do. A local
probe confirmed this. This explains a fixed cadence of 15 s plus the
query time (about 27 s on staging). After this change each such pass
costs milliseconds, not seconds.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL benchmarks and
tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
behavior or command change; no doc update needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.3
|
||
|
|
3ae7aba643 |
perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server uses `withDedicatedDbConnection` (packages/db) when an action must hold one database session outside the main pool: advisory transaction locks, lock keepalive, and `pg_settings` reads for workspace lifecycle, finalization, source locks, skill caches and status cards > - Each call built a new postgres.js client, opened a new physical connection (TCP, TLS and startup through the pooler), built a new drizzle client with the full schema, and then ended all of it > - On staging the dedicated application name shows about 21 connection setups per second, and the main thread is pinned (event-loop utilization 0.965) > - This pull request keeps a small set of idle single-connection clients per owner and reuses them > - The benefit is that a dedicated action costs about 0.2 ms of main-thread time instead of 4 to 8 ms, with the same exclusive-session guarantee ## Linked Issues or Issue Description **What existing behavior does this improve?** `withDedicatedDbConnection` opens and closes one new database client and one new physical connection for every call. **Subsystem affected** Database client (`packages/db`) and the server services that take dedicated advisory-lock sessions. **Current behavior** Each call builds a new postgres.js client, opens a new connection (TCP, TLS and startup through the pooler), and builds a new drizzle client with the full schema. On staging (Neon statement statistics, 326 s window) `SET application_name='paperclip-workspace-finalization-lock'` ran 21.4 times per second. Only this factory uses that application name. In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2% inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that is this per-call setup. **Proposed behavior** A caller still gets exclusive use of one session for the whole action. The session and its drizzle client are reused by later actions, so the per-call cost is the action's own queries, not a full connection setup. **Reason and benefit** The server main thread is saturated on staging (event-loop utilization 0.965). Reuse removes about 5 ms of main-thread work per call. **Breaking changes** None. The exported function keeps its signature. `closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are new exports. ## What Changed - `packages/db/src/client.ts`: each `createDb` owner now has one set of dedicated single-connection clients, shared with every private client derived from that owner (so nested dedicated actions use the same set). - An action takes an idle client (last in, first out) or opens a new one. It never waits. Callers hold advisory locks and nest dedicated actions, so a waiting queue could deadlock. - On success the client returns to the idle set if fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed. - When the action throws, the client is closed and never reused. This covers a failed or cancelled transaction, a broken socket, and an interrupted lock wait. - When an action ends the client itself (the source-lock cancel path calls `$client.end`), the client is never reused. - Idle clients keep the owner's idle timeout and max lifetime, so postgres.js still closes idle physical connections after `DATABASE_IDLE_TIMEOUT_SECONDS`. - New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle set and stops retention. Ending the owner `$client` does the same. A later action still works with a connection that closes on release. - `server/src/index.ts`: shutdown closes the dedicated set next to the other private pools. - `doc/DATABASE.md`: documents the idle cap, the "not an admission limit" rule, and the transaction-scoped session state contract. - Session state audit: every current caller uses `pg_advisory_xact_lock` / `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or temporary tables. A reset round trip per call is therefore not added. ## Verification - New `packages/db/src/dedicated-connection-pool.test.ts` (embedded PostgreSQL, 10 tests). - Red on `origin/master`: 200 sequential calls used 200 distinct sessions (`expected 200 to be 1`). All 10 tests failed. - Green with this change, 3 runs in a row: 200 sequential calls use 1 session; 7 concurrent callers each hold a different session for the whole transaction, and afterwards at most 4 stay open and later calls reuse them; a failed transaction's connection is closed; a terminated socket during the action is not reused; an idle connection terminated by the server reconnects; a client the action ended is not reused; transaction-local settings do not leak; shutdown and owner `$client.end` close every retained connection; nested actions do not block. - Focused suites pass: packages/db `work-signals`, `prepared-queries-postgres`, `response-capture-postgres`, `checkpoint-pool`, `client`, `client-options`, `client-teardown-registry`, `postgres-connection-recovery`, `client-disconnect-replay` (132 tests). Server suites that use dedicated connections: `skill-package-store`, `status-card-evaluation-work` (updated: it now proves advisory locks are released, since sessions are reused), `agent-instruction-working-copies`, `workspace-recovery-retry-copyback`, `native-execution-lease-batch`, `company-skills-empty-runtime`, `decisions-service`, `legacy-controller-lease`, `server-startup-feedback-export` (467 tests). - `pnpm verify --explain` selects 2779 files for a `packages/db/src/client.ts` change. Per AGENTS.md I ran the focused checks above and let CI give the broad coverage. - Benchmark: 200 calls of `withDedicatedDbConnection(db, d => d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))` against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating before/after process pairs per mode. Main-thread time is `eventLoopUtilization().active`. | Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call (median, range) | | --- | --- | --- | --- | | TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) | | TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) | | Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) | | Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) | At the staging rate of about 21 calls per second, this removes roughly 100 ms of main-thread work per second (TLS figures; staging CPUs may be slower). ## Risks - A retained connection that the server or network drops while idle: postgres.js reconnects on the next query when it has seen the close (tested). If the drop and the reuse race, the first query of that action fails like a main-pool query would. Before this change, a dedicated action always started on a fresh connection. - Through a transaction-mode pooler, `SET application_name=...` is a server-side parameter switch, so its count may not drop as much as the client-side connection count. Measure the deployed effect with pooler client connections and main-thread CPU, not only that statement. - Concurrency above 4 opens extra connections exactly as before; they close on release. Total retained connections per owner grow by at most 4 idle clients, which still close after the idle timeout. - An action that sets session-level state would now leak it to the next action. The contract is documented on `withDedicatedDbConnection` and in `doc/DATABASE.md`; no current caller does this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c36b96ae00 |
fix(recovery): hold the start of queued automatic runs while a company's automatic-run brake is on (#21561)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The recovery system starts automatic runs: continuations, retries, and sweep replacements > - #21443 added a per-company automatic-run brake. It stops recovery from creating new automatic runs while a company's runs keep failing before they start > - The brake did not hold automatic runs that were already queued. The dispatcher still started them, so a burst of queued recovery runs kept failing in the same way > - This pull request adds the brake's start gate to the queued-run dispatch pass, and wakes a held company at its next try without a polling loop > - The benefit is that a failing company stops spending runs on automatic work, people's work still starts at once, and held work starts again on its own when the brake releases ## Linked Issues or Issue Description Refs #21443 Refs #21433 Refs #21427 Refs #21505 **What happened?** While the automatic-run brake was open for a company, its already-queued automatic runs (continuations and retries) still started. Each one failed before it started, for the same cause that opened the brake. **Expected behavior** A queued automatic run must stay queued while the brake is open. Only the brake's single probe, and then the half-open release rate (2, 4, 8 per window), can start one. A person's work (a Retry, or a continuation of a person's request) must never be held. Held runs must start again on their own, with nobody acting. **Steps to reproduce** 1. Make 20 automatic runs in one company fail before they start, so the brake opens. 2. Queue automatic continuations for that company. 3. Run a dispatch pass. Before this change, the continuations start. ## What Changed - `server/src/services/heartbeat/queue.ts`: the dispatch pass asks `reserveAutomaticRunStart` after the tried/backoff skip and the claim-attempt budget check, and before the budget increment and the full row read. A held run spends no claim attempt, does no row read, and does not enter the declined-claim backoff. If the brake grants the start (its probe or a half-open release start) but the claim then returns null, throws, or finds the row gone, the pass gives the start back, so another automatic run can use it in the same pass. The dispatcher's person-origin answer is passed as is, and it is `undefined` only when the dispatcher did not check origin (an agent with one queued run). Then the brake's own cached person check decides. - `server/src/services/heartbeat/queued-run-line.ts`: `queuedRunDispatchFacts` also returns `personOriginChecked`, so the queue knows when the dispatcher's negative answer is real. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch deadline worker adds companies whose next try is due to its scopes, and includes the next try time in its next wake. A held company wakes for its next try without a polling loop. If the brake's bounded wake list had to drop a company, the worker wakes all companies at the earliest dropped time. - `server/src/services/recovery/automatic-run-brake.ts`: when the brake holds a run, it records the company's next try time (probe time or the next release window) in a bounded map. A probe or half-open release grant returns a release that gives the start back with a compare-and-set on its own run or window. It exports `reserveAutomaticRunStart`, `takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and `isRestartClassFailureCode` from `continuation-cause.ts` directly. The rules are the same as before; the lazy loader keeps only the imports that would make an import cycle. - `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests through the real heartbeat queue): bounded queries for 250 held runs over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over 3 passes; a declined probe claim gives the probe back and another automatic run starts in the same pass, and the probe stays with that run on a later pass; an unused half-open start goes back to its window; with 276 held companies (past the 256-company wake list) every company is still woken; a work run behind 250 held runs starts in the same pass; a person's Retry is never held; the only queued run is not held when it continues a person's Retry; exactly one probe starts across two server instances; the release gives 2, 6, 14 cumulative starts over three windows; held runs are not in the declined-claim backoff after the pause; the dispatch worker wakes the company at the next try; and an end-to-end test that a held automatic run stays queued while the brake is open, then starts and succeeds on its own at the next try. - Two small test updates in `automatic-run-brake.test.ts` and `automatic-run-brake-import.test.ts` for the new cache size field and the moved rule. - `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`: one test for the sweep path from #21505 (continue a blocked task behind a run refused at dispatch). The sweep only records the continuation (no wake, no new run), and `deliverReconciledExecutions`, which the brake already gates, creates the run only after the pause. No second gate is needed on that path. ## Verification - `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11 passed. - Red before for the review fixes: on the first PR head, the 1,500-run cost test made 3,848 queries (bound 130), the declined-probe test started 0 runs (expected 1), and the 276-company wake test missed 20 companies. - Red before: with the `queue.ts` gate line removed, 5 of the 8 tests fail (bounded pass, single probe, release rate, backoff, end-to-end). - Red before for the person case: with the dispatcher's `false` passed through as final, "never holds the only queued run when it continues a person's Retry" fails (0 started). It passes after the change. - `automatic-run-brake.test.ts` 19 passed, `automatic-run-brake-import.test.ts` 1 passed, `automatic-run-brake-recovery-passes.test.ts` 28 passed, `failed-continuation-continues-again.test.ts` 32 passed (after merging master with #21505). - Nearby queue tests (on the final head): `heartbeat-run-dispatch-work` 15, `queued-run-dispatch-stacking` 6, `heartbeat-queued-run-claim-isolation` 10, `assigned-work-stalled-queued-run` 12, all passed. - `server` `tsc --noEmit` is clean. ## Risks - Behavior change only while a company's brake is open or half-open. When the brake is closed, the gate returns from its cache with no query. - The first time the gate sees a non-person automatic run while the brake is on, the brake does one person-origin query for that run. The result is cached per run (bounded to 1,024 runs). - No migration. No API or UI change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
84cb28f9b7 |
fix(connections): keep machine sign-in secrets and settings in one locked commit (#21566)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21473 added machine sign-in (MCP OAuth client credentials) for organization connections > - Auto-merge squashed #21473 from an earlier head, so its last review fixes did not reach master > - Those fixes close a P1 race and two smaller setup gaps that Greptile found on #21473 > - This pull request applies that last commit to current master, with its tests > - The benefit is that a client ID and its secret always stay together, and key switching and app selection work as intended ## Linked Issues or Issue Description Refs #21473 **What existing behavior does this improve?** Machine sign-in setup and rotation, added in #21473. **Problem** - A connection test wrote a replacement client secret before checking whether the connection changed. If the save was then refused, the secret stayed and could pair with another client ID, so the next token request failed. - Switching a client-secret connection to a Paperclip key always created an RS256 key, even when the sign-in service accepts only ES256. - Machine sign-in dropped the setup's app ID and created a separate app. **Proposed improvement** Write the secret and the settings in one locked transaction after the change check. Record the accepted signing algorithms and use them. Pass the app ID through. ## What Changed - `client-credentials-service.ts`: `saveConnection` locks the row, checks that the sign-in method and credential references are unchanged and that the connection is not removed, then writes any new credential and the settings in the same transaction. Test, secret rotation and key staging all use it. - Setup records `tokenEndpointAuthSigningAlgValuesSupported`. Switching from a secret to a key uses `chooseAssertionAlgorithm` on it. - UI: `MachineSignInSetup` takes `applicationId`, and the custom-server flow passes `prefill.applicationId`. - Tests: two new server tests and one UI expectation. ## Verification - `server/src/__tests__/tool-access-client-credentials.test.ts`: 25/25 pass. The 2 new tests fail on master's service code ("never leaves a refused test's secret behind with another client ID", "creates the kind of key the sign-in service accepts when switching from a secret"). - `MachineSignInSetup.test.tsx` and `GenericMcpConnectStep.machine.test.tsx`: 8/8. The app ID expectation fails on master's component. - Server `tsc --noEmit` clean. UI `tsc` clean except the pre-existing generated Storybook catalog imports. ## Risks Low. The change is limited to machine sign-in. A concurrent change now returns 409 `client_credentials_changed` instead of saving partly. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bbed32450c |
perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how many concurrent agents one server process can carry > - It assumed 20 text deltas/s and 1 other event/s per agent, and every other event was an `item.completed` > - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other events/s, with a mix in which 39% of the other events are server trace spans that take a cheaper path > - This pull request makes the recorded rates and mix the model's default, keeps the old rates as an explicit stress profile, adds a burst profile, and mirrors the delta batching of #21520 > - The benefit is a model that measures what production does, so the next optimizations target the real per-agent cost ## Linked Issues or Issue Description Refs #21418 (load model and plan). Refs #21520 (delta batching that the model now mirrors). **What existing behavior does this improve?** The load model's default rates, its event mix and its delta batching rule, and the plan document's measurements. **Subsystem affected** `scripts/benchmark-agent-scaling-load.ts`, its test `server/src/__tests__/agent-scaling-load-model.test.ts`, and `doc/plans/2026-11-13-thousand-concurrent-agents.md`. **Current behavior** The model defaults to 20 deltas/s and 1 other event/s per agent. Every other event is a native `item.completed` with a chat-progress wake and a checkpoint save. It cuts a delta batch on the timer even while the previous batch commits. **Proposed behavior** - `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute. `PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each rate can still be set on its own. - `EVENT_MIX=recorded` (default) sends other events in the recorded proportions. A `run.performance.span` is an ordinary append with no wake and no checkpoint, as the server's run trace writes it. `tool.execution.*` and `item.completed` are native appends with a chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one type only. - Delta batching follows #21520: deltas that arrive while a batch commits all join one next batch. - The report adds `profile` and `eventsIssuedByType`. **Reason and benefit** At the recorded rates the per-agent cost is the non-delta runner events and their checkpoint saves, not deltas. The plan and the next optimizations need that picture. **Breaking changes** None for the product. Anyone who runs the model with no options now gets the recorded profile; `PROFILE=stress` gives the earlier load. ## What Changed - Profiles, the recorded event mix (a 100-slot cycle per agent, offset by agent so the fleet sends the mix at every moment), the span append path, and the coalesced delta batches. - The model test now checks the profile in the report and that both server trace spans and runner events with a progress wake are in the mix, for the recorded profile and for the stress profile with one transaction per delta. - The plan has a new section "Load model at recorded rates": the recorded source (40 recent native runs on one hosted workspace, as aggregates), the profiles, the mix and its server paths, fresh runs at 500, 1,000 and 2,000 agents plus burst and stress, and what they show. It records the 3-connection checkpoint pool as a computed per-process limit on a hosted database, and that SQL building in Drizzle costs about as much main-thread time as result parsing on the hosted instance. A new section "Considered and not done" records why `run.performance.span` rows stay (about 3% of the statements per agent, with readers) and why checkpoint saves per critical event only are not done now. ## Verification - `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2 tests). The new assertions fail against the previous script: it reports no `profile` and no `eventsIssuedByType`. - `server/src/__tests__/agent-scaling-load-model.test.ts` also checks each profile's rates with no overrides, and that a rate set on its own wins. - Load model runs, recorded in the plan (60 s, main pool of 10, embedded PostgreSQL on the same machine, each run started below a 1-minute load average of 15): recorded rates at 1,000 agents (event-loop utilization 0.61) and 2,000 (0.85) finish every operation with no failure; 500 agents all in their busiest minute run at 0.95 with p99 lateness under 0.4 s. About 4.4 statements per agent per second at recorded rates, 35% of them on the checkpoint pool. - Earlier runs at load averages of 20–55 had shown one process falling behind at 500 burst agents and at 2,000 recorded; the quieter runs after the model fixes (event order, spans off the runner stream) do not, and the plan says so. The checkpoint pool as a per-process limit is now stated as a computed limit for a hosted round trip (about 400 agents in a busy minute at 1 ms), not as measured. - Greptile findings fixed: a waiting delta batch no longer takes deltas sent after the next runner event, takes every delta queued when it starts, and spans no longer join the runner event stream. ## Risks - Low: a benchmark script, its test and a plan document. No product code changes. - The machine that measured the plan's table was shared; the plan marks the rows that the machine distorted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6dd0349d52 |
fix(ui): name a queued turn's hold once, in the turn header (#21564)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's chat shows its run as a turn: a header line with the run's
state, then the turn body
> - #21405 and #21447 made a queued turn name why it waits: "Waiting for
Codie · …", "Waiting for PAP-5656 to finish", "Waiting to start"
> - On staging, the turn body repeated that label under the header
("Waiting for PAP-5656 to finish" twice). The tail used by other
adapters showed a fixed "Waiting to start..." even while the pill above
said "Working" or named a hold
> - #21447 removed the repeat only for the agent's waiting line
> - This pull request names every queued hold once, in the turn header
> - The benefit is one clear statement of why a task waits, with no
repeat and no contradicting line
## Linked Issues or Issue Description
Refs #21405, #21447
**What happened?**
On staging (revision
|
||
|
|
351b69941c |
perf(checkpoint): write a routine controller checkpoint save in one statement (#21553)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote runner's controller saves a checkpoint after each runner event it commits, on a 3-connection pool, before it acknowledges the event > - Each save is a transaction of 4 statements (5 where commit notifications are on): BEGIN, the owner lock with the transaction id, the write, COMMIT > - What forces the transaction is the live-run recovery wake that every save registers. Recovery does not react to a routine save > - This pull request writes a routine save (new or changed event, command and cursor records) with one statement that checks the owner and writes, and keeps today's transaction for every other save > - The benefit is one statement instead of four for most checkpoint saves, on the pool that limits one process in a busy minute ## Linked Issues or Issue Description Refs #21528 (owner lock returns the transaction id). Refs #21549 (load model at recorded rates; the plan records the checkpoint pool as a per-process limit). **What existing behavior does this improve?** `nativeRunnerCheckpointWriter` in `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Subsystem affected** Controller checkpoint persistence for remote native runs. **Current behavior** Every save locks the coordinator row (`native_run_finalizations` `for update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery queues), and upserts the changed parts, in one transaction. **Proposed behavior** A routine save is one statement: a CTE locks the coordinator row only while the lease owner, boot id, generation and an unexpired lease (`clock_timestamp()`) all match, and the upsert writes only through that row. It registers no recovery wake. If the owner does not match, it writes nothing and the save takes today's full path, which runs the complete owner check (successor rules included) and fails with `native_runner_checkpoint_owner_lost` when it should. A save is routine when it has changed parts, no reset, no removed parts, no header change, no predecessor, and the run is its own authority. Every other save (the first save of a controller, which is a reset, a save after a failed save, header or retirement changes, removals, source-run consumers) is unchanged. **Reason and benefit** At recorded rates about a third of one process's statements are checkpoint saves. In a busy minute the 3-connection pool cannot keep up at 500 agents (#21549). **Breaking changes** None. No schema change. ## What Changed - `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in `nativeRunnerCheckpointWriter`. ## Why a routine save needs no recovery wake I read the six consumers of `notifyLiveRunRecovery`: - `nativeProcessSessions`, `nativeQuestionCancellations`, `activeExecutionRecovery`, `settledNativeProcesses` (`live-recovery-work.ts`): each selects candidates from `heartbeat_runs` status, runtime mode, `native_run_finalizations` phase, lease and retry times, the context snapshot and process identity. None reads checkpoint rows to decide eligibility. - `strandedAgentStatus` (same file, `agents` rows) and `activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`, `heartbeat_runs.last_output_at`): neither reads checkpoint rows. - The checkpoint rows matter to recovery in two places: `listExpiredNativeRunnerAuthorities` requires that a checkpoint exists, and the remote bootstrap proof reads its parts. A checkpoint first appears with a reset save, which keeps its wake. A routine save only adds event and command parts, which can only take a run further from the bootstrap proof; recovery reads the current rows when its own deadline (lease expiry) fires. - The intent of the wake also fenced a recovery pass against a save in flight. The routine save is one atomic statement and holds the coordinator row lock while it writes, so a recovery claim that takes that row waits for it, and a reader sees the save whole or not at all. ## Verification - Red/green on embedded PostgreSQL (`native-runner-checkpoint-save.test.ts`); the new tests failed against `origin/master` first: - A routine save is one statement (`with owner as ... for update ... insert into native_runner_checkpoints`) and registers no recovery wake (before: 4 statements and a wake). - A reset save keeps BEGIN, the owner lock with the transaction id, and COMMIT, and still registers its recovery wake. - A routine save that waits while another connection takes over the lease (new owner and generation) falls back to the full path and fails with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged. - The same race where the new owner also resets the checkpoint (deletes every part and writes its own header): the save fails the same way and the checkpoint holds only the new owner's header. - A lease that expires while the routine save waits for the coordinator row lock (the holder does not change the row) writes nothing: the statement judges the lease after the lock, as `lockNativeRunnerCheckpointOwner` does. My first version checked the lease in the locking CTE itself, and PostgreSQL does not re-check a condition after a lock wait on an unchanged row; the existing `native-runner-restart-recovery.integration.test.ts` test "fences checkpoint delivery when ownership expires during an unchanged row-lock wait" failed on it, and so does the new test. Both pass now. - The existing test "refuses a save after the owner's lease expired and writes nothing" passes through the new fallback. - Mutation checks: with the owner conditions removed from the statement, both race tests and the expired-lease test fail; with the lease check moved back into the locking CTE, the lock-wait expiry test fails. - Other focused suites, one file at a time: `native-runner-restart-recovery.integration` (231 passed, 14 skipped), `native-runner-restart-checkpoint-cost.integration` (1; it greps for `insert into "native_runner_checkpoints"`, so the statement quotes the table name), `native-recovered-owner-followup` (97), `native-controller-recovery.integration` (30), `native-runner-checkpoint-authority.integration` (1), `native-restart-recovery-isolation` (1), the port suite with conformance (105), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51 tests pass; on this busy machine its teardown hook timed out, as it has before on master). `pnpm --filter @paperclipai/server run typecheck` passes. - Statements per routine save: 4 → 1 (5 → 1 where commit notifications are on). I did not rerun the load model for this change: the machine is too busy for a fair timing comparison today, and the statement count is exact. - CPU per operation (a local script, not committed: 600 routine saves of one event part and the cursor for one run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after (encryption included in both); wall time per save 0.45–0.73 ms before and 0.22–0.30 ms after. ## Risks - A routine save no longer wakes the six recovery queues. Their own deadlines and the 30 s reconstruction still run; see the reading above for why no decision depends on a routine save. - The owner check of the routine statement accepts only the exact active owner. Any other state goes to the unchanged full path, so a successor or a finished result keeps today's rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
67652580ee |
perf(tool-access): index enabled policies so each decision checks only candidates (#21563)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).
**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).
**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).
**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.
**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.
**Breaking changes**
None. The policy data model, API, and listing contract do not change.
## What Changed
- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.
## Verification
- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.
| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
| --- | --- | --- |
| Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
| Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
| Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
| Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
| Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
| Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
| Selector evaluations | 305,600 | 1,164 |
| Raw selector property reads | 6,155,830 | 9,490 (compile only) |
```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
```
- Not verified: the hosted staging profile after deploy.
## Risks
- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
82ce8e64ba |
feat(runner): let the session driver admit a turn that the provider started (#21453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Core reads Runner events through the Codex app-server driver, also for ACPX agents. The driver refuses a turn that the controller did not start, except a goal turn of Codex. > - This pull request lets the driver accept a turn that the provider started, only directly after a turn that the Runner marked as waiting. > - The benefit is that the next parts can use that turn. Nothing reaches this code until the last part sends the opt-in. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 1. Its base branch is `feat/adopt-claude-provider-started-turn`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the driver does with a turn that the provider started. **Subsystem affected** Paperclip Runner package, the controller side: `src/drivers/codex` (the driver that Core uses to read a native session). **Current behavior** - A `turn/started` that the controller did not ask for fails the session with `turn_binding_mismatch`, except for an active Codex goal. - An interrupt with no active turn fails with "already terminal". **Proposed behavior** - A turn terminal that the Runner marked with `awaitingProviderTurn` and that has no result puts the driver in a waiting state. The driver event `turn.completed` carries the marker. - In that state, a `turn/started` with `origin: "provider"` is admitted the same way as a goal turn, and the driver event carries `origin: "provider"`. In each other state it still fails closed. - In that state with no active turn, an interrupt gives an acknowledgement item with the status `provider_turn_wait_ended`. It sends nothing to the Runner. The driver also gives that acknowledgement when it gets the internal `paperclip/runTerminal` notification with a private marker; a notification without the marker does nothing. - An interrupt for a provider-started turn that is running is sent to the Runner only when it names that turn. - The driver saves the waiting state in the persisted session (`awaitingProviderTurn`, written only during a wait). After a controller restart, the restored session still admits the provider-started turn and still acknowledges an interrupt during the wait. The driver restores the state only when the session has no active turn, no stored result and at least one finished turn, so a new run cannot take the wait of the run before it. - When a restart finds that the saved active turn is finished, the turn read of the transport and `reconcile()` of the driver carry the same marker as the live `turn/completed` notification. `runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from the saved `turn.completed` event to the turn object of a read. - The driver ends the waiting state when the Runner reports the run terminal. **Reason and benefit** The consumer in the next part needs these three signals: a marked terminal, an admitted turn, and a clear end of the wait. ## What Changed - `codex-session-terminal.ts`, `codex-session-notifications.ts`, `codex-session-state.ts`, `codex-harness-session.ts`, `codex-driver-values.ts`: 37 lines added, 4 removed. - `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines). - `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`, `backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`: the optional field in the two persisted shapes and its copy in each direction. - `live/runnerd-codex-transport.ts`: the marker on a recovered turn terminal. The loop that builds the earlier terminals of a `thread/read` is now the function `runnerdPriorTurnTerminals`, with the same logic, so that a test can call it. ## State saved by the previous release - One new optional field in the persisted driver session: `awaitingProviderTurn`. A snapshot outside a wait has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so the driver does not wait. This is the behavior of today. - **State from this release, read by the previous release:** the previous reader copies the fields that it knows and ignores this one. No reader rejects it, and the integrity digest covers the stored object as written. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - 6 driver tests: a provider-started turn after a marked terminal is admitted and can report a result; a provider-started turn with no marked terminal fails closed; an interrupt during the wait is acknowledged; a running provider-started turn is not interrupted by the end of the wait; the marked notification ends the wait and a notification with no marker does not; a normal terminal is not changed. - Restart tests: a session restored from a snapshot with the waiting state admits the provider-started turn and settles its result; a snapshot outside a wait has no new field; a restored session with no waiting state refuses a provider-started turn as today; an interrupt during a restored wait is acknowledged. A saved active turn that the read reports finished with the marker gives a `turn.completed` with `awaitingProviderTurn: true`; with no marker it is an ordinary terminal. - `harness-driver-backend.test.ts`: the field goes through a serialized checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324 passed): the marker is on a recovered terminal only for a `turn.completed` with a boolean `true`. - `src/native-session-runtime.test.ts`: 297 passed, with the real `paperclip-runnerd`. - Mutation checks for the restart work, each made a named test fail: no restore of the state; no copy in the backend; no copy in the transport; no marker argument in `reconcile()`; no end of the wait at the run terminal. - Mutation checks: the acknowledgement branch removed, the admission removed, and the marker check removed each made a test fail. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** The Runner sends the marker only after the opt-in of part 4. - **The admission rule is shared with Codex goal turns.** The existing goal tests pass with no change. - **Known limit after a controller restart.** If the Runner already runs a provider-started turn and the saved active turn is the turn before it, recovery fails closed, as today for each unexpected active turn. - **A controller turn that starts during a wait** clears the waiting state when its `turn/started` arrives, not when it is sent. A snapshot in that short time can hold an active turn and the field together; the restore rule drops the field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
ca7d42ba71 |
fix(heartbeat): refresh queued dispatch when task drains end (#21562)
## Thinking Path > - Paperclip runs agent work for a company. > - A task drain holds new run starts while maintenance or idle handoff is in progress. > - A queued dispatcher can keep an unreconciled request and a 30-second retry while the drain is active. > - Ending the drain can resume domain work without refreshing that dispatcher's saved deadline. > - This change refreshes the pending queued owner when a drain ends or its expiry is observed. > - Normal admission checks and real retry deadlines still apply. ## Linked Issues or Issue Description Related: #20640. That change refreshes deferred receipt workers. This change refreshes the separate queued-run dispatcher. **What happened?** A queued-work notification received during a drain could leave a 30-second fallback in the dispatcher after the drain ended. The owner stayed unreconciled after useful work had settled. This delayed its next pass and the instance's idle state. **Expected behavior** Refresh an unreconciled queued owner when normal dispatch becomes allowed. Keep other drain holds, admission rules and scheduled retry deadlines. **Steps to reproduce** 1. Let the normal dispatcher complete its first pass. 2. Start an idle task drain and commit a company-scoped queued-work notification. 3. Stop the drain, or observe its expiry, before the saved fallback is due. 4. Check that the scoped queued pass runs before that fallback. The two regression cases fail on the unchanged owner and pass with the fix. **Paperclip version or commit** The main-specific patch is based on `f7b1651799d8c0c660a76266878c420a12f0beca`. The bug was found in assembled sleep acceptance. The affected queued-dispatch, drain and notification modules match main. ## What Changed - Listen for normal drain changes and observed expiry in the queued-run owner. - Wake that owner only when dispatch is allowed and its reconciliation is pending. - Forward the deadline worker's optional stop callback after its stop fence. - Remove both subscriptions when the owner or coordinator stops. - Add release, expiry, remaining-gate, shutdown, retry-deadline and idle controls. ## Verification Local checks used the integrated feature branch that exposed the issue. The main-specific port preserves main's existing `canRun` logic and excludes the private branch's extra cancellation hook. - Both new release and expiry cases failed at the intended scoped-resume assertion before the product change. - Six dispatch cases pass with the normal package import. - Fourteen existing deadline controls and nineteen queue-order controls pass. Their inputs did not change during the test-import correction. - The complete normal Server typecheck and its three affected-file program checks pass. - The first typecheck caught a test import outside the Server root. The import now uses the existing public package wildcard. The corrected six-case suite and normal typecheck pass. - Normal Source SDK input verification passes. - Exact-head CI at `fd623d3ba9345e5ec8f252c55932ff797cd57e8f` completed: 54 passing checks, four expected skips and one neutral result; Snyk passed. Greptile reviewed this exact head at 5/5 with no actionable findings. All review threads were read; none remain open. A full local repository build and suite were not run in this checkout. The existing admission-drain contract in `doc/execution-semantics.md` already requires prompt admission after drain release. This fix restores that contract. No configuration or command documentation changes are needed. ## Risks Drain events can request an earlier pass for a pending owner. The normal permission, reconciliation, coalescing and earliest-time checks remain active. The scheduled retry owner has no new listener. Shutdown must remove both listeners; the coordinator-stop control covers that path. This PR does not claim deployed suspension or complete transport sleep acceptance. ## Model Used OpenAI `gpt-6.1-sol` assisted with implementation and independent review. The Codex managing agent performed technical review and verification. The host does not expose its exact runtime model ID or context-window size. Tools used include source inspection, code edits and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (not required: the existing contract remains accurate) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
9824277f65 |
perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Most runner events that are not text deltas (`tool.execution.started/progressed/completed`, `item.completed`) wake the chat progress projection > - That wake is registered inside the append's transaction, so each such event takes 4 statements (BEGIN, the run lock read, the append, COMMIT), where an event without a wake takes 1 (#21511) > - The chat progress projection rebuilds itself from the stored events; the wake only tells it when to look > - This pull request writes an event whose only wake is chat progress with the one-statement append, and sends the wake after that statement commits > - The benefit is 1 statement instead of 4 for most non-delta runner events, with response, reassessment, retention and recovery wakes unchanged ## Linked Issues or Issue Description Refs #21511 (one-statement append). Refs #21528 (lock read returns the transaction id). Refs #21549 (load model at recorded rates). **What existing behavior does this improve?** `appendHeartbeatRunEvent` for native source events whose only wake is chat progress. **Subsystem affected** Server run event ingest (`heartbeat-run-events.ts`), the chat progress notification helper, and two additive exports in `packages/db/src/work-signals.ts`. **Current behavior** A progress event opens a transaction, reads the run lock with `runtime_mode`, the bound task and the transaction id, probes for a replay, registers the chat-progress wake as a tracked intent, appends, and commits. **Proposed behavior** When chat progress is the event's only wake, the append takes the one-statement path of #21511. That statement also returns the run's `runtime_mode` and bound task id from the row it locks, and, where commit notifications are on, calls `pg_notify` inside the same statement for runs that are native and bound to a task, so other server processes get the wake at commit. After the statement returns, this process publishes the wake as settled to its own listeners. A replay, a gap, a changed run or any other fallback takes the existing transaction, which registers the wake as before. **Reason and benefit** At recorded rates these events are about 61% of the non-delta events (#21549), so this cuts about 3 statements per such event. **Breaking changes** None. No schema change. ## What Changed - `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other wake clear) also qualifies for the one-statement path. - `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`, rendered once) from the locked row, plus a conditional `pg_notify`. - `chat-native-progress-notifications.ts`: `chatNativeProgressCommitNotification` and `publishChatNativeProgressSettled`. - `work-signals.ts` (additive): `databaseWorkCommitNotification(owner, topic)` returns the cross-process payload when notifications are on; `publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled wake to this process's listeners. ## Verification - Red/green on embedded PostgreSQL. Each new test failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "appends an event whose only wake is chat progress in one statement and wakes progress after it commits": 1 statement (before: a transaction), the wake is published once as settled with no intent, and a replay publishes nothing. - "keeps the transaction for the last events of a turn and a run, and one statement for progress-only events": `tool.execution.started/progressed/completed` and a non-final `item.completed` take one statement; a final `item.completed`, `run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled` and `turn.interrupted` keep the transaction. - "sends the progress wake to other server processes with the one statement's commit": with commit notifications on, another connection that listens on the work channel receives exactly one chat-progress notification, and a replay sends none. - `chat-native-progress-work.test.ts` "wakes the projection from a one-statement progress event after the statement commits": the real chat progress worker publishes the progress for an event appended in one statement. - Mutation check: without the settled publish after the statement, that test fails. - Lost-wake repair (condition b): an event row stored without any wake models a lost hint. - "repairs a lost progress wake with the run's next progress event": nothing is published for the stored event; the run's next progress event publishes the progress. - "repairs a lost progress wake with the startup reconcile": a worker that starts after the stored event publishes it. - The existing response-wake test now uses a final reply and still shows the intent before the write, and the race test of #21528 uses a final reply so it still covers the transaction path. - Other focused suites, one file at a time: the port suite with conformance (105), `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25), atomic-append (31), `chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - CPU per operation (a local script, not committed: 800 `tool.execution.progressed` appends to one native task run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms after in the warm rounds; wall time per event 0.84–1.04 ms before and 0.52–0.53 ms after. - Inside a caller's transaction: new test "keeps a progress-only event inside a caller's transaction on the locked path, and settles its wake only when that transaction ends". `publishDatabaseWorkSettled` publishes at once only outside a transaction; inside one it waits for the outermost commit and drops the wake on rollback (new `work-signals.test.ts` test, which failed with the unguarded publish). The one-statement path is only taken when the owner is not a transaction (`!is(db, PgTransaction)`, #21511), so the two guards agree. ## Risks - **Lost wake.** If the statement's outcome is unknown (the connection drops after the server ran it), this process sends no wake for that event. In a chat channel (Slack, Telegram) the progress line for that task can then stay one step behind (for example it does not move from "working" to the latest tool step) until the run's next runner event, a run milestone, or a server restart. A turn normally ends with an event that keeps the transaction (`turn.completed/failed/cancelled/interrupted`), and the run's completion notifies chat run milestones, which include chat progress (`notifyChatRunMilestones` in `run-completion.ts` and `native-run-finalizer.ts`). So progress cannot stay stale after a run ends; in the worst case it lags one step for the rest of a turn. The projection rebuilds from stored events, so nothing is lost or duplicated. - The chat-progress wake no longer publishes an intent before the write. Intents let the coordinator fence a pass against an in-flight write; the settled wake after the commit makes the projection run again after the row is visible. - `packages/db/src/work-signals.ts` gets two additive exports and no change to existing behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c0c7c6abb |
perf(ui): load the @task mention pool after the task page settles; add a cold-load perf spec (#21557)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People often open a task by its URL, for example from a link in a message or after a reload > - #20592 made a hard load of a task URL read the chat's first frame from one `chat-bootstrap` request > - The task page still sends a large read as it mounts: 100 full recently updated tasks for the `@task` mention picker. On staging this read returned 395 KB and took 3.9 s of server time > - The hosted database is CPU-bound, so reads that do not affect the first frame must not compete with the reads that do > - Also, no test measured a cold direct load under realistic latency or guarded the one-bootstrap request shape > - This pull request starts the mention pool after the task page settles. It also adds a cold-load perf spec with a delay on each API request and an always-on request-shape test > - The benefit is less concurrent server work during a cold task open, and a regression guard for the one-round-trip first frame ## Linked Issues or Issue Description Refs #20592 (hard-load chat bootstrap that this pull request measures and guards). Refs #20355 (the `chat-bootstrap` endpoints). **What existing behavior does this improve?** The task page loads the `@task` mention pool when it mounts. **Subsystem affected** Board UI task page (`ui/src/pages/IssueDetail.tsx`) and the issue-detail perf harness (`tests/perf/issue-detail`). **Current behavior** `GET /api/companies/:id/issues?limit=100&sortField=updated&sortDir=desc` starts with the chat's first-frame reads. It starts before the first chat frame in 14 of 14 local cold loads. The picker needs it only after the person types `@`. **Proposed behavior** The read starts after the page has painted and its own reads have finished. On a page that keeps reading, it starts after at most 5 s. **Reason and benefit** The read never affects the first frame. Deferring it removes a large concurrent read from the busiest part of a cold open. **Breaking changes** None. If the person types `@task` in the first moments after a cold load, the task suggestions can appear about half a second later. ## What Changed - `ui/src/hooks/usePageSettled.ts`: new hook. It returns `true` once `whenPageSettles` runs (the page has painted, the tab is visible, no query is fetching, and the browser is idle), or after `PAGE_SETTLED_MAX_WAIT_MS` (5 s). The cap keeps a long-running read from keeping the picker empty. - `ui/src/pages/IssueDetail.tsx`: the mention-pool query is enabled only when `usePageSettled()` is true. - `ui/src/hooks/usePageSettled.test.tsx`: tests that the hook waits for in-flight reads and that the cap applies. - `ui/src/pages/IssueDetail.test.tsx`: mocks the hook to `true`, so existing tests keep their behavior. - `tests/perf/issue-detail/cold-bootstrap.perf.spec.ts`: new spec. It delays every API response (300 ms by default) and opens the task URL in a new browser context, with the JavaScript in the HTTP cache and no app storage. The finish event is the first readable frame: the history is revealed, the latest saved message is visible and the composer is editable. - The request-shape test always runs. It requires one `chat-bootstrap` request and no per-endpoint first-frame read before the frame. Reads that start after the live socket is created are excluded, because they are its catch-up refresh. - The benchmark runs only when `PAPERCLIP_ISSUE_PERF_COLD_RUNS` is set. It writes the median, the spread, the request count before the frame, the last requests before the frame, the JavaScript bytes transferred and a request timeline. - The delay uses CDP `Fetch` interception instead of Playwright routing, because routing disables the HTTP cache. - `tests/perf/issue-detail/README.md`: documents the spec. - My earlier draft also added a deadline for a stalled bootstrap. Master already has one (`CHAT_BOOTSTRAP_SHARE_DEADLINE_MS`, 5 s), so this pull request does not include it. ## Verification Measurement setup: - Production UI builds of master `4a78124a3e` and of this branch, served by one local server (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`, embedded PostgreSQL). - A delay of 300 ms on each API request. - Cold direct task URL with 24 saved comments and a plan. - Master and the branch alternated: 2 rounds of 7 runs each. - The host was shared with other agent sessions. | Build | Runs | First readable frame, median | p25–p75 | Min–max | API requests before frame (median) | Mention pool started before frame | |---|---:|---:|---:|---:|---:|---:| | master | 14 | 1,422 ms | 1,336–1,525 | 1,284–7,398 | 50 | 14 / 14 | | this branch | 14 | 1,390 ms | 1,338–1,457 | 1,272–2,207 | 49 | 0 / 14 | - The spec adds the API delay through CDP request interception, so the HTTP cache stays on. The measured loads downloaded 0 KiB of JavaScript in every run. The spec records this as `jsTransferBytes`. Each build has one outlier (7,398 ms on master, 2,207 ms on this branch), from host contention. - With this branch, the mention pool starts after the frame in every run (350–400 ms after it in an earlier set). - The first-frame timing is neutral locally. The local database answers the pool read in milliseconds, so it does not compete with other reads. The intended benefit is lower concurrent server load on a busy database. This pull request does not show it as a local latency gain. - On master and on this branch, the hard load adopts the bootstrap and the frame is no longer held by task reads. The frame now comes right after the shell reads return: `auth/get-session` and `instance/settings/experimental`, then `companies` and `workspace-preferences/me`. Profiling marks confirm this: the chat thread mounts after `workspace-preferences/me` returns, and all readiness gates are ready within 30 ms. - The request-shape test passed 5 of 5 on master and 5 of 5 on this branch. - `layout-stability.spec.ts`, `plan-startup.spec.ts` and `cold-bootstrap.perf.spec.ts` against this branch's build: 18 passed. - `ui` Vitest for `usePageSettled.test.tsx` and `IssueDetail.test.tsx`: 278 passed. The `ui` typecheck passed. `pnpm check:token-gates` passed. - `pnpm verify` exits 2 because `tests/perf/issue-detail/README.md` is a global CI input. Required CI supplies the broad coverage. To reproduce the measurement: ```sh PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:3287 PAPERCLIP_ISSUE_PERF_COLD_RUNS=7 PAPERCLIP_ISSUE_PERF_LABEL=branch \ pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts cold-bootstrap -g benchmark ``` ## Risks - Low risk. The only product change is when one read starts. - If the page keeps a query in flight, the pool waits up to 5 s. During that time, `@task` suggestions list only agents, projects and members. - The request-shape test treats reads that start after the live socket is created as catch-up refreshes. If the socket connects before the task page mounts, such a test would not see a first-frame read that started later. The test still requires exactly one bootstrap request before the frame. - The perf spec is not in the default CI browser suites. It runs when the issue-detail harness runs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing, Playwright browser runs). It wrote the change, the tests and the measurements. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.2 |
||
|
|
54c3616ad9 |
fix(recovery): ignore the non-assignee comment wake marker in the failed-preparation proof (#21550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a Runner run fails in setup before any provider work, a person's Retry must start a new run > - Admission proves "no provider work" for a failed preparation by checking that the run record holds only known, harmless fields > - The run profile allowlist did not include `aiConnectionNonAssigneeCommentWake`, a boolean that dispatch writes when it admits a run from a non-assignee comment > - So a run with that marker never passed the proof. Its Retry waited on `previous_execution_unverified` forever, even after saved-work recovery finished and the workspace was handed on unchanged > - This pull request accepts the marker (boolean only) in the proof, and in the related workspace-wait parent proof > - The benefit is that these tasks get a working Retry, and a Retry that was saved earlier starts by itself once the proof passes ## Linked Issues or Issue Description Refs #21481, #21497. **What happened?** On a hosted staging instance, a Runner run failed in setup ("RPC call environmentExecute timed out") while it prepared. Saved-work recovery finished ("Saved work recovered."). The workspace was handed on unchanged for this run. A person pressed Retry two times. Both Retries waited on `previous_execution_unverified`, and no run started. **Expected behavior** A run that failed in setup before any provider work (`executionRecovery.providerWorkStarted: false`, no process, no execution events, stopped sandbox) gets a working Retry. A saved Retry starts by itself when its gate clears. **Steps to reproduce** 1. Let dispatch admit a Runner run from a non-assignee comment, so the run profile has `aiConnectionNonAssigneeCommentWake: true`. 2. Make the run fail in setup during preparation. 3. Press Retry. See that the request waits on `previous_execution_unverified`. **Paperclip version or commit** Observed on staging at `e82c4f0a31`; still present on `master` at `6b631dd477`. ## What Changed - `server/src/services/cancelled-native-startup.ts`: the failed-preparation proof accepts `aiConnectionNonAssigneeCommentWake` when its value is a boolean. The workspace-wait parent proof ignores the same key. - `server/src/services/legacy-preparation-retry-hold.test.ts`: staging-shape tests for a recovered setup failure (failed restore, a workspace that still names its location, the marker, all together), and a Retry that attaches to an earlier refused request and then starts. ## Verification - `cd server && npx vitest run src/services/legacy-preparation-retry-hold.test.ts`: 12 passed. Without the fix, the `comment_wake_marker`, `all` and `attached_after_refusal` cases fail (red), and the other cases pass. - Focused test file only (the machine was loaded). CI supplies typecheck, build and the full suite. ## Risks - Low risk. The marker only records why dispatch admitted the run. Every other proof clause (no process, no execution events, no native result, stopped leases, settled preparation evidence, workspace repair) is unchanged. A non-boolean value still fails the proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6b9e45787f |
fix(recovery): continue a task whose system retry was refused at claim behind a saved-work hold (#21505)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, automatic recovery continues the task, through the fresh-continuation fallback, in a new run that starts from the saved work > - A follow-up run (for example one started by a new comment or by finished subtasks) can fail before any provider work and leave a saved-work hold. The system then queues its own retry, and the dispatch gate cancels that retry at claim with `execution_reconciliation_required` because the hold is still in place > - The fallback takes over the hold when that happens, but it continued only "recent interruptions". That test required the failed run to be a recovery run itself or to fail with a restart code. A first follow-up run that failed with `adapter_failed` passed neither test, even though the system had already decided to retry it. So the fallback answered "a person must press Retry". The backlog pass used the same test, and the assigned-work sweep never listed the task. The task stayed blocked with no run, no pending path and no Retry button > - This pull request treats a system retry refused at dispatch as proof that the system continues the task. It also stops cancelled-at-dispatch attempts from counting against the hourly budget, lets the sweep continue a blocked task whose latest run was refused at dispatch, and keeps recovery notices from asking for a Retry that the task does not offer > - The benefit is that such a task continues by itself in exactly one fresh run from the saved work, whether the saved work was recovered before or after the refusal, and a person's own failed Retry keeps today's behaviour ## Linked Issues or Issue Description Refs #21394, #21348, #21363, #21478 **What happened?** A task's follow-up run, a legacy run that was not a recovery run, failed before any provider work with `adapter_failed`. Its files could not be restored, so it left a saved-work hold (`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure: restore_failed`). The system queued a retry of that run. The dispatch gate cancelled the retry at claim with `execution_reconciliation_required`, and the "older run hold" path handed the hold to the fresh-continuation fallback (`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback found the refused run and asked `interruptionEligibility` whether the interruption was recent. That predicate required `continuesAutomaticallySql(source)` (a recovery run, or a restart, deploy or bootstrap code), so it returned `person_owned` / `retry_needed`. The stranded reconciler then blocked the task with "Press Retry", but the saved-work hold reports that Retry is not available. This happened twice in a local restart-chaos wave. In one case the saved work was recovered two minutes before the claim. In the other it was recovered thirty seconds after the claim. **Expected behavior** A task never rests blocked with no live run, no pending path and no Retry. While the saved-work recovery runs, that recovery is the pending path. When it finishes, one continuation starts from the saved work through the fallback (isolation, the shared hourly cap). A run that was cancelled at dispatch before it started does not use up the automatic budget. A person's own failed Retry is still left for that person. **Steps to reproduce** 1. Fail a follow-up run (wake reason `issue_children_completed`) before provider work with `adapter_failed` and `workspaceRestoreFailure: restore_failed`. 2. Queue the system's retry of that run (`retryOfRunId`, wake reason `issue_continuation_needed`) and claim it. The dispatch gate cancels it with `execution_reconciliation_required`. 3. Before this change, no continuation is recorded, the stranded reconciler blocks the task, and nothing ever continues it. **Paperclip version or commit** master at |
||
|
|
f7b1651799 |
feat(acpx-sidecar): adopt Claude's own cycle as a provider-started turn when the Runner arms it (#21316)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The parts before this one give the adapter an ordered stream of those cycles (#21421), declare the contract (#21422), and make `paperclip-runnerd` wait for such a turn (#21423). > - This pull request is the last part in the Runner. The sidecar reads the stream, and gives a cycle to `paperclip-runnerd` as a provider-started turn only when `paperclip-runnerd` armed it. > - The benefit is that a later change in Core can let a run continue when background work finishes. Until Core sends the opt-in, a production run does the same as today. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request was first opened with the full change (49 files). It is now split. The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. This pull request: the sidecar adopts a cycle. Its base branch is `feat/runnerd-await-provider-turn` (#21423). The code at the head of this branch is the same as before the split; only the base changed. Review comments made before the split are about code that is now in the parts above. Refs #19912 (design). **What existing behavior does this improve?** What the ACPX sidecar does with a cycle that Claude starts by itself. **Subsystem affected** Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts`. Claude only. **Current behavior** - The sidecar does not read the cycle stream of the adapter (#21421), so each cycle is declined and the background state becomes unknown. - The sidecar does not know the command `session.provider_turn`, so `paperclip-runnerd` (#21423) never waits. **Proposed behavior** - The command `session.provider_turn` with `armed: true` arms the sidecar for one turn. The sidecar accepts it only for the terminal that reported a running background shell, with no active turn and a provider that is alive. One flag holds the arming. It is cleared at each turn start and turn end, on suspend and close, and when the provider is lost. - One loop, `consumeProviderCycles`, reads the stream. For each cycle: - if the sidecar is armed and no turn is active, the sidecar adopts the cycle: it emits `runtime.turn_started` with `{ "origin": "provider" }`, then the events, the usage and the terminal of the cycle, as for a normal turn; - if the cycle cannot be decided yet (the prompt turn is still settling, or `paperclip-runnerd` did not arm yet), the sidecar holds it for a maximum of 5 seconds, and then adopts or drops it; - in each other case the sidecar drops the cycle. The stream ends, and the background state becomes unknown, so `paperclip-runnerd` restarts the provider at the end of the turn as today. - A Paperclip tool call or a permission request of a held cycle waits for the decision of its own cycle. It is never bound to the prompt turn or to a different cycle. - If an adopted cycle proposes a plan (`ExitPlanMode`), the sidecar publishes the same `plan.proposed` event as for a controller turn, and the turn ends with the same terminal: completed, with no `backgroundTasks`, so `paperclip-runnerd` does not wait again. A controller turn is not changed. - If the provider exits while the sidecar is armed with no turn active, the sidecar sends a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`, and `paperclip-runnerd` ends the wait. - `turn.cancel` for a provider-started turn cancels the cycle and closes the provider. - With no arming, nothing is shown. No controller sends the opt-in yet, so `paperclip-runnerd` never arms. **Reason and benefit** - Only `paperclip-runnerd` decides that a cycle becomes a turn. The sidecar cannot start a turn by itself. - One reader of one ordered stream means that two cycles cannot overlap. ## What Changed - `src/cli/acpx-runtime-sidecar.ts` (127 lines added): the command, the arming flag, `consumeProviderCycles`, `adoptProviderCycle`, `pumpProviderTurn`, and the wait of `waitForTool` for a cycle that is not decided. - `src/cli/acpx-runtime-sidecar.provider-turn.test.ts` (new, 320 lines) and two fake hosts in `src/test-support` (227 lines). One host follows a script. The second runs the real cycle stream and the real background task tracker. - `src/cli/acpx-runtime-sidecar.test.ts` (2 lines): one more parameter for a test harness. No schema, no Rust and no persisted state changes in this part. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `pnpm run typecheck:typescript`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript`: no errors. The server check `node scripts/verify-runner-vendor-dependencies.mjs`, the cache guard and the contract tests: passed. - `npx vitest run src/cli src/drivers/acpx src/live/runnerd-acpx-provider-turn-wait.test.ts src/drivers/runner-tool-bridge.test.ts src/native-session-runtime.provider-turn.test.ts`: `128 passed (128)` files, `2147 passed`, 9 skipped. - `cargo test -p paperclip-runner-core --lib` and the `acpx_*` suites: all passed. No Rust changed in this part. - The sidecar tests of this part: a cycle that starts before the arming command, or while the prompt turn settles, is held and then adopted with its tool call; a cycle is never shown when `paperclip-runnerd` did not arm, disarmed, armed for a different terminal, armed for a quiet terminal, or armed after the provider exited; two cycles in sequence with `{ live: 2 }`, then `{ live: 1 }`, then an unknown state; cancel; a lost provider; Codex gets no change. - Mutation checks, each made the named tests fail: the sidecar arms with no matching terminal; no hold; the old hold condition that drops a cycle while the prompt turn settles; no wait of a tool call for the decision. - Live run with a real Claude account, not in CI, with the real ACP runtime and the real Anthropic API (model `claude-sonnet-5-5`), at the adapter level: one background shell gave `{ live: 1 }`, one cycle with its own usage, then an unknown state. Two shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown. All Paperclip tool calls were admitted, and none was refused. Not run: a live run through the real sidecar process, `paperclip-runnerd` and Core (that path needs an API key, not a local sign-in); Linux; `pnpm verify --full`. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in, so the sidecar is never armed. The new code that runs today is the loop that reads the stream: with no arming it drops each cycle, and the background state becomes unknown as before. - **A follow-up for a command that finishes while Claude reports on an earlier one is lost.** The Runner waits again only while a background command is still running. The run ends as it does today. The reason is in #21421. - **A cycle that starts in the last second of a prompt turn** can have its first messages in the prompt turn and the rest in the provider-started turn. Tool calls stay safe: the admission rule of #21278 refuses a call that the active turn did not announce. - **The 5 second hold** is not tested under load. A cycle that is held longer is dropped, and the run takes the path of today. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
87d363a5c0 |
perf(ui): reveal a returning Mine visit from the saved cache (#20594)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Mine inbox is the board's main list of work for one person; the
app saves its query cache in `localStorage` so a returning visit can
paint from it
> - On a hard load the saved cache gave no benefit: Mine restored its
rows but still held them until the first approvals, join-request and
dismissal reads answered, because those items sort in among task rows by
time and a late one would push rows down (#20407)
> - Those three reads were not saved, so every returning visit waited on
the network for items it showed a minute before
> - This pull request saves what Mine showed from those reads, as one
record per user and company, and lets it answer the wait on a returning
visit while every read refreshes in the background
> - The benefit is that a returning visit shows its rows with no API
response at all; the first visit keeps its bounded wait unchanged
## Linked Issues or Issue Description
**What happened?**
A returning hard load of `/:companyPrefix/issues?view=mine` with the
saved cache present (`paperclip:query-cache:v1:<user>:<company>`) showed
rows no sooner than with the cache cleared. On the hosted staging
instance, rows appeared about 40 ms after `GET
/companies/:id/join-requests?status=pending_approval` returned in 7 of
10 runs.
**Steps to reproduce**
1. Open Mine once so the app saves its query cache.
2. Delay API responses (for example 300 ms) and reload.
3. Rows appear only after the approvals and join-request reads answer.
**Expected behavior**
Rows show from the saved cache at once. The reads refresh in the
background.
Related: #20407 (kept approvals and join requests in the reveal gate),
#20408 and #20419 (Mine snapshot and its perf guard), #20591 (removes
the second wave of reads; merged), #20592 (task chat preload; merged).
## What Changed
- `ui/src/lib/inbox-mine-reveal-sources.ts` (new): the record `["inbox",
companyId, "mine-reveal-sources", userId]` holds the approvals visible
in Mine, the pending join requests, and the dismissals of those items
and of Mine's failed runs. A record over 256 KB is replaced by an empty
`incomplete` record that never answers the wait. `readMineRevealSources`
accepts a record only for the same user and company.
- `ui/src/lib/persisted-query-cache.ts`: the saved cache keeps that
record per user and company, only when its data names the same user and
company as its key.
- `ui/src/pages/Inbox.tsx`:
- Mine writes the record once approvals, join requests and dismissals
all have server answers.
- On a returning visit with a restored Mine snapshot, the saved
approvals, join requests and dismissals stand in while their reads are
pending, and after a transient failure. A refusal (401/403/404) or a
server answer replaces them. The first-attempt wait is skipped only when
saved values stand in; the first visit is unchanged.
- Selection fix: the selected row's key is recorded only when the
selection moves. Before, every list change recorded the row now at the
old index before the re-anchor ran, so a row inserted above the
selection (a late approval, or any live insert) moved the selection to
another row.
- `ui/src/hooks/useInboxBadge.ts`: `useInboxDismissals` takes an
optional saved `fallback` and returns `hasServerData`. Other callers are
unchanged.
- Tests:
- `Inbox.test.tsx`: a returning visit reveals rows and the saved
approval while the reads are pending, then keeps selection on the same
row when a newer approval arrives above it; a transient failure keeps
the saved approval and a 403 drops it; a record stamped for another user
is ignored and the page waits as a first visit does.
- `inbox-mine-reveal-sources.test.ts`: projection, size limit,
user/company checks, persistence scope.
- `persisted-query-cache.test.ts`: the record round-trips for the same
user, restores as invalidated, and another user's record is not saved.
- `useInboxDismissals.test.tsx`: fallback while pending, after a
transient failure, dropped after a refusal, and no fallback.
- `tests/perf/collections/mine-returning-visit.spec.ts` +
`returning-visit-fixture.ts`: production-build browser checks and the
timing case; README updated.
## Verification
- `cd ui && pnpm exec vitest run src/pages src/lib src/hooks
src/components/coordinator`: 453 files, 5574 tests pass.
- Browser checks on a disposable local instance (production UI build,
`local_trusted`, embedded PostgreSQL, Chromium headless, 1440×900):
`PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec
playwright test --config tests/perf/collections/playwright.config.ts
mine-returning-visit.spec.ts`
- With every read of the company held, task rows and saved approvals
show and no company read has answered. Fails on `master`, passes here.
- An approval dismissed after the cache was saved stands in, then leaves
when the reads answer. Fails on `master` (no saved approvals), passes
here.
- Another company's Mine never shows the saved company's rows. Passes on
both.
- Timing (5 fresh contexts per arm, alternating saved/cleared, every API
response +300 ms, same instance and fixture, host load average about
20). "Responses before rows" counts API responses that finished before
the first row:
| Build | Saved cache: median (range) | Responses before rows | Cleared
cache: median (range) |
| --- | --- | --- | --- |
| `master` (`99edcd28a7`) | 1028 ms (906–3011) | 16–19 | 1502 ms
(1423–1592) |
| This PR | 625 ms (589–672) | 0 | 1632 ms (1359–1692) |
| This PR + #20591 | 625 ms (604–671) | 0 | 1495 ms (1398–1585) |
Saved cache: difference of medians 403 ms. The remaining 625 ms is
JavaScript download and route-chunk loading in a fresh browser context;
no API response precedes the rows. Cleared cache: no change is claimed;
the spread overlaps.
- Existing collection specs on this build: `parent-context.spec.ts`
passes; `startup-regressions.spec.ts` has 4 failures (partial
failure/context error and two 429 cases) that fail the same way on the
`master` build. `mine-workspaces.spec.ts` was not run (needs a database
URL for its fixture).
- Review follow-up (
|
||
|
|
b7922570e1 |
fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.
Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.
Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on
|
||
|
|
4a78124a3e |
feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn that ended with no result while a background shell is running, this stops the shell and the run fails. > - #21278 binds each tool call to its turn, and #21293 reports how many background shells are running at the end of a turn. Part 4 declared the contract. > - This pull request makes `paperclip-runnerd` wait for a turn that Claude starts by itself, when the controller asks for it, and admit that turn into the same run. > - The benefit is that the Runner has one place that decides to wait and one place that ends a wait. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 5. Its base branch is `feat/prp-provider-turn-contract`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** What `paperclip-runnerd` does after a Claude turn ends with no result while a background shell is running. Nothing changes until a controller sends the opt-in. **Subsystem affected** Paperclip Runner: the ACPX provider backend in `runner-core` (Rust). Claude only. **Current behavior** - After each completed turn the Runner emits `run.terminal`. A turn with a running background shell gets the provider restart, and the shell stops. **Proposed behavior** Nothing changes unless the controller sends `providerStartedTurns: true` in the payload of `turn.start`. No controller sends it yet. With that field: - The Runner keeps the provider and waits when all of these are true: the agent is Claude; the turn completed; the run has no accepted `paperclip_finish` or `paperclip_block` result; the terminal event reported `backgroundTasks` with `live` above zero; each condition of #21293 for a kept process is true; the sidecar accepted `session.provider_turn` with `armed: true`. - `turn.completed` then has `awaitingProviderTurn: true`, and no `run.terminal` follows. The Runner holds that `run.terminal`. - When the sidecar reports a turn that the provider started, the Runner admits it only during a wait and publishes `turn.started` with `origin: "provider"`. In each other case that report fails the session closed. - When a wait ends with no turn active, the Runner publishes the held `run.terminal`. This applies when the provider process exits, the sidecar exits, the controller cancels or stops the run, the session is closed or suspended, and after a Runner restart. - The held `run.terminal` and the removal of the wait are saved in one write, and the controller gets the event from the next poll. A crash after a cancel or a stop cannot lose it, and it is never sent two times. - A controller `turn.start` during a wait replaces the provider first, as before, and the held event is dropped. - A `run.attach` from a different run during a wait ends the wait when the attach succeeds. The Runner then discards the held `run.terminal`, because that event is of the old run and must not end the new run. This is the same rule that a Runner restart applies. The wait is removed only in the state that the attach saves for the new run. If the attach fails, for example when the old session does not stop, the old run keeps the wait and gets its held `run.terminal` one time, from the next poll or from a cancel or a stop. A `run.attach` from the same run publishes the held event to that run. - The Runner waits at most 5 seconds for the sidecar to answer a request to arm or to disarm a provider-started turn. This is the limit that turn cancellation already uses. Before, the limit was the 30 seconds of an ordinary request, and a cancel, a stop or a shutdown had to wait behind it. No answer is handled as a lost sidecar: the session fails closed, and the Runner still publishes the held `run.terminal` one time. - The sidecar of this branch does not know `session.provider_turn`, so the arming request fails and the Runner does not wait. The behavior comes with part 6. **Reason and benefit** - The wait is one entry in the saved session state. One rule ends it at each save, so each new way to leave the open session ends the wait with no new code. - A wait must not outlive the provider or the sidecar, and a cancel must stop it. ## What Changed - `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests): `provider_turn_wait { turn_id, held_run_terminal }` in the saved state; the rule that ends a wait at save; `settle_turn_events`, which decides at one place to arm, to create the wait and to put `awaitingProviderTurn` on `turn.completed`. - `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its terminal, the arming request, and `admit_provider_turn`. - `acpx_provider_session.rs`, `acpx_provider_backend.rs`, `acpx_provider_state.rs`, `provider_events.rs`, `acpx_park_preparation.rs`: the opt-in from `turn.start`; the record that the run has a result; the provider-started turn in the state and in the PRP events; cancel and stop with no turn; no turn-end restart and no park during a wait. - `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no event also checks that the sidecar process is alive. A child process can keep the output stream of a dead sidecar open. - `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and failures for tests. - Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`, `tests/support/acpx_quiet_turn_restart.rs`, `src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon), `src/native-session-runtime.provider-turn.test.ts` (the controller of today). - `runner/DURABLE_TRANSPORT.md`: the rules. ## State saved by the previous release - One new optional field in the saved ACPX session state: `providerTurnWait`. It is written only during a wait. In each other state the saved file has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so there is no wait. A unit test covers this. - **State from this release, read by the previous release (a rollback):** outside a wait, the file loads as before. During a wait, the previous Runner refuses the file with its "ACPX provider state is malformed" error, because its loader refuses each unknown field. Only a run that waits at the moment of a rollback is in this case, and no run waits before a controller sends the opt-in. - **Runner restart during a wait:** the wait is restored. The provider is a new process, so the wait is over, and the controller receives the held `run.terminal` one time. A test with the real daemon covers this. - **Controller of today or of the previous release:** `turn.start` has no opt-in. A test runs a turn with a running shell and a turn with an unknown background state, and compares the events, the commands after the turn, the restart, and the park state before and after the restart. They are the same. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored. All 17 `acpx_*` suites passed. - `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends with exactly one `run.terminal` when the sidecar exits while a child process keeps its output open, when the provider is lost, when the controller cancels, and after a Runner restart; with no opt-in, `run.terminal` follows the turn and the sidecar is never armed. - Rust tests: wait and admit; no wait with a result; no wait with no opt-in; an active goal neither marks nor arms a wait; a sidecar that refuses to arm; a provider-started turn that the Runner did not wait for fails the session closed; cancel and stop during a wait; provider and sidecar loss; restart; the saved shape in both directions. - `src/native-session-runtime.provider-turn.test.ts`: the real `consumeTurn` of today gets the event sequence with no opt-in and finalizes it as `native_finalization_missing`, as today. - Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a different run attaches after the snapshot reports `warmAttachReady: true`; the new run gets no `run.terminal`, the saved state has no wait, and the old session is closed. A same-run attach gives exactly one `run.terminal`. An attach that fails when the old sidecar is dead leaves the wait with the old run, and the old run then gets exactly one `run.terminal` from a poll, `run.cancel`, `turn.interrupt` or `turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in the pending events of the new run; with the wait ended before the attach is complete, the old run gets no `run.terminal` after a failed attach. - No answer from the sidecar, in `tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not answer the disarm request; `run.cancel` and `turn.interrupt` each return `already_settled` in less than 10 seconds, exactly one `run.terminal` reaches the run, `session.close` completes in less than 2 seconds with no second `run.terminal`, and the provider process is gone. With no answer to the arm request, the turn ends with its `run.terminal` and with no wait. With the limit removed, each test fails at 30 seconds. - Mutation checks, each made the named tests fail: the rule that ends a wait at save; no `run.terminal` after a restart; arming with no opt-in; the marker set with no wait; the sidecar process check removed. Not run: a rollback with the binary of the previous release; a run with the opt-in and a sidecar that answers the arming request with an error (the code reads each error as not armed); Linux; the full suite. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in. - **A rollback of the Runner during a wait** makes the previous Runner refuse that saved session. See "State saved by the previous release". - **The wait has no time limit in the Runner.** The controller must close, suspend or cancel. Core will own that limit. - **Park is refused during a wait.** The controller must end the wait before it parks. - **A new run can attach during a wait.** The attach is not blocked, because the Runner has no time limit on the wait and a new run can send only `run.attach`. After an attach that succeeds, the old run gets no `run.terminal` from the Runner; the controller must not depend on that event. The request that disarms a sidecar that the attach keeps has no test, because no test can keep a session across an attach during a wait. - **After a sidecar that does not answer, `runner.suspend` returns an error at once** ("provider lifetime cleanup is not yet proven"), and `session.close` completes. This is the behavior of today for each lost sidecar; this PR does not change it. - **The shared code that changed** runs for each Claude turn: the record of a run result, and the settle step at each turn end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.1 |
||
|
|
689f3b190e |
fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Automatic recovery continues interrupted and failed tasks without a person, with a per-task limit of 6 automatic attempts per hour > - There is no company-level limit. When the sandbox provider refuses to create sandboxes, every task fails the same way, and recovery keeps creating new runs for each task > - On a hosted instance, one hour of a provider outage made 510 automatic runs, 587 failures and 0 successes across 289 tasks. No task had more than 4 runs, so the per-task limit did not engage. Each start asked a refusing provider for a sandbox and used server memory, which made the outage longer > - Recovery must still continue every task by itself. The answer is to wait while the infrastructure fails, not to give the task to a person > - This pull request adds a company-scoped pause for automatic work: it opens when recent automatic runs fail before the agent starts, the recovery passes stop creating new automatic runs while it is open, and it closes by itself when runs start normally again > - The benefit is that an infrastructure outage no longer multiplies itself through recovery, and recovery resumes with no person acting ## Linked Issues or Issue Description **What happened?** During a sandbox provider outage, automatic recovery created hundreds of new runs per hour across hundreds of tasks. All of them failed before the agent did any work. The per-task limit did not engage because no single task reached it. **Expected behavior** When most recent automatic runs of a company fail before they start, recovery waits instead of creating more runs. It tries again at a small rate. It resumes at full rate when runs start normally again. A person's work is never held. **Steps to reproduce** 1. Make the sandbox provider refuse lease creation (HTTP 503) for a company with many assigned tasks. 2. Let recovery continue the failed tasks. 3. Count automatic runs created per hour: it grows with the number of tasks, and every run fails at sandbox creation. Refs #21394 (continuation classification this change will reuse when it lands), #21433 (start-pass rewrite that the start hook will build on). ## What Changed - New table `automatic_run_brakes` (one row per company, cascade delete with the company; migration `0480`). Migration `0481` builds `heartbeat_runs_company_finished_idx (company_id, finished_at desc) where finished_at is not null` concurrently, so the signal reads the runs that finished in the window directly. It stores the state (closed, open, half_open), the reason with the window numbers, and the compare-and-set fields that two servers share. A company with no row is closed. - New module `server/src/services/recovery/automatic-run-brake.ts`: - Signal: on the database clock, the automatic runs of the company that finished in the last 10 minutes (and since the last state change), newest 500 by finish time, plus up to 200 running runs. A run counts as a failure only when it ended in its own start phase: no provider work (`executionRecovery.providerWorkStarted = false` or the native `nativePreDispatchFailure` record), a legacy run still preparing, or a transient setup code. A run interrupted by a server stop, restart or lease loss is neutral, so a release swap does not open the pause. A run that succeeded, or ended for a person-owned or task reason, counts as healthy. - Opens at 12 or more counted runs with 80% or more failures, or on 5 sandbox refusals (`environment_provider_unavailable`) in a row. - Each outcome counts once, at its own time: a finished run at its finish time, a running run that got past setup at its start time. Read time never orders evidence. - Closes in two steps: open moves to half_open after at least 2 minutes when the newest 2 counted runs are healthy or the newest one succeeded; half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64) and then closes. It reopens from half_open if the signal trips again. It closes by itself after 15 minutes with no automatic activity. - Reads are one indexed query per company, cached for 5 seconds in a bounded in-process map (256 companies). Each state change writes one structured log line (`automatic_run_brake.opened|half_open|closed|reopened`) with the numbers. - It also contains the start gate `holdAutomaticRunStart` (one probe at a time, at most one new probe per 60 s, release budget per window by compare-and-set, no I/O while the cached "held until" time has not passed, person origin taken as an argument with a cached fallback). This pull request does not call it from the start pass; see Risks. - Recovery passes defer creating runs while the pause is open. The gate is only where a run is created: - `deliverReconciledExecutions`: after the agent-room check and right before the wake, the delivery asks for a creation slot; when the pause holds it, the continuation stays pending. The held backlog (`continueHeldExecutions`) is not gated: recording a pending continuation creates no run and costs nothing, and its delivery passes this gate once. - Assigned-work sweep `act()`: after the process rate limit and before it claims a continuation, it asks for a creation slot; when held, it returns `deferred_rate_limit` with the next try time, so nothing is written to the per-task attempt ledger. New assignments with no earlier run are not deferred. - A reserved slot is given back (compare-and-set on its own timestamp) when no run is created: the wake is refused or throws, the task is capped, the claim changed, or the task went back to review. The give-back is conditional in SQL on no run for that task having been created since the reservation, so a wake that commits its run and then throws keeps the slot. - The sweep's overdue saved-work recovery path returns before the slot is asked for: it starts a recovery attempt, not a run, and is never held. - Person work is never deferred. The gate is called only at the delivery (`agentHasRoom`) and the sweep (`act()`), and each call passes the task and the run it continues. While the pause is open, the gate does not hold the record when any of these is true; it holds only when none is. With no task or no source run it does not hold (fails open). | Shape | How it is recognised | | --- | --- | | A Retry or "move forward" waiting for the task (`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an unstarted wake request from a user for the task | | A Retry folded (`coalesced`) into the continuation, or a Retry while the delivery is pending | a wake request from a user for the task, requested after the source run finished (any status) | | A person's message or a new assignment by a person after the last run | the same: a wake request from a user for the task after the source run finished | | A held task selected because a person pressed Retry (the refused Retry run) | a run for the task created after the source run finished whose wake is a person reason (`task_retry`, chat message), or that has `taskRetryRequestId` or `explicitUserContinuation` | | A continuation of a person's Retry (any depth) | the source run's chain reaches a person's run (`listPersonOriginSourceRunIds`, the dispatch rule) | | A Retry during pending delivery that becomes the continuation itself (`deliverContinuationAsRun`) | not gated: that run is created by the person's own wake, never by these passes | | The held backlog's saved-input path | not gated: recording or releasing a held continuation creates no run here | - To prevent a deadlock, a pass may still create one automatic run per minute per company when no automatic run of the company started in the last 2 minutes, however many runs wait (compare-and-set, shared by servers). - `GET /api/companies/:id/assigned-work-invariant` now returns `automaticWork` with the state, the time it paused, the window numbers and a plain sentence, for example "Automatic work is paused for this company since T because 47 of 50 recent automatic runs failed before starting." - `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is now exported, unchanged, so the gate uses the same person-origin rule as dispatch. - The outcome rules (person-owned codes, restart codes, transient setup codes, native pre-dispatch record) come from their owning modules through one memoised dynamic import. A static import of `heartbeat/retries.ts` made a module cycle at server start. ### What the pause consumes, and how waste is bounded | Token | Taken | Given back or bounded | | --- | --- | --- | | Creation slot (one per minute per company while open) | right before the run is created, after every other check | released only when no run for the task was committed after the reservation | | Probe slot (start gate) | when the start gate lets one automatic run through | another run may take it when the reserved run has not started within 10 seconds (claim declined, agent paused or full) or was cancelled, deleted or finished | | Gradual-release start (start gate) | one per run the start gate lets through | a declined claim spends one start of that window; the window ends within 60 seconds and the next one doubles | | Probe success count | once per probe run, by compare-and-set on its id | a probe run counts once; cancelled or deleted probes count as neither success nor failure | ### When the pause opens, stays open and closes | State | Leaves when | Bound | | --- | --- | --- | | closed | opens when, in the last 10 minutes since the last change, 12 or more counted automatic runs finished and 80% or more failed before starting, or the newest 5 counted runs were sandbox refusals | none needed | | open | moves to half_open when it has been open 2 minutes or more and the newest 2 counted runs are healthy, or the newest one succeeded | evidence arrives at least once a minute: when no automatic run started in the last 2 minutes, a pass may create one per minute | | open | closes when no automatic run started or finished for 15 minutes (queued runs do not count) | 15 minutes after the last activity | | open | stays open only while new automatic runs keep failing before starting | each new try is spaced by the existing passes and by one creation per minute when nothing starts | | half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6 minutes | | half_open | reopens when the signal trips again, counted only since half_open began | none needed | Restart-class endings (server stop, restart, lease loss, process loss, interrupted native session) are neutral in every state, so a release swap does not open the pause. ## Verification - `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests, pass): classification; opens on 20 failures before start; 50 queued automatic runs give exactly one probe and no failure or attempt; a person's Retry, a new assignment, a run a person joined and the continuation of a person's Retry are not held; mixed and person-owned outcomes stay closed; old data is ignored; 40 working runs interrupted by a restart stay closed, then the same continuations failing at sandbox acquisition open it; 5 refusals in a row open it; failed probe waits for the interval, two probes past setup move to half_open, the release gives exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen from half_open; probe slot frees after cancel, delete or no start; quiet close after 15 minutes; two instances on one database agree and start one probe; one creation per minute while open across servers; 250 held runs over 100 passes use at most 5 queries (at most 255 with the person-origin fallback, one per run, once); caches stay at 256 companies and 1024 runs. - `npx vitest run src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one test per person shape above at the delivery and the sweep, with the pause open and no creation slot, plus a control that still holds a continuation no person asked for): includes "no pause record: every pass behaves as before", "the waiting run cannot start: a pass continues a task on another agent, then the pause closes", "one held task spends one creation slot", and "a person's Retry or message is never paused at the delivery, the sweep or the held backlog".: sweep, delivery and held backlog create nothing while open, write no attempts, and continue by themselves after recovery; the sweep still continues one task a minute when nothing automatic waits. With the three deferral lines removed, all 4 fail (red), and they pass with them (green). - The pause reads as closed and logs once when its run rules cannot load (test in `automatic-run-brake.test.ts`). - `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the module loads after the heartbeat service, in the server's import order. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify`: see the comment below for the result. CI supplies the full suite. ## Risks - This pull request does not hold starts. Already queued runs and `transient_failure_retry` runs still start while the pause is open; they act as the natural tries. The start gate goes in as a follow-up on top of #21433, at a point before that pass counts a claim attempt and reads the full run row, so a held run costs no attempt and no read. - The sweep reuses the `deferred_rate_limit` outcome for a deferral, to keep its outcome type unchanged while #21394 edits that file. - The automatic-run predicate is built from master exports (`RECOVERY_WAKE_REASONS`, retry markers). It will switch to `isAutomaticContinuationRun` when #21394 lands. - Migration `0480` adds one small table. - Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`) runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement: it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in `packages/db/src/client.ts`, like 0470 and 0476, and the runner drops and rebuilds an invalid index after an interrupted build. The index is declared in the Drizzle schema, so `pnpm db:generate` reports no changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On a large `heartbeat_runs` table the build is online (no write lock) but takes time proportional to the table. After it exists, it is one more index on a hot table: each run that finishes writes one more index entry. - No user-visible text changes outside the operator report. ## Model Used - Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell, file edits, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
6826800425 |
feat(ci): report the required checks on merge-queue commits (#21545)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each merge to `master` publishes the standard image, and the time from merge to published image limits how fast a fix reaches users > - #21096 lets `docker.yml` push an image that the merge queue already built, but the merge queue is not on > - The `master` ruleset requires `ci / verify` and `ci / e2e`. Only `pr.yml` reports them, and only for pull requests > - A merge queue waits for the same two names on its merge-group commit. No workflow reports them there, so every queued merge would stall > - This pull request adds a workflow that reports both names on merge-group commits > - The benefit is that the owner can turn the merge queue on with one ruleset change ## Linked Issues or Issue Description Refs #21096 **What existing behavior does this improve?** The merge-queue image build from #21096. It cannot run until the merge queue is on, and the merge queue cannot be turned on safely. **Current behavior** `ci / verify` and `ci / e2e` come from `pr.yml`, which runs on `pull_request` only. With a merge queue, GitHub waits for both checks on the merge-group commit. They never arrive. Each queued pull request would time out and leave the queue. **Proposed behavior** `merge-queue-checks.yml` runs on `merge_group` and reports `ci / verify` and `ci / e2e`. It runs no tests. It does not check out the queued commit. It has no token permissions and no secrets. **Reason and benefit** A pull request enters the queue only after both checks pass on the pull request. A queued merge therefore has the same protection as a direct merge today. The exact-source gate still tests each `master` commit before publication. Real tests on the combined tree can replace these jobs later under the same names. ## What Changed - Added `.github/workflows/merge-queue-checks.yml`. It reports `ci / verify` and `ci / e2e` on merge-group commits. - Added `scripts/__tests__/merge-queue-checks-workflow.test.mjs`. It fails if the names differ from the names that `pr.yml` reports, or if the workflow gains a checkout, an action, a secret, a token permission or a self-hosted runner. - `pr-trusted.yml` now runs that test and `scripts/merge-group-image.test.mjs` in a step that skips test files an older checkout does not have. The second test came with #21096 but no CI step ran it. - `scripts/merge-group-image.mjs find` now waits up to 15 minutes for a merge-queue image build that is still queued or running. The queue can land a commit before its image build ends. Without the wait, the `master` push always builds from source. - `doc/RELEASING.md` explains the workflow and lists the ruleset settings that turn the queue on and off. ## Verification - `node --test scripts/__tests__/merge-queue-checks-workflow.test.mjs`: 2 pass. - `node --test scripts/merge-group-image.test.mjs`: 10 pass, with new cases for unfinished builds. - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: pass. - `actionlint` on both changed workflows: no findings. - Not verified: no `merge_group` event can fire before the queue is on. The first queued merge is the live test. `pr.yml` calls `pr-trusted.yml@master`, so the new CI step runs only after this merges. Owner steps after merge: 1. In the `master` ruleset, add **Require merge queue**: Squash, build concurrency 5, group size 1 to 5, status check timeout 60 minutes. 2. After the first queued merge, confirm that the `master` commit SHA equals the SHA of its merge-group runs. ## Risks - Low risk while the queue is off. The workflow has no trigger other than `merge_group`. - With the queue on, the queue does not test the combined tree. Two pull requests that pass alone can still break `master` together. This is the same as today. - A merge-queue image build that hangs delays the source build by up to 15 minutes. A build that fails does not. - If GitHub does not accept a job named `ci / verify` for the required context, queued merges time out. Rollback: remove the merge queue rule. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |