mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
b9cf64f5bf88f66f02e695f8f7bc326e004961ba
2576
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5b750fdf25 |
feat(ui): add foldable sidebar subtasks with bounded refreshes (#21649)
## Thinking Path > - Paperclip helps people manage AI agents for work. > - The sidebar helps people return to recent tasks. > - Recent parents need foldable child tasks. > - Busy companies must not refresh every child group for unrelated work. > - Exact unread counts must preserve personal reads and task access. > - This pull request adds child groups and bounds their refresh work. > - People can scan subtasks without repeated background requests. ## Linked Issues or Issue Description **Subsystem affected** Task navigation in the standard and Classic sidebars. **Problem or motivation** Recent parent tasks did not show their subtasks. The first implementation refreshed every previously viewed group. One unrelated update caused 52 child-page requests and one summary request after viewing 50 groups. Exact child counts also repeated privacy and unread checks per child. **Proposed solution** Show direct subtasks with independent folding and ten-row pages. Refresh affected groups and summary batches. Stop offscreen reads. Bound child-request concurrency. Aggregate viewer facts once and preserve the existing inbox unread rule. **Roadmap alignment** This extends existing task navigation. The existing feature PR is reused for its performance repair. No new PR duplicates it. ## What Changed - Add shared parent groups to both sidebar layouts. Preserve Classic row actions and active-child visibility. - Keep unread state personal to each task. Folding and paging do not mark tasks read. Counts include readable children beyond ten. - Target live refreshes with authorized current and previous parent IDs. Reuse parent metadata from the narrow run-binding read. Resolve fallback lookups once per delivery batch. Keep the previous summary fallback for older servers. - Limit child requests to three at a time. Cancel active and queued reads when groups leave the viewport. - Discard extra cached pages on Show less and collapse. - Compute exact navigation counts in one SQL statement. Materialize readable candidate IDs once. Use indexed participation sources and conservative unread candidates before the shared inbox rule. - Skip recursive privacy walks for standalone open tasks without a project. Keep the recursive rule for all restricted or inherited shapes and all delegated principals. - Add request-count, cancellation, reparenting, read-receipt, inbox-parity and privacy regressions. Include a disposable PostgreSQL benchmark. - Update product and API documentation. Retain the production Storybook states. ## Verification - Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`: `a23533e93d3d155918f0254d296a7fd2f0c7c12f`. - Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote publication preserves prior feature commits. - The request regression failed before the fix and passes after it. An unrelated event now sends zero child requests. Current servers also avoid the unrelated summary request. Old-server events keep one summary fallback. - A relevant update after Show less reads one page. Reparenting refreshes two affected groups. Offscreen reads stay at zero and resume once on return. Active and queued cancellation tests pass. - The same synthetic authenticated HTTP workload with 50,000 read children improved from 767 ms to 92.84 ms median. Five warm samples per case. The all-unread case improved from 463.85 to 351.65 ms. Untouched children improved from 208.01 to 131.09 ms. The old-receipt case improved from 799.70 to 245.80 ms. These local measurements are not a production capacity guarantee. - Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests passed: 38 tests. - All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy and WebSocket tests passed. UI typecheck and low-memory server declaration checks passed. - All 41 Chromium Storybook checks passed. They cover 20 states in both themes and keyboard, folding, paging and unread behavior. They use production components with HTTP fixtures. - Both new review findings failed as regressions before their fixes and pass afterward. Explicit child reads include waiting external conversations. Deleting a child publishes a separate parent-authorized update without deleted-child content. Both review threads are resolved. - `pnpm verify --explain` returned exit 2 because product documentation requires broader CI. [The prior CI run](https://github.com/paperclipai/paperclip/actions/runs/43711489306) found type errors in the new test fixtures. The same errors failed locally. The fixture fix passes `node server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21 issue summary route tests. Production code is unchanged. That CI run then found a live-delivery read regression. The existing test failed locally with eight statements against its six-statement budget. A new PostgreSQL regression also found three parent lookups for two same-batch calls. Run delivery now reuses parent metadata from its narrow run-binding query. Fallback lookups resolve to one promise. Both regressions pass without increasing the budgets. All 74 targeted route, WebSocket and privacy tests pass. The affected below, above and test-tree-0 typecheck layers pass. [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2) passes on the published head. All 62 check runs are terminal and clean: 57 succeeded, two were neutral and three were skipped. Typecheck, build, policy, all general and serialized server suites, E2E and the [final verify gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231) pass. The separate security status passes. Attempt 1's server shard 7/12 reached its 20-minute job limit without a test verdict. One retry of that job passed without a source change. [Greptile completed successfully on this head](https://github.com/paperclipai/paperclip/runs/132816163906) with a [5/5 score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606). All five review threads are resolved. The final CI gate is complete. - A full local Storybook build exceeded available memory while server typechecking ran. Browser verification uses the current-source development server. Rebuilding stale Runner and SDK declarations resolved the initial server check errors. The first browser attempt lacked Chromium; all checks passed after installation. - Reproduce the database measurements with `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`. - Extra local shard diagnosis ran 164 suites. Under this shell's `0077` creation mask, 157 suites passed and three had fixture permission or timing failures. Rerunning those three with `0022` resolved the permission failures. The remaining ten cases passed with one worker: two workspace cases and eight archive cases. The eight archive cases also passed on the pre-performance-fix revision, whose archive source is identical. The broad local shard is not claimed as one passing execution. The same-head CI retry supplies complete final CI coverage. - Nothing is merged or deployed. ## Risks - Exact counts still grow with the readable hierarchy and comment history. The change reduces repeated work; it does not cap or approximate counts. - A relevant summary batch can contain several recent parents. Older servers retain a broader summary fallback until upgraded. - Live delivery adds bounded, permission-checked parent lookups. Short delivery-window reuse limits repeated reads. - The authorization fast path applies only to open tasks with no privacy parent and no project. Restricted cases retain the shared predicate. - No migration is required. ## Model Used OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools, PostgreSQL and Chromium verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked a public issue or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
348ed0ae10 |
fix(runner): keep closing tool-call tags and model control tokens out of OpenCode chat (#21761)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner OpenCode driver streams model text into the
chat as progress and selects the final reply
> - DeepSeek v4.1 flash (through OpenRouter) wrote tool-call scaffolding
and tokenizer control tokens as ordinary text
> - #21361 hides tool-call markup, but its detector matched only opening
DSML tags, and nothing recognised control tokens
> - A person saw raw `</|DSML| invoke>` lines in a progress message, and
in another conversation got a degenerate reply instead of an answer
> - This pull request extends the same detector to closing DSML tags and
to model control tokens, on the streaming path and at completion
> - The benefit is that scaffolding never reaches the chat, and a
degenerate reply goes to the existing correction instead of being posted
## Linked Issues or Issue Description
Refs #21361
**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master
|
||
|
|
e7f31705da |
perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push runs a canary, and only one canary runs at a time, so its duration sets how often merged work ships > - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`, and 4 min 40 s of that is an uncached `cargo build --release` of `paperclip-runnerd` > - The release then replaces that binary with the exact-source daemons that `release_runner_binaries` already built and verified for all four targets > - This pull request stages the verified daemon for the publisher's own target during a release, instead of compiling it > - The benefit is about 4.5 min less per canary, with the same verified bytes published ## Linked Issues or Issue Description **What happened?** In canary run 43487884481 (2026-11-13), the `Publish canary` step took 28.6 min. Its `pnpm build` ran 19:42-19:50. The `packages/paperclip-runner build` took 5.3 min, and its `cargo build` reported `Finished release profile ... in 4m 40s`. Every other package waited for it. `release.sh` then runs `assemble-daemons`, which overwrites `dist/bin` with the four exact-source release daemons. **Expected behavior** A release does not compile a binary that it discards. The Runner build stamp stays valid, so the server build still reuses the Runner build. Refs #21554 (the other canary speed-up) ## What Changed - `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new): when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the release daemons with the existing `releaseDaemonManifest` (source revision equal to `HEAD`, digest, architecture, build metadata) and stages the host target's daemon at `dist/bin` with the existing `stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as before. - The Runner `build` chain calls this script in place of `pnpm run build:binary`. `build:binary` itself and the image build are unchanged. - `stage-release-runner-binaries.mjs` exports `metadataContract`. - Tests: `stage-runner-binary.test.mjs` (staging per target, wrong source, bad digest, missing target, non-release fallback); `runner-image-identity.test.ts` step list. - `doc/RELEASING.md` describes the behavior. ## Verification - `node --test packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9 pass. - `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42 of 42 pass. - With the real daemon artifacts of canary run 43487884481 (receipts pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=... pnpm run build` in `packages/paperclip-runner` finished in 15 s with no `cargo` line. The build stamp was recorded, and `server/scripts/prepare-runner-vendor.mjs` then reported `action:reuse,reason:inputs_unchanged`. The staged daemon runs. - After merge: the next canary's `packages/paperclip-runner build` log shows "Staged the verified exact-source linux-x64 release daemon" and no `Compiling` lines. ## Risks - Low. A release still fails closed: a daemon with the wrong source, digest, architecture or metadata stops the build, and `assemble-daemons` checks all four again afterwards. - Local, Docker and CI builds without the variable are unchanged. - On macOS, staging re-signs the binary, as it does for a compiled one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
689d62661d |
fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The delivery work coordinator wakes background tasks when a database work topic settles, and passes the settled scope hints to the task's `onSettled` > - The coordinator calls `onSettled(hints)` before `wake()` > - If `onSettled` throws on a hint that it does not recognise, the task is not woken. Only a generic listener warning is logged > - Hints now cross server processes (#21681), so a hint shape from another server version can arrive during a deploy overlap > - This pull request makes the coordinator log a warning, apply the settlement unscoped once, and always wake the task > - The benefit is that a hint that a task cannot read never makes the task miss committed work ## Linked Issues or Issue Description Follow-up to #21681 (merged). This branch is based on master. **What existing behavior does this improve?** How the delivery work coordinator handles an `onSettled` callback that throws. **Subsystem affected** `server/src/services/delivery-work-coordinator.ts`. **Current behavior** `subscribeDatabaseWork` calls the coordinator listener, which calls `task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws, `wake()` does not run. The work-signal publisher catches the error and logs "Database work listener failed". The task stays asleep until a different event or deadline wakes it. **Proposed behavior** If `onSettled(hints)` throws, the coordinator logs a warning and calls `onSettled(undefined)` once, which is the unscoped settlement. If that also throws, it logs a second warning. In all cases it then wakes the task. **Reason and benefit** A settled event means committed work can be ready. A task must never miss it because a hint could not be applied. **Breaking changes** None. ## What Changed - `delivery-work-coordinator.ts`: a settled event goes through `applySettled(hints)`. It tries `onSettled(hints)`. If that throws and hints were given, it tries `onSettled(undefined)` one time. Each failure logs `Delivery settlement could not be applied for <queue>` with `process.emitWarning`. Then `wake()` runs. - Unscoped means `onSettled(undefined)`. Each registered task already defines what that means. For example, scoped deadline queues, dispatch work, muse maintenance and the revocation index mark `*` dirty. Counter-style tasks set their request flag. `chat-action-work` does nothing. The fallback is bounded: it calls `onSettled` at most twice per event, it does not recurse, and it does not schedule another settlement. So it cannot loop. - `work-signals.test.ts`: a new test pins one rule of #21681 for the local path and the remote path. An unhinted `signalDatabaseWork` in the same transaction as a hinted one gives a settlement that is scoped to the hinted set. The owner of `work-signals.ts` confirmed that this is deliberate. The repository does not allow code comments in pull requests, so this sentence and the test document it. ## Verification - `cd server && npx vitest run src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests: - An `onSettled` that throws on an unknown hint shape: the task gets `[hint]` and then the unscoped settlement, it is woken once, and one warning is logged. `onError` is not called. - An `onSettled` that always throws: the task gets 2 calls and is woken once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1 warning. No more runs happen after 60 seconds, so there is no loop. - A normal `onSettled`: it gets the hints once, the task is woken once, and no warning is logged. - Red/green: with the coordinator from master, the 2 throwing tests fail. `onSettled` is called only with the hints, and there is no unscoped settlement and no wake. With this branch, all tests pass. - `cd packages/db && npx vitest run src/work-signals.test.ts -t alongside`: passes on real PostgreSQL. The local listener and the receiving process both get `["company-1"]`. ## Risks - Low risk. The change applies only when `onSettled` throws. Before, that path skipped the wake. Now it wakes with an unscoped settlement. For scoped deadline queues an unscoped settlement is a `*` reload. That is the existing safe fallback, and the reconstruction interval of the queue still throttles it. - A task whose `onSettled` throws on every event now runs on every settled event and logs a warning each time. Before, it did not run. The warning makes such a defect visible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
07d5a6ee17 |
perf(db): carry scope hints in cross-process work notifications (#21681)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Server processes share durable queues and wake each other with a
commit-bound PostgreSQL NOTIFY
> - The NOTIFY payload carries only `{version, origin, topic}`, so the
receiver gets an unscoped "settled" event
> - In `scoped-deadline-work.ts` an unscoped settle becomes `*`, which
reloads every candidate of that queue on the whole instance
> - On a large instance one cross-process notification can re-check
about 850 open tasks, at about 22 statements each
> - This blocks more than one server process per stack, and it costs
work during every deploy overlap
> - This pull request puts the topic's hints in the NOTIFY payload, so
the receiver reloads only the hinted scopes
> - The benefit is that a scoped commit in one process costs the same
scoped reload in every other process
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Cross-process work notifications between server processes on one
database.
**Subsystem affected**
`packages/db` work signals (`work-signals.ts`) and their consumers, such
as `server/src/services/scoped-deadline-work.ts`.
**Current behavior**
`receiveDatabaseWorkNotification` publishes "settled" with no hints. A
scoped deadline queue then reloads all of its candidates, throttled only
by its reconstruction interval.
**Proposed behavior**
The NOTIFY payload carries the deduplicated hints of its topic. The
receiving process publishes "settled" with the same hints that a local
commit publishes. Overflow and malformed input fall back to the old
unscoped behavior.
**Reason and benefit**
A scoped commit no longer causes an instance-wide reload in other
processes. This is necessary before a stack runs more than one server
process.
**Breaking changes**
None. Old and new processes interoperate in both directions (see Risks).
## What Changed
- `packages/db/src/work-signals.ts`: the per-level pending notifications
are now a map of topic to `{ hints, unscoped }`. Savepoint merge and
rollback keep the same rules as before.
- The hints follow the local "settled" semantics: the union of the hints
from `signalDatabaseWork` and `publishDatabaseWorkSettled`. An unhinted
`publishDatabaseWorkSettled`, `notifyDatabaseWorkOnCommit`, or a topic
with no hints sends the unscoped payload.
- A hinted payload is `{version: 1, origin, topic, hints: [...]}`. The
field is additive and the version stays 1.
- Each payload is at most 1,024 bytes
(`DATABASE_WORK_NOTIFICATION_MAX_BYTES`), because current receivers drop
any payload longer than 1,024 characters before they parse it. Many
hints split into several payloads. The payloads of one topic are at most
7,000 bytes in total (`DATABASE_WORK_TOPIC_NOTIFICATIONS_MAX_BYTES`). If
a topic exceeds this cap, or one hint alone does not fit, the topic
sends one unscoped payload.
- All payloads still go in the one batched `select pg_notify(...) from
jsonb_array_elements_text(...)` statement at commit. There are no added
statements or awaits on the commit path.
- `receiveDatabaseWorkNotification` accepts payloads up to 8,000
characters (the PostgreSQL limit). It keeps the drop of own-origin
payloads and of payloads without a usable origin or topic. A missing,
malformed, or empty hint list, or a version other than 1, gives an
unscoped wake. It never throws.
- The `remote_notification` debug event now logs the received hints
(`hints`: a JSON array, or `null` when unscoped).
- Tests: new payload, compatibility and fallback tests in
`packages/db/src/work-signals.test.ts`, and a new two-process test on
real PostgreSQL in
`server/src/__tests__/scoped-deadline-work-notifications.test.ts`.
## Verification
- `cd server && npx vitest run
src/__tests__/scoped-deadline-work-notifications.test.ts`: two pools on
one embedded PostgreSQL database. Process A commits work. Process B runs
a real delivery coordinator with `registerScopedDeadlineWork` over 800
candidate rows in 20 companies.
- Master (red): a commit hinted with one company makes B load `*`: 1
load, 801 rows.
- This branch (green): B loads only `company-7`: 1 load, 41 rows. A
commit with two hints loads only those two companies (80 rows). An
unscoped commit still loads all 801 rows.
- `cd packages/db && npx vitest run src/work-signals.test.ts` (32 tests,
real PostgreSQL). New tests:
- deduplicated hints per topic, including a released savepoint and a
rolled-back savepoint
- split into payloads of at most 1,024 bytes, at most 7,000 bytes in
total, and the union of hints is complete
- overflow and one oversized hint fall back to one unscoped payload
- malformed hint lists and unknown versions wake unscoped; unusable
payloads do not wake; own-origin payloads are dropped; nothing throws
- an old-format payload `{version, origin, topic}` gives an unscoped
wake
- every new payload passes a copy of the receiver from master, which
reads it as an unscoped wake for the correct topic
- the debug event logs the received hints
- With master's `work-signals.ts`, 5 of these tests fail (red). With
this branch, all pass.
- Also run: `scoped-deadline-work.test.ts`,
`delivery-work-coordinator.test.ts`, `deployment-notifications.test.ts`,
`run-start-handover.test.ts`, `heartbeat-run-start-handover.test.ts`,
`heartbeat-run-event-atomic-append.test.ts` (111 tests pass),
`work-signal-debug.test.ts` (12 pass), and `tsc --noEmit` for
`packages/db`.
## Risks
- Mixed deploys, new sender to old receiver: every payload is version 1
and at most 1,024 bytes. The old receiver ignores the `hints` field and
wakes unscoped, as today. A commit with many hints sends several
payloads, so an old receiver can get up to about 7 unscoped wakes for
one commit. The coordinator merges them into one `*` reload.
- The sender bounds bytes, and old receivers compare `.length` in UTF-16
code units. A string never has more UTF-16 code units than UTF-8 bytes,
so a payload of at most 1,024 bytes always passes the old check.
- PostgreSQL collapses identical NOTIFY payloads in one transaction. The
chunks of one topic always carry different hints, so PostgreSQL drops
none of them.
- Mixed deploys, old sender to new receiver: the old payload has no
`hints`, so the new receiver wakes unscoped, as today.
- I did not use one large payload of about 7 KB: current receivers drop
any payload over 1,024 characters before they parse it, so they would
miss the wake during a deploy overlap.
- Consumers that read hints (for example `chat-action-work`,
`approval-queue-seeding-work`) now get the same hints from another
process that they already get from a local commit. Before, they got no
hints from another process.
- The receiver accepts hint strings from the database channel. Consumers
decode them in the same way as local hints. `scoped-deadline-work`
already turns a decode failure into a `*` retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
01c1e708e9 |
fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself
## Linked Issues or Issue Description
**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.
**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.
**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.
**Paperclip version or commit**
`master` at
|
||
|
|
a78be8b601 |
feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On each warm turn Core sends the plugin a `prepare` operation with the full launch requirements for the new run > - The requirements' `requirementsDigest` covers the run identity, and the run identity changes on every turn > - So a plugin cannot tell that only the run moved, and it must redo its whole preparation every warm turn > - This pull request gives the plugin two digests: one that stays the same while its preparation stays valid, and one that names the run > - The benefit is that a plugin can transfer only the run identity when nothing it reconciles changed ## Linked Issues or Issue Description **What existing behavior does this improve?** The `prepare` operation that Core sends to an environment plugin for a plugin Runner task, on cold starts and on warm attaches. **Subsystem affected** Native runtime, plugin Runner tasks, the Runner launch contract, and the plugin SDK environment task contract. **Current behavior** A warm attach sends the full requirements. The only digest available, `requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it changes on every turn even when the launch is otherwise the same. **Proposed behavior** Core computes two digests for every `prepare`: - `preparationDigest`: the launch requirements an environment reconciles when it prepares a Runner: - the runner, lease and session identity; - provider, model and settings; - the prompt and instruction text; - the runtime context's instruction and skill bundles, working copy and connection instructions. It leaves out the `runId`, `turnId` and `itemId`, and the runtime context's MCP assignment and binding (with `aggregateDigest`, which hashes the assignment). MCP reaches the Runner through Core's tool authority on every run attachment, never through files the environment publishes. Core already continues a warm session across an MCP-only change (`nativeRuntimeContextsMatch` with tool refresh). The binding is also named per run (`native-mcp:<runId>`). So the digest stays the same across warm turns of one Runner, lease and session, including when the agent's MCP access changes. - `runDigest`: `preparationDigest` plus everything it leaves out: `runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is unique to one run's launch. A plugin that advertises `launchDigestVersion: 1` in its task description gets both fields on `prepare`. **Reason and benefit** A plugin can compare `preparationDigest` with its saved preparation and transfer only the run identity when they match, instead of a full preparation. **Breaking changes** None. Plugins without `launchDigestVersion` get no new fields. ## What Changed - `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add `runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The preparation digest drops the MCP assignment and binding; the run digest takes the full requirements and includes them. Both use the same canonical JSON as `runnerLaunchRequirementsDigest`, with their own schema tags so the three digests never collide. Export them from the package and the server's vendored runner shim. - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `preparationDigest` and `runDigest` to the `prepare` operation, and optional `launchDigestVersion: 1` to the task description. The capability gates both digests. - `server/src/services/native-runtime/plugin-runner-task.ts` and `native-session-executor.ts`: send both digests on `prepare` when the description advertises `launchDigestVersion: 1`. - `server/src/services/environment-task-runtime.ts`: recompute both digests from the requirements, and refuse a `prepare` whose digests are missing in part or do not match. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK public source pin. - Tests: - the preparation digest is stable across runs and changes when the runner, lease, session, instructions, model, approval policy or settings change; - with a runtime context, an MCP-only change (the run's binding, no binding, a changed access set) keeps the preparation digest and changes the run digest; - a change to the instruction bundle, instruction entry, a skill bundle, the skill set, the working copy or the connection instructions changes the preparation digest; - the run digest changes per run; - Core sends the digests only when advertised; - the runtime refuses forged or partial digests before it calls the plugin. ## Verification - `npx vitest run src/live/runner-launch-preparation.test.ts` in `packages/paperclip-runner`: 26 passed. - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed. - `npx vitest run source-sdk-public-inputs.test.mjs source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in `tests/runner-e2e`: 59 passed. - `pnpm typecheck` in `server` and in `packages/paperclip-runner`: clean. - Not run locally: the full suite. CI covers it. ## Risks Low risk. The digests are new optional fields behind a plugin capability version. Core computes them and checks them again before it calls the plugin, so a plugin can rely on them. They prove only that the launch requirements are unchanged. A plugin that skips work on a match must still check its own state, such as files it published into the sandbox. If the runtime context carries values that change per run, the preparation digest changes too, so a plugin does its full preparation. That is safe, but the plugin saves nothing. The MCP binding is the one such value Core sets per run today, and MCP is excluded entirely. A plugin that does publish MCP configuration into its sandbox must not rely on this digest for it. Values outside the requirements are not covered at all: the account identity and credentials, the GitHub broker, project IDs, and the plugin's own runtime revision. A plugin must keep checking those itself. A match lets a plugin skip publishing its launch configuration, not computing the descriptor: `prepare` must still return a launch whose `requirementsDigest` matches the current requirements, which carry the current MCP assignment and binding. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b1e70ab6dd |
fix(db): recover prepared statements that a pooler or schema change invalidates (#21718)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs every database statement from one Node main thread.
With `prepare: true`, postgres.js sends a statement in one round trip
instead of two, and the CPU cost of each statement is about half
(benchmark from #21707).
> - `DATABASE_PREPARED_STATEMENTS=true` turns on preparation for ORM
queries (#17673). Hosted instances connect through a transaction-mode
pooler (PgBouncer 1.22 or later, Neon `-pooler` hosts), which supports
protocol-level prepared statements.
> - I tested the flag locally through PgBouncer 1.26.1 in transaction
mode. Evictions, server connection churn, `RECONNECT`, transactions,
pipelining, `SET LOCAL`, advisory transaction locks and savepoints all
work. One failure is not acceptable: after a migration changes a column
type, every prepared statement that reads that column fails, for every
client and for every new process.
> - PgBouncer shares one server-side prepared statement by SQL text.
PostgreSQL never repairs a named statement whose result type changed, so
it fails with `cached plan must not change result type` until PgBouncer
drops it. The postgres.js retry prepares the same text again, and
PgBouncer gives it the same broken statement. postgres.js also caches
the failed statement without columns, so the connection throws
`TypeError: Cannot read properties of undefined (reading 'length')` for
that text even after the schema is correct again.
> - This pull request makes the client recover. When a prepared
statement fails with a prepared-statement error, the client stops
preparing that text, retries once without preparation where that is
safe, and counts it. It also caps the number of prepared texts.
> - The benefit is that the flag can be turned on behind a pooler
without a deploy migration that breaks reads. Behavior with the flag
unset or `false` does not change.
## Linked Issues or Issue Description
Refs #17673 (the opt-in), #10795 (the flag), #21707 (the CPU benchmark).
**What happened?**
With `DATABASE_PREPARED_STATEMENTS=true` behind PgBouncer 1.26.1 in
transaction mode (`max_prepared_statements=5`, local PostgreSQL 17), a
change of a column type made every later execution of an affected
statement fail. A new client process failed in the same way. Measured
with the local probe, 20 executions after the change and 10 from a new
process:
| Change | Before: failures | After: failures |
| --- | --- | --- |
| `int` to `bigint`, ORM read outside a transaction | 20/20, new process
10/10 (0A000) | 0/20, 0/10 |
| `varchar` length change, ORM read | 20/20, 10/10 | 0/20, 0/10 |
| Add column, `select *` | 20/20, 10/10 | 0/20, 0/10 |
| `int` to `bigint`, first statement of a transaction | 20/20, 10/10 |
0/20, 0/10 |
| `int` to `bigint`, second statement of a transaction | 20/20, 10/10 |
1/20, 1/10 |
| Table recreated with its original shape, same client | 10/10
(TypeError in `DataRow`) | 0/10 |
| Add column, explicit-column ORM read | 0/20 | 0/20 |
**Expected behavior**
A statement whose pooled prepared statement is stale or lost runs
anyway, and the failure does not reach the caller.
**Steps to reproduce**
1. Run PostgreSQL behind PgBouncer 1.22 or later with `pool_mode =
transaction` and `max_prepared_statements` above 0.
2. Start Paperclip with `DATABASE_PREPARED_STATEMENTS=true` and run a
read of a table a few times.
3. Change the type of a column that the read returns, for example `alter
table ... alter column n type bigint`.
4. Run the read again from the same process or from a new process: it
fails with `cached plan must not change result type`.
## What Changed
- `packages/db/src/prepared-queries.ts`:
- The prepare decision for each `unsafe` query is made when the driver
builds the query, not when the caller creates it, so a query that waits
on a busy connection sees a text that was demoted after it was queued.
- A failure with 26000 (`prepared statement does not exist`), 42P05
(duplicate statement name), or 0A000 `cached plan must not change result
type` demotes the text: the client does not prepare it again in that
process.
- A pooled statement that fails in that way is retried once with
`prepare: false`. Statements in a reserved connection, cursors,
`forEach`, `describe` and queries with `onexecute` are not retried.
- When the failed statement is the first statement that the transaction
ran (only `SET TRANSACTION` before it), the client restarts the
transaction on the same connection: `ROLLBACK`, `BEGIN` with the same
options, the same `SET TRANSACTION` statements, then the statement
unprepared. The callback runs once and receives the result. Tagged
statements and statements in savepoints count toward the execution
order. The client does not restart when another statement of the
transaction is queued, and it never restarts inside a savepoint.
- At most 1,000 distinct texts are prepared for each client and its
private pools. Texts longer than 8 KB are not prepared. Explicit
per-query `prepare` options, simple queries and the flag-off paths are
unchanged.
- `packages/db/src/client.ts`, `packages/db/src/index.ts`: a client and
its dedicated, liveness, checkpoint, file-storage and listener pools
share one state. `preparedStatementStats(db)` returns the prepared,
unprepared, demoted and retried counts.
- `packages/db/src/prepared-queries-postgres.test.ts`: nine tests
against embedded PostgreSQL: restart after a stale plan and after a lost
statement (the callback runs once and its later work commits),
`repeatable read read only` kept after a restart for driver and ORM
transactions, no restart after a later statement, after a tagged first
statement, or for an out-of-order await, a savepoint whose first
statement fails while the outer transaction continues once, the pooled
retry with `.values()`, and the cap.
- `doc/DATABASE.md`, `docs/deploy/database.md`: the cap and the
recovery.
## Verification
- Red/green, `pnpm --filter @paperclipai/db exec vitest run
src/prepared-queries-postgres.test.ts`. With the old
`prepared-queries.ts` (stub stats exports only), the five new tests
fail: two transactions fail with the stale-plan and the
missing-statement errors, the ORM read fails with 42P05, the invalidated
text is prepared again, and 5 texts are prepared where the cap is 2.
With this change, 23/23 pass.
- `pnpm --filter @paperclipai/db exec vitest run` on
`prepared-queries-postgres`, `client-options`, `client`,
`dedicated-connection-pool`, `dedicated-connection-admission`,
`response-capture-postgres`, `lazy-transaction-queries`,
`client-disconnect-replay`: 91/91 pass on the final revision. `tsc
--noEmit -p packages/db` passes.
- 31 server test files with `DATABASE_PREPARED_STATEMENTS=true`
(services, routes, inbox, recovery, heartbeat events, agent auth,
prepared-statement helpers): 829/830 pass. The one failure
(`execution-workspaces-service`, a git-scan timing test) passes alone
with the flag on and off. The machine load average was 9 to 20.
- Local PgBouncer probe (not in CI, because CI has no PgBouncer):
PgBouncer 1.26.1, PostgreSQL 17, postgres.js 3.4.9 (patched), Node 26.4.
The results in the table above, plus 0 failures for 4,800 statements
over 60 distinct texts with `max_prepared_statements=5`, 0 failures with
`pg_terminate_backend` and `RECONNECT` every 25 statements, and 0
failures for transactions with `SET LOCAL`, pipelines of 20 statements,
`pg_advisory_xact_lock` and savepoint rollback.
- Through PgBouncer 1.26.1 (transaction mode,
`max_prepared_statements=5`, 4 server connections) and PostgreSQL 17, at
head
|
||
|
|
8cc26221af |
perf(native): accept a plugin's in-preparation project-auth check on warm attach (#21710)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On a warm turn Core attaches the kept Runner to the new run: the plugin prepares the new run, then Core checks managed project authentication > - A warm attach has no submit step, so Core sends a separate `check_project_auth` operation after `prepare` > - That separate operation is one more plugin dispatch and Launcher round trip on every warm turn, about 0.12–0.18 s, and it is serial > - This pull request lets the plugin do the check inside `prepare` and say so, so Core does not send the second operation > - The benefit is a faster warm turn with the same proof and the same failure behavior ## Linked Issues or Issue Description **What existing behavior does this improve?** The warm attach of a plugin Runner task (`plugin-runner-task.ts` `prepareLaunch`). When `launched` is true, Core always calls `checkProjectAuth` after `prepare`. **Subsystem affected** Native runtime, plugin Runner tasks, and the plugin SDK environment task contract. **Current behavior** Cold starts fold the check into submit (`projectAuthCheckAtSubmit`). Warm attaches send `prepare`, then a separate `check_project_auth`. Measured on a plugin sandbox, the separate operation costs 0.12–0.18 s on each warm turn. **Proposed behavior** The `prepared` result has a new optional flag, `projectAuthChecked`. A plugin sets it only when it ran the managed project-authentication check inside this preparation, for this launch's configuration, provider and projects, and the check allowed them. Core then records the same lease proof that an allowed `check_project_auth` records, and skips the separate check. **Reason and benefit** One fewer serial round trip on every warm turn. The plugin already runs the check for this task; this removes a dispatch, not a check. **Breaking changes** None. The flag is optional. A plugin that does not set it, or that saw a refusal, gets the current behavior. Plugins detect the flag with the SDK schema before they send it. ## What Changed - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `projectAuthChecked: true` to the `prepared` result, with a doc comment that defines it. - `server/src/services/environment-task-runtime.ts`: when a `prepared` result has `projectAuthChecked`, record `runnerTaskProjectAuthValidation` for the launch's configuration, provider and projects, without `atSubmit`. This is the proof that `submit` and `read_file` require. - `server/src/services/native-runtime/plugin-runner-task.ts`: do not send `check_project_auth` when the prepared result has `projectAuthChecked`. - Tests: a warm attach with the flag makes only `prepare`; without it, `prepare` then `check_project_auth`. The runtime test covers all four combinations of `projectAuthCheckAtSubmit` and `projectAuthChecked`. ## Verification - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server/`: 157 passed. - Not run locally: full typecheck and full suite. CI covers them. ## Risks Low risk. Core trusts the flag the same way it trusts a plugin's `check_project_auth` answer and `projectAuthCheckAtSubmit` today. A refusal still goes through Core's own check, so the conflict error is unchanged. The plugin side ships separately and sends the flag only when its SDK accepts it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, tracing queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b5ba548112 |
feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line that a real-agent ramp harness checks (#21695) > - Four finish-line criteria need numbers from inside each server process: main-thread CPU, run event lateness, agent API time before the handler, and missed lease renewals > - The server had no way to give them, so every harness run was at best "incomplete" > - This pull request adds one instance-admin endpoint that returns those numbers per process over a short sliding window, off by default, and connects the harness to it > - The benefit is that a ramp run can measure every server criterion it can see, with near-zero cost when on and no cost when off ## Linked Issues or Issue Description Refs #21695 **Subsystem affected** Server (instance routes, run event append, native lease renewal), database client (statement counter), and the scale-ramp harness in `scripts/scale-ramp/`. **Problem or motivation** The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the server has no endpoint for it. The criteria `main_thread_cpu`, `run_event_lateness_p99`, `agent_api_pre_handler_p99` and `lease_renewals_missed_zero` are therefore always "not measured", and no run can pass. **Proposed solution** Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of one server process over the last 50 to 60 s. Only an instance admin can read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start, default off) turns on collection and the route. Off, the path answers 404 and no collector starts. **Alternatives considered** - The postgres.js `debug` option for the statement counter: rejected, because it captures a stack (`new Error()`) for every query. - Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected, because it includes GC and libuv threads. It is returned as `processCpu` instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux only, null elsewhere). - Default on: the measured cost is negligible (numbers below), but the event-loop monitor wakes the process every 20 ms when idle, and this is a new instance-wide surface. It stays off; the runbook says how to turn it on for a ramp run. ## What Changed - `server/src/services/server-metrics.ts`: a per-process collector. It uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20 ms), `process.cpuUsage()`, the main thread's `/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native `perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing on the hot path awaits, allocates per call or does I/O. - `server/src/routes/instance-server-metrics.ts`: `GET /api/instance/server-metrics`, guarded by the existing `assertInstanceAdmin` (the check that `/instance/database-backups` and other instance routes use). It returns numbers, a random per-process `processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no SQL, names, IDs or hostnames. - `server/src/app.ts`: when the flag is on, two middlewares time agent API requests from arrival to the API router, the route is mounted, and shutdown stops the collector. When the flag is off, nothing is mounted. - The snapshot also carries `dedicatedConnections` from `dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit` since start. The harness ignores it today; it shows dedicated-connection queueing in a restart burst. - `packages/db/src/statement-count.ts` and `lazy-transaction-queries.ts`: a Drizzle logger that increments one counter per statement replaces Drizzle's no-op logger. No edit to `client.ts`. - `paperclip-control-plane-port.ts`: one call records runner `emittedAt` to committed row for each committed (not duplicate) runner event. - `native-execution-lease.ts`: one call counts a lease whose renewal deadline passed. - Harness: missed lease renewals are summed per `processId` (several processes, or a restart, no longer cancel out). A metrics URL on the target origin gets the board token in the `Authorization` header; `PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin gets no token. - Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the serializer's own output. A server test asserts the serializer produces exactly this file, and a harness test parses it and checks every field. - `rules/company-scope-exceptions.mdc`: adds the snapshot as an instance-wide exception with its reason. The runbook in the plan doc documents the path, the flag, the credential and each field. | Field | Provided | Source | | --- | --- | --- | | `mainThreadCpu` | yes (Linux; null elsewhere) | `/proc/self/task/<pid>/stat` utime+stime | | `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` | | `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s p99 in the window | | `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT or raw driver calls) | | `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue time | | `runEventLatenessP99Ms` | yes (null with no events) | runner event append | | `agentApiPreHandlerP99Ms` | yes (null with no agent requests) | arrival and router-entry middlewares | | `leaseRenewalsMissed` | yes, counter since process start | native lease deadline timer | ## Verification - `cd server && npx vitest run src/__tests__/instance-server-metrics-routes.test.ts src/__tests__/server-metrics.test.ts`: through `createApp` on the embedded test database. 200 with the exact field set for an instance admin; 403 for a company owner who is not an instance admin, for an agent key, and unauthenticated; 404 and no collector when the flag is off; values move after agent requests, statements, a late run event and a missed renewal; the body has no IDs, keys or SQL. - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks (red/green checked for the lease hook by removing the call). - `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts src/client-options.test.ts`: the counter counts statements inside transactions and savepoints. - `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture contract and the board-token header. - `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in `packages/db`: pass. Overhead, measured on a shared, loaded laptop (load average 8 to 20), median with min and max: | Measure | Off | On | | --- | --- | --- | | Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) | 3.36 ns (3.32 to 6.16) | | CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us (20.3 to 42.4) | 32.3 us (20.7 to 45.5) | | CPU per HTTP request, client and server in one process (2,000 x 6) | 67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) | | Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08% (0.05 to 0.09) | | Record one run event (Date.parse + histogram) | - | 148 ns (135 to 188) | | One snapshot | - | 55 us | The statement and request differences are inside the run-to-run spread. The direct cost is about 3 ns per statement, against about 30 us of CPU per statement. ## Risks - Low risk. Off by default; off, only the statement counter runs, and it costs one increment more than Drizzle's no-op logger. - `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other platforms. The harness then uses `eventLoopUtilization`. - Run event lateness compares the runner's clock with the server's clock, so clock skew adds to it. Negative values count as zero. - With several processes, each request reaches one process. The harness keeps the highest gauge and sums counter growth per process, but a process it never samples is not seen. - The statement counter does not count BEGIN/COMMIT or raw postgres.js calls. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8adf0a463a |
perf(db): build the agent request authentication statements once per client (#21707)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - One Node process controls every run of a workspace. At 1,000
concurrent agents, main-thread CPU for each database statement is the
scarce resource.
> - A CPU profile of the hosted server shows the cost is per statement:
building SQL text in Drizzle costs about as much as parsing the result
in postgres.js.
> - Every authenticated agent API request (about 0.3 per agent per
second, plus tool calls) runs three fixed statements in
`actorMiddleware`: the agent row, the run authentication projection, and
the responsible user's company memberships. The server built the SQL
text of all three again for every request.
> - A local micro-benchmark shows that building the text is 15 to 40 µs
of CPU per statement for these shapes, and that a reused prepared query
costs the same as a raw driver call. Drizzle's `tracing.ts` wrapper is a
no-op (0.1 µs).
> - This pull request prepares the three statements once per database
client with placeholders, and executes them with the request's values.
The driver receives the same SQL text and parameters as before.
> - The benefit is about 40% less main-thread CPU for each authenticated
agent request in a local benchmark.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The CPU cost of agent JWT and agent key authentication on every agent
API request.
**Subsystem affected**
Server request authentication (`server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts`).
**Current behavior**
For every agent request, `actorMiddleware` builds three Drizzle queries
from scratch: `agents` by id, the `heartbeat_runs` authentication
projection (with `jsonb_build_object` and two `run_identity_contexts`
subqueries), and the `company_memberships` read with an `exists` on
`auth_users`. Building and preparing them costs 14, 38 and 40 µs of CPU
(median of 7 passes of 5,000 builds).
**Proposed behavior**
Build each statement one time for each database client, with
`sql.placeholder` values. Execute the prepared query with the request's
values. The SQL text, the parameters, and the returned rows stay the
same.
**Reason and benefit**
At 1,000 agents, agent API traffic is several hundred authenticated
requests per second on one main thread. Measured CPU per authenticated
request (local, embedded PostgreSQL, `process.cpuUsage()`, 2,000
requests per pass):
| Run (interleaved) | Before: median (min–max) µs | After: median
(min–max) µs | Load average |
| --- | --- | --- | --- |
| 1 | 436 (363–599) | 200 (183–224) | 10.8 / 10.1 |
| 2 | 489 (430–535) | 312 (212–392) | 11.6 / 13.4 |
| 3 | 461 (442–562) | 281 (247–409) | 13.1 / 12.9 |
| separate | 383 (322–810), 301 (268–476) | 169 (156–439), 193 (174–215)
| 9.5 / 9.0 |
The machine is shared (18 cores), so the spread is wide. Every pair
shows the after run lower. The drop is larger than the build cost alone,
because the request also allocates less (fewer builder objects to
collect).
**Breaking changes**
None. The statements, their parameters and the result shapes do not
change.
**Additional context**
Per-statement cost attribution from the new
`packages/db/scripts/benchmark-statement-cpu.ts` (CPU µs per statement,
median of 5 passes of 2,000, two runs, concurrency 1, load average
20–26):
| Statement | Drizzle builder | Build only (`toSQL`) | Drizzle
`.prepare()` reused | `unsafe(text, params)` | `unsafe(..., {prepare:
true})` |
| --- | --- | --- | --- | --- | --- |
| `issues` parent by id | 105 / 122 | 17 / 18 | 59 / 75 | 57 / 63 | 30 /
33 |
| `issues` full row by id | 277 / 326 | 136 / 153 | 96 / 81 | 97 / 77 |
48 / 33 |
| `company_memberships` by principal | 128 / 103 | 35 / 33 | 71 / 66 |
80 / 63 | 37 / 32 |
| join with an in-list of 10 | 169 / 173 | 57 / 57 | 83 / 88 | 109 / 80
| 42 / 40 |
| update by id | 110 / 106 | 25 / 26 | 75 / 80 | 76 / 73 | 39 / 40 |
| insert `activity_log` | 116 / 125 | 34 / 35 | 65 / 57 | 64 / 55 | 41 /
30 |
Drizzle's no-op tracer (`startActiveSpan`) measured 0.1 µs per call.
## What Changed
- `server/src/services/prepared-statements.ts`: add
`preparedStatement(prepare)`, the single-statement form of the existing
`preparedStatements` helper (one prepared query for each `Db`, held in a
`WeakMap`).
- `server/src/middleware/agent-run-authentication.ts`: split the query
into `agentRunAuthenticationQuery(db, binding)`;
`loadAgentRunAuthentication` executes the query prepared once with
placeholders.
- `server/src/middleware/auth.ts`: add `agentAuthenticationQuery` and
`responsibleUserMembershipsQuery`; the JWT path, the agent key path and
`loadResponsibleUserMemberships` execute their prepared forms.
- `server/src/__tests__/agent-auth-prepared-statements.test.ts` (new):
proves that each prepared statement sends the same SQL text and
parameters and returns strictly equal rows, and that repeated agent
requests do not build the statements again.
- `server/src/__tests__/agent-auth-middleware.test.ts`: the test
double's select chain now supports `.prepare().execute()`.
- `server/src/__tests__/prepared-statements.test.ts`: covers
`preparedStatement`.
- `packages/db/scripts/benchmark-statement-cpu.ts` (new): CPU per
statement for the builder, build only, a reused prepared query, `unsafe`
with and without `prepare: true`, and the no-op tracer. Run with `cd
packages/db && ITERATIONS=2000 PASSES=5 node --import tsx
scripts/benchmark-statement-cpu.ts` (`CONCURRENCY`, `ONLY`, `VARIANTS`
are optional).
- `scripts/benchmark-agent-request-auth-cpu.ts` (new): CPU per
authenticated agent JWT request through `actorMiddleware`. Run with
`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-request-auth-cpu.ts`.
Both benchmarks start their own embedded PostgreSQL in a temporary
directory.
## Verification
- Red/green: the "builds the request authentication statements once per
database client" test failed on `origin/master` (`select` called 12
times in 3 requests) and passes with this change.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-auth-prepared-statements.test.ts
src/__tests__/prepared-statements.test.ts
src/__tests__/agent-auth-middleware.test.ts
src/__tests__/agent-run-authentication.test.ts
src/__tests__/agent-jwt-request-authentication.test.ts
src/__tests__/auth-session-route.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/local-agent-process-boundary.test.ts
src/__tests__/liveness.test.ts src/__tests__/inbox-mine-routes.test.ts
src/__tests__/board-bootstrap-routes.test.ts
src/__tests__/agent-auth-identity-transfer.test.ts
src/__tests__/prepared-statements-driver.test.ts
src/services/fresh-continuation-fallback.test.ts`: all pass.
- CPU before and after: the tables above.
## Risks
- Low risk. The prepared query uses Drizzle's own `PgPreparedQuery`, the
same execute path that the builder uses. The test pins the SQL text, the
parameters and the rows against the builder.
- Drizzle's `.prepare(name)` on postgres.js does not send a statement
name to the database (see #21201). With `DATABASE_PREPARED_STATEMENTS`
unset, the driver sends the same unnamed statement as before, so
connection poolers see no change.
- A test double for `Db` must now support `.prepare().execute()` on
these selects. One test double changed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
82b243ab07 |
perf(server): bound dedicated database connections with a fair queue (#21693)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server holds long advisory-lock transactions on dedicated
database connections (`withDedicatedDbConnection`): workspace source and
lifecycle locks, the skill cache lock, agent-directory operations,
status cards and others
> - `withDedicatedDbConnection` kept at most 4 idle connections, but it
had no limit on connections in use. A run start can hold two at once,
because the source lock nests the lifecycle lock
> - After a restart or deploy, a burst of run starts can open hundreds
of connections. That can exceed the database's `max_connections` at the
moment recovery needs the database most. The documented per-process
total (`DATABASE_POOL_MAX` + 2) was also wrong
> - This pull request bounds the dedicated connections in use with a
fair queue, keeps nested and in-transaction callers deadlock-free, and
states the real connection budget
> - The benefit is that a restart burst of 200 or 1,000 run starts
completes within the limit (at most 48 in use with the default limit of
24), where before most of the starts failed with "too many clients"
## Linked Issues or Issue Description
**What existing behavior does this improve?**
`withDedicatedDbConnection`, which gives long advisory-lock transactions
a connection outside the main pool.
**Subsystem affected**
Database client (`packages/db/src/client.ts`) and the connection-sizing
documentation.
**Current behavior**
Nothing bounds the dedicated connections in use; only 4 idle ones are
kept. A run start can hold two at once (the source lock nests the
lifecycle lock). A burst of run starts after a restart can open hundreds
of connections and exceed `max_connections`. The documented per-process
total (`DATABASE_POOL_MAX` + 2) is wrong.
**Proposed behavior**
A per-process limit on dedicated connections in use
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24) with a first-in,
first-out queue. A caller that already holds a dedicated connection, or
runs inside an open transaction, never waits, because its wait could
deadlock with the holder that blocks it. After
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (default 30 s), a queued caller
gets a connection over the limit, so a lock cycle that PostgreSQL cannot
see ends in a delay, not a hang. The documentation states the real
per-process budget.
**Reason and benefit**
At 50 agents, staging showed about 9 new connections per second from
this function. At 1,000 agents, a restart burst can take the database
down when recovery needs it most. With the default limit of 24, a burst
of 200 or 1,000 starts completes with at most 48 dedicated connections
in use.
**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). Reusing the
outer session for a nested lock was considered. The locks are
transaction-level advisory locks (`pg_advisory_xact_lock`), so a nested
lock in the outer transaction would be held until the outer transaction
ends, which changes lock lifetimes and order.
## What Changed
- `packages/db/src/client.ts`: `withDedicatedDbConnection` admits
callers against a per-owner limit (`DATABASE_DEDICATED_CONNECTION_MAX`,
option `dedicatedConnectionMax`, default 24) with a first-in, first-out
queue. The slot is released after the connection is returned or closed.
- Nested callers (an active dedicated hold of the same owner in the
async context) and callers inside an open transaction of a client from
this process are admitted without a wait. The transaction marker is an
`AsyncLocalStorage` scope that ends when the transaction callback ends.
- A queued caller is admitted over the limit after
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (option
`dedicatedConnectionWaitMs`, default 30000).
- `dedicatedDbConnectionStats(db)` reports the limit, in-use, peak,
waiting, queued, over-limit and nested-over-limit counts.
- `PRIVATE_POOL_MAX_CONNECTIONS` names the private pool sizes that were
literals, and `databaseConnectionBudget()` returns the per-process worst
case: main pool + 9 private + 2 listeners + 2 x limit + 4 idle dedicated
= 73 with the defaults.
- `server/src/services/skill-package-store.ts`: an isolated package read
takes its package I/O permit before its dedicated connection, and takes
a second dedicated connection only for the cache trim after a publish.
Before, each of up to 32 queued readers held a dedicated connection
while it waited for one of 2 permits. `skillPackageStorageIoDepth()`
reports the queue for the store's test.
- `withDedicatedDbConnection(db, action, { admitWithoutWaiting: true })`
admits a caller at once. The skill package store passes it for the
dedicated connections it takes while it holds a package permit, so a
permit holder never waits for a slot that a cache-lock owner holds while
that owner waits for the permit. The overshoot is bounded by the permits
(2 storage, 2 response).
- `doc/DATABASE.md` and `docs/deploy/database.md`: document the
admission rules, the two new variables and the real connection budget,
which replaces "`DATABASE_POOL_MAX` + 2".
## Verification
- New `packages/db/src/dedicated-connection-admission.test.ts` (embedded
PostgreSQL, `max_connections` 100). A restart burst with the default
limit: each start holds a source advisory lock on a dedicated connection
and nests a lifecycle lock on a second one, 40 ms in all.
- Before (same burst, scratch test on master): 200 starts gave 151
failures ("sorry, too many clients already") and 99 dedicated sessions
open. 1,000 starts gave 823 failures and 99 sessions.
- After: 200 starts gave 0 failures, 45 sessions, peak in use 48, queue
wait p50 541 ms / p99 815 ms. 1,000 starts gave 0 failures, 40 sessions,
peak in use 48, wait p50 1,767 ms / p99 3,422 ms. The wait is the queue
of this synthetic burst of 40 ms holds. Before, the same callers waited
about as long for connection setup, and most then failed.
- Steady state: 20 concurrent holders under the limit are never queued
(`queued: 0`).
- Nested acquisition: with a limit of 2, two holders each take a nested
connection while a third caller is queued. All finish (no deadlock); the
queued caller runs after them.
- In-transaction caller: with a limit of 1, a pooled transaction holds a
row lock and then asks for a dedicated connection, while the one
dedicated holder waits for that row lock. The caller is admitted at once
and both finish. A hard limit would deadlock here.
- Resource holder: with a limit of 1, the slot holder waits for a
resource that a second caller holds. The second caller, with
`admitWithoutWaiting`, is admitted at once and both finish.
- Wait bound: with a limit of 1 and a 100 ms bound, a queued caller is
admitted over the limit after the bound.
- Budget: the sum of the actual `max` of every private pool equals
`databaseConnectionBudget().privatePools`, and the default total is 73.
- `skill-package-store.test.ts` ("does not load queued legacy bodies for
files reads"): 32 readers queue for package I/O while only the 2 active
readers hold dedicated connections, and the 33rd reader still gets
"busy" at once. With the old order and a limit of 24, CI showed only 22
readers reaching that queue. `skill-package-store`,
`skill-version-cache`, `company-skills-empty-runtime`,
`runtime-skill-preservation`, `company-skills-runtime-concurrency` and
`company-skills-service` pass locally. Two `company-skills.test.ts`
workspace-discovery tests fail locally on master too (same result with
this change), so they are not caused by it.
- `packages/db`: client, lazy-transaction, prepared-query,
response-capture, work-signal and dedicated-pool suites, 134 passed.
- `server`: skill-version-cache, status-card-evaluation-work,
git-workspace-expiration, execution-workspace-runtime-control-conflict,
agent-instruction-working-copies, company-skills-empty-runtime and
decisions-service suites: 413 passed. In one combined run under machine
load, git-workspace-expiration failed at file level with all its tests
passing; alone it passed 96 of 96.
- `npx tsc --noEmit` in `packages/db`: passes.
## Risks
- A caller that holds an application-level lock (for example a
filesystem lock) and then waits in the queue can block a dedicated
holder that needs that lock. The 30 s wait bound ends such a cycle with
a delay. The over-limit counter shows it.
- Nested and in-transaction callers do not wait, so the in-use count can
reach twice the limit with one level of nesting. The documented budget
counts this.
- In a large restart burst, run starts now queue for a slot instead of
opening new connections. Steady state (about 10 to 20 dedicated
connections at 1,000 runs) stays under the default limit of 24, so a
run's first turn does not wait there.
- The checkpoint pool (3 connections) can limit throughput at 1,000 runs
on a hosted database. This pull request does not change it; it needs an
owner decision on the hosted connection budget.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
c603f591dc |
fix(runner): bound ACPX instructions only where the sandbox delivers them (#21646)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner prepares a private sandbox for each ACPX agent before it
opens the provider session
> - `validateProviderPolicy` in that sandbox stage refuses system
instructions over 32 KiB for every ACPX agent
> - Only Pi gets the text from the sandbox stage. Claude, Codex, Grok
and Antigravity do not use it there. The host accepts 256 KiB
> - An agent with a large instruction bundle fails every run at
`session.open`. The error says only `classification=admission_sandbox`,
so the operator cannot see the cause
> - This pull request applies the 32 KiB sandbox bound only to Pi, and
gives every instructions rejection its own closed admission code
> - The benefit is that Claude and Codex agents with large instructions
can start, and an over-bound rejection names its real cause
## Linked Issues or Issue Description
Fixes: #20814
Refs #14430 (this pull request added the 32 KiB check, with no stated
reason for the other providers)
Related: #21644 (also fixes #20814, but only for Claude, and does not
change the classification)
**What happened?**
Every run of a Claude ACPX agent failed with
`provider_session_open_rejected`. The message was `ACPX sidecar command
session.open was rejected (retryable=false,
classification=admission_sandbox)`. A reproduction in a sandbox from the
same image showed that all the filesystem steps of
`prepareAcpxRuntimeSandbox` succeed. The real cause: the agent's
instructions entry file was 54,734 bytes. That is more than the 32 KiB
check in `validateProviderPolicy`. The host bound (`boundedInstructions`
in `runtime-host.ts`) is 256 KiB. Agents with smaller instructions (12.7
KB) work.
**Expected behavior**
The sandbox stage must refuse only the text that it delivers. #14430
added the check when only Pi consumed `systemInstructions` in the
sandbox (as the `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var). The check
runs before any per-provider branch, so it also refuses Claude, Codex,
Grok and Antigravity. Those providers get the text in the session
request. When a bound refuses the text, the error must name the
instructions, not the sandbox.
**Steps to reproduce**
1. Create a Claude ACPX agent whose composed instructions are more than
32 KiB (for example, an `AGENTS.md` of 54 KB).
2. Start a run of the agent on the Paperclip Runner.
3. See that `session.open` fails with
`classification=admission_sandbox`.
**Paperclip version/commit**
`master` at
|
||
|
|
ddfbbfff20 |
fix(runner): recover provider-started turns after a controller restart (#21612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - The Runner can keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started. Core can wait for that turn. #21607 lets a caller ask the transport for it. > - A controller can restart at any time, for example at a deploy. A wait can last 10 minutes and a provider-started turn can last longer, so a restart during one of them is a normal event. > - Tests with the real `paperclip-runnerd` showed that such a restart failed the session. The checkpoint that Core had was the checkpoint of the turn before, and the recovery of the driver refused or mislabelled the turns that it did not know. > - This pull request makes that restart safe. It changes no production run, because no caller asks for provider-started turns before part 3. > - The benefit is that part 3 can turn the behavior on without a failed run at each deploy. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. #21607 (merged): the transport can ask the Runner for provider-started turns. No caller asks. 2. **This pull request:** the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. This is part 2. Its base branch is `master`. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It makes a controller restart safe for a run that asked for provider-started turns. **Subsystem affected** Paperclip Runner package: the consumer (`src/native-session-runtime.ts`), the session backend (`src/backends/harness-driver-backend.ts`), the driver (`src/drivers/codex/`) and the transport (`src/live/`). Only for a run that asked for provider-started turns. **Current behavior** - The consumer saves no checkpoint when a wait starts or when a provider-started turn starts. During a wait and during a provider-started turn, the checkpoint that Core has is the checkpoint from the start of the turn before. It names that turn as active. - A restart from that checkpoint fails: the driver has the old turn as active, gets the events of a provider-started turn, and stops the session with `turn_binding_mismatch`. - The recovery of the driver adopts one turn that it does not know as a disposition recovery turn, and refuses two or more. With two or more, it closes the transport, and that stops the Runner and Claude. - The Runner does not send again an event that the controller acknowledged. If the start of a provider-started turn was acknowledged before the restart, the new driver gets the other events of that turn with no start. - A turn that ends waiting for a provider-started turn sets the terminal of the session, as a turn that ends the run does. After a restart, if the provider-started turn already has its result when recovery completes, the consumer finds a result and a terminal, settles the run, and does not read the events of that turn. The start, the items, the usage event and the terminal of that turn do not reach Core. - The recovered driver reads notifications from its construction, and `reconcile()` maps the terminal of the saved turn only after its reads. If the saved checkpoint is still the checkpoint of the controller turn and the start of the provider-started turn is already queued, the driver judges that start too early and stops the session with `turn_binding_mismatch`. - A checkpoint that waits can also have a consumed disposition marker (a recovery turn after an earlier restart ended waiting). The driver restores that session as closed and reads no notifications, and the consumer settles the run on the earlier terminal with no result. **Proposed behavior** - **Checkpoints.** The consumer saves a checkpoint when a wait starts and when a provider-started turn starts, with the function that it already uses for a governed wait. A checkpoint that fails stops the turn, as for a governed wait. - **A checkpoint as of the event that Core stored.** The driver gives each event its sequence when it reads it from the provider, before the consumer stores it. A snapshot that is taken later can be ahead of Core. For these two checkpoints, the driver captures its snapshot when it has handled the notification that made the event (the `turn.completed` that waits, or the `turn.started` of the provider), with the sequence of that event. The consumer asks for the snapshot as of the event that it has just appended. The saved state and cursor are then those of an event that Core has. - **Session terminal.** A turn that ends waiting for a provider-started turn does not set the terminal of the session (`HarnessNativeSession`). The terminal of the session is the end of the run, and that turn does not end the run. A checkpoint that is saved during a wait or during a provider-started turn has no terminal, so a recovered session settles only from the events that it reads, in order, to the terminal of the turn that has the result. - **Driver, a checkpoint that waits.** Recovery keeps the wait. It does not adopt and does not refuse the turns after the last terminal that it knows. The Runner sends again, in order, each event that was not acknowledged, and the driver admits each provider-started turn from that stream, as it does with no restart. - **Driver, a checkpoint with an active turn.** `reconcile()` accepts one turn in progress that comes after the saved active turn, when the saved turn ended with the wait marker. It refuses each other unexpected active turn as before, and it refuses this one too when the reconciled turn settled a result. - **Driver, a start that is queued before recovery reconciled the saved turn.** A `turn/started` of the provider that arrives while the session does not wait for one first waits until `reconcile()` has settled, and is then judged as before. `reconcile()` judges the live state after its reads; if the stream already settled the saved turn into a wait, it accepts exactly one turn in progress after the last terminal turn and does not adopt it. The driver admits that turn at its start. - **A wait with a consumed disposition marker.** The driver keeps reading notifications for a restored session that waits, as it does for a wait with no restart. It still refuses a second controller-started turn. The consumer does not take the consumed-marker shortcut for a recovered snapshot that waits, so the wait is resumed. - **Transport, the start comes first.** For each driver, the transport gives the `turn/started` of a provider-started turn before each other notification and before a tool call of that turn. If it did not give that start to this driver, it builds it from the `turn.started` event in its stored events. It does not do this for the turn that the recovered driver already has as active (`resumeActiveTurnId`). It binds the events and the tool call of a provider-started turn to that turn, not to the turn that the Runner reports as active at the time of the recovery read. - **Transport, which start owns an event.** The stored events can have gaps: the window removes old events but keeps the latest of each anchor type, and the controller checkpoint drops streamed deltas. The owner of an event is the last stored `turn.started` before it, if it is a provider start, and if the first turn terminal that is stored after that start is not before the event and is the terminal of the same provider turn (`providerTurnId`). If the first stored terminal after the start is the terminal of a different turn, the end of the provider turn is not known, and the transport builds no start. **Reason and benefit** - The checkpoint is the cause. With a true checkpoint, the two states of a restart are states that recovery can handle: a wait, or an active turn. - The Runner already sends again each event that was not acknowledged. Recovery uses that stream and does not build the turns again from a read. - The window removes a terminal only while a later terminal of the same type is stored. Thus the first stored terminal after a start shows if a terminal of that turn was removed. A rule that only ignores the gaps is not safe: the Runner can make a `turn.failed` for a turn that did not emit its start (`provider_turn_lost_on_restore`). ## What Changed Production code: 113 lines added, 19 removed. - `src/native-session-runtime.ts` (15 added, 4 removed): the two checkpoints in `consumeTurn`, with the sequence of the appended event, and the resumed wait with a consumed disposition marker. - `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed), `codex-session-terminal.ts` (2 added), `codex-session-notifications.ts` (2 added), `codex-harness-session.ts` (24 added, 7 removed), `codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at the event, and the recovery of the driver. - `src/contracts/harness-driver.ts` and `src/contracts/native-session-backend.ts` (1 added each): an optional `boundarySnapshot` on a driver session, and an optional sequence in the snapshot options. Other drivers do not implement it and are not changed. - `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the session terminal, and the snapshot as of a sequence. - `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and `src/live/provider-turn-tool-order.ts` (18 added): the start-first rule and the owner of an event. - Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5 removed), `src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348 added), `src/live/runnerd-codex-transport.test.ts` (137 added), `src/control-plane/durable-prp-control-plane.test.ts` (84 added), `src/live/provider-turn-tool-order.test.ts` (new, 61), `src/native-session-runtime.provider-turn.test.ts` (92 added, 6 removed), `src/backends/harness-driver-backend.test.ts` (56 added). - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only, 123 added, 14 removed): a delay before a provider-started turn, a goal state at its terminal, a list of provider-started turns for one session, a gap between them, a turn that stays open until the new controller reads the snapshot, and a turn that starts after that read. The modes that existed are not changed, and the fake reads its input on one thread as before. ## State saved by the previous release - No new persisted field. The driver session already has `awaitingProviderTurn` (#21453). - **More checkpoints:** one when a wait starts and one when a provider-started turn starts. An ordinary turn saves the same number of checkpoints as before (4 in the unit test, measured before and after). - **No terminal in a checkpoint of a wait.** A checkpoint that is saved during a wait or during a provider-started turn has `terminal: null`. When a wait ends with no provider-started turn (the limit, a comment of a person, or the Runner ends the run), the last checkpoint of the run also has `terminal: null`; before, it had the terminal of the turn that waited. One reader uses that field with no result: the text of a cleanup failure (`nativeCheckpointRecordsCompletedTurn`), which then does not say that the turn completed. In the tests with the real daemon (the limit, the provider process lost, the sidecar exits), the run still ends on the path of today (`native_finalization_missing`). - **A checkpoint of the previous release** has no waiting state. Recovery takes the path of today. - **Rollback of the Runner package during a wait:** the previous code reads the checkpoint, ignores `awaitingProviderTurn`, and takes its recovery path of today. ## Verification With nothing from part 3. The Rust checks and the build checks in the first item were run before the last seven commits; these commits change no Rust. The other items were run at the head of this branch (`36d5e5e587`), or at an earlier commit of this branch for a suite whose files the later commits do not change. - Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm run build:typescript` and `node --test test/acpx-sidecar-contract.test.mjs` (18 passed) in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt --check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the `acpx` unit tests (102 passed): all passed. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in `server`: all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the production code of this head, 26 of 26 passed in each. The Core of these tests follows the production append (`appendNativeSourceEvent`): it stores an event that is out of order, keeps its contiguous cursor below a gap, and reports a duplicate or a conflict for an event that it has. It does not reject a gap in the sequence; production does not. The restart cases, all with the real `paperclip-runnerd`: - **From the checkpoint that the real consumer saved.** The first `executeNativeSession` is fenced and detached as the server does for a restart, and a second one recovers from the last checkpoint that the first one saved. (a) Restart during the wait; the provider-started turn starts after the new controller is attached. (b) Restart while a provider-started turn is in progress. In each: the test asserts the shape of that checkpoint, the run settles with the result of the provider-started turn, each event is appended one time and in order, the turn has one `turn.started` with `origin: "provider"`, and the session closes with no error. - **Without the two checkpoints, each of these cases fails:** the saved checkpoint is the checkpoint of the turn before, and the second controller fails with `turn_binding_mismatch` or with a replay conflict. - **A slow `appendEvent`, with events that arrive behind the stored event** (3 cases, each run 3 times with the fix and 3 times without it, with the same result in each run). R7: the append of the `turn.completed` that waits is slow, and the provider-started turn starts behind it. With the snapshot at the event, the checkpoint is the waiting state at the sequence that Core has, and Core gets the `turn.started` of the turn one time. Without it, the checkpoint names the provider-started turn as active, the transport builds no start, and Core never gets that `turn.started`. R8: the first controller answers the `paperclip_finish` call while the append of the provider `turn.started` is slow. With the snapshot at the event, the checkpoint has no result and the run ends with `native_finalization_missing` (see Risks). Without it, the checkpoint has a result that Core does not have, and recovery stops with "session cannot start another turn". R6: the append of the provider `turn.started` is slow and a delta is queued behind it. With the snapshot at the event, the new controller continues the sequence that Core has. Without it, the only difference is one sequence number that Core never gets. - **The start was acknowledged before the restart** (3 cases: one turn open across the restart; two turns; a turn that waits for the answer to its `paperclip_finish` call). The new driver gets the start of each turn one time before its other events, the finish call is answered exactly one time, and the session closes with no error. - **The turn ran with no controller attached** (2 cases, with and with no result) and **two turns in one outage** (2 cases): the turns arrive in order after the recovery. - **A provider-started turn that is active in the checkpoint** gets no second start, and the next provider-started turn arrives after it. - **The turn already has its result when recovery completes** (3 cases: the turn made its `paperclip_finish` call with no controller attached; the turn waited for the answer to that call at the restart; two turns across the restart). The test asserts the exact list of events that Core gets: each provider-started turn has one start, its usage item, its message item and its terminal, and then `run.result.accepted` and `run.terminal`. `completeRun` is called one time with the turn that has the result. Before the change to the session terminal, the second controller appended only `run.result.accepted` and `run.terminal` in each of these cases. - **The wait checkpoint is not saved, then a restart** (R10). The last checkpoint is the checkpoint of the controller turn, Core has the terminal of that turn, and the start of the provider-started turn is queued. The run settles with the result of the provider-started turn, Core has one `turn.started` for it, and the finish call is answered one time. Without the wait in the driver, recovery stops with `turn_binding_mismatch`. - **An ordinary turn**, with no option. One that waits for the answer to its finish call at a restart: Core gets its result, its terminal and the run terminal one time. One whose checkpoint after `startTurn` is ahead of Core at a restart (R9): the run settles, and Core does not have only the one event that the first controller did not store. These cases pass with and without this pull request. - **No checkpoint with a terminal and no result** is saved by the two controllers in the cases above. - `src/live/provider-turn-tool-order.test.ts` and `src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests cover a stored start with removed events after it, a dropped delta, the terminal of the turn before and after the event, a terminal of a different turn after a removed terminal, and a controller start. With no option, with a different agent or provider, and for a turn that the controller started, the notifications of the transport are the same as before this pull request. - `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed. Three new tests use the real commit path: a full window of 4,096 events removes the events after a provider start and the start is found; the window removes the `turn.failed` of a provider turn after a later `turn.failed` with no start, and no start is found; a restored checkpoint that dropped the deltas of the turn, and the start is found. - `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`: 491 passed. The driver tests cover a provider start that is queued before `recoverSession()` is called, the wait state with one turn in progress at `reconcile()` (and the two cases that it must still refuse), a restored wait with a consumed disposition marker (the provider turn is admitted and `startTurn` is refused), the snapshot at the two events (also after the driver moved on, and for a restart continuation that ends waiting), one later turn in progress, one completed and a second in progress, two completed with a result, and the cases that `reconcile()` must still refuse. The backend tests cover the session terminal and the snapshot as of a sequence. - `src/native-session-runtime.provider-turn.test.ts` and `src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a restored wait with a consumed disposition marker: the provider-started turn arrives and settles the run, or the wait ends at its limit and the run takes the path of today with no new turn. - `server`: `native-session-executor.test.ts`, `native-provider-turn-wait.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21 skipped. - Mutation checks, each made named tests fail: no checkpoint at the start of a wait; no checkpoint at the start of a provider-started turn; the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in the sequence, no `turn.started` in Core, a different failure); the backend ignores the sequence; the driver has no snapshot at the event; no start-first rule; no exclusion of the active turn of the recovered driver; the turn binding only at the first delivery; the contiguous sequence required again; no check of the first terminal after the start; the adoption of later turns put back; no exception in `reconcile()`; no check after the settlement in `reconcile()`; the terminal of the session set again by a turn that ends waiting; no wait of the provider start for `reconcile()`; no exception in `reconcile()` for a session that waits; the pump deferred for a restored wait; the consumed-marker shortcut taken for a snapshot that waits. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** Each change is behind the waiting state, a provider `turn.started`, or the option of the transport. No caller sets the option before part 3. - **An event that the first controller received but Core did not store is not sent again after a restart.** This is the same on `master` and is not changed here. Only the terminal of a turn is built again. If the lost event is the result of a provider-started turn (the first controller answered the `paperclip_finish` call and stopped before Core stored the proposal), the run ends on the path of today for a turn with no result. R8 shows this. R9 shows the same loss for one event of an ordinary turn. - **A second `turn.started` for one provider-started turn can reach Core.** This occurs if the restart comes after Core stored the start and before the checkpoint of that start is saved (the time of one checkpoint write). The last checkpoint then waits, and the transport builds the start again for the new driver. Seen with the real daemon when the checkpoint write is held; not in a committed test. The usage accounting of the server uses the turn id as its key, so there is no second count, but the second start resets the "restored" mark of that turn. The transcript does not read `turn.started`. - **The older checkpoints can also save a cursor that is ahead of Core** (measured for the checkpoint after `startTurn` of an ordinary turn). Production accepts the gap in the sequence that follows, so this is not a failure there. It is on `master` and is not changed here. - **When the terminal of a provider-started turn was removed from the stored events,** the transport builds no start for an event of that turn, and the session stops on that event, as before. This needs a later terminal of the same type with no start of its own (a turn that was lost while it started) and a full window. - **The last checkpoint of a wait that ends with no provider-started turn has no terminal,** as the section above says. The change is only in a run that waits, and no run waits before part 3. - **A wait with a consumed disposition marker has no test with the real daemon.** The state needs two restarts in one run, and the test helper does one. Unit tests of the driver and of the consumer cover it. - **A provider start that waits for `reconcile()`** waits as long as the reads of `reconcile()` take. Each read ends at the command deadline of the Runner (30 seconds by default) or when the transport closes. - **`item.delta` events across a restart are not asserted.** A streamed delta that the first controller received is not sent again. The completed items are asserted. - **The driver keeps a maximum of 8 snapshots at events** and drops each one when the consumer asks for it or for a later one. If the consumer is more than 8 such events behind, the checkpoint uses the current snapshot, as before this change. - **A second provider-started turn that also ends during the same outage** is not tested with the real daemon: the fake gives each terminal the same goal revision, and the Runner then holds the second terminal until a controller is attached. The end of the reconnect time of the Runner is not tested here. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, decided the designs, reviewed the diffs, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with tool use (shell, file edits, local test runs), wrote the change and the tests, reproduced the review findings with the real daemon, and verified this slice alone. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
d4498ae7ac |
fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.
Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.
**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.
**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.
**Paperclip version or commit**
`master` at `df1b12ad8e`.
## What Changed
- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.
## Verification
- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.
## Risks
- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9b581d689b |
fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In agent chat, a finished delegated task or a declined tool action wakes the chat agent so it tells the person > - #21401 stopped a second completion report when an earlier reply already gave it > - But the suppressed completion still created a zero-second heartbeat run, and declined tool actions still got a second report > - Run accounting must count only real turns, and the person must get each outcome once > - This pull request settles an already-reported outcome before it becomes a run, and applies the same rule to declined tool actions > - The benefit is one report per outcome, no phantom run, and no extra model call ## Linked Issues or Issue Description Refs #21401 **What happened?** Chat eval cohort 13 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`) showed three problems after #21401. 1. `questions-and-approvals.duplicate-answer`: the completion wake was suppressed, but it still created a 0-second run (`runtimeMode: legacy`, no usage). The bounded-native-runs check rejected that run. 2. `skills.studio-test`: the CEO's single reply reported the Studio result after it read the test task done (0 passed, 1 failed, links COM-2). The completion was then suppressed, but the eval waited for a separate completion run and timed out after 240 s. 3. `questions-and-approvals.decline-tool`: the person declined the tool review one second after the turn for their message started. That turn read the declined action and said so. The decline continuation, which waited behind that turn, then posted the same declined outcome again. **Expected behavior** Each finished task or declined tool action is reported once in the conversation. An outcome that an earlier reply already reported creates no heartbeat run. **Steps to reproduce** 1. In chat, delegate a short task, or start a governed tool read that needs approval. 2. While it waits, send a message that wakes the chat agent. 3. Let the task finish, or decline the tool review, while that turn runs. The turn reads the outcome and reports it. 4. See the completion wake become a 0-second run, or see a second decline report from the decline continuation. **Paperclip version or commit** master `df1b12ad8e` ## What Changed - `server/src/services/chat-completion-delivery.ts`: `settleReportedChatCompletions` marks each undispatched completion that an earlier reply reported as delivered, with that reply as its response. It returns true only when nothing else is owed. The outbox `deliver` also checks this before it requests a wake, so no wake and no run exist for an already-reported completion. - `server/src/modules/wake-queue`: a new transaction port, `skipAlreadyReportedChatWake`, runs in the release drain before a deferred wake is promoted. When the earlier reply already reported every outcome of the wake, the wake ends `skipped` ("Already reported by an earlier reply in this conversation") and no run is created. Wakes that carry queued comments or a failed-chat retry are never skipped. - `server/src/services/chat-interaction-outcome-report.ts` (new): the same structural rule for declined tool actions. A conversation agent run that lists its interactions records `chat_interaction.declined_read` for each declined tool action. The decline continuation is settled when the same agent replied in the conversation after the decline, from a run with that receipt, posted after the read. The rule covers only declined tool-action confirmations. Answers, plan reviews and accepted actions keep their own turn. - `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls the recorder for agent runs. - The run-start check from #21401 stays as a fallback for a completion turn that was admitted before the earlier reply landed. - `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`: the Studio oracle (v5) accepts either the completion turn's reply or an earlier CEO reply as the delivery. The earlier reply counts only from saved state: no completion run exists, the replying run has a `chat_handoff.completed_read` receipt for the test task, the reply was posted after that read, it is the only such reply, and it links the saved result. No wording check is added. - `doc/run-log-events.md`: documents both read receipts and the skipped wake outcome. ## Verification - `cd server && npx vitest run src/__tests__/chat-completion-delivery.test.ts src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts src/modules/wake-queue/application/use-cases.test.ts`: all pass. - Red: with the drain check and the outbox check disabled, 5 tests fail: "starts no completion wake or run ...", "settles a deferred completion wake without a run ... (true)", "settles the decline continuation with no run ...", and both "settles an already-reported chat wake without a run" cases. - Negative controls stay green: a linking reply without a done read, a decline the earlier turn never read, a non-tool-action card, and an accepted card all still run their continuation. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`: 34 pass. Red on the old oracle: "accepts the earlier reply the server took as the completion report" fails. New negative cases reject no done read, a read of another task, a reply before the read, a reply from another run, two replies, an extra completion run, and a missing task link. - A narrow paid eval run of `duplicate-answer`, `studio-test` and `decline-tool` on this branch follows in a comment. ## Risks - A completion turn admitted before the earlier reply landed (a narrow race) still ends as a short succeeded run with no model call, as in #21401. - If an earlier reply read the outcome but did not mention it, the person does not get a second message. The read receipt, the reply-after-read order and, for completions, the task link make this unlikely. - The decline rule covers only declined tool-action confirmations. Other interaction outcomes are unchanged. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
95a6ab31e6 |
feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.
## Linked Issues or Issue Description
No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).
**Current behavior**
A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.
**Proposed behavior**
An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.
**Reason and benefit**
Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.
## What Changed
- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
- `signal` in `signalDatabaseWork`;
- `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
- `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
- each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
- `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
- the process instance is disabled without the env;
- allowlist filtering, and spec parsing including colon topics;
- per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
- real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
- suppressed records never capture a stack;
- buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
- no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.
## Verification
- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:
```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```
## Risks
- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #21621.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
68d5c9c120 |
feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner can already keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started (#21423, #21316). Core can already wait for that turn (#21453, #21454, #21455). No run does this yet, because no controller asks the Runner for it. > - The request goes through the transport of the Runner package. This pull request gives the transport an option for it and the behavior that the option needs. > - No caller sets the option in this pull request. It changes no production run. > - The benefit is a small base for the two pull requests after it: the recovery after a controller restart, and the switch that turns the behavior on. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. **This pull request:** the transport can ask the Runner for provider-started turns. No caller asks. 2. #21612: the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It adds the support that parts 2 and 3 use. **Subsystem affected** Paperclip Runner package, the controller side: `src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a caller that sets the new option. **Current behavior** - The transport has no way to ask the Runner for provider-started turns. **Proposed behavior** - The transport options have a new field, `providerStartedTurns`. The transport uses it only for ACPX with the agent `claude`. In each other case the field has no effect. - With the option, the transport sends `providerStartedTurns: true` in `turn.start`. It does not send it for a compaction turn. - With the option, a tool call in a provider-started turn waits until the transport gave each earlier event to the driver. Without this, a `paperclip_finish` of that turn could reach the driver before the `turn.started` of that turn, and the driver would refuse it. - With the option, the transport gives a `run.terminal` of the Runner to the driver as a marked internal notification (`paperclip/runTerminal`). The driver uses it only during a wait, so a wait ends at once when the Runner says that the run is over. - With no option, the transport does the same as before this pull request. Live sessions, evals, the CLI runner and the probes do not set the option. **Reason and benefit** - The opt-in is a choice of the caller, not a property of the agent. Only a consumer that can wait for a provider-started turn must ask for one. `CapabilityLiveSession` completes a turn at the first `turn/completed`, so it must not get the behavior because its agent is Claude. ## What Changed - `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the option, the field in `turn.start`, the order of the tool call, and the `paperclip/runTerminal` notification. - `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the source sequence of a tool call that belongs to a provider-started turn. - `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7 cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the real consumer of Core. - `src/live/live-session.test.ts` (12 lines added): a Claude live session gives the transport no such option. - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only): the mode `provider-result-resolved`. In this mode the provider-started turn calls `paperclip_finish` and waits for the answer of Core through the Runner (`tool.resolve`), as the real sidecar does. The mode `provider-result` is not changed. ## State saved by the previous release - No persisted state changes in this part. - **A Runner that does not know the field** ignores it. - **Rollback of the Runner package:** the previous transport does not send the field, so the Runner does not wait. ## Verification At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and 3: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node --test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed in each. The cases: - background work, then the turn of Claude calls `paperclip_finish`; Core answers; the Runner gives the answer to the sidecar with exactly one `tool.resolve`; the turn completes; the run finalizes with one `run.result.accepted`; - a Claude run with no option from the caller takes the path of today: no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`, no wait, and no `session.provider_turn` to the sidecar; - the limit ends the wait, the run takes the path of today, and the provider processes had exited before the cleanup of the test; - the provider process is lost during the wait, and the run settles in seconds; - the sidecar exits during the wait, and the run settles in seconds; - a turn with no background work does not wait; - an ACPX agent that is not Claude is never opted in, also when the caller sets the option. - Each case asserts the result of the close of the session: clean in 6 cases, and a named failure in the case where the sidecar is killed. Each clean case also asserts that there is no `native_session_settlement_incomplete` diagnostic and no "conflicting duplicate tool result" in the durable state. - `src/live/live-session.test.ts`, `src/live/runnerd-codex-transport.test.ts`, `src/live/runnerd-acpx-provider-turn-wait.test.ts`, `src/native-session-runtime.provider-turn.test.ts`: 434 passed. - Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102 passed). - Mutation checks, run on the full change before the split: with the option ignored, 6 of the 7 assertions of the case with no option fail; with the old mode of the fake, the case with the result fails at the close of the session. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No caller sets the option. Core sets it in part 3. - **The transport is shared by each provider.** The three new behaviors are behind one condition: the option, ACPX, and the agent `claude`. With the condition false, the code path is the path of today. - **The tool-call ordering uses a 10 ms poll** while earlier events are not delivered, only in a provider-started turn. - **A controller restart during a wait or during a provider-started turn is not handled in this part.** Part 2 handles it. Part 3 must not merge before part 2. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d25667f370 |
perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly
## Linked Issues or Issue Description
**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.
The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.
**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.
**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.
**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.
## What Changed
- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:
```diff
remove: (x) => {
- const index = xs.indexOf(x)
- return index === -1
+ const at = xs.indexOf(x, index)
+ return at === -1
? null
- : (xs.splice(index, 1), x)
+ : (xs.splice(at, 1), x)
},
...
if (index === xs.length) {
index = 0
xs = []
+ } else if (index > 1024 && index * 2 > xs.length) {
+ xs = xs.slice(index)
+ index = 0
} else {
xs[index - 1] = undefined
}
```
The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:
| Measurement | Before | After |
| --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |
## Risks
- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
f452b449fd |
fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.
**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.
**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.
`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.
**Paperclip version or commit**
`master` at
|
||
|
|
b1f608296e |
feat(runner): keep a run open for a limited time while the Runner waits for the provider's own turn (#21454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - `consumeTurn` reads the events of one turn and returns at the terminal. It already continues past a terminal for a Codex goal. > - This pull request makes it continue past a terminal that waits for a turn of the provider, and it limits that wait. > - The benefit is that a run continues in the turn that Claude starts, and a wait always ends. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 2. Its base branch is `feat/core-provider-turn-driver`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the consumer of a native session does after a turn that ended with no result while the Runner waits. **Subsystem affected** Paperclip Runner package, the controller side: `src/native-session-runtime.ts` and a new module `src/native-provider-turn-wait.ts`. **Current behavior** - `consumeTurn` returns at the first terminal. With no result, the run is finalized as `native_finalization_missing`. **Proposed behavior** - At a terminal with `awaitingProviderTurn` and no result, `consumeTurn` continues to read. It consumes the provider-started turn as the next turn of the same run. A turn that reports a result finalizes as a normal run. - The wait has a limit of 10 minutes from the first terminal that waits. A later terminal that waits does not start the clock again. The limit is never later than 5 seconds before the timeout of the run. - The limit and an early end stop the wait only. They never interrupt a provider-started turn that is running; that turn stays under the timeout of the run. If that turn ends and the Runner waits again, the run takes the path of today at once. - When the wait ends with no turn active, the consumer returns the last terminal, and the existing code for a run with no result decides (`resolveMissingResult`, then `native_finalization_missing`). No new failure code. - The wait ends at once when the driver gives the acknowledgement of part 1. This covers Stop, the limit, and a Runner that says the wait is over. - `executeNativeSession` gets one new option, `providerTurnWait`: the start time of a wait that an earlier controller process saw, and a hook that ends the wait early. - When the wait ends a turn, the consumer first waits for the receipts of the text deltas that are committed in batches (#21460), as each other return path of `consumeTurn` does. Without this, a delta that arrived during the wait was not in the event count of the turn, and a delta that failed to commit was not reported. - When the interrupt that ends the wait is not available or fails, the limit forces the end of the wait. If that forced end starts a new turn (the continuation that a restart owes), the consumer continues to read that turn. The forced-end signal is used one time only, and the read that is open at that moment is kept until it gets its event. Before this fix, the consumer returned the terminal of the old turn and lost the first event of the new turn, and the run could end as `native_finalization_missing`. - If the controller restarts during the wait, the run continues to wait. When the saved checkpoint still shows the first turn as active, the recovered session gives the waiting terminal again, and the wait starts again with the start time that was recorded. When the checkpoint was saved after the waiting terminal, the consumer finds that terminal in the journal, starts the wait again, and does not send the disposition-only recovery prompt. The consumer does this only if two conditions are true: the recovered session has the waiting state (`awaitingProviderTurn`, saved by the driver since #21453), and the terminal in the journal has the marker. If one condition is false, the run uses the recovery path of today. When a wait that started again ends with no turn, the existing code for a run with no result decides, as for each other wait. **Reason and benefit** - The 10 minutes are the maximum time of a foreground command in the pinned Claude Code. - A limit on the wait, not on the work, means that a long test run in the turn of Claude is not cut. ## What Changed - `src/native-provider-turn-wait.ts` (new, 129 lines) and `src/native-session-runtime.ts` (58 lines added, 4 removed), `src/index.ts` (1 export). - `src/native-session-runtime.provider-turn.test.ts`: written again for the new behavior (557 lines added, 61 removed). ## State saved by the previous release - This part adds no persisted state. The wait is in memory. Part 3 reads its start time from an event row that is already saved. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - `src/native-session-runtime.provider-turn.test.ts`: 30 tests. They cover: a controller restart with the first turn still active in the checkpoint (the wait continues; a provider-started turn settles its result; with no turn the limit ends the wait on the recorded clock); a controller restart with the checkpoint saved after the waiting terminal (the same two results, no recovery prompt, no event of the journal appended again); a limit that passed before the restart ends the wait at once, with and without an interrupt; the recovery prompt of today when the terminal in the journal has no marker, when the journal has no terminal, and when the recovered session has no waiting state; a second restart during the wait; a forced end that starts a new turn settles the result of the new turn, with each event of that turn appended one time and in order; an event that reaches the open read after the forced end is appended exactly one time; a delta that arrives during the wait is counted when the limit ends the turn (the test fails when the new wait for receipts is removed); continue and settle the result of the provider-started turn; a second wait does not start the clock again; the limit with no turn active; the limit and a comment during a running turn (not interrupted); a running turn that ends and waits again after the limit; Stop; a `run.terminal` that the Runner held; the Runner stream ends during the wait; the limit under the timeout of the run; a start time from an earlier controller; a plan that a provider-started turn proposes settles as the plan review wait. - `src/native-session-runtime.test.ts`, not changed by this PR: 297 passed. - Mutation checks, each made tests fail: no continue at a waiting terminal; no limit; no early end; no clock from an earlier controller; the clock starts again at each wait; a running turn is interrupted at the limit. - Mutation checks for the forced end, each made the two new tests fail: the forced-end signal used on each loop (the run ends as `native_finalization_missing`); a new read started after the forced end (the first event of the new turn is lost). - Mutation checks for the restart, each made named tests fail: the recovery prompt not held back; the wait not started again in `consumeTurn`; the wait started again on the session state alone (the test with no marker in the journal fails). Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No terminal has the marker before part 4. - **`consumeTurn` runs for each native turn of each agent.** It makes one small object for each turn and reads one more payload field at a terminal. - **If the interrupt that ends the wait is not available or fails,** the limit forces the end: the consumer returns the last terminal, or continues to read when the forced end starts a new turn. This is a fallback path. On a forced return, one read of the event stream stays open until the session closes, as before this PR. - **A wait that starts again after a restart holds back the recovery prompt.** This is the one change to the recovery flow. It needs the two conditions above, and both exist only for a run that asked for provider-started turns. - **Known limit.** The driver does not clear its waiting state when the consumer ends a wait with an interrupt. A restart in the short time after that can start the wait again; the recorded clock and the comment check then end it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3ae7aba643 |
perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server uses `withDedicatedDbConnection` (packages/db) when an action must hold one database session outside the main pool: advisory transaction locks, lock keepalive, and `pg_settings` reads for workspace lifecycle, finalization, source locks, skill caches and status cards > - Each call built a new postgres.js client, opened a new physical connection (TCP, TLS and startup through the pooler), built a new drizzle client with the full schema, and then ended all of it > - On staging the dedicated application name shows about 21 connection setups per second, and the main thread is pinned (event-loop utilization 0.965) > - This pull request keeps a small set of idle single-connection clients per owner and reuses them > - The benefit is that a dedicated action costs about 0.2 ms of main-thread time instead of 4 to 8 ms, with the same exclusive-session guarantee ## Linked Issues or Issue Description **What existing behavior does this improve?** `withDedicatedDbConnection` opens and closes one new database client and one new physical connection for every call. **Subsystem affected** Database client (`packages/db`) and the server services that take dedicated advisory-lock sessions. **Current behavior** Each call builds a new postgres.js client, opens a new connection (TCP, TLS and startup through the pooler), and builds a new drizzle client with the full schema. On staging (Neon statement statistics, 326 s window) `SET application_name='paperclip-workspace-finalization-lock'` ran 21.4 times per second. Only this factory uses that application name. In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2% inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that is this per-call setup. **Proposed behavior** A caller still gets exclusive use of one session for the whole action. The session and its drizzle client are reused by later actions, so the per-call cost is the action's own queries, not a full connection setup. **Reason and benefit** The server main thread is saturated on staging (event-loop utilization 0.965). Reuse removes about 5 ms of main-thread work per call. **Breaking changes** None. The exported function keeps its signature. `closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are new exports. ## What Changed - `packages/db/src/client.ts`: each `createDb` owner now has one set of dedicated single-connection clients, shared with every private client derived from that owner (so nested dedicated actions use the same set). - An action takes an idle client (last in, first out) or opens a new one. It never waits. Callers hold advisory locks and nest dedicated actions, so a waiting queue could deadlock. - On success the client returns to the idle set if fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed. - When the action throws, the client is closed and never reused. This covers a failed or cancelled transaction, a broken socket, and an interrupted lock wait. - When an action ends the client itself (the source-lock cancel path calls `$client.end`), the client is never reused. - Idle clients keep the owner's idle timeout and max lifetime, so postgres.js still closes idle physical connections after `DATABASE_IDLE_TIMEOUT_SECONDS`. - New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle set and stops retention. Ending the owner `$client` does the same. A later action still works with a connection that closes on release. - `server/src/index.ts`: shutdown closes the dedicated set next to the other private pools. - `doc/DATABASE.md`: documents the idle cap, the "not an admission limit" rule, and the transaction-scoped session state contract. - Session state audit: every current caller uses `pg_advisory_xact_lock` / `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or temporary tables. A reset round trip per call is therefore not added. ## Verification - New `packages/db/src/dedicated-connection-pool.test.ts` (embedded PostgreSQL, 10 tests). - Red on `origin/master`: 200 sequential calls used 200 distinct sessions (`expected 200 to be 1`). All 10 tests failed. - Green with this change, 3 runs in a row: 200 sequential calls use 1 session; 7 concurrent callers each hold a different session for the whole transaction, and afterwards at most 4 stay open and later calls reuse them; a failed transaction's connection is closed; a terminated socket during the action is not reused; an idle connection terminated by the server reconnects; a client the action ended is not reused; transaction-local settings do not leak; shutdown and owner `$client.end` close every retained connection; nested actions do not block. - Focused suites pass: packages/db `work-signals`, `prepared-queries-postgres`, `response-capture-postgres`, `checkpoint-pool`, `client`, `client-options`, `client-teardown-registry`, `postgres-connection-recovery`, `client-disconnect-replay` (132 tests). Server suites that use dedicated connections: `skill-package-store`, `status-card-evaluation-work` (updated: it now proves advisory locks are released, since sessions are reused), `agent-instruction-working-copies`, `workspace-recovery-retry-copyback`, `native-execution-lease-batch`, `company-skills-empty-runtime`, `decisions-service`, `legacy-controller-lease`, `server-startup-feedback-export` (467 tests). - `pnpm verify --explain` selects 2779 files for a `packages/db/src/client.ts` change. Per AGENTS.md I ran the focused checks above and let CI give the broad coverage. - Benchmark: 200 calls of `withDedicatedDbConnection(db, d => d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))` against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating before/after process pairs per mode. Main-thread time is `eventLoopUtilization().active`. | Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call (median, range) | | --- | --- | --- | --- | | TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) | | TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) | | Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) | | Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) | At the staging rate of about 21 calls per second, this removes roughly 100 ms of main-thread work per second (TLS figures; staging CPUs may be slower). ## Risks - A retained connection that the server or network drops while idle: postgres.js reconnects on the next query when it has seen the close (tested). If the drop and the reuse race, the first query of that action fails like a main-pool query would. Before this change, a dedicated action always started on a fresh connection. - Through a transaction-mode pooler, `SET application_name=...` is a server-side parameter switch, so its count may not drop as much as the client-side connection count. Measure the deployed effect with pooler client connections and main-thread CPU, not only that statement. - Concurrency above 4 opens extra connections exactly as before; they close on release. Total retained connections per owner grow by at most 4 idle clients, which still close after the idle timeout. - An action that sets session-level state would now leak it to the next action. The contract is documented on `withDedicatedDbConnection` and in `doc/DATABASE.md`; no current caller does this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
82ce8e64ba |
feat(runner): let the session driver admit a turn that the provider started (#21453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Core reads Runner events through the Codex app-server driver, also for ACPX agents. The driver refuses a turn that the controller did not start, except a goal turn of Codex. > - This pull request lets the driver accept a turn that the provider started, only directly after a turn that the Runner marked as waiting. > - The benefit is that the next parts can use that turn. Nothing reaches this code until the last part sends the opt-in. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 1. Its base branch is `feat/adopt-claude-provider-started-turn`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the driver does with a turn that the provider started. **Subsystem affected** Paperclip Runner package, the controller side: `src/drivers/codex` (the driver that Core uses to read a native session). **Current behavior** - A `turn/started` that the controller did not ask for fails the session with `turn_binding_mismatch`, except for an active Codex goal. - An interrupt with no active turn fails with "already terminal". **Proposed behavior** - A turn terminal that the Runner marked with `awaitingProviderTurn` and that has no result puts the driver in a waiting state. The driver event `turn.completed` carries the marker. - In that state, a `turn/started` with `origin: "provider"` is admitted the same way as a goal turn, and the driver event carries `origin: "provider"`. In each other state it still fails closed. - In that state with no active turn, an interrupt gives an acknowledgement item with the status `provider_turn_wait_ended`. It sends nothing to the Runner. The driver also gives that acknowledgement when it gets the internal `paperclip/runTerminal` notification with a private marker; a notification without the marker does nothing. - An interrupt for a provider-started turn that is running is sent to the Runner only when it names that turn. - The driver saves the waiting state in the persisted session (`awaitingProviderTurn`, written only during a wait). After a controller restart, the restored session still admits the provider-started turn and still acknowledges an interrupt during the wait. The driver restores the state only when the session has no active turn, no stored result and at least one finished turn, so a new run cannot take the wait of the run before it. - When a restart finds that the saved active turn is finished, the turn read of the transport and `reconcile()` of the driver carry the same marker as the live `turn/completed` notification. `runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from the saved `turn.completed` event to the turn object of a read. - The driver ends the waiting state when the Runner reports the run terminal. **Reason and benefit** The consumer in the next part needs these three signals: a marked terminal, an admitted turn, and a clear end of the wait. ## What Changed - `codex-session-terminal.ts`, `codex-session-notifications.ts`, `codex-session-state.ts`, `codex-harness-session.ts`, `codex-driver-values.ts`: 37 lines added, 4 removed. - `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines). - `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`, `backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`: the optional field in the two persisted shapes and its copy in each direction. - `live/runnerd-codex-transport.ts`: the marker on a recovered turn terminal. The loop that builds the earlier terminals of a `thread/read` is now the function `runnerdPriorTurnTerminals`, with the same logic, so that a test can call it. ## State saved by the previous release - One new optional field in the persisted driver session: `awaitingProviderTurn`. A snapshot outside a wait has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so the driver does not wait. This is the behavior of today. - **State from this release, read by the previous release:** the previous reader copies the fields that it knows and ignores this one. No reader rejects it, and the integrity digest covers the stored object as written. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - 6 driver tests: a provider-started turn after a marked terminal is admitted and can report a result; a provider-started turn with no marked terminal fails closed; an interrupt during the wait is acknowledged; a running provider-started turn is not interrupted by the end of the wait; the marked notification ends the wait and a notification with no marker does not; a normal terminal is not changed. - Restart tests: a session restored from a snapshot with the waiting state admits the provider-started turn and settles its result; a snapshot outside a wait has no new field; a restored session with no waiting state refuses a provider-started turn as today; an interrupt during a restored wait is acknowledged. A saved active turn that the read reports finished with the marker gives a `turn.completed` with `awaitingProviderTurn: true`; with no marker it is an ordinary terminal. - `harness-driver-backend.test.ts`: the field goes through a serialized checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324 passed): the marker is on a recovered terminal only for a `turn.completed` with a boolean `true`. - `src/native-session-runtime.test.ts`: 297 passed, with the real `paperclip-runnerd`. - Mutation checks for the restart work, each made a named test fail: no restore of the state; no copy in the backend; no copy in the transport; no marker argument in `reconcile()`; no end of the wait at the run terminal. - Mutation checks: the acknowledgement branch removed, the admission removed, and the marker check removed each made a test fail. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** The Runner sends the marker only after the opt-in of part 4. - **The admission rule is shared with Codex goal turns.** The existing goal tests pass with no change. - **Known limit after a controller restart.** If the Runner already runs a provider-started turn and the saved active turn is the turn before it, recovery fails closed, as today for each unexpected active turn. - **A controller turn that starts during a wait** clears the waiting state when its `turn/started` arrives, not when it is sent. A snapshot in that short time can hold an active turn and the field together; the restore rule drops the field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
9824277f65 |
perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Most runner events that are not text deltas (`tool.execution.started/progressed/completed`, `item.completed`) wake the chat progress projection > - That wake is registered inside the append's transaction, so each such event takes 4 statements (BEGIN, the run lock read, the append, COMMIT), where an event without a wake takes 1 (#21511) > - The chat progress projection rebuilds itself from the stored events; the wake only tells it when to look > - This pull request writes an event whose only wake is chat progress with the one-statement append, and sends the wake after that statement commits > - The benefit is 1 statement instead of 4 for most non-delta runner events, with response, reassessment, retention and recovery wakes unchanged ## Linked Issues or Issue Description Refs #21511 (one-statement append). Refs #21528 (lock read returns the transaction id). Refs #21549 (load model at recorded rates). **What existing behavior does this improve?** `appendHeartbeatRunEvent` for native source events whose only wake is chat progress. **Subsystem affected** Server run event ingest (`heartbeat-run-events.ts`), the chat progress notification helper, and two additive exports in `packages/db/src/work-signals.ts`. **Current behavior** A progress event opens a transaction, reads the run lock with `runtime_mode`, the bound task and the transaction id, probes for a replay, registers the chat-progress wake as a tracked intent, appends, and commits. **Proposed behavior** When chat progress is the event's only wake, the append takes the one-statement path of #21511. That statement also returns the run's `runtime_mode` and bound task id from the row it locks, and, where commit notifications are on, calls `pg_notify` inside the same statement for runs that are native and bound to a task, so other server processes get the wake at commit. After the statement returns, this process publishes the wake as settled to its own listeners. A replay, a gap, a changed run or any other fallback takes the existing transaction, which registers the wake as before. **Reason and benefit** At recorded rates these events are about 61% of the non-delta events (#21549), so this cuts about 3 statements per such event. **Breaking changes** None. No schema change. ## What Changed - `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other wake clear) also qualifies for the one-statement path. - `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`, rendered once) from the locked row, plus a conditional `pg_notify`. - `chat-native-progress-notifications.ts`: `chatNativeProgressCommitNotification` and `publishChatNativeProgressSettled`. - `work-signals.ts` (additive): `databaseWorkCommitNotification(owner, topic)` returns the cross-process payload when notifications are on; `publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled wake to this process's listeners. ## Verification - Red/green on embedded PostgreSQL. Each new test failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "appends an event whose only wake is chat progress in one statement and wakes progress after it commits": 1 statement (before: a transaction), the wake is published once as settled with no intent, and a replay publishes nothing. - "keeps the transaction for the last events of a turn and a run, and one statement for progress-only events": `tool.execution.started/progressed/completed` and a non-final `item.completed` take one statement; a final `item.completed`, `run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled` and `turn.interrupted` keep the transaction. - "sends the progress wake to other server processes with the one statement's commit": with commit notifications on, another connection that listens on the work channel receives exactly one chat-progress notification, and a replay sends none. - `chat-native-progress-work.test.ts` "wakes the projection from a one-statement progress event after the statement commits": the real chat progress worker publishes the progress for an event appended in one statement. - Mutation check: without the settled publish after the statement, that test fails. - Lost-wake repair (condition b): an event row stored without any wake models a lost hint. - "repairs a lost progress wake with the run's next progress event": nothing is published for the stored event; the run's next progress event publishes the progress. - "repairs a lost progress wake with the startup reconcile": a worker that starts after the stored event publishes it. - The existing response-wake test now uses a final reply and still shows the intent before the write, and the race test of #21528 uses a final reply so it still covers the transaction path. - Other focused suites, one file at a time: the port suite with conformance (105), `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25), atomic-append (31), `chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - CPU per operation (a local script, not committed: 800 `tool.execution.progressed` appends to one native task run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms after in the warm rounds; wall time per event 0.84–1.04 ms before and 0.52–0.53 ms after. - Inside a caller's transaction: new test "keeps a progress-only event inside a caller's transaction on the locked path, and settles its wake only when that transaction ends". `publishDatabaseWorkSettled` publishes at once only outside a transaction; inside one it waits for the outermost commit and drops the wake on rollback (new `work-signals.test.ts` test, which failed with the unguarded publish). The one-statement path is only taken when the owner is not a transaction (`!is(db, PgTransaction)`, #21511), so the two guards agree. ## Risks - **Lost wake.** If the statement's outcome is unknown (the connection drops after the server ran it), this process sends no wake for that event. In a chat channel (Slack, Telegram) the progress line for that task can then stay one step behind (for example it does not move from "working" to the latest tool step) until the run's next runner event, a run milestone, or a server restart. A turn normally ends with an event that keeps the transaction (`turn.completed/failed/cancelled/interrupted`), and the run's completion notifies chat run milestones, which include chat progress (`notifyChatRunMilestones` in `run-completion.ts` and `native-run-finalizer.ts`). So progress cannot stay stale after a run ends; in the worst case it lags one step for the rest of a turn. The projection rebuilds from stored events, so nothing is lost or duplicated. - The chat-progress wake no longer publishes an intent before the write. Intents let the coordinator fence a pass against an in-flight write; the settled wake after the commit makes the projection run again after the row is visible. - `packages/db/src/work-signals.ts` gets two additive exports and no change to existing behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f7b1651799 |
feat(acpx-sidecar): adopt Claude's own cycle as a provider-started turn when the Runner arms it (#21316)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The parts before this one give the adapter an ordered stream of those cycles (#21421), declare the contract (#21422), and make `paperclip-runnerd` wait for such a turn (#21423). > - This pull request is the last part in the Runner. The sidecar reads the stream, and gives a cycle to `paperclip-runnerd` as a provider-started turn only when `paperclip-runnerd` armed it. > - The benefit is that a later change in Core can let a run continue when background work finishes. Until Core sends the opt-in, a production run does the same as today. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request was first opened with the full change (49 files). It is now split. The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. This pull request: the sidecar adopts a cycle. Its base branch is `feat/runnerd-await-provider-turn` (#21423). The code at the head of this branch is the same as before the split; only the base changed. Review comments made before the split are about code that is now in the parts above. Refs #19912 (design). **What existing behavior does this improve?** What the ACPX sidecar does with a cycle that Claude starts by itself. **Subsystem affected** Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts`. Claude only. **Current behavior** - The sidecar does not read the cycle stream of the adapter (#21421), so each cycle is declined and the background state becomes unknown. - The sidecar does not know the command `session.provider_turn`, so `paperclip-runnerd` (#21423) never waits. **Proposed behavior** - The command `session.provider_turn` with `armed: true` arms the sidecar for one turn. The sidecar accepts it only for the terminal that reported a running background shell, with no active turn and a provider that is alive. One flag holds the arming. It is cleared at each turn start and turn end, on suspend and close, and when the provider is lost. - One loop, `consumeProviderCycles`, reads the stream. For each cycle: - if the sidecar is armed and no turn is active, the sidecar adopts the cycle: it emits `runtime.turn_started` with `{ "origin": "provider" }`, then the events, the usage and the terminal of the cycle, as for a normal turn; - if the cycle cannot be decided yet (the prompt turn is still settling, or `paperclip-runnerd` did not arm yet), the sidecar holds it for a maximum of 5 seconds, and then adopts or drops it; - in each other case the sidecar drops the cycle. The stream ends, and the background state becomes unknown, so `paperclip-runnerd` restarts the provider at the end of the turn as today. - A Paperclip tool call or a permission request of a held cycle waits for the decision of its own cycle. It is never bound to the prompt turn or to a different cycle. - If an adopted cycle proposes a plan (`ExitPlanMode`), the sidecar publishes the same `plan.proposed` event as for a controller turn, and the turn ends with the same terminal: completed, with no `backgroundTasks`, so `paperclip-runnerd` does not wait again. A controller turn is not changed. - If the provider exits while the sidecar is armed with no turn active, the sidecar sends a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`, and `paperclip-runnerd` ends the wait. - `turn.cancel` for a provider-started turn cancels the cycle and closes the provider. - With no arming, nothing is shown. No controller sends the opt-in yet, so `paperclip-runnerd` never arms. **Reason and benefit** - Only `paperclip-runnerd` decides that a cycle becomes a turn. The sidecar cannot start a turn by itself. - One reader of one ordered stream means that two cycles cannot overlap. ## What Changed - `src/cli/acpx-runtime-sidecar.ts` (127 lines added): the command, the arming flag, `consumeProviderCycles`, `adoptProviderCycle`, `pumpProviderTurn`, and the wait of `waitForTool` for a cycle that is not decided. - `src/cli/acpx-runtime-sidecar.provider-turn.test.ts` (new, 320 lines) and two fake hosts in `src/test-support` (227 lines). One host follows a script. The second runs the real cycle stream and the real background task tracker. - `src/cli/acpx-runtime-sidecar.test.ts` (2 lines): one more parameter for a test harness. No schema, no Rust and no persisted state changes in this part. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `pnpm run typecheck:typescript`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript`: no errors. The server check `node scripts/verify-runner-vendor-dependencies.mjs`, the cache guard and the contract tests: passed. - `npx vitest run src/cli src/drivers/acpx src/live/runnerd-acpx-provider-turn-wait.test.ts src/drivers/runner-tool-bridge.test.ts src/native-session-runtime.provider-turn.test.ts`: `128 passed (128)` files, `2147 passed`, 9 skipped. - `cargo test -p paperclip-runner-core --lib` and the `acpx_*` suites: all passed. No Rust changed in this part. - The sidecar tests of this part: a cycle that starts before the arming command, or while the prompt turn settles, is held and then adopted with its tool call; a cycle is never shown when `paperclip-runnerd` did not arm, disarmed, armed for a different terminal, armed for a quiet terminal, or armed after the provider exited; two cycles in sequence with `{ live: 2 }`, then `{ live: 1 }`, then an unknown state; cancel; a lost provider; Codex gets no change. - Mutation checks, each made the named tests fail: the sidecar arms with no matching terminal; no hold; the old hold condition that drops a cycle while the prompt turn settles; no wait of a tool call for the decision. - Live run with a real Claude account, not in CI, with the real ACP runtime and the real Anthropic API (model `claude-sonnet-5-5`), at the adapter level: one background shell gave `{ live: 1 }`, one cycle with its own usage, then an unknown state. Two shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown. All Paperclip tool calls were admitted, and none was refused. Not run: a live run through the real sidecar process, `paperclip-runnerd` and Core (that path needs an API key, not a local sign-in); Linux; `pnpm verify --full`. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in, so the sidecar is never armed. The new code that runs today is the loop that reads the stream: with no arming it drops each cycle, and the background state becomes unknown as before. - **A follow-up for a command that finishes while Claude reports on an earlier one is lost.** The Runner waits again only while a background command is still running. The run ends as it does today. The reason is in #21421. - **A cycle that starts in the last second of a prompt turn** can have its first messages in the prompt turn and the rest in the provider-started turn. Tool calls stay safe: the admission rule of #21278 refuses a call that the active turn did not announce. - **The 5 second hold** is not tested under load. A cycle that is held longer is dropped, and the run takes the path of today. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
4a78124a3e |
feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn that ended with no result while a background shell is running, this stops the shell and the run fails. > - #21278 binds each tool call to its turn, and #21293 reports how many background shells are running at the end of a turn. Part 4 declared the contract. > - This pull request makes `paperclip-runnerd` wait for a turn that Claude starts by itself, when the controller asks for it, and admit that turn into the same run. > - The benefit is that the Runner has one place that decides to wait and one place that ends a wait. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 5. Its base branch is `feat/prp-provider-turn-contract`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** What `paperclip-runnerd` does after a Claude turn ends with no result while a background shell is running. Nothing changes until a controller sends the opt-in. **Subsystem affected** Paperclip Runner: the ACPX provider backend in `runner-core` (Rust). Claude only. **Current behavior** - After each completed turn the Runner emits `run.terminal`. A turn with a running background shell gets the provider restart, and the shell stops. **Proposed behavior** Nothing changes unless the controller sends `providerStartedTurns: true` in the payload of `turn.start`. No controller sends it yet. With that field: - The Runner keeps the provider and waits when all of these are true: the agent is Claude; the turn completed; the run has no accepted `paperclip_finish` or `paperclip_block` result; the terminal event reported `backgroundTasks` with `live` above zero; each condition of #21293 for a kept process is true; the sidecar accepted `session.provider_turn` with `armed: true`. - `turn.completed` then has `awaitingProviderTurn: true`, and no `run.terminal` follows. The Runner holds that `run.terminal`. - When the sidecar reports a turn that the provider started, the Runner admits it only during a wait and publishes `turn.started` with `origin: "provider"`. In each other case that report fails the session closed. - When a wait ends with no turn active, the Runner publishes the held `run.terminal`. This applies when the provider process exits, the sidecar exits, the controller cancels or stops the run, the session is closed or suspended, and after a Runner restart. - The held `run.terminal` and the removal of the wait are saved in one write, and the controller gets the event from the next poll. A crash after a cancel or a stop cannot lose it, and it is never sent two times. - A controller `turn.start` during a wait replaces the provider first, as before, and the held event is dropped. - A `run.attach` from a different run during a wait ends the wait when the attach succeeds. The Runner then discards the held `run.terminal`, because that event is of the old run and must not end the new run. This is the same rule that a Runner restart applies. The wait is removed only in the state that the attach saves for the new run. If the attach fails, for example when the old session does not stop, the old run keeps the wait and gets its held `run.terminal` one time, from the next poll or from a cancel or a stop. A `run.attach` from the same run publishes the held event to that run. - The Runner waits at most 5 seconds for the sidecar to answer a request to arm or to disarm a provider-started turn. This is the limit that turn cancellation already uses. Before, the limit was the 30 seconds of an ordinary request, and a cancel, a stop or a shutdown had to wait behind it. No answer is handled as a lost sidecar: the session fails closed, and the Runner still publishes the held `run.terminal` one time. - The sidecar of this branch does not know `session.provider_turn`, so the arming request fails and the Runner does not wait. The behavior comes with part 6. **Reason and benefit** - The wait is one entry in the saved session state. One rule ends it at each save, so each new way to leave the open session ends the wait with no new code. - A wait must not outlive the provider or the sidecar, and a cancel must stop it. ## What Changed - `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests): `provider_turn_wait { turn_id, held_run_terminal }` in the saved state; the rule that ends a wait at save; `settle_turn_events`, which decides at one place to arm, to create the wait and to put `awaitingProviderTurn` on `turn.completed`. - `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its terminal, the arming request, and `admit_provider_turn`. - `acpx_provider_session.rs`, `acpx_provider_backend.rs`, `acpx_provider_state.rs`, `provider_events.rs`, `acpx_park_preparation.rs`: the opt-in from `turn.start`; the record that the run has a result; the provider-started turn in the state and in the PRP events; cancel and stop with no turn; no turn-end restart and no park during a wait. - `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no event also checks that the sidecar process is alive. A child process can keep the output stream of a dead sidecar open. - `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and failures for tests. - Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`, `tests/support/acpx_quiet_turn_restart.rs`, `src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon), `src/native-session-runtime.provider-turn.test.ts` (the controller of today). - `runner/DURABLE_TRANSPORT.md`: the rules. ## State saved by the previous release - One new optional field in the saved ACPX session state: `providerTurnWait`. It is written only during a wait. In each other state the saved file has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so there is no wait. A unit test covers this. - **State from this release, read by the previous release (a rollback):** outside a wait, the file loads as before. During a wait, the previous Runner refuses the file with its "ACPX provider state is malformed" error, because its loader refuses each unknown field. Only a run that waits at the moment of a rollback is in this case, and no run waits before a controller sends the opt-in. - **Runner restart during a wait:** the wait is restored. The provider is a new process, so the wait is over, and the controller receives the held `run.terminal` one time. A test with the real daemon covers this. - **Controller of today or of the previous release:** `turn.start` has no opt-in. A test runs a turn with a running shell and a turn with an unknown background state, and compares the events, the commands after the turn, the restart, and the park state before and after the restart. They are the same. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored. All 17 `acpx_*` suites passed. - `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends with exactly one `run.terminal` when the sidecar exits while a child process keeps its output open, when the provider is lost, when the controller cancels, and after a Runner restart; with no opt-in, `run.terminal` follows the turn and the sidecar is never armed. - Rust tests: wait and admit; no wait with a result; no wait with no opt-in; an active goal neither marks nor arms a wait; a sidecar that refuses to arm; a provider-started turn that the Runner did not wait for fails the session closed; cancel and stop during a wait; provider and sidecar loss; restart; the saved shape in both directions. - `src/native-session-runtime.provider-turn.test.ts`: the real `consumeTurn` of today gets the event sequence with no opt-in and finalizes it as `native_finalization_missing`, as today. - Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a different run attaches after the snapshot reports `warmAttachReady: true`; the new run gets no `run.terminal`, the saved state has no wait, and the old session is closed. A same-run attach gives exactly one `run.terminal`. An attach that fails when the old sidecar is dead leaves the wait with the old run, and the old run then gets exactly one `run.terminal` from a poll, `run.cancel`, `turn.interrupt` or `turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in the pending events of the new run; with the wait ended before the attach is complete, the old run gets no `run.terminal` after a failed attach. - No answer from the sidecar, in `tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not answer the disarm request; `run.cancel` and `turn.interrupt` each return `already_settled` in less than 10 seconds, exactly one `run.terminal` reaches the run, `session.close` completes in less than 2 seconds with no second `run.terminal`, and the provider process is gone. With no answer to the arm request, the turn ends with its `run.terminal` and with no wait. With the limit removed, each test fails at 30 seconds. - Mutation checks, each made the named tests fail: the rule that ends a wait at save; no `run.terminal` after a restart; arming with no opt-in; the marker set with no wait; the sidecar process check removed. Not run: a rollback with the binary of the previous release; a run with the opt-in and a sidecar that answers the arming request with an error (the code reads each error as not armed); Linux; the full suite. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in. - **A rollback of the Runner during a wait** makes the previous Runner refuse that saved session. See "State saved by the previous release". - **The wait has no time limit in the Runner.** The controller must close, suspend or cancel. Core will own that limit. - **Park is refused during a wait.** The controller must end the wait before it parks. - **A new run can attach during a wait.** The attach is not blocked, because the Runner has no time limit on the wait and a new run can send only `run.attach`. After an attach that succeeds, the old run gets no `run.terminal` from the Runner; the controller must not depend on that event. The request that disarms a sidecar that the attach keeps has no test, because no test can keep a session across an attach during a wait. - **After a sidecar that does not answer, `runner.suspend` returns an error at once** ("provider lifetime cleanup is not yet proven"), and `session.close` completes. This is the behavior of today for each lost sidecar; this PR does not change it. - **The shared code that changed** runs for each Claude turn: the record of a run result, and the settle step at each turn end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
689f3b190e |
fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Automatic recovery continues interrupted and failed tasks without a person, with a per-task limit of 6 automatic attempts per hour > - There is no company-level limit. When the sandbox provider refuses to create sandboxes, every task fails the same way, and recovery keeps creating new runs for each task > - On a hosted instance, one hour of a provider outage made 510 automatic runs, 587 failures and 0 successes across 289 tasks. No task had more than 4 runs, so the per-task limit did not engage. Each start asked a refusing provider for a sandbox and used server memory, which made the outage longer > - Recovery must still continue every task by itself. The answer is to wait while the infrastructure fails, not to give the task to a person > - This pull request adds a company-scoped pause for automatic work: it opens when recent automatic runs fail before the agent starts, the recovery passes stop creating new automatic runs while it is open, and it closes by itself when runs start normally again > - The benefit is that an infrastructure outage no longer multiplies itself through recovery, and recovery resumes with no person acting ## Linked Issues or Issue Description **What happened?** During a sandbox provider outage, automatic recovery created hundreds of new runs per hour across hundreds of tasks. All of them failed before the agent did any work. The per-task limit did not engage because no single task reached it. **Expected behavior** When most recent automatic runs of a company fail before they start, recovery waits instead of creating more runs. It tries again at a small rate. It resumes at full rate when runs start normally again. A person's work is never held. **Steps to reproduce** 1. Make the sandbox provider refuse lease creation (HTTP 503) for a company with many assigned tasks. 2. Let recovery continue the failed tasks. 3. Count automatic runs created per hour: it grows with the number of tasks, and every run fails at sandbox creation. Refs #21394 (continuation classification this change will reuse when it lands), #21433 (start-pass rewrite that the start hook will build on). ## What Changed - New table `automatic_run_brakes` (one row per company, cascade delete with the company; migration `0480`). Migration `0481` builds `heartbeat_runs_company_finished_idx (company_id, finished_at desc) where finished_at is not null` concurrently, so the signal reads the runs that finished in the window directly. It stores the state (closed, open, half_open), the reason with the window numbers, and the compare-and-set fields that two servers share. A company with no row is closed. - New module `server/src/services/recovery/automatic-run-brake.ts`: - Signal: on the database clock, the automatic runs of the company that finished in the last 10 minutes (and since the last state change), newest 500 by finish time, plus up to 200 running runs. A run counts as a failure only when it ended in its own start phase: no provider work (`executionRecovery.providerWorkStarted = false` or the native `nativePreDispatchFailure` record), a legacy run still preparing, or a transient setup code. A run interrupted by a server stop, restart or lease loss is neutral, so a release swap does not open the pause. A run that succeeded, or ended for a person-owned or task reason, counts as healthy. - Opens at 12 or more counted runs with 80% or more failures, or on 5 sandbox refusals (`environment_provider_unavailable`) in a row. - Each outcome counts once, at its own time: a finished run at its finish time, a running run that got past setup at its start time. Read time never orders evidence. - Closes in two steps: open moves to half_open after at least 2 minutes when the newest 2 counted runs are healthy or the newest one succeeded; half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64) and then closes. It reopens from half_open if the signal trips again. It closes by itself after 15 minutes with no automatic activity. - Reads are one indexed query per company, cached for 5 seconds in a bounded in-process map (256 companies). Each state change writes one structured log line (`automatic_run_brake.opened|half_open|closed|reopened`) with the numbers. - It also contains the start gate `holdAutomaticRunStart` (one probe at a time, at most one new probe per 60 s, release budget per window by compare-and-set, no I/O while the cached "held until" time has not passed, person origin taken as an argument with a cached fallback). This pull request does not call it from the start pass; see Risks. - Recovery passes defer creating runs while the pause is open. The gate is only where a run is created: - `deliverReconciledExecutions`: after the agent-room check and right before the wake, the delivery asks for a creation slot; when the pause holds it, the continuation stays pending. The held backlog (`continueHeldExecutions`) is not gated: recording a pending continuation creates no run and costs nothing, and its delivery passes this gate once. - Assigned-work sweep `act()`: after the process rate limit and before it claims a continuation, it asks for a creation slot; when held, it returns `deferred_rate_limit` with the next try time, so nothing is written to the per-task attempt ledger. New assignments with no earlier run are not deferred. - A reserved slot is given back (compare-and-set on its own timestamp) when no run is created: the wake is refused or throws, the task is capped, the claim changed, or the task went back to review. The give-back is conditional in SQL on no run for that task having been created since the reservation, so a wake that commits its run and then throws keeps the slot. - The sweep's overdue saved-work recovery path returns before the slot is asked for: it starts a recovery attempt, not a run, and is never held. - Person work is never deferred. The gate is called only at the delivery (`agentHasRoom`) and the sweep (`act()`), and each call passes the task and the run it continues. While the pause is open, the gate does not hold the record when any of these is true; it holds only when none is. With no task or no source run it does not hold (fails open). | Shape | How it is recognised | | --- | --- | | A Retry or "move forward" waiting for the task (`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an unstarted wake request from a user for the task | | A Retry folded (`coalesced`) into the continuation, or a Retry while the delivery is pending | a wake request from a user for the task, requested after the source run finished (any status) | | A person's message or a new assignment by a person after the last run | the same: a wake request from a user for the task after the source run finished | | A held task selected because a person pressed Retry (the refused Retry run) | a run for the task created after the source run finished whose wake is a person reason (`task_retry`, chat message), or that has `taskRetryRequestId` or `explicitUserContinuation` | | A continuation of a person's Retry (any depth) | the source run's chain reaches a person's run (`listPersonOriginSourceRunIds`, the dispatch rule) | | A Retry during pending delivery that becomes the continuation itself (`deliverContinuationAsRun`) | not gated: that run is created by the person's own wake, never by these passes | | The held backlog's saved-input path | not gated: recording or releasing a held continuation creates no run here | - To prevent a deadlock, a pass may still create one automatic run per minute per company when no automatic run of the company started in the last 2 minutes, however many runs wait (compare-and-set, shared by servers). - `GET /api/companies/:id/assigned-work-invariant` now returns `automaticWork` with the state, the time it paused, the window numbers and a plain sentence, for example "Automatic work is paused for this company since T because 47 of 50 recent automatic runs failed before starting." - `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is now exported, unchanged, so the gate uses the same person-origin rule as dispatch. - The outcome rules (person-owned codes, restart codes, transient setup codes, native pre-dispatch record) come from their owning modules through one memoised dynamic import. A static import of `heartbeat/retries.ts` made a module cycle at server start. ### What the pause consumes, and how waste is bounded | Token | Taken | Given back or bounded | | --- | --- | --- | | Creation slot (one per minute per company while open) | right before the run is created, after every other check | released only when no run for the task was committed after the reservation | | Probe slot (start gate) | when the start gate lets one automatic run through | another run may take it when the reserved run has not started within 10 seconds (claim declined, agent paused or full) or was cancelled, deleted or finished | | Gradual-release start (start gate) | one per run the start gate lets through | a declined claim spends one start of that window; the window ends within 60 seconds and the next one doubles | | Probe success count | once per probe run, by compare-and-set on its id | a probe run counts once; cancelled or deleted probes count as neither success nor failure | ### When the pause opens, stays open and closes | State | Leaves when | Bound | | --- | --- | --- | | closed | opens when, in the last 10 minutes since the last change, 12 or more counted automatic runs finished and 80% or more failed before starting, or the newest 5 counted runs were sandbox refusals | none needed | | open | moves to half_open when it has been open 2 minutes or more and the newest 2 counted runs are healthy, or the newest one succeeded | evidence arrives at least once a minute: when no automatic run started in the last 2 minutes, a pass may create one per minute | | open | closes when no automatic run started or finished for 15 minutes (queued runs do not count) | 15 minutes after the last activity | | open | stays open only while new automatic runs keep failing before starting | each new try is spaced by the existing passes and by one creation per minute when nothing starts | | half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6 minutes | | half_open | reopens when the signal trips again, counted only since half_open began | none needed | Restart-class endings (server stop, restart, lease loss, process loss, interrupted native session) are neutral in every state, so a release swap does not open the pause. ## Verification - `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests, pass): classification; opens on 20 failures before start; 50 queued automatic runs give exactly one probe and no failure or attempt; a person's Retry, a new assignment, a run a person joined and the continuation of a person's Retry are not held; mixed and person-owned outcomes stay closed; old data is ignored; 40 working runs interrupted by a restart stay closed, then the same continuations failing at sandbox acquisition open it; 5 refusals in a row open it; failed probe waits for the interval, two probes past setup move to half_open, the release gives exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen from half_open; probe slot frees after cancel, delete or no start; quiet close after 15 minutes; two instances on one database agree and start one probe; one creation per minute while open across servers; 250 held runs over 100 passes use at most 5 queries (at most 255 with the person-origin fallback, one per run, once); caches stay at 256 companies and 1024 runs. - `npx vitest run src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one test per person shape above at the delivery and the sweep, with the pause open and no creation slot, plus a control that still holds a continuation no person asked for): includes "no pause record: every pass behaves as before", "the waiting run cannot start: a pass continues a task on another agent, then the pause closes", "one held task spends one creation slot", and "a person's Retry or message is never paused at the delivery, the sweep or the held backlog".: sweep, delivery and held backlog create nothing while open, write no attempts, and continue by themselves after recovery; the sweep still continues one task a minute when nothing automatic waits. With the three deferral lines removed, all 4 fail (red), and they pass with them (green). - The pause reads as closed and logs once when its run rules cannot load (test in `automatic-run-brake.test.ts`). - `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the module loads after the heartbeat service, in the server's import order. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify`: see the comment below for the result. CI supplies the full suite. ## Risks - This pull request does not hold starts. Already queued runs and `transient_failure_retry` runs still start while the pause is open; they act as the natural tries. The start gate goes in as a follow-up on top of #21433, at a point before that pass counts a claim attempt and reads the full run row, so a held run costs no attempt and no read. - The sweep reuses the `deferred_rate_limit` outcome for a deferral, to keep its outcome type unchanged while #21394 edits that file. - The automatic-run predicate is built from master exports (`RECOVERY_WAKE_REASONS`, retry markers). It will switch to `isAutomaticContinuationRun` when #21394 lands. - Migration `0480` adds one small table. - Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`) runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement: it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in `packages/db/src/client.ts`, like 0470 and 0476, and the runner drops and rebuilds an invalid index after an interrupted build. The index is declared in the Drizzle schema, so `pnpm db:generate` reports no changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On a large `heartbeat_runs` table the build is online (no write lock) but takes time proportional to the table. After it exists, it is one more index on a hot table: each run that finishes writes one more index entry. - No user-visible text changes outside the operator report. ## Model Used - Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell, file edits, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f6c5986b61 |
feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service
## Linked Issues or Issue Description
**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.
**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.
**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.
**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.
**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.
## What Changed
- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.
## Verification
- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.
## Risks
- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1dee321844 |
perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every native event that announces work to another part of the server (chat progress, a final response, an evidence outcome, recovery) registers a wake in its transaction before it writes the event > - Such an event takes six statements: BEGIN, the run lock read, a replay probe, a read of the transaction id for the wake, the append, and COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock, the transaction id, the write, COMMIT > - In the agent scaling load model these two paths are now the largest per-agent cost after #21511 and #21520, and other events fall behind first at 600 agents > - This pull request returns the transaction id from the lock read that each path already sends, and moves the replay probe into the run lock read when it is safe > - The benefit is four statements for such an event and four for a checkpoint save, with the same locks, the same wake order and the same replay rule ## Linked Issues or Issue Description Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The statement count of native events that register a wake (`appendNativeSourceEvent`) and of controller checkpoint saves (`nativeRunnerCheckpointWriter`). **Subsystem affected** `packages/db/src/work-signals.ts` (one new export), `server/src/services/heartbeat-run-events.ts`, `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Current behavior** `signalDatabaseWork` sends `select pg_current_xact_id()` for the first wake of each transaction. A native event with a wake reads the run lock, then probes for its replay in a separate statement, because a statement that waits for the lock keeps its earlier snapshot. **Proposed behavior** - The run lock read and the checkpoint owner lock read also return `pg_current_xact_id()::text`. Both already lock a row, so the transaction has an id at that point. `primeDatabaseWorkTransactionId` gives it to the transaction's wake state, and the wake sends no statement for it. - For an event that may register a progress, response, reassessment or retention wake, the run lock read also reads `next_event_seq` from its own snapshot and checks for a replay in that snapshot. Every event append advances `next_event_seq` in its transaction. So when the locked row has the same `next_event_seq` as the snapshot, no append committed in between, and the probe result is current. Otherwise, or when the probe finds a replay, the separate probe runs as before. **Reason and benefit** Fewer round trips while a pool connection is held in a transaction. In the load model these paths hold the main pool and the three-connection checkpoint pool longest per event. **Breaking changes** None. No schema change. ## What Changed - `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`, exported from `@paperclipai/db`): inside a transaction whose id is not yet known, it records the id. Outside a transaction it does nothing. An uncertain commit still resolves with `pg_xact_status` on the same id. - `appendNativeSourceEvent`: the lock read returns the transaction id when the event has any wake. For a probing wake it is one statement: a CTE that reads `next_event_seq` from the snapshot, a CTE that locks the run row (`for no key update`, unchanged), and `exists (...)` for the replay. The wake is still registered after the replay check and before the append. - `lockNativeRunnerCheckpointOwner` returns the transaction id with its owner lock and primes the wake state. Its return value and every caller are unchanged. ## Verification - Red/green on embedded PostgreSQL; each new assertion failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "registers a native event's wakes before its write and only when the event is new": four statements (before: six), the wake's intent and settlement are published, and a replay publishes no wake. - New: "finds a replay that another writer committed while the lock read waited, and registers no wake for it". Another connection holds the run row (the test waits until PostgreSQL reports the append waiting for that lock), stores the same event and moves `next_event_seq`; the waiting append returns `duplicate`, runs the separate probe, and publishes no wake. With the `next_event_seq` comparison removed, this test fails. - New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the owner lock with the transaction id, the write, COMMIT (before: five statements), and its recovery wake is published; a save after the lease expired writes nothing and fails with `native_runner_checkpoint_owner_lost`. - `work-signals.test.ts`: a primed transaction sends no id statement for its wakes; a primed transaction whose connection is killed after its write stays pending and then resolves through `reconcileDatabaseWork`, with no row stored. - Added after merging `origin/master` (with #21511, #21520 and #21436) into this branch: - "gives the wake the transaction id of the transaction that took the run lock": inside a caller's transaction, an event with a progress wake sends no separate `pg_current_xact_id()` statement; the transaction's connection is then killed, and `reconcileDatabaseWork` checks `pg_xact_status` with exactly the id that `pg_current_xact_id()` returns on that same transaction. Priming a wrong id makes it fail. - "primes nothing when the lock read fails, so the transaction is not left uncertain": a lock read that fails on `lock_timeout` leaves no pending work; after the same failure inside a savepoint, the next signal on that transaction still sends its own `pg_current_xact_id()`. Priming before the lock read makes it fail. - Both call sites pass only the value returned by the statement they just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)` then `primeDatabaseWorkTransactionId(tx, run.xid)`) and `lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then `primeDatabaseWorkTransactionId(tx, xid)`). - Retest on the combined tree, one file at a time: atomic-append 28 (including the equivalence test and both race tests), the port suite with conformance 105, checkpoint-save 2, `live-recovery-work` 51, `work-signals` 25, runner `native-session-runtime` 297; server typecheck passes. No code change was needed for the combination. - Other focused suites pass, one file at a time: the port suite with its conformance suite (100), `chat-native-progress-work` (35), `native-observer-replay` (1), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-recovered-owner-followup` (97), `native-restart-recovery-isolation` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25). - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). Stacked: this pull request merged locally with #21511, and the model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model applies the port's batching rules itself; that mode is not in any pull request). Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. "Before" rows are `origin/master` from the same session. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 / 12,000 | 1 / 70 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 / 24,000 | 2 / 1,274 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 / 72,000 | 1,136 / 4,980 | | 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 | 71,958 / 72,000 | 1,319 / 5,712 | | 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 / 48,000 | 1,974 / 3,835 | With the three changes, one process finishes 99.9% of the offered work at 600 agents, but its event-loop utilization is 0.90–0.93, above the 0.75 target, and p99 lateness is about 5 s. It saturates between 600 and 800 agents. - Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5 where commit-time notifications are on). ## Risks - The transaction id is read in a statement that already assigns one (a row lock), so no transaction gets an id it did not have before. - The probe in the lock read is used only when `next_event_seq` did not change between the snapshot and the lock. Every writer of `heartbeat_run_events` moves `next_event_seq` in the same transaction (native and ordinary appends, the run coordinator store, the retry queue). A future writer that inserts events without it would make this probe miss a concurrent replay; the append statement then still finds the replay and stores nothing, but the wake would already be registered (a spurious wake, not a lost or duplicate event). - This pull request and #21511 both edit `heartbeat-run-events.ts`, in different functions. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a726dce4f0 |
feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools from connected apps through the Paperclip MCP gateway > - Some tool calls take a long time: exports, builds, large searches > - The MCP progress utility lets an app report how far a call is, but Paperclip ignored it > - So the task chat showed only a spinner until the call finished, with no sign of life > - This pull request reads the app's progress and shows it on the tool's row while the call runs > - The benefit is that people can see that a long call is working, how far it is, and how long it has run ## Linked Issues or Issue Description **Subsystem affected** Connected apps (MCP gateway) and the task chat. **Problem or motivation** A long tool call from a connected MCP app shows only a spinner in the task chat. Paperclip does not send a `progressToken`, and it drops `notifications/progress` from the app. **Proposed solution** Send a `progressToken` on each `tools/call` that an agent run makes. Read the app's progress notifications from the call's response stream. Show a progress bar (or an indeterminate sweep), the latest message, and elapsed time on the running tool row. Keep the latest state so that a reload mid-call shows it. **Alternatives considered** A new transport for progress. The existing company live-events socket already carries per-run events with run read checks, so this change uses it. **Additional context** Spec: https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress ## What Changed - `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`. `readMcpHttpResponse` gives each notification on the response stream to an observer before the result. The observer cannot fail or delay the call. New `mcpProgressNotificationOf` matches a notification to its token. - `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an agent run send a per-call token and forward progress to a reporter. Calls without a run send no token. - New `server/src/services/tool-call-progress.ts`: checks each update (finite, increasing, plain single-line text of at most 200 characters). It publishes a `heartbeat.run.tool_progress` live event at most every 250 ms (latest wins) and saves the latest update on the invocation at most every 1 s. Nothing is awaited by the call. A final `done` event goes out when the call ends. - DB: nullable `tool_invocations.progress` jsonb column (migration `0479`, `ADD COLUMN` only). A save never writes onto a finished call. - Live delivery: the new event follows its run like transcript output. A client gets it only for runs it follows, after the run read check, in the run's company only. - New route `GET /api/issues/:issueId/tool-call-progress` (task read check) returns the latest progress of the task's running calls, for a reload mid-call. Documented in `docs/api/issues.md`. - UI: `ToolCallProgressProvider` (one per chat) keeps the latest update per call. A running tool row subscribes and shows a bar or sweep, the message, the percent and elapsed time (design tokens, reduced motion honored). Finished rows drop it and never fetch or re-render for progress. `LiveUpdatesProvider` ignores the event for query invalidation. - `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool (`steps`, `stepMs`, `withTotal`) in its own file. - Storybook: `Tasks/Tool progress` stories (running with total, running without total, done). - Docs: new `doc/connections/MCP-PROGRESS.md`; `MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported. ## Verification - Red/green: on the original source, 5 of the new server tests fail (token in `_meta`, SSE with interleaved progress and result, notification matching, run-interest delivery, gateway end-to-end). With this change they pass. - `server/src/__tests__/mcp-http.test.ts` (13), `tool-call-progress.test.ts` (9: throttling, latest wins, bounded text, out-of-order drop, finish, failures never throw, company/run/tool scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres: company and task scoping, no write onto a finished call), `realtime/live-event-delivery.test.ts` (run interest), `tool-gateway.test.ts` end-to-end against the real SDK kv demo server over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7 throttled events, all for the right company, run, task and tool, the last one `done`; no event reaches another company; the invocation keeps the progress; the result is unchanged. - UI: `tool-call-progress.test.tsx` (10: name matching, row states, plain-text rendering, drop on complete, restore after reload, no fetch for finished rows). All 67 task-chat test files pass. - `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP with and without a total. - Storybook: checked the three stories in a browser. The bar fills to 40%, the sweep animates, the elapsed time ticks, and the done row shows no progress. - `pnpm verify`: see the CI checks for the final revision. ## Risks - Low. The migration adds one nullable column. - An SSE response is now bounded per message, not per stream, so handled notifications (such as progress) do not use up the result's size limit. Each message is still bounded. - Progress code runs only while a run's tool call is in flight. It adds a few bytes to the request and throttled, fire-and-forget work. Agent boot and time to first token do not change. - A row finds its call by run and tool name. If one run calls the same tool twice at the same time, both rows show the newer call's progress. - Progress from a call that became an MCP Task is not shown. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81d06cfc97 |
feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools in connected apps through the tool gateway, which speaks MCP to each app > - MCP Tasks (#21263) lets an app answer a long tool call with a durable task. It shipped behind the experimental setting `enableMcpTasks`, off by default > - No public app returns tasks without credentials, so nobody had seen the feature work against a real server, and it was hard to understand > - This pull request adds a small demo app that uses Tasks, tests the gateway against it over HTTP, and records a live end-to-end run > - The live run found one bug: every task stop sent the call's audit record, with a server stack trace, to the agent. This pull request fixes it > - The benefit is a repeatable way to try and test Tasks, plain-language docs, and evidence for the decision to turn the setting on ## Linked Issues or Issue Description Refs #21389 Refs #21263 ## What Changed - Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin `paperclip-tasks-demo-mcp-server`; run `node packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is stateless MCP 2026-07-28 with `server/discover`, and it declares `io.modelcontextprotocol/tasks`. It has six tools: `slow_report`, `needs_approval` (form question), `will_fail`, `cancellable`, `expires`, and `quick_echo`. It saves task state to a file, so it can restart mid-task. It needs no credentials. It has a status page at `/` and JSON at `/api/state`. - I put it in the existing demo package, so there is no new workspace package and no `pnpm-lock.yaml` change. - Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the real gateway against the real demo server over HTTP, with one test per tool plus a restart. - Fix: `server/src/routes/tool-gateway.ts` no longer sends `details.execution` to callers. That record has request IDs and the server-side error stack. The audit still keeps it. Add `server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the fix, green after). - Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a person sees", "Try it yourself", the live-run results, and a recommendation. The package README documents the demo server. - `enableMcpTasks` stays off by default. ## Verification - `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13 new demo-server tests). `tsc -p tsconfig.test.json` is clean. - `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass. `tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2 before the fix. - Existing suites that cover the changed route and Tasks: `mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results in a comment below). - `pnpm verify` exits 2, because the doc is a global CI input. Required CI gives the broad coverage. - Live end-to-end run on a local `test-drive` instance with `enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The demo was connected through "Connect your own MCP server": | Behaviour | Result | `tools/call` count at the app | | --- | --- | --- | | Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 | | Question (`needs_approval`) | The card was answered in the UI. `tasks/update` was sent, and the task finished | 1 | | Failure (`will_fail`) | `provider_task_failed` with the app's message | 1 | | Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 | | Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 | | Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s. The result was delivered once, and the agent was woken once | 1 | After the run, all 6 `tool_mcp_tasks` rows were settled and none had a next poll. ## Risks - Low risk. The demo server is new code in a demo package, and the bin is additive. - The route change removes `execution` from tool-gateway error responses. No UI code or test reads that field from these responses. The audit and tool-call events still record it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a4c570aede |
feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use connected MCP apps through the tool gateway. > - An MCP app can ask the person a question in the middle of a tool call (MCP elicitation). There are two kinds: a form, or a link to open. > - Today a form question is a dead end: > - A 2026-07-28 server's form `input_required` stops with "needs information Paperclip can't collect yet". > - An older server's `elicitation/create` makes a card, but the answer is never sent back, and the server waits on an open stream. > - Paperclip does not declare form support, so compliant servers never ask. > - This pull request shows the question as a card in the task chat. The card names the app, and only a person can answer it. When the person answers, the agent wakes. The agent's repeated call sends the answer to the app exactly once, for both protocol eras. > - The benefit: apps that need a choice or a detail from the person now finish their work. Restarts lose nothing, and no call is left waiting. ## Linked Issues or Issue Description Refs #21140 (stateless MCP client; this PR closes its "form elicitation" follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs #21389 (MCP Tasks follow-ups: the card answer validation item). **Is your feature request related to a problem?** When a connected MCP app asks the person a form question during a tool call, the call stops and cannot continue. A 2026-07-28 server gets a "needs input" error. An older server gets a card whose answer never reaches it. **Describe the solution you'd like** Show the question in the task chat. Let a person answer, skip, or leave it. Wake the agent when the question is resolved. Send the answer on the agent's repeated call, once, with the server's exact state. **Describe alternatives you've considered** Holding the tool call open until the person answers. We rejected this: it holds a stream and a run slot, and it does not survive deploys or restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records nine design decisions with their alternatives. ## What Changed - **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec, the inventory, the decisions, the journey, tests, and risks. - **New module `server/src/services/mcp-elicitation.ts`.** It is the single mapping for direct calls and MCP Tasks: - It turns each spec form field into a real card field: text with defaults, number, yes/no, single choice (`enum` and `oneOf`), and multiple choice (`items.enum` and `items.anyOf`). - It turns card answers back into typed `content`. - Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or expired sends `cancel`. - It refuses forms that ask for passwords, keys, tokens, card numbers, or one-time codes. - It keeps app text plain, so links in it are not clickable. - It seals `requestState` with AES-256-GCM, bound to the company and connection. - **New table `tool_mcp_elicitations`** (migration 0479). It is company-scoped, with one row per question round. It holds a keyed call identity, the sealed state, the questions and their cards, a claim time, and a 24-hour expiry. - **Gateway, stateless servers:** - Paperclip declares `elicitation: {form, url}` on task tool calls. - A form-only `input_required` saves the round, makes the cards, and stops with a wait message. - A repeated call before the person answers stops again without contacting the app. - After the answer, the repeated call claims the round and sends `inputResponses` and the exact `requestState`, once. - A failed send releases the claim. A crash mid-send lets the claim lapse. - **Gateway, older servers:** - Paperclip declares form support at `initialize`. - The first question gets `cancel` at once, so the server never waits. - When the same question (same words and schema) arrives on the repeated call, Paperclip answers it from the card. - **Gateway, mixed rounds and retries:** - A form beside a URL step shows the link first. - A repeated call that stopped for a question runs again in the same run. It is no longer replayed by the per-run idempotency key. - **Cards:** the title is "*App* needs your input". The resolver policy is `human_only`. "Send to *App*" sends the answer. A user comment does not supersede the card. - **Activity log:** new events `tool_gateway.elicitation_sent`, `tool_gateway.elicitation_refused`, and `tool_gateway.elicitation_expired`. They record the action only, never the values. - **MCP Tasks:** task cards use the same mapping and gain the app name, real fields, and `human_only`. A secret or an unsupported form stays unsupported. - **Storybook:** `Tasks and conversations/Questions and approvals/MCP App Questions` has five states: Asking, Answered, Skipped, Expired, and TwoQuestions. The fixtures are generated from the server mapper's output. - **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are updated. ## Verification - `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass on the branch. - **Red/green:** 9 of the 9 gateway tests fail with master's `tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new module. - The tests use a fake 2026-07-28 server and a fake 2025-06-18 server that asks over the open stream and waits for the reply. - Covered: answer, skip (`decline`), withdraw (`cancel`), two questions at once, a restart while waiting, a crash mid-send, another company with the same app and arguments, expiry, a refused password field, an agent that tries to answer (rejected), and capabilities. - Existing suites: `mcp-tasks`, `mcp-protocol-versions`, `tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See the results comment below. - Storybook: the five stories render in headless Chromium, and the play tests pass. The only console errors are the 404s from the backend-less API calls that every story makes. - `tsc --noEmit` is clean for `server` and `ui`. `node scripts/storybook-catalog.mjs` passes. ## Risks - **Older servers may start asking questions.** Declaring form support at `initialize` can make older servers ask where they did not before. Mitigation: the first question always gets `cancel` at once, so no call hangs, and the agent gets a clear next step. - **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one indexed lookup for a waiting round. Nothing is added before the first token or to agent boot. - **Migration.** One additive table with no backfill, so it is safe to apply online. - **Changed Tasks behavior.** A task card that is withdrawn or closed now sends `cancel` instead of `decline`, as the spec defines. A skipped card still sends `decline`. - **Stale cards.** Expiry is checked only when the agent calls again. A card can sit past 24 hours until the next call. Then it is withdrawn and the app asks again. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use (shell, file edits, headless browser), with extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a96f972e1 |
Connect Tailscale networks through device sign-in (#21526)
## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - SSH environments can reach machines through a company Tailscale connection. > - The initial connector requires a managed OAuth app that Tailscale limits to its own tailnet. > - Customers need to connect their own tailnets without registering an app or supplying a key. > - This pull request uses Tailscale's normal device sign-in for new connections. > - Customers approve the Paperclip device and then select their machines in SSH environments. ## Linked Issues or Issue Description **Problem or motivation** The current Tailscale enrollment path cannot serve independent customer tailnets through the documented self-service OAuth app model. It also requires Cloud broker configuration and pilot admission before a customer can start. **Proposed solution** Start a private Tailscale device in the tenant and open its normal sign-in link. Save its encrypted device state before reporting a successful connection. Retain the existing SSH machine selection and agent account behavior. **Alternatives considered** Auth-key entry would require customers to copy a credential. A public OAuth integration still needs a confirmed registration path. Device sign-in uses a documented Tailscale flow without either requirement. See [device authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication) and [OAuth app limitations](https://tailscale.com/docs/features/oauth-apps). **Related PR search** Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for `tailscale`. This follows the merged network connector in [#20617](https://github.com/paperclipai/paperclip/pull/20617) and preserves the SSH agent-account support in [#21248](https://github.com/paperclipai/paperclip/pull/21248). The open [#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes storing an OAuth client and checking its health; this change instead enrolls a device through interactive sign-in so customers do not need an OAuth app. ## What Changed - New connections use device sign-in. Existing connected devices and pending legacy OAuth callbacks remain supported. - Pending sign-ins are bound to their company and initiating user. They have a fixed ten-minute deadline. Device state uses server-owned vault secrets with trusted version and digest checks. - The active Tailscale CLI waits for consent and device approval. Cancellation, expiry, secret deletion, and stale completion cannot restore a cancelled connection. - The connector stays open during sign-in. It provides a link when popups are blocked and shows device approval separately. Progress checks stop on completion, failure, expiry, or unmount. - Shared response types, OpenAPI contracts, SDK source pins, and the API/setup documentation describe the new flow. No database migration or Cloud provider configuration is required. ## Verification - 109 focused tests cover network processes, the real database and encrypted vault, routes, UI, and legacy connection behavior. - Server and UI typechecks pass. - Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure locally. All 70 OpenAPI and Tailscale route tests and server typecheck pass after synchronizing the API contract. - Follow-up `79de05f38b9`: the transient-recovery regression failed before the fix; all 20 connector UI tests and UI typecheck pass after it. The query-error policy check reports no findings. - The public SDK input registry check passes. - [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2) passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57 successful checks, no failures or pending checks. The interrupted canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile is 5/5 on this head with no unresolved review threads. - `pnpm verify --explain` exits 2 because the documentation change selects broad CI coverage. Focused local checks passed; broad CI passed on `29cf296a597e470d432b84767050bc539caa280e`. - Live local qualification on `29cf296a597e470d432b84767050bc539caa280e`: native device sign-in completed with interactive user approval. An isolated Ubuntu 24.04 ARM64 EC2 machine appeared in discovery and passed SSH pairing and workspace probes. The assigned Claude Code agent used a Paperclip-managed subscription, created `hello_world.txt` containing exactly `Hello, world!` plus a newline, posted a completion comment, and marked the task done. An independent read through the private connector verified the bytes, remote user, and hostname. The remote machine had no stored Claude login and no leftover temporary credential directories. - Restarted the local Paperclip server and verified the same Tailscale device identity, online peer discovery, successful SSH probe, and unchanged file without new consent. Temporary public SSH ingress was removed. - Local qualification covers one user tailnet and the final Core revision. Browser UI acceptance, two unrelated tailnets, a hosted restart, and an agent task on the final staging revision are not yet verified. ## Risks - The enrollment method changes. It must pass the live qualification above before broad rollout claims. - Existing encrypted device state and the vault master key must survive a restart or redeploy. Device expiry or removal still requires reconnecting. - The existing one-serving-process-per-instance constraint remains. Concurrent replicas need durable network ownership. - Pending setup uses bounded progress requests and an event-driven CLI with a deadline. It adds no idle database scan. ## Model Used OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed by the runtime. No subagents were used. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
493519fc88 |
fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (runnerd) relays provider events to the server and waits for cumulative ACKs before it polls the provider again. > - After a control frame, runnerd uses a 1 ms queued read only when the executor reports a live provider poll. Otherwise it waits up to the 250 ms runtime read timeout, and any frame that arrives in that time skips the poll. > - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex through ACPX and others) always wait the full 250 ms after the last ACK of every batch, and steady control traffic starves their provider polls. > - This pull request lets an ACPX executor report a live poll while its restored session has an active turn. > - The benefit is that ACPX output is relayed without the 250 ms idle wait, and tool-heavy turns no longer starve provider polls. ## Linked Issues or Issue Description **What happened?** A routine run started several subagents that made about 190 tool calls in about 10 minutes. On the server, consecutive runner events arrived at least 576 ms apart (median about 1.5 seconds), one event at a time, while the time from each event's `emittedAt` to its insert was 0 to 0.1 seconds. The runner fell several minutes behind the provider. The backlog then overflowed the transport's event buffer (fixed separately in #21524) and the turn failed. **Expected behavior** A live ACPX turn is polled again as soon as its outstanding events are acknowledged and no control frame is queued, the same as a Codex facade turn. **Steps to reproduce** 1. Run an ACPX turn that produces events while the controller sends commands and ACKs. 2. After each ACK, runnerd calls `receive_json`, which waits up to 250 ms before it polls the provider. A frame that arrives in that time skips the poll. **Relevant logs or output** ``` min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched insert time - emittedAt: 0.0 to 0.1 s ``` Related: #21524 bounds the receive buffer that this backlog overflowed. ## What Changed - `acpx_provider_backend.rs`: `AcpxCommandExecutor::provider_poll_is_live` returns true only when the session was restored, has no persistence error, does not wait for a next-turn generation, and its state is `turn_starting` or `turn_active` with an active turn, no unconfirmed provider exit, and the run it owns. - `native_provider_backend.rs`: `SelectedExecutor` forwards `provider_poll_is_live` to the ACPX executor. - `tests/acpx_warm_attachment.rs`: new test through `NativeProviderCommandExecutor`. The session is not live before `run.prepare` or after `session.open`, is live after `turn.start`, and a new executor that has not restored its session is not live. What does not change: - runnerd still polls only when the read returns no frame, so a queued stop or suspend still runs before the next provider poll. - The ACK gate (no poll while sent events are not acknowledged) does not change. - The 1 ms read is used only while the source sequence advanced since the last quiet poll, as for Codex today, so an idle turn does not busy-loop. ## Verification - `cargo test --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test acpx_warm_attachment` (in `packages/paperclip-runner`): 7 passed. The new test fails without the new method, because the default returns false. - `cargo fmt --all -- --check` and `cargo check --locked --workspace` pass. - `cargo test --release --locked -p paperclip-runner-core --no-fail-fast`: 1,127 passed, 3 failed. The failures are `codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames` ("failed to signal owned process group: Operation not permitted") and `stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement` and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay exit timing). They also fail without this change in the macOS sandbox where I ran the suite, and none of them uses the ACPX executor. CI covers them on Linux. - The existing `durable::tests::ack_release_poll` cases cover the runner side of the live read: an ACK releases the next poll without the read timeout, and a command queued behind the ACK runs first. - I did not run a live provider. After the deploy, the gap between runner events of an ACPX run with queued output should drop below 250 ms. ## Risks - Low risk. The change only selects the existing 1 ms queued read for ACPX during an active turn. - A turn that produces events continuously now polls more often. Each poll still writes the same durable state as before, so disk writes per event do not change. - Analysis also found that runnerd waits for every sent event to be acknowledged before it polls again. A bounded in-flight window would raise throughput further, but it changes the tested rule that control queued behind an ACK runs before the next poll. That is left out of this pull request on purpose. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
254744f77c |
fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.
## Linked Issues or Issue Description
**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.
**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.
**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.
**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```
No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.
## What Changed
- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.
## Verification
- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.
## Risks
- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
|
e82c4f0a31 |
fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**
This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it
## Linked Issues or Issue Description
**What happened?**
Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.
- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.
**Expected behavior**
A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.
**Steps to reproduce**
1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.
**Paperclip version or commit**
`master` at
|
||
|
|
1332ab987f |
feat(prp): declare the contract for provider-started turns (#21422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The Runner, its sidecar and the controller speak two protocols with schemas: PRP between the controller and the Runner, and the sidecar protocol between `paperclip-runnerd` and the ACPX sidecar. > - A turn that the provider starts needs new fields and one new command in those protocols. > - This pull request declares them and generates the readers again. It changes no behavior. > - The benefit is that the next two pull requests show behavior only, with the contract already in review. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 4. Its base branch is `feat/claude-provider-cycle-stream`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production. The protocol schemas. **Subsystem affected** Paperclip Runner: `protocol/` and the generated TypeScript and Rust contract code. **Current behavior** - PRP has no way to say that a turn was started by the provider, or that the Runner waits for such a turn. - The sidecar protocol has no command to arm the sidecar and no event for a turn that the provider starts. **Proposed behavior** - PRP (`schemas/provider-turn.schema.json`, new): `providerStartedTurns` (boolean) in the payload of `turn.start`, `origin: "provider"` on `turn.started`, and `awaitingProviderTurn: true` on `turn.completed`. `command-v3` and `event-v3` refer to it. - Sidecar protocol (`provider-schemas/acpx-sidecar.schema.json`): the command `session.provider_turn` with `{ turnId, armed }`, the event `runtime.turn_started` with `{ origin: "provider" }`, and the payload of the diagnostic for a lost provider. The schema enforces that payload for a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`. Diagnostics with other codes keep their open payload. - `paperclip-runnerd` decodes `runtime.turn_started` and refuses it. No sidecar sends it yet. The Runner of today refuses an unknown event too. - No controller, Runner or sidecar uses the new fields in this pull request. **Reason and benefit** The repository rule for Runner contracts says that new fields must be in the schema with generated readers. A separate pull request lets a reviewer read the contract alone. ## What Changed - `protocol/schemas/provider-turn.schema.json` (new), `command-v3.schema.json`, `event-v3.schema.json`, `protocol/provider-schemas/acpx-sidecar.schema.json`, `protocol/manifest.json`. - `scripts/generate-acpx-sidecar-contract.mjs` and `scripts/generate-protocol-schema-module.mjs`. - Generated: `src/drivers/acpx/generated-sidecar-contract.ts`, `src/protocol/generated/schema-bundle.ts`, `src/protocol/generated/standalone-validators.ts`, `runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs`. - `runner/crates/runner-core/src/acpx_event_payload.rs` (3 lines): the Rust `match` on the event type must name the new event to compile. - `test/protocol-contract.test.mjs`. - The contract digests of the pack change (`prp.acpx-sidecar`, `prp.command-v3`, `prp.event-v3`, `rust`, and the new `prp.provider-turn`). No stored file refers to these digests. No provider `commandDigest` and no profile version changes. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. The contract tests: 32 passed. - The sidecar suite and the tracker tests: 141 passed. Not run: the full suite. CI supplies that coverage. ## Risks - **The pack contract digest changes.** Older image packs are not used again. - **The opt-in field is accepted by the schema but has no effect yet.** A controller that sends it before part 5 gets the behavior of today. - **State saved by the previous release:** this pull request changes no persisted shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e5ad224cc5 |
feat(acpx): give Claude's own cycles to the adapter as one ordered stream (#21421)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - Claude Code starts a cycle of its own when a background task finishes and no prompt is in flight. The Runner sees the raw messages of that cycle. > - #21293 only marks the background state as unknown when it sees such activity. > - This pull request gives each such cycle to the caller of the adapter as one ordered stream: started, its events, ended with its result. Nothing reads the stream yet. > - The benefit is one place that knows where a cycle starts and ends, with cycles that cannot overlap. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 3. Its base branch is `feat/acp-session-update-event-converter`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. How the ACPX adapter sees the cycles that Claude starts by itself. **Subsystem affected** Paperclip Runner: `src/drivers/acpx` (the adapter, the runtime host, the Claude background task tracker). Claude only. **Current behavior** - The raw ACP observer gives each message to the background task tracker. - Activity of Claude with no client request in flight makes the background state unknown. The messages of that cycle are not kept. **Proposed behavior** - `ClaudeProviderCycles` puts each cycle on one ordered stream. A cycle starts at the first update of the own session with no client request and no prompt in flight. It ends at the `usage_update` that has `_meta._claude/origin`. - A caller gets the stream with `providerCycles()`, reads it with `next()`, and calls `adopt()` or `drop()` for each cycle. A permission request waits for the decision of its own cycle. The stream records the cycle of each tool call, and `decisionFor(toolCallId)` gives that decision. The binding that a caller gives to `adopt()` can also carry `onPlanProposal`, so a plan that an adopted cycle proposes goes to the caller as it does for a normal turn. - With no caller, a cycle is declined and the background state becomes unknown, as before. The sidecar does not call `providerCycles()` in this pull request, so production does the same as today. - The tracker reports `{ live: N }` only while N shells are running, `{ live: 0 }` only if the process never had a background task and Claude never acted between prompts, and unknown in each other case. A dropped, failed or cancelled cycle makes the state unknown. - An empty result that directly follows the end of a cycle (zero usage, same context use and cost) is ignored. Claude Code sometimes sends it. It must not become a cycle, and its usage must not be reported two times. **Reason and benefit** - One producer and one reader make the order of cycles a property of the structure. - The tracker does not read the order of messages to decide which finished task a cycle reported on. Captures showed that Claude Code can put two notifications in one cycle, start a second cycle, or give a notification to a cycle in progress, and the ACP stream does not say which. ## What Changed - `src/drivers/acpx/claude-provider-turns.ts` (new, 303 lines): `ClaudeProviderCycles`. - `acpx-runtime-adapter.ts` (78 added, 8 removed) and `runtime-host.ts` (24 added): `providerCycles()` and `providerTurnsAvailable()`, and the permission path that waits for the decision. - `claude-background-tasks.ts` (16 added, 7 removed): the rule above. - Tests (764 lines): `claude-provider-turns.test.ts`, `claude-provider-turn-tasks.test.ts`, `claude-background-tasks.test.ts`, a permission test in `acpx-runtime-adapter.test.ts`, two tests with the real `claude-agent-acp` (`acpx-runtime-adapter.claude-provider-turn.test.ts`, `acpx-runtime-adapter.claude-overlapping-shells.test.ts`), and a captured raw stream as a fixture. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - 10 test files, 294 passed. This includes the test of #21293 with the real `claude-agent-acp`, not changed: with no caller of the stream it still gets `{ live: 1 }` at the end of the turn and an unknown state after the cycle of Claude. - Real `claude-agent-acp` with a local fake Anthropic API: a background shell finishes, and the stream gives one cycle with its own usage. With two shells where the second finishes before the first cycle sends its first update, the first cycle ends with an unknown state. - Mutation checks, each made the named tests fail: an end item before the last message of a cycle; a drop that does not make the state unknown; a finished task that reports `{ live: 0 }`; the empty result not ignored; a permission request that does not wait. - Live run with a real Claude account, not in CI, on the full change: one background shell gave `{ live: 1 }`, one cycle, then unknown. Two shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown. Not run: the full suite. CI supplies that coverage. ## Risks - **The time at which the state becomes unknown changes a little.** With no caller, a cycle makes the state unknown at its first message. Before, it was at the first message of an activity kind. The result for the Runner is the same: a restart at the end of the turn. - **Limit, by design.** After a cycle, `{ live: N }` counts only running shells. A command that finishes while Claude already reports on an earlier one gets no cycle that the Runner waits for. - **The rule for the empty result is narrow** and comes from captures of one Claude Code version. The reason that Claude Code sends it is not known. - **A queue of 4,096 messages for one cycle.** A cycle that is larger fails, and the state becomes unknown. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1140984b82 |
feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.
**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.
**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.
**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.
## What Changed
- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.
## Verification
- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.
## Risks
- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
21bf7ebcee |
feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path > - Paperclip manages work for AI agents. > - SSH environments run tasks on computers that users own. > - A Paperclip server needs network access before it can reach a private computer. > - A reusable Tailscale connection belongs in Connectors. > - Connected machines should become choices in the existing SSH environment form. > - This change connects a tailnet and resolves selected machines through a private network client. > - Agents retain their choice of harness and AI account. ## Linked Issues or Issue Description **Subsystem affected** Connectors, environment settings, SSH execution, company secret storage, and runtime packaging. **Problem or motivation** Entering a Tailscale address in the SSH form does not connect the Paperclip server to that tailnet. Requiring a separate device key for every environment also duplicates network setup. **Proposed solution** Add a Tailscale Connector with **Connect Tailscale Network**. Use brokered OAuth consent to enroll one private network client per connection. Let SSH environments select a connected network and stable machine identity. Preserve the enrolled node state through the company vault so a restart does not reuse the one-time token. **Alternatives considered** A separate Tailscale environment type would duplicate SSH behavior and mix network access with harness selection. This change uses the existing `ssh` driver. Related #15648 adds OAuth client credentials and a health check; this change uses user consent and supplies outbound task execution. Related #11743 concerns HTTPS brokering. **Roadmap alignment** This extends the existing Connected Apps, environment, and Personal & Shared AI Accounts capabilities. This PR is stacked on #21248, which provides SSH account selection independently of Tailscale. The hosted OAuth broker is a companion control-plane change. ## What Changed - Preserve Cloud reauthentication in the current browser tab. Explain canceled or failed consent in the connector and offer a retry of the same authorized saved connection. - Record Cloud enrollment starts initiated through the Tailscale connector in the company activity log. - Check install-preparation descendant liveness through the kernel process table in the Runner support test. Keep the direct child’s strict reaping assertion and reject any non-zombie descendant. - Align the application image stage contract with the Tailscale client stage after the upstream Runner-image refactor. Keep the exact stage list and check every application stage, including `tailscale-client`, for prohibited Runner compiler commands. - Group Tailscale consent, discovery, daemon lifecycle, pairing, and deletion under `server/src/services/connectors/tailscale/`, with their service tests. Update existing route, runtime, environment, vault, and company-deletion imports. Keep the existing contracts, authorization, audit, and transaction owners. - Handle early stdin closure in the Kubernetes real-shell test helper. Stop the input stream on `EPIPE`, preserve nonzero child exits, and reject a zero exit when input was lost. Add two real-process regression cases. Production transfer code and confinement checks are unchanged. - Read configured Tailscale executable paths when a client starts, after supported `.env` loading. Keep explicit command overrides available. - Discard pending pairing results after switching to Direct SSH or closing the form. Retain the existing company, connection, machine, and port checks. - Store network credentials with an immutable server-owned purpose from their first vault row. Exclude them from generic catalogs, bindings, environment projection, and value reads. Use a dedicated audited server consumer for pinned connector reads. Block unclassified legacy serialized node state at the same value boundary. Keep SSH private-key binding separate from connector credentials. - Gate network queries and setup on verified local-implicit or instance-admin access. Hide Tailscale catalog rows for other members, show an admin requirement on direct setup navigation, and avoid device discovery before management access is known. - Keep failed-client cleanup in the network manager. A failed request cannot disconnect a newer retry for the same identity; explicit connector disconnect and enrollment retirement still stop their networks. - Check for readable Ed25519, ECDSA, or RSA host public keys before pairing edits `authorized_keys`. Print a known-hosts entry for every available key, or fail before changing authorized keys if none are available. - Return self-hosted Cloud enrollment to the Tailscale connector. Save the unprefixed setup path and let the shared callback add the verified company prefix. Keep external and unrelated destinations rejected. - Hide pairing instructions after an SSH port change. Ignore late key-selection results for the old port. Operators prepare again for matching known-hosts output. - Retire connector references before deleting a company or its vault credential. Stop company daemons after the database commit. Keep failed secret retirement retryable and prevent late node-state writes from restoring deleted connections. Discard retired state without blocking behind the vault deletion lock. - Reload a mismatched credential before replacing a ready daemon. Recheck the active entry and manager generation after the read. An older request cannot stop a newer healthy socket or cross shutdown. - Pin connector credentials to the exact vault version and SHA-256 digest recorded by trusted enrollment and node-state writes. Ignore generic secret rotations. Advance both fields with a compare-and-set guard. Reject unpinned or tampered credentials before network startup. - Save an explicit Direct SSH selection and clear old Tailscale fields after config merging. Preserve the operator's entered direct host. - Route all Tailscale SSH runtime access through the connection service after SSH-key loading. Pin access to the environment's saved connector identity and recheck after device discovery. Remove standalone key-based daemon startup and reject raw enrollment keys or credential-only network configurations. Align the shared config type and SDK pins. - Correct two E2E assumptions exposed during connector qualification. Keep the two-window chat fixture stale until its own mutation. Validate the original failed run and its permitted write-revocation timestamp in the legacy recovery browser test. - Add the Tailscale Connector page, catalog entry, OAuth return flow, connected-network management, device discovery, and disconnect. Document all six routes in OpenAPI and the API reference, including their instance-admin requirement. - Add company-scoped connection storage in `0480_lazy_lethal_legion`, trusted credential version/digest fields in `0481_round_pet_avengers`, and server-owned secret purpose/backfill in `0482_aberrant_rumiko_fujikawa`. Generate all three migrations through the maintained command and retain the five newest snapshots. - Add machine pairing that generates a fresh SSH key per request, stores its private half in the company vault, and uses the generated public half in the install command. Do not trust editable saved metadata. Preserve existing authorized keys, including a file without a final newline. Keep SSH username, workspace, and trusted host keys explicit. - Use the existing SSH driver. Resolve the machine's current address from its stable device ID. Reject disconnected, removed, offline, and wrong-company selections. - Hide Network when Direct is the only option. Select a sole connected tailnet automatically and hide its selector. Remove the requested setup helper copy. Do not restrict the harness to Claude. - Run a private userspace Tailscale daemon with a separate socket. Persist its node state before execution. Restore it after restart, bound startup, serialize reconnect cleanup, and close it during server shutdown. - Guard OAuth completion with a durable attempt marker. Disconnect and newer authorization invalidate older callbacks. Close replaced daemons and remove cancelled enrollment secrets without changing newer connection state. - Preserve SSH host verification and tailnet-specific session identity. Honor the configured Tailscale executable path, including spaces and OpenSSH percent escaping. - Package checksum-pinned Tailscale clients and their license in the standard runtime image. Exercise the client build stage in PR CI and include it in image smoke checks. ## Verification - Merge status: all CI and image checks pass, and current-head Apex is 5/5 with no unresolved comments. GitHub rejected a direct merge because the active master ruleset requires code-owner approval for `.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3) applies; eligible maintainers already have review requests. Auto-merge is enabled with the squash method for the verified head. The PR remains open pending that required approval; no branch policy was bypassed. - Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI verification passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320). All required checks pass, all applicable test/build gates are green, and the branch is mergeable against master. SSH prerequisite #21248 and Cloud broker paperclipai/paperclip-cloud#1314 are already merged. - [Apex reviewed this exact revision at 5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437). There are no unresolved review threads. The PR is classified as Critical impact. - Final image qualification passes for the [application Runner path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267), [amd64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198), and [arm64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156). Typechecks, Runner support, native/Rust tests, provider tests, serialized-server suites, browser shards, and the remaining applicable CI checks pass. This is source/image CI qualification, not a production rollout. - The last master conflict affected only Docker stage declarations and their boundary test. Both the checksum-pinned Tailscale stage and upstream `runner-lockfile` guard are retained, including the upstream `deps` inheritance. The Tailscale client stage is byte-identical. All 180 cases across five Docker/image contract suites pass locally, along with server and UI typechecks. The maintained 543-input SDK check has no drift. Runtime and lock pins are unchanged. Secret/PII and whitespace scans pass. - Seven OAuth regression cases failed before the fixes. The current-tab test uses the real handoff helper, checks that the session is stored before reauthentication navigation, mounts the shared return page, resumes consent, and verifies handoff cleanup. Callback tests check trusted saved-connection context, fixed safe outcomes, failure audit metadata, denied membership and expired-state rejection, JSON API errors, and no provider text in browser returns. UI retries require a connection in the current company's catalog. All 80 OAuth route/service/UI/shared-handoff cases pass; server and UI source/test typechecks pass. - [Server shard 4/12 on the prior refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436) passed 2,784 cases and skipped 21, then failed a reassignment fixture's `TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its automatic retry found the surviving `REA` prefix. The unchanged test passes all three cases locally. A controlled reader/cleanup overlap reproduces the same deadlock, but the exact competing CI query was not logged. Each case now has a distinct company prefix and identifier; the owned database is removed at suite end. The three test bodies and assertions are byte-identical. Eight local reassignment/lock-release cases pass, and the [final server shard 4/12 passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392). Production reassignment logic, deadlines, and retries are unchanged. - The initial master integration passed 330 cases across 17 suites. Coverage includes connector state, device discovery, network lifecycle, pairing, routes, SSH binding and exports, environment persistence, and connector UI. All three unpublished migrations were regenerated after master's `0479_short_darkstar`: `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL body is byte-identical to its earlier unpublished migration. Generated snapshots preserve existing master tables and add only the intended Tailscale table and secret-purpose column. Exactly five snapshots remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift checks pass. The later Docker refresh did not alter this migration sequence. - Earlier attempts remain part of the verification record. On `1c020a0`, Rust cache restoration and server shard 5/12 were interrupted by shutdown signals, Runner shard 2/3 lost communication with no retrievable log, and an amd64 provider download failed with npm `ECONNRESET`. Same-head retries recovered, without establishing the cause of the communication loss. Later superseded runs also had cancellations with incomplete logs. These interrupted attempts are not counted as passes. The final revision has fresh successful CI. - Local platform limits remain recorded. The Kubernetes suite on macOS passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and skipped four Linux-only cases. Unchanged source reproduced the same 12 failures. An initial run also lacked provider dependencies until the maintained preparation command installed them. Those local attempts are not green qualification. Final Linux provider CI passes. Local Docker qualification was unavailable; final image checks passed in CI. - Before PR separation, a live tailnet/SSH Claude Code task completed on Ubuntu ARM64 without provider login for the remote user. The selected Paperclip subscription ran it, wrote the expected file, posted completion, and marked the task done. File contents and private runtime cleanup were checked. This is prior live evidence; the final revision was not live-retested. Other providers have automated coverage only. Hosted registration for unrelated tailnets and production concurrency remain qualification limits below. - The Superagent security and supply-chain reports remain neutral and inconclusive. Earlier attempts to restart the failed security scan returned HTTP 404. Supply-chain intelligence lacked coverage for extracted pinned dependencies and the shared workflow reference. These reports do not establish successful security qualification. The required CI and review gates pass; this external coverage gap remains disclosed. ## Risks - The Superagent security and supply-chain reports are inconclusive. The security scan re-run request returned HTTP 404, and the supply-chain scanner lacks coverage for the extracted dependencies and shared workflow reference. These neutral results are not successful security qualification. - The server-owned secret purpose is internal database metadata, not an HTTP-writable field. The generic secret service owns enforcement so direct API reads and runtime projection share the same restriction. Previously copied node state remains valid outside Paperclip until the enrolled device is revoked in Tailscale; a local disconnect cannot stop an external copy. - Tailscale network configurations now require a connected Connector and selected machine. The unreleased key-only path is removed; any saved configuration from that path needs Connector setup. - The connector remains a pilot. The local test used an OAuth app registered in the consenting user's tailnet. Qualify hosted app registration for unrelated customer tailnets before general availability. - This implementation supports one serving process per instance. Concurrent replicas need exclusive ownership of the persisted network identity before production rollout. Preserve the database and company vault master key across restarts. - Each machine-pairing preparation creates a new vault secret. Unused generated keys follow the existing secret deletion lifecycle. A saved SSH key remains available as an explicit choice. - Apply additive migrations `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before application rollout. The new credential version and digest columns are nullable. Existing unpinned connections require fresh OAuth consent; the migration deliberately does not trust or backfill a generic secret value. Regenerate or renumber migrations if master advances the sequence before merge. - The target needs an SSH server, its chosen harness, and allowed tailnet access. A reachable network does not grant SSH or provider authorization. Pairing reads standard `/etc/ssh` host-public-key files; custom host-key locations require manual known-hosts configuration. - Roll back the application revision to remove new entry points. Existing connection secrets and registered Tailscale nodes retain their separate lifecycle. Do not drop the table or delete node state as part of an application rollback. - Production qualification still needs denied-policy, revocation, cancellation, and deployment-overlap checks. Local Docker and a production deployment were not performed. ## Model Used OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and this extends existing connector and environment features - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the feature request fields - [x] I have not referenced internal issues, local URLs, or private machine identifiers - [x] My branch name describes the change - [x] I have run focused tests locally and they pass - [x] I have added tests for new behavior and failure paths - [x] I have updated documentation - [x] I have documented migration and runtime risks - [x] All Paperclip CI gates are green - [ ] Required code-owner approval received for the workflow change - [x] Greptile Apex is 5/5 on the final head with no unresolved comments - [x] I will address CI and review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1d112a504f |
feat(acpx): convert raw Claude session updates the way ACPX does (#21420)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The Runner sees each raw ACP message before ACPX does. A cycle that Claude starts by itself is not a part of an ACPX prompt, so ACPX gives no events for it. > - To show that cycle in the same form as a normal turn, the Runner must convert the raw messages itself. ACPX 0.22 does not export its converter. > - This pull request adds a copy of that converter with a test that compares it with ACPX. Nothing uses it yet. > - The benefit is that a later pull request can show the cycle of Claude with the same events as a normal turn. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 2. Its base branch is `refactor/acpx-sidecar-runtime-event-step`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing yet. It adds a module that a later part uses. **Subsystem affected** Paperclip Runner: `src/drivers/acpx`. One new direct dependency in the Runner and the server. **Current behavior** - ACPX converts `session/update` messages to runtime events inside a prompt. The function is private. - The Runner has no way to make the same events from a raw message. **Proposed behavior** - `acp-session-update-event.ts` converts a raw `session/update` to the event that ACPX 0.22 makes for the same update. - It validates the message with `@agentclientprotocol/sdk` 1.9.0. This is the version that ACPX 0.22 already installs. - No production code imports the module in this pull request. **Reason and benefit** A copy with a comparison test is a smaller risk than a patch of ACPX. The test fails when the two differ. ## What Changed - `src/drivers/acpx/acp-session-update-event.ts` (new, 367 lines). - `acp-session-update-event.test.ts` and `acpx-runtime-adapter.claude-session-update-events.test.ts` (new, 363 lines). - `packages/paperclip-runner/package.json` and `server/package.json`: `@agentclientprotocol/sdk` 1.9.0 as a direct dependency. The server needs the line because it copies the Runner build with no `node_modules`. - `pnpm-lock.yaml` is not in this pull request. The lockfile workflow owns it. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - The two new test files and the sidecar suite: 130 passed. - The comparison test runs the real ACPX runtime with the pinned `claude-agent-acp` and a local fake Anthropic API, and gives the same raw updates to the copy. The events are the same. - A second test fails if the SDK version of the Runner is not the version that ACPX resolves. With 1.4.0 in `package.json` it fails. Not run: the full suite. CI supplies that coverage. ## Risks - **A new direct dependency.** It is already installed through ACPX, so no new package is downloaded. The version test keeps the two the same. - **The converter is a copy of ACPX code.** An ACPX upgrade must run the comparison test. - **The module is not used yet.** It is used in part 3. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
055a0a3c54 |
refactor(acpx-sidecar): extract the runtime event step and the test process helper (#21419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The last part of that stack adds a second kind of turn to the sidecar. It needs two pieces of the code of a normal turn. > - This pull request moves those two pieces into functions, and moves a test helper to a file of its own. It changes no behavior. > - The benefit is that the later pull request shows only new behavior, not moved code. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 1. Its base branch is `refactor/remove-direct-gateway-path`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing for a user. The structure of the ACPX sidecar code. **Subsystem affected** Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts` and its test. **Current behavior** - `pumpTurn` has the code that prepares each runtime event for the Runner, and the code that sends the terminal frame of a turn, inline. - The sidecar test file has the `SidecarProcess` helper that starts a sidecar process. **Proposed behavior** - `createRuntimeEventStep` has the event code. `emitTurnTerminal` has the terminal code. `pumpTurn` calls both. - `src/test-support/acpx-sidecar-process.ts` has the helper, so a second test file can use it. - No protocol, event or timing change. **Reason and benefit** A later pull request adds a turn that Claude starts by itself. It uses the same two functions, and its tests use the same helper. ## What Changed - `src/cli/acpx-runtime-sidecar.ts`: 74 lines added and 56 removed. The code moved with its comments, with no change. - `src/cli/acpx-runtime-sidecar.test.ts` and `src/test-support/acpx-sidecar-process.ts`: the helper moved, 138 lines added and 132 removed. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `npx vitest run src/cli/acpx-runtime-sidecar.test.ts`: 125 passed. The tests are the same as before, apart from the import of the helper. Not run: the full suite. CI supplies that coverage. ## Risks - **A refactor of the turn loop of each ACPX agent.** The moved code runs for Claude, Codex, Pi, Kimi, Cursor, Copilot and Grok. The existing sidecar suite covers it. No new test is necessary, because no behavior changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0ead74b4f3 |
feat(slack): show useful progress without extra messages (#20827)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Chat connectors show agent work in the source conversation. > - Slack progress uses a small set of general status messages. > - Native runtimes already mark some completed assistant notes as public progress. > - The server can select those notes and control their size and rate. > - This pull request keeps one progress message and sends the final answer as a new reply. > - Users can follow useful changes without reading each tool event. ## Linked Issues or Issue Description **What existing behavior does this improve?** Automatic progress updates in Slack task threads. **Current behavior** Users see general messages such as “Maya is making progress”. The first answer can replace that message. A source update that arrives during the progress delay can be lost if no later event arrives. **Proposed behavior** Show the latest short public note in one message. Wait 10 seconds before the first update and at least 30 seconds between updates. Recover pending work after a restart. Send the final answer as a fresh reply. **Reason and benefit** Users can see what the agent checks and what it found. The thread stays short. The model needs no new tool call or summary model. Related change: [resolved-card chat bindings](https://github.com/paperclipai/paperclip/pull/20825). This branch includes that master revision. The progress change uses the existing source authority checks. ## What Changed - Select completed `agentMessage` items from the normalized PRP `progress` channel. Keep unknown channels, reasoning, tool output, and raw provider fields out of progress text. - Sanitize each note. Keep at most two sentences and 400 characters. Add a task link when a public board URL is available. - Prefer public notes over general tool status. Remove repeated text and combine bursts into the latest pending note. - Use run events and the publication outbox as saved state. Use the shared deadline worker for the grace period and update delay. A delivery receipt wakes the worker when an earlier update is still in flight. Settled history does not poll. - Apply the run's registered-secret filter before text selection or storage. Reuse the latest usable note within a sweep. Invalidate the cache when the source event or secret registry changes. - Reuse the exact source-bound Slack fast-response receipt when it replaced the working placeholder. Check its current outbound message link before editing it. - Send Slack answers as new replies. After delivery, queue a separate cleanup edit for a generic working message. Keep its current outbound link as the edit authority. Leave useful public notes in place. Cancel pending public progress when an answer, question, or terminal run takes over. Keep the existing delivery policy for other connectors. - Add one short Slack prompt instruction. Update product documentation and regression tests. The Slack module owns public-note selection, cadence, lane eligibility, and cleanup. The shared progress service holds its existing issue and run locks and calls a narrow provider policy. The secret registry supplies redaction and a version token for the per-sweep note cache. The existing outbox owns delivery, retries, and authority checks. The Slack module owns receipt reuse. Shared delivery changes supply the receipt wake and select the Slack reply policy. No schema or configuration change is required. ## Verification - The initial content-selection regression failed before the implementation and passed after it. - The follow-up reproduced two review defects before the fixes. A registered synthetic secret appeared in saved progress. A generic working message stayed active after the answer. Both regressions now pass. - The final content, database-worker, query, protocol-boundary, and redaction selection passes 78 tests across four files. - The earlier affected transport selection passes all 65 cases. It covers public progress, native cadence, separate answers, Slack and Telegram retry isolation, FIFO restart, and cleanup ownership. Its first run passed 64 cases and found one old assertion that rejected the intended cleanup edit. The corrected assertion checks the exact successor message ID. - Full server typecheck passes: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm --filter @paperclipai/server typecheck`. An earlier attempt found two transaction type errors. The helper contracts now accept their actual database operations. The final run passes all source and test layers. - The broad Slack selection stopped after 189 passing cases when the sandbox killed its worker with SIGKILL. That attempt is not a pass. The focused transport run above supplies affected local coverage. Required CI must supply broad coverage on the final head. - `pnpm verify --explain` returns exit 2 because the product documentation change requires broad CI coverage. No full local suite was started. - Preflight installed dependencies and built the SDK. Cargo is absent, so local Rust checks could not run. No Rust source changed. - [CI on `02552bb584`](https://github.com/paperclipai/paperclip/actions/runs/43493597242) passed build, typecheck, policy, all browser shards, Runner/Rust checks, and the other test lanes. One chat shard failed in three Slack cases. Each expected one edit after an answer was already sent. The intended cleanup adds a “Response sent.” edit before the later failure notice. All three failures reproduced locally. The test assertions now check both exact edit texts and the original run's message ID. They also check that another sweep adds no edits. - The latest local run passes all 85 selected transport cases. It includes all 18 Slack/Telegram answer and failure orders, plus progress, deferred FIFO, retry isolation, and cleanup coverage. Command: `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'preserves selected answer and failure lanes|public progress|native progress|keeps intermediate.*agent comments|keeps a.*retry.*queue|FIFO'`. - [Final CI](https://github.com/paperclipai/paperclip/actions/runs/43499797545) passed on `55ec4c053fe5263d9033d807777dbf2e679407e3`. All 52 executed jobs passed. This includes build, typecheck, general tests, serialized tests, all eight browser shards, Runner/Rust checks, and the required aggregates. The affected chat shard now passes. The test-file selection job was skipped because this run used full coverage. - [Greptile](https://github.com/paperclipai/paperclip/runs/132142301124) completed on the same head with 5/5 and no new findings. All four inline review threads are resolved. The review job and Socket checks passed. Of 61 check runs, 57 succeeded, three were skipped, and one was neutral. No checks remain pending or failed. The PR is mergeable with a clean state. - [Superagent Security Scan](https://github.com/paperclipai/paperclip/runs/132142345827) returned a neutral result with “PR scan workflow failed”. It supplied no security verdict. Do not count it as a successful scan. - Live Slack notification and unread behavior has not been tested. Service tests use real PostgreSQL and a fake provider runtime. ## Risks - Only runtimes that map commentary to the explicit public `progress` channel supply contextual prose. Other runtimes retain general status messages. - Slack final answers now create a fresh reply. This is an intended delivery change. - Public commentary must suit the current conversation audience. Text sanitization adds protection; it does not authorize sharing unrelated private material. - Source events and outbox rows supply recovery state. No new timer table or recurring idle scan is added. - The existing large integration suite retains its provider fixtures and lifecycle assertions. No source file crosses the 1,000-line threshold in this change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code tools. The exact deployment ID was not supplied. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
5df1ada9c1 |
feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21335 lets a plugin task provider that restarts the Runner on `resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant run's harness credential > - A provider restarting the Runner also needs the agent's configured environment, which Core sends on `resume` only during a session handoff > - Without it, a Runner a provider moves to a new computer starts without the agent's configured environment variables > - This pull request widens that opt-in to both launch inputs, under a name that says so, before any provider adopts the old name > - The benefit is that a provider can restart a parked Runner on resume with the same credential and environment a `submit` would carry ## Linked Issues or Issue Description Refs #21289 Follows: #21335 **What existing behavior does this improve?** The opt-in resume launch inputs added in #21335. **Subsystem affected** Plugin SDK environment task schema (`describe` capability), the plugin Runner task owner's `resume`, and the plugin environment task docs. **Current behavior** With `resumeCredentialVersion: 1`, `resume` carries `credential`. `configuredEnvironment` is sent on `resume` only for a session handoff. **Proposed behavior** The capability is renamed `resumeLaunchInputsVersion: 1`. With it, `resume` carries `credential` as before and also `configuredEnvironment` (same builder as `submit`) when the provider advertises `configuredEnvironmentVersion: 1`. Session handoff is unchanged. **Reason and benefit** A provider that restarts the Runner on resume gets both launch inputs it needs. **Breaking changes** The capability name changes from `resumeCredentialVersion` to `resumeLaunchInputsVersion`. No provider advertises the old name yet (it merged in #21335 and has no adopter), so nothing changes behavior. ## What Changed - Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the SDK `describe` result schema, executor plumbing, owner input, docs and tests. - With the capability, `resume` also adds `configuredEnvironment` when the provider advertises `configuredEnvironmentVersion: 1`. A normal resume does not mark the variables as delivered, since a live Runner keeps the environment it started with. - Docs: the "Release and resume" section describes both fields; the "Configured environment" section points to it. - The runner-e2e SDK public-input pin is updated. ## Verification - SDK vitest `tests/environment-task-release-resume.test.ts` and `tests/environment-tasks.test.ts`: 68 passed. - Server vitest `plugin-runner-task.test.ts` and `environment-task-runtime.test.ts`: 153 passed, including a normal resume with the capability (credential sent, variables added only with `configuredEnvironmentVersion: 1`, delivered flag unset) and redaction of both fields from metadata writes and sanitized errors. - `native-session-executor.test.ts -t "uses a task-only"`: 5 passed. - runner-e2e `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts`: 46 passed. - `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed. - Not run locally: the full suite and `pnpm verify`; required CI covers them. ## Risks - A provider restarting the Runner gets the claimant run's current variables, which can differ from the ones the parked Runner started with. That is intended. - Rollback: revert this pull request. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs), implemented by a sub-agent and reviewed before opening. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
342fb308bd |
perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - Text deltas (`item.delta`) are about 85% of that process's SQL statements: each delta is one transaction of four statements on the shared 10-connection pool > - At 50 agents and 20 deltas per second, the process falls behind and unrelated agent requests wait for the pool > - This pull request commits text deltas in batches: one transaction for all deltas that arrive within 250 ms, with one multi-row insert > - The benefit is about 7 times fewer transactions and 5–6 times fewer statements per delta, with no delay for the first text of an item ## Linked Issues or Issue Description Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a of the agent scaling plan in #21418. **What existing behavior does this improve?** The persistence of native text deltas. `native-session-runtime` awaited `PaperclipControlPlanePort.appendEvent` for each event, and the port ran one transaction per event. **Subsystem affected** Server run event ingest (`paperclip-control-plane-port.ts`, `heartbeat-run-events.ts`) and the runner package's turn consumer (`native-session-runtime.ts`). **Current behavior** Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In the load model with 50 agents (deltas only), that is about 4.0 statements and 1.0 transaction per delta. **Proposed behavior** The turn consumer starts the next append without waiting for a delta's receipt. The port collects deltas and commits them in one transaction with one insert. In the same load model, that is about 0.7 statements and 0.14 transactions per delta. **Reason and benefit** Fewer transactions free the shared pool and the main thread for other agents' requests. This is the largest per-agent cost in the scaling plan. **Breaking changes** None. The row shape does not change. A port without the new capability keeps the earlier serial behavior. ## What Changed - `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends native source events of one run and one source in one transaction. When the events continue the stored source cursor (from #21435), it sends four statements for the whole batch: the run lock, one replay probe, one `insert ... select from jsonb_array_elements(...)` of every row, and the commit. Any other batch (a replay, a gap, an event with a wake, or a run without a cursor) appends its events one by one in the same transaction, with the single-append checks. - **Storage choice: one row per delta, written by one statement per batch.** I did not store one row per batch. The transcript, live events, run log export, chat progress, replay (`replayEvents`), recovery cursors and the UI all read one row per delta. One statement per batch keeps every reader unchanged, and rows written before this change read the same. - `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An `item.delta` appended with the new `streaming` option goes into a queue: - It commits at once until a delta with text commits after the latest other event. So the first text of each item, and the time to first token, never wait for the timer. - Later deltas commit after at most 250 ms, or at once when the batch has 100 deltas. - Every other event, `checkpointSession`, `completeRun`, `replayEvents` and `provesNoRunProviderWork` first commit the queue. A non-streaming event is never committed before the deltas that preceded it. - A delta appended without `streaming` commits at once, so callers that wait for each receipt are not slowed. - The receipt of a delta settles only after its batch commits and its observers (live event, executor callback) run, in order. - If a batch fails, every delta in it rejects and nothing from it is stored. - `native-session-runtime` `consumeTurn` appends deltas with `streaming` and does not wait for their receipts (at most 256 pending). Before any other event, any result, and the governed-wait settlement, it calls `flushStreamingEvents()` and then waits for all pending receipts, so a boundary never waits for the timer. A failed delta stops the turn before a later event is appended. The consumer joins every pending delta append before it settles, also when the provider stream fails. - If one observer of a committed batch fails, only that delta's receipt rejects. The other rows are still published and observed. - `ControlPlanePort` contract: optional `pipelinesStreamingEvents`, `flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`. ## Verification - Red/green. The new tests failed before the change and pass after it: - Runner: "starts text delta appends without waiting and flushes them before any other event" (`native-session-runtime.test.ts`). The port in the test has no timer, so the turn finishes only if the boundary flushes. Before: the appends were serial, and later the boundary waited for the timer. - Runner: "joins pending delta appends before a failed turn settles". Before: the turn settled while a delta commit still ran. - Port: "observes every committed delta of a batch when one observer fails". - Port: "commits the first text delta at once and later deltas in one batch before the next other event". It checks one transaction and one insert for five deltas, flush before a non-streaming event, receipts only after commit, and immediate commit of the first delta with text. Before: each delta committed alone. - Port: "flushes queued deltas on the timer and before a replay read, and rejects every delta of a failed batch". It checks the crash-before-commit case: a revoked writer fails the batch, nothing is stored, and the resent deltas commit idempotently (the stored one is a duplicate). - Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`: one lock, one probe and one insert per batch; replay idempotency and payload conflict; a batch that fails before commit stores nothing and its resend commits; the one-by-one path for no cursor and for gaps. - Focused suites pass: `heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`, `paperclip-control-plane-port` (including the package conformance suite), `paperclip-semantic-conformance`, `native-observer-replay`, `native-event-boundary-golden`, `live-recovery-work`, `native-interaction-bridge`, `chat-native-progress-work`, and `native-session-runtime.test.ts` in the runner package. - `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript` and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm check:module-boundaries` passes. - Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418, `AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls the store directly, so for this run I added a local `BATCH_MS` mode that applies the port's batching rules (first delta at once, 250 ms timer, 100 cap, flush before other events) and does not wait for delta receipts. That mode is not in this pull request. The machine was very busy (load average 40–60), so absolute figures are noisy; per-delta ratios are the stable result: | Deltas only | Statements/s | Transactions/s | Deltas done of offered | Statements per delta | Transactions per delta | Lateness p50 / p99 | | --- | --- | --- | --- | --- | --- | --- | | One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 | 1.00 | 3,074 / 5,384 ms | | Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 / 2,858 ms | | One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 | 3.99 | 1.00 | 6,558 / 11,464 ms | | Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17 | 187 / 556 ms | With all sources on, the machine was saturated (event-loop utilization 0.96–0.98) in both modes and the results were not usable (764 and 12,800 of 20,000 deltas done). ## Risks - **Ack and crash (read first).** In the live runnerd path the runner's ack does not wait for the `heartbeat_run_events` row today. `DurablePrpControlPlane.#event` acks after the in-memory journal commit. Streaming events are not in the controller checkpoint. So a delta that is acked but not yet appended is lost if the server crashes, and the runner does not resend it. This is the earlier behavior. This pull request makes that window up to 250 ms longer. The final text still survives through `item.completed`, which commits immediately and in order. This pull request does not change `durable-prp-control-plane.ts`. - Live text in the UI arrives in steps of up to 250 ms after the first text of an item. - A delta failure now surfaces at the next non-delta event or at the end of the turn, not at the delta. The turn stops before it appends a later event. - The governed-wait drain and the control-event replay still append one event at a time. They do not carry deltas. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0aaab644fe |
refactor(runner): remove the direct assigned-gateway path (#21311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent reaches the tools of its app connections through an MCP gateway. > - Before #21274, some native harnesses got the gateway URL and a bearer token and connected to the gateway directly. The Runner has code for that: an HTTP MCP server for ACPX providers, a server entry in the Codex `config.toml`, a server entry in the OpenCode config, and the lists that pass the three `PAPERCLIP_NATIVE_MCP_*` variables to a sidecar. > - #21274 gives each native harness its gateway tools through the PRP relay. Core removes the three variables from each Runner environment. The direct code path gets no binding. > - Code that cannot run still needs maintenance, and it can still put a token in a provider process if a variable arrives by another way. > - This pull request deletes the direct path from the Runner. > - The benefit is one path for assigned gateway tools, and no Runner code that can give a gateway token to a provider. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is in a stack. Its base branch is `fix/keep-quiet-claude-provider-between-turns` (#21293). It depends on #21274, which is the bottom of the stack. On `master` without #21274, the direct path is still in use. **What existing behavior does this improve?** The Runner code that connects a provider directly to the assigned MCP gateway. **Subsystem affected** Paperclip Runner: the TypeScript drivers and the Rust `runner-core` environment lists. **Current behavior** After #21274, Core never gives `PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` or `PAPERCLIP_NATIVE_MCP_TOKEN` to a Runner. The Runner still has the code that reads them and configures a direct gateway server for ACPX providers, Codex and OpenCode. **Proposed behavior** The Runner has no code that reads the three variables to configure a provider. If the variables are in the environment of a Runner, no provider environment, sidecar, Codex `config.toml` or OpenCode config gets them. **Reason and benefit** Less code, one path, and no way for a gateway token to reach a provider process through the Runner. ## What Changed - Deleted `packages/paperclip-runner/src/drivers/native-mcp.ts` (`nativeMcpLaunchBinding`). - `drivers/acpx/runtime-host.ts`: removed the native gateway server entry and the name conflict check. The admission gate keeps its message. A run with a binding id is admitted when the sidecar uses the PRP transport with its tool bridge, as the production sidecar always does. - `drivers/acpx/environment.ts`: the name and the URL are not in the provider allowlist. - `drivers/runtime-context-materializer.ts` and `live/runnerd-codex-transport.ts`: the Codex home functions have no `nativeMcp` parameter. The sidecar token pass-through is gone. - `drivers/opencode/opencode-server-driver.ts`: removed the read of the binding, its trace redaction value, and the gateway server entry in the OpenCode config. - `drivers/stdio-mcp.ts` and `runner-core/src/stdio_mcp.rs`: removed the reservation of the gateway server name. No gateway server exists, so no name can conflict. The names `paperclip` and `paperclip_runner_*` stay reserved. - `runner-core/src/acpx_sidecar_transport.rs` and `codex_provider.rs`: the three variables are not in the pass-through lists. - `control-plane/durable-prp-control-plane.ts`: the three variables are not in the platform environment list. - `packages/paperclip-runner/README.md`: the text about direct callers and assigned servers. - Kept: the three names stay in the reserved lists (`src/configured-environment.ts`, `runner-core/src/configured_environment.rs`). A user cannot configure them. Tests in both languages now check all three names. - No server code changed. The relay needs the gateway token and URL on the control plane. 23 files, 73 lines added and 211 lines removed. ## State saved by the previous release - This pull request changes no persisted shape. The provider server lists, the launch profile digests, the protocol versions and the sidecar contract are the same as with #21274. - A Runner or sidecar that the previous release started keeps its own code until its run ends, as #21274 describes. - A kept Codex home that has the old gateway server in `config.toml` is written again at the next start and at a recovery of the same run. A test covers this. - If the old variables reach a new Runner, no provider gets them. Tests cover this for ACPX, Codex and OpenCode. ## Verification From `packages/paperclip-runner`: - `npx vitest run src/drivers/acpx/runtime-host.test.ts src/live/runnerd-codex-transport.test.ts src/drivers/acpx/environment.test.ts src/drivers/opencode/opencode-server-driver.test.ts src/live/runnerd-codex-transport-warm-attachment.test.ts src/drivers/runtime-context-materializer.test.ts src/drivers/agent-observability.test.ts src/drivers/stdio-mcp.test.ts src/control-plane/durable-prp-control-plane.test.ts src/configured-environment.test.ts`: `10 passed (10)` files, `807 passed` tests. - `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: no errors. - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no changes. - `cargo test -p paperclip-runner-core --test acpx_sidecar_transport`: 20 passed. `cargo test -p paperclip-runner-core --lib`: 639 passed, 4 ignored. Tests that failed with the old code and pass now: - `runtime-host.test.ts`: "admits a relayed … run with a binding id and ignores a previous-release gateway binding", for Claude, Codex and Cursor. - `runnerd-codex-transport.test.ts`: "drops a previous-release gateway binding from the … runner environment", for ACPX Claude, Codex, Pi, Cursor, Copilot and Kimi, and for OpenCode. - `opencode-server-driver.test.ts`: the OpenCode config has only the task bridge. - `runnerd-codex-transport-warm-attachment.test.ts`: a restored Codex home has no gateway server and no gateway credential in `config.toml`. - `environment.test.ts`: a provider environment has none of the three variables. - Rust `previous_release_gateway_binding_and_unrelated_secrets_never_reach_the_sidecar`: a Runner that starts with the old variables gives none of them to the sidecar, for five agents. Mutation check: with the three names put back in the Rust pass-through list, the Rust test fails. Not run: a live run with a real provider. Not run: `pnpm verify` and the full suite. CI supplies that coverage. ## Risks - **This pull request is safe only after #21274.** The stack order keeps that. - **A developer who sets the three variables by hand for `eval-session`** no longer gets direct gateway tools. No document describes that use. - **The warm attachment test uses the staged Runner binary from preflight.** That binary does not have the Rust changes of this pull request. The Codex path of that test does not use the changed lists. - The evidence that no entry point supplies a binding comes from a read of the code: Core removes the variables before it builds each Runner environment, and a native run cannot use the in-process path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff, and ran the verification and a mutation check again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), traced each code path and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
12aa08721c |
feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.
Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on
|
||
|
|
2a8b485140 |
fix(runner): accept object fields sent as JSON text and name wrong-type rejections (#21485)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip runner checks each tool call from the model against the tool's input schema before it sends the call to the control plane > - Some model hosts send an object argument as its JSON text. On the DigitalOcean host, `paperclip_finish` arrived with `continuation` as a string > - The runner rejected these calls, and the repeated-rejection guard then told the model that its arguments "did not arrive", which is wrong when every field arrived with the wrong type > - This pull request makes the runner parse such a string back into its object when that is safe, records each coercion as a run event, and makes the guard name the actual problem > - The benefit is fewer stuck finish calls, and guidance that points the model at the real fault ## Linked Issues or Issue Description Refs #21398 **What happened?** Chat-eval cohort 13 ran OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`. On the DigitalOcean host, 6 of the 8 remaining "Invalid tool input" rejections were `paperclip_finish` calls with `continuation` sent as a JSON string (`/continuation must be object,null`). Examples: `gateways-profiles.rotate-token`, `instance-operations.restart-service`, and `questions-and-approvals.multi-part-question`. The run diagnostic recorded `continuationShape: "string"`. After the second identical rejection, the guard from #21398 said "If you meant to send other arguments, they did not arrive". Every argument had arrived; one had the wrong type. **Expected behavior** When a schema field is object-typed, does not allow text, and the text parses to a plain JSON object that makes the call valid, the runner accepts the call with the parsed object and logs a run event. When a call stays invalid, the guard says which required fields are missing and which fields have the wrong type. **Steps to reproduce** 1. Start the OpenCode MCP bridge. 2. Call `paperclip_finish` with `reportedWorkDisposition: "yielded"` and `continuation` as `JSON.stringify({ kind: "response_wake", ... })`. 3. Before this change, the bridge returns `Invalid tool input: /continuation must be object,null`. ## What Changed - Added `packages/paperclip-runner/src/drivers/tool-input-coercion.ts`. `admitToolInput` validates the arguments. If they are invalid, it parses a string in an object-typed field back into its object. It does this only when the field's schema declares `object`, does not allow `string`, the text parses to a plain JSON object, and the whole coerced call validates. It applies to any object-typed field at any depth, not only `continuation`. - The OpenCode MCP bridge and the runner tool bridge use `admitToolInput`. They dispatch the coerced arguments, and they send one `paperclip.runner.tool-input-coerced/v1` diagnostic per accepted coercion. The OpenCode driver already records bridge diagnostics as run events. The diagnostic names the tool and the field paths. It never contains the values. The runner tool bridge has a new optional `onDiagnostic` option. - When parsed text is still invalid, the rejection describes what is wrong inside the parsed object, for example `/options/mode must be string`. - The repeated-rejection guard now says "Required fields X did not arrive" only for missing required fields. For fields with the wrong type, it says "Every field you sent arrived, but one has the wrong type: continuation must be an object or null, not text". ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/mcp-bridge.test.ts src/drivers/runner-tool-bridge.test.ts` (63 tests pass). - Red/green: with the source changes reverted and the new tests kept, 5 tests fail: coercion acceptance and its run event (both bridges), no coercion for text fields or non-object text, the wrong-type guard wording, and the missing-field guard wording. With the change, all of them pass. - `npx tsc --noEmit -p packages/paperclip-runner` passes. - A narrow paid run of `questions-and-approvals.multi-part-question`, `runs-adapters.inspect-failed-run` and `budgets-costs.explain-hard-stop` ran on a branch that combines this PR with the two sibling cohort-13 PRs. The results are in a PR comment. ## Risks - Low risk. Coercion runs only after a call fails validation, and the call is accepted only if the coerced form validates. A field that allows text is never parsed, so a valid string never changes meaning. - Previous-release story (upgrade-safe run state): this change adds no persisted run, session, launch or pack state. It changes only how one in-flight tool call is admitted. Saved state carries forward unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0ebe6d846d |
perf(runner): keep a quiet Claude provider between turns of a run (#21293)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. The Runner stops and starts the sidecar and Claude at each turn boundary. > - The reason was that a late tool call from an earlier turn could run in a later turn. #21278 removes that reason for Claude: the sidecar refuses a call that the active turn did not announce. > - A second reason remains. Claude Code works by itself between prompts. When a background task finishes, Claude starts a cycle of its own. If the process stayed alive with a live background task, that cycle could start between turns or mix with the next turn. > - So the Runner can keep the process only when it knows that Claude has no background work. > - This pull request adds that knowledge and uses it. After a Claude turn that completed with no background work, the Runner keeps the process for the next turn of the same run. > - The benefit is that a later turn of the same run starts with no provider restart. This is also the base for the next step, which lets a run continue when background work finishes. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is stacked on #21278. Its base branch is `fix/bind-claude-tool-calls-to-turn`. Refs #19912 (design), #12419 (restart at turn start), #20145 (restart at park) and #20190 (restart at turn end). **What existing behavior does this improve?** The provider restart between turns of the same run, for Claude through ACPX. **Subsystem affected** Paperclip Runner: the ACPX sidecar (TypeScript) and the ACPX provider session in `runner-core` (Rust). Claude only. **Current behavior** - When a completed turn settles, the Runner restarts the sidecar and Claude (`restart_session_after_turn`, #20190). - If that restart did not run, the next turn restarts them at its start (`start_turn`). - The Runner does not know if Claude has background tasks. **Proposed behavior** - The sidecar asks `claude-agent-acp` for its async task reports. This is an optional capability of the pinned `claude-agent-acp` 0.73.0. The patch does not change. - A tracker reads those reports and the id of each background Bash command. It reports `{ live: 0 }` only when it knows that this process never had a background task and that Claude did not act between prompts. In all other cases the state is unknown. - The sidecar adds `backgroundTasks` to the terminal event of a completed Claude turn, and to `session.snapshot`. - The Runner keeps the process between two turns only when all of these are true: - the agent is Claude; - the turn completed (not failed, cancelled or interrupted); - the terminal event reported `backgroundTasks: { "live": 0 }`; - each existing condition of the turn-end restart is true; - before the next turn, a new `session.snapshot` reports the same identity and run, no active turn, no pending work, and `{ "live": 0 }`; - the settled-turn ledger and the tool replay ledgers are not full. - In each other case the Runner restarts as before. - A kept process keeps its tool call-id tombstones and its settled turn identities. **Reason and benefit** - A turn that follows a quiet turn in the same run starts without a restart. - The decision is safe by default: unknown state means a restart. ## What Changed TypeScript (`packages/paperclip-runner/src`): - `drivers/acpx/claude-background-tasks.ts` (new): `ClaudeBackgroundTaskTracker`. It reads raw ACP messages. It reports unknown when the capability is not negotiated, when a task finished, when a live task is not a shell command, when an update is malformed, when a limit is reached, when Claude acts with no client request in flight, or when Claude calls `CronCreate` or `ScheduleWakeup`. - `drivers/acpx/provider-failure.ts` and `acpx-runtime-adapter.ts`: Claude sessions also advertise the `asyncTasks` capability. The raw ACP observer gives each message to the tracker. ACPX 0.22 drops the `async_task_*` updates before the turn event stream, so only the raw observer sees them. - `drivers/acpx/runtime-host.ts`: `backgroundTaskState()`. - `cli/acpx-runtime-sidecar.ts`: the terminal event of a completed Claude turn carries `backgroundTasks` in the same payload as `goalAtTerminal`, with the same size limit. `session.snapshot` carries it when no turn is active. - `drivers/acpx/claude-tool-use-admission.ts`: `hasRoomForAnotherTurn()`. When the admitted-id ledger cannot hold one more full turn, the sidecar reports unknown, so the Runner restarts before the ledger is full. Rust (`packages/paperclip-runner/runner/crates/runner-core`): - `acpx_provider_session.rs`: the session records a quiet turn from the terminal event. `start_turn` continues in the same process only under the conditions above, and then uses `prepare_turn`, which keeps the tombstones. - `acpx_provider_generation.rs`: `can_continue_after_turn`, `continue_after_quiet_turn`, and the snapshot check `background_work_is_quiet`. - `acpx_park_preparation.rs`: the turn-end restart does not run for a quiet Claude turn. - `acpx_provider_backend.rs`: `session.reconciled` reports a replaced process only when the turn start replaced it. - `bin/fake-acpx-sidecar.rs`: new option `--background-tasks` for tests. - `runner/DURABLE_TRANSPORT.md`: the rule for a kept sidecar. Contract (`packages/paperclip-runner/protocol`): - The `backgroundTasks` report is `$defs.backgroundTasks` in `provider-schemas/acpx-sidecar.schema.json`: an object with only `live`, a non-negative integer. The schema validates it on `runtime.turn_terminal`. - `scripts/generate-acpx-sidecar-contract.mjs` generates the TypeScript type and reader and the Rust reader from it. The tracker, the sidecar and the Rust session use the generated code. - `fixtures/provider-boundary/acpx-background-tasks.json` has shared test documents. The TypeScript reader, the Rust reader and the schema are tested with the same accepted and refused shapes. - This changes the contract digest of the provider pack (`prp.acpx-sidecar` and `rust`). Older image packs are not reused. No provider `commandDigest` and no profile version changes. Not changed: park and run attachment. A process that ran a turn is still restarted at park, and only a process that ran no turn is bound to the next run. Other ACPX agents and Codex do not change. The `claude-agent-acp` patch does not change. ## State saved by the previous release - This pull request adds no persisted state. The three new fields of the session are in memory only. - A terminal event or a snapshot without `backgroundTasks` means unknown. An older sidecar therefore gets the restart, as before. - An older Runner with a new pack ignores the field and restarts at each turn, as before. - After a Runner restart, the in-memory state is gone. The next turn restarts the provider, as before. A test covers this. ## Verification From `packages/paperclip-runner`: - `npx vitest run src/drivers/acpx/claude-background-tasks.test.ts src/drivers/acpx/acpx-runtime-adapter.claude-background-tasks.test.ts src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts src/drivers/acpx/claude-tool-use-admission.test.ts src/cli/acpx-runtime-sidecar.test.ts src/drivers/acpx/acpx-runtime-adapter.test.ts src/drivers/acpx/runtime-host.test.ts`: `7 passed (7)` files, `363 passed` tests. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json`: no errors. - `pnpm run typecheck:typescript` (it includes the contract checks): no errors. `node --test test/acpx-sidecar-contract.test.mjs`: 17 passed. - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no changes. - `cargo test -p paperclip-runner-core --test acpx_quiet_turn_continuation --test acpx_park_preparation --test acpx_park_readiness --test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment --test native_provider_backend`: 4, 33, 8, 31, 12, 6 and 47 passed. No test failed. - `cargo test -p paperclip-runner-core --lib`: 637 passed, 4 ignored. - Mutation check: when the Rust code accepts each live count as quiet, two tests in `acpx_park_preparation` fail. New tests: - **Real ACP runtime** (`acpx-runtime-adapter.claude-background-tasks.test.ts`), with the pinned `claude-agent-acp`, the pinned Claude Code client and a local fake Anthropic API. A foreground Bash turn reports `{ live: 0 }`. A background Bash in the same process reports `{ live: 1 }` at the terminal event, and unknown after Claude ran its own cycle for the finished task. Each turn has its own usage receipt. One process serves all turns. - **Tracker unit tests** (13): spawn, completion, failure, stop, pause, a task that is not a shell, a subagent, no negotiation, malformed updates, activity between prompts, scheduling tools, session load replay, limits. - **Sidecar tests**: the terminal event carries the state for Claude and not for Codex. A full ledger reports unknown. - **Rust** `tests/acpx_quiet_turn_continuation.rs` (4) and `tests/support/acpx_quiet_turn_restart.rs` (7): - a quiet Claude turn continues in the same process, and a replay of a settled call id is refused; - live, unknown and malformed reports restart; - background work that appears after the terminal event restarts at turn start; - failed, cancelled and interrupted turns restart; - Codex with a quiet report restarts; - an active goal restarts; - 4,096 kept turns, then the next turn restarts; - park after a kept process still prepares a new process, and the next run attaches to it; - a Runner restart after a kept process resumes in a new process. - The existing restart tests did not change. The fake sidecar sends no report by default, so they are the tests for the unknown case. Live run with a real Claude account, not in CI: the real ACP runtime (ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code client), a real tool bridge, and the real Anthropic API with model `claude-sonnet-5-5`, on the code at the top of this stack. A turn that started one background shell reported `{ live: 1 }`. A turn that started two reported `{ live: 2 }`, then `{ live: 1 }` and `{ live: 0 }` as each shell finished. A turn in which Claude waited for its background shell in the foreground reported the unknown state. Not run: server integration tests (the fake sidecar there sends no report, so their process counts do not change), Linux, and a live run through the real sidecar process and runnerd. ## Risks - **All Claude sessions now negotiate the `asyncTasks` capability.** `claude-agent-acp` then sends `async_task_*` updates and adds a marker to a background Bash update. ACPX drops the updates. No other behavior of `claude-agent-acp` changes, from a read of its code. - **Park after a kept process pays the restart again**, as it did before #20190. With real Claude that restart was about 1.5 s. - **A small window remains** between the snapshot and the prompt of the next turn. Claude can start a cycle of its own in that window only without a background task, because a process that had a background task is not kept. - **Activity between prompts is found only when no client request is in flight.** A request that never gets a response would hide such activity. - **Background watchers cannot be told apart from other tasks**, so each task that is not a shell command makes the state unknown. - **Each kept turn adds one `session.snapshot` request.** - A multi-turn run with real Claude now uses one provider process. This path has tests with the fake sidecar and with the real ACP runtime. It has no test with the real sidecar host. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff, and ran the TypeScript and Rust verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), investigated the signals and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
969cf246ae |
perf(run-events): keep each native source's cursor on the run row (#21435)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - Each native source event append counted all retained events of its source (`count(*)`, `max(source_seq)`) to compute the acknowledgement cursor > - So the cost of each append grew with the length of the run, and long runs made every text delta slower > - This pull request keeps the gap-free cursor of each source on the run row and moves it in the same UPDATE that reserves the event sequence > - The benefit is that an in-order append stops reading other events, so its cost does not grow with run length ## Linked Issues or Issue Description Refs #21131 (earlier reduction of statements per event). Item 1b of the agent scaling plan in #21418. **What existing behavior does this improve?** The append of a native source event (`appendNativeSourceEvent` in `server/src/services/heartbeat-run-events.ts`). It ran a `retained` CTE that counted every event of the source on every append. **Subsystem affected** Server run event ingest (`heartbeat_run_events`, `heartbeat_runs`). **Current behavior** On a run that already holds 50,000 events, one in-order append takes about 10–13 ms (p50) on a local PostgreSQL. On a run with 400 events it takes about 3.3 ms. **Proposed behavior** The append reads the stored cursor in the run lock statement that it already sends. An in-order event moves the cursor and does not count events. On the same 50,000-event run an append takes about 1.7–3.3 ms (p50). **Reason and benefit** Text deltas are about 85% of the server's statements in the load model. Their cost must not grow with run length. **Breaking changes** None. The new column is nullable. The previous release ignores it. ## What Changed - Add the nullable column `heartbeat_runs.native_source_cursors` (`jsonb`, `{"<source instance id>": <seq>}`). Migration `0479` is `ADD COLUMN` only and classifies as `online_compatible`. - The run lock read of a native append also reads the cursor of its source. - When the event sequence is at most cursor + 1, the append statement has no `retained` CTE. It moves the cursor in the UPDATE that reserves the sequence, and the receipt uses the cursor. - Every other case (no cursor, a gap, or a cursor behind the stored events) runs the earlier recount. In the same UPDATE it stores the cursor when the source has no gap, and removes it when the source has a gap. So the event that fills a gap recounts. - The cursor is a lower bound. A writer that does not keep it (the previous release during a deploy overlap, or `NativeRunCoordinatorStore`) can only make it lag. The next out-of-order event then recounts and corrects it. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-event-atomic-append.test.ts src/__tests__/heartbeat-run-event-sequencing.test.ts` passes (19 tests). - Red/green: before the change, 3 tests failed: the two new tests and the updated lock-read statement test. After the change, all pass. - New: "keeps the source cursor on the run so an in-order append does not count the run's events" checks that the append statement has no `count(` and that a replay returns the cursor. - New: "recounts a source whose run has no cursor or a cursor behind its stored events" covers rows saved before the change, rows from a writer that does not keep the cursor, a gap, and the event that fills the gap. - Focused related tests: `paperclip-control-plane-port.test.ts`, `native-observer-replay`, `native-event-boundary-golden`, `chat-native-progress-work`, `live-recovery-work`, `native-interaction-bridge`. - `pnpm check:module-boundaries` passes. `pnpm verify` returned exit 2 (broader CI coverage required) because of local benchmark scripts that are not in this pull request. Required CI supplies the broad coverage. - Append cost on a run with many events (local embedded PostgreSQL, 1,000 in-order appends, busy machine, load average about 20–40): | Events already in run | Before p50 / p90 | After p50 / p90 | | --- | --- | --- | | 50,000 (run 1) | 10.4 / 15.8 ms | 1.7 / 3.7 ms | | 50,000 (run 2) | 12.9 / 21.6 ms | 3.3 / 5.9 ms | | 400 | 3.3 / 6.2 ms | 3.7 / 7.7 ms (noise) | - Statements per event do not change: BEGIN, run lock read, one append statement, COMMIT (4). The append statement no longer scans the source's events. - Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418, `AGENTS=50 SECONDS=20`): the machine was too busy for a stable result. Deltas only: before 20,000 of 20,000 done at 3,999 statements/s. After: 13,796 and 14,817 of 20,000 in two runs. All sources: before 6,346 of 20,000 deltas done. After: 20,000 and 2,083 in two runs. A 20-second window gives only 400 events per run, so it cannot show this change. The long-run figures above are the evidence. ## Risks - Low. The column is nullable and the previous release does not read it. Runs that are in flight during a deploy have no cursor. Their first append recounts once and stores the cursor. - A stale cursor can only make the acknowledgement lag, never pass the stored events. Event rows are deleted only with their company or agent. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |