mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
b1f608296ef6cf8a5b77f4c42cc9a29a204ea499
8001
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b1f608296e |
feat(runner): keep a run open for a limited time while the Runner waits for the provider's own turn (#21454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - `consumeTurn` reads the events of one turn and returns at the terminal. It already continues past a terminal for a Codex goal. > - This pull request makes it continue past a terminal that waits for a turn of the provider, and it limits that wait. > - The benefit is that a run continues in the turn that Claude starts, and a wait always ends. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 2. Its base branch is `feat/core-provider-turn-driver`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the consumer of a native session does after a turn that ended with no result while the Runner waits. **Subsystem affected** Paperclip Runner package, the controller side: `src/native-session-runtime.ts` and a new module `src/native-provider-turn-wait.ts`. **Current behavior** - `consumeTurn` returns at the first terminal. With no result, the run is finalized as `native_finalization_missing`. **Proposed behavior** - At a terminal with `awaitingProviderTurn` and no result, `consumeTurn` continues to read. It consumes the provider-started turn as the next turn of the same run. A turn that reports a result finalizes as a normal run. - The wait has a limit of 10 minutes from the first terminal that waits. A later terminal that waits does not start the clock again. The limit is never later than 5 seconds before the timeout of the run. - The limit and an early end stop the wait only. They never interrupt a provider-started turn that is running; that turn stays under the timeout of the run. If that turn ends and the Runner waits again, the run takes the path of today at once. - When the wait ends with no turn active, the consumer returns the last terminal, and the existing code for a run with no result decides (`resolveMissingResult`, then `native_finalization_missing`). No new failure code. - The wait ends at once when the driver gives the acknowledgement of part 1. This covers Stop, the limit, and a Runner that says the wait is over. - `executeNativeSession` gets one new option, `providerTurnWait`: the start time of a wait that an earlier controller process saw, and a hook that ends the wait early. - When the wait ends a turn, the consumer first waits for the receipts of the text deltas that are committed in batches (#21460), as each other return path of `consumeTurn` does. Without this, a delta that arrived during the wait was not in the event count of the turn, and a delta that failed to commit was not reported. - When the interrupt that ends the wait is not available or fails, the limit forces the end of the wait. If that forced end starts a new turn (the continuation that a restart owes), the consumer continues to read that turn. The forced-end signal is used one time only, and the read that is open at that moment is kept until it gets its event. Before this fix, the consumer returned the terminal of the old turn and lost the first event of the new turn, and the run could end as `native_finalization_missing`. - If the controller restarts during the wait, the run continues to wait. When the saved checkpoint still shows the first turn as active, the recovered session gives the waiting terminal again, and the wait starts again with the start time that was recorded. When the checkpoint was saved after the waiting terminal, the consumer finds that terminal in the journal, starts the wait again, and does not send the disposition-only recovery prompt. The consumer does this only if two conditions are true: the recovered session has the waiting state (`awaitingProviderTurn`, saved by the driver since #21453), and the terminal in the journal has the marker. If one condition is false, the run uses the recovery path of today. When a wait that started again ends with no turn, the existing code for a run with no result decides, as for each other wait. **Reason and benefit** - The 10 minutes are the maximum time of a foreground command in the pinned Claude Code. - A limit on the wait, not on the work, means that a long test run in the turn of Claude is not cut. ## What Changed - `src/native-provider-turn-wait.ts` (new, 129 lines) and `src/native-session-runtime.ts` (58 lines added, 4 removed), `src/index.ts` (1 export). - `src/native-session-runtime.provider-turn.test.ts`: written again for the new behavior (557 lines added, 61 removed). ## State saved by the previous release - This part adds no persisted state. The wait is in memory. Part 3 reads its start time from an event row that is already saved. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - `src/native-session-runtime.provider-turn.test.ts`: 30 tests. They cover: a controller restart with the first turn still active in the checkpoint (the wait continues; a provider-started turn settles its result; with no turn the limit ends the wait on the recorded clock); a controller restart with the checkpoint saved after the waiting terminal (the same two results, no recovery prompt, no event of the journal appended again); a limit that passed before the restart ends the wait at once, with and without an interrupt; the recovery prompt of today when the terminal in the journal has no marker, when the journal has no terminal, and when the recovered session has no waiting state; a second restart during the wait; a forced end that starts a new turn settles the result of the new turn, with each event of that turn appended one time and in order; an event that reaches the open read after the forced end is appended exactly one time; a delta that arrives during the wait is counted when the limit ends the turn (the test fails when the new wait for receipts is removed); continue and settle the result of the provider-started turn; a second wait does not start the clock again; the limit with no turn active; the limit and a comment during a running turn (not interrupted); a running turn that ends and waits again after the limit; Stop; a `run.terminal` that the Runner held; the Runner stream ends during the wait; the limit under the timeout of the run; a start time from an earlier controller; a plan that a provider-started turn proposes settles as the plan review wait. - `src/native-session-runtime.test.ts`, not changed by this PR: 297 passed. - Mutation checks, each made tests fail: no continue at a waiting terminal; no limit; no early end; no clock from an earlier controller; the clock starts again at each wait; a running turn is interrupted at the limit. - Mutation checks for the forced end, each made the two new tests fail: the forced-end signal used on each loop (the run ends as `native_finalization_missing`); a new read started after the forced end (the first event of the new turn is lost). - Mutation checks for the restart, each made named tests fail: the recovery prompt not held back; the wait not started again in `consumeTurn`; the wait started again on the session state alone (the test with no marker in the journal fails). Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No terminal has the marker before part 4. - **`consumeTurn` runs for each native turn of each agent.** It makes one small object for each turn and reads one more payload field at a terminal. - **If the interrupt that ends the wait is not available or fails,** the limit forces the end: the consumer returns the last terminal, or continues to read when the forced end starts a new turn. This is a fallback path. On a forced return, one read of the event stream stays open until the session closes, as before this PR. - **A wait that starts again after a restart holds back the recovery prompt.** This is the one change to the recovery flow. It needs the two conditions above, and both exist only for a run that asked for provider-started turns. - **Known limit.** The driver does not clear its waiting state when the consumer ends a wait with an interrupt. A restart in the short time after that can start the wait again; the recorded clock and the comment check then end it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3c15acd3fc |
perf(recovery): look up held runs by key so the fresh-continuation deadline stops scanning run history (#21568)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off, the fresh-continuation fallback continues the
task in a new run. The execution-disposition worker schedules its next
pass with `nextFreshContinuationCheckAt` and lists due work with
`heldExecutionsAwaitingFreshContinuation`
> - Both queries share two predicates (`heldExecutionWhere` and
`recentInterruptionSourceSql`). These predicates matched the run as
`source.id::text = <evidence text>` and the task's runs as
`native_issue_id::text = <issue id>::text`. A cast on an indexed column
prevents an index lookup
> - So for each held record, PostgreSQL read the company's whole run
history (two times), and it decoded each run's wide context JSON. On
staging, the deadline query took about 13.5 s of database time in each
27 s worker cycle. It was the largest statement by time
> - This pull request compares the uuid columns directly. It turns the
recorded run id into a uuid only when the text is the canonical
lowercase form, which is exactly when `id::text = text` can be true
> - The benefit is that each held record costs a few index lookups. The
same records are selected, and the worker cycle no longer keeps a
database session busy
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The fresh-continuation scheduler queries: `nextFreshContinuationCheckAt`
(the disposition worker's deadline),
`heldExecutionsAwaitingFreshContinuation` (the backlog listing, also
probed by the schedule), and the "left alone" count. They are in
`server/src/services/fresh-continuation-fallback.ts`.
**Subsystem affected**
Server: recovery and the execution-disposition worker
(`executionControlSchedule().disposition` in
`execution-control-work.ts`).
**Current behavior**
Neon statement statistics on staging (window of 160 s on 2026-11-14):
the `select
min((issue_recovery_actions.evidence#>>'{freshContinuationCheck,nextCheckAt}')::bigint)
…` statement ran 6 times at about 13.5 s each. That is 81 s of database
time in 160 s, about half of one busy session. All other statements in
the window used less than 35 s together. Staging has 31,399 runs (1.96
GB, wide `context_snapshot`), 75,285 recovery records, and 282 records
with a check deadline.
The local plan shows the cause. For each candidate record:
- `source.id::text = coalesce(evidence->>'runId',
evidence->>'sourceRunId')` cannot use the primary key. PostgreSQL
filters the company's failed runs (6,767 rows removed in each loop, and
6,798 again in the outer semi-join).
- In the recent-interruption check, `max(held.finished_at)` matches runs
of the task with `native_issue_id::text = source_issue_id::text or
issue_id::text = … or context_snapshot->>'issueId' = …`. The casts
cannot use the task-link indexes, so PostgreSQL filters the company's
run history and decodes each run's context JSON (29,937 rows removed in
each loop).
**Proposed behavior**
The queries look up the source run by its primary key and the task's
runs by the existing `(company_id, native_issue_id, …)`, `(company_id,
issue_id, …)` and `(company_id, context_snapshot->>'issueId', …)`
indexes. The results do not change.
**Reason and benefit**
On a staging-shaped local fixture, the deadline query goes from 2,330 ms
to 27 ms (87x), and the backlog listing and count go from 3.7-5.7 s to
19-27 ms. Buffer accesses go from 2.15 M to 28 k.
**Breaking changes**
None. No schema change and no migration.
## What Changed
- `server/src/services/fresh-continuation-fallback.ts`:
- Add `sourceRunKey`. It compares the recorded run id with
`heartbeat_runs.id` through the existing `canonicalExecutionUuid` guard
(cast only when the text matches `^[0-9a-f]{8}-…-[0-9a-f]{12}$`).
`uuid::text` always gives that lowercase form, so this matches exactly
the rows `id::text = text` matched. Uppercase, braced, padded, numeric
and invalid ids still match nothing, and no cast error is possible.
- Use it in `heldExecutionWhere`, `interruptionEligibility` and the
listing's order expression.
- `runOfTask` compares `native_issue_id` and `issue_id` with the issue
id as a uuid. Both callers pass a uuid: the `source_issue_id` column, or
the id of a task row read in the same transaction.
- Export `heldExecutionWhere` and `recentInterruptionSourceSql` so the
regression test can compare them with the previous predicates row by
row.
- Add `server/src/services/fresh-continuation-schedule-queries.test.ts`
(regression test, see Verification).
- Add `scripts/benchmark-fresh-continuation-deadline.ts`. It seeds a
staging-shaped fixture, captures the SQL that the baseline revision and
the working tree send, compares complete positional rows, and reports
timings and plans.
## Verification
- Red/green: `fresh-continuation-schedule-queries.test.ts` "finds the
deadline by run-key and task-link lookups, not a scan of run history per
record" counts the `heartbeat_runs` rows that the deadline query's plan
reads (loops x (rows + rows removed)). With the previous predicates it
fails: 549,166 rows read against a bound of 9,442 (two times the run
count). With this change it passes.
- Equivalence: the same file seeds 900 held records with a seeded random
generator. The records cover these cases: `runId`, `sourceRunId`, both
keys, uppercase, braced, padded, numeric, invalid and unknown ids; a
source run in another company; source statuses (failed, timed_out,
interrupted, cancelled, succeeded, running) with and without
`finished_at`; the assignee and other agents; newer runs linked by
native id, legacy issue id, or context only; unrelated runs;
cancelled-at-dispatch runs; live (queued, running, scheduled_retry)
runs; earlier and later creation times; retries of the source; person
Retry wakes; old and recent finish times; restart, person-owned and
sandbox-resume error codes; and issue statuses and assignees. For every
record it compares `heldExecutionWhere()`, `heldExecutionWhere({
includeActive: true })` and `recentInterruptionSourceSql(now)` with
verbatim copies of the previous predicates. It also compares
`nextFreshContinuationCheckAt` and the full ordered
`heldExecutionsAwaitingFreshContinuation` rows with the previous
queries. The test checks that each predicate is true and false for more
than 5 records, so the comparison is meaningful.
- Focused suites pass locally: `fresh-continuation-fallback.test.ts` and
`fresh-continuation-schedule-queries.test.ts` (82 tests). Related
recovery suites pass: `fresh-continuation-retry-delivery`, `task-retry`,
`fresh-continuation-move-forward-offer`,
`execution-recovery-dead-controller`,
`automatic-run-brake-recovery-passes`, `execution-control-work`,
`recovery-wake-herd` (with the new test: 8 files, 252 tests). `node
server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a global CI input),
so CI supplies the broad coverage.
- Benchmark: local embedded PostgreSQL 18.1, `jit=on`,
`shared_buffers=128MB`, macOS arm64, a shared machine with other work.
Baseline `67652580ee`. Fixture: 5,669 issues, 31,351 runs with 60 KiB
incompressible context JSON (1,962 MB), 75,285 recovery records (282
with a check deadline, a third overdue). One warm-up and 5 alternating
timed pairs. Both versions return identical positional rows for every
query.
| Query | Baseline median (samples, ms) | This PR median (samples, ms) |
Speedup | Shared buffers |
| --- | --- | --- | --- | --- |
| `nextFreshContinuationCheckAt` | 2,330 (1,962-2,421) | 27 (21-28) |
87x | 2,149,802 -> 28,227 |
| `heldExecutionsAwaitingFreshContinuation` limit 5 | 3,692
(3,296-4,014) | 19 (18-24) | 194x | 2,831,539 -> 23,970 |
| same, limit 1 (schedule probe) | 3,554 (3,210-4,861) | 20 (18-24) |
179x | 2,831,539 -> 23,970 |
| left-alone count | 5,746 (5,090-5,992) | 27 (25-28) | 213x | 5,435,614
-> 40,953 |
With 1 KiB contexts (rows stored inline) the deadline query goes from
4,096 ms to 19 ms. Staging measured 13.5 s for the deadline query. The
local fixture reproduces the same plan shape, but the staging data
distribution and Neon storage are not reproduced exactly, so the staging
gain is not verified yet.
```sh
pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-fresh-continuation-deadline.ts \
--baseline-ref=67652580ee --context-kib=60 --samples=5 --warmups=1
--print-plans
```
- Plan after the change: the source run is an `Index Scan using
heartbeat_runs_pkey` (0 rows removed), and each task-run probe is a
`BitmapOr` over the native-id, issue-id and context-issue indexes. The
newer-run `not exists` already used that `BitmapOr` before. It does not
change.
## Risks
- Low risk. The SQL change is limited to how two keys are compared. The
regression test compares the old and new predicates for every record of
a varied fixture.
- `runOfTask` now casts its issue id to uuid. A caller that passes a
non-uuid text would get an error instead of no match. The two callers
pass a uuid column and a task row's id.
- Follow-up, not changed here: the disposition schedule calls
`deadline()`, which returns "retry in 15 s" when the earliest check is
already due. An active hold that the disposition pass does not pick up
(for example, one that records its run only as `sourceRunId`, which
`unresolvedExecutionDispositions` does not read) keeps the deadline
overdue, so the worker runs again every 15 s with nothing to do. A local
probe confirmed this. This explains a fixed cadence of 15 s plus the
query time (about 27 s on staging). After this change each such pass
costs milliseconds, not seconds.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL benchmarks and
tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
behavior or command change; no doc update needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.3
|
||
|
|
3ae7aba643 |
perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server uses `withDedicatedDbConnection` (packages/db) when an action must hold one database session outside the main pool: advisory transaction locks, lock keepalive, and `pg_settings` reads for workspace lifecycle, finalization, source locks, skill caches and status cards > - Each call built a new postgres.js client, opened a new physical connection (TCP, TLS and startup through the pooler), built a new drizzle client with the full schema, and then ended all of it > - On staging the dedicated application name shows about 21 connection setups per second, and the main thread is pinned (event-loop utilization 0.965) > - This pull request keeps a small set of idle single-connection clients per owner and reuses them > - The benefit is that a dedicated action costs about 0.2 ms of main-thread time instead of 4 to 8 ms, with the same exclusive-session guarantee ## Linked Issues or Issue Description **What existing behavior does this improve?** `withDedicatedDbConnection` opens and closes one new database client and one new physical connection for every call. **Subsystem affected** Database client (`packages/db`) and the server services that take dedicated advisory-lock sessions. **Current behavior** Each call builds a new postgres.js client, opens a new connection (TCP, TLS and startup through the pooler), and builds a new drizzle client with the full schema. On staging (Neon statement statistics, 326 s window) `SET application_name='paperclip-workspace-finalization-lock'` ran 21.4 times per second. Only this factory uses that application name. In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2% inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that is this per-call setup. **Proposed behavior** A caller still gets exclusive use of one session for the whole action. The session and its drizzle client are reused by later actions, so the per-call cost is the action's own queries, not a full connection setup. **Reason and benefit** The server main thread is saturated on staging (event-loop utilization 0.965). Reuse removes about 5 ms of main-thread work per call. **Breaking changes** None. The exported function keeps its signature. `closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are new exports. ## What Changed - `packages/db/src/client.ts`: each `createDb` owner now has one set of dedicated single-connection clients, shared with every private client derived from that owner (so nested dedicated actions use the same set). - An action takes an idle client (last in, first out) or opens a new one. It never waits. Callers hold advisory locks and nest dedicated actions, so a waiting queue could deadlock. - On success the client returns to the idle set if fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed. - When the action throws, the client is closed and never reused. This covers a failed or cancelled transaction, a broken socket, and an interrupted lock wait. - When an action ends the client itself (the source-lock cancel path calls `$client.end`), the client is never reused. - Idle clients keep the owner's idle timeout and max lifetime, so postgres.js still closes idle physical connections after `DATABASE_IDLE_TIMEOUT_SECONDS`. - New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle set and stops retention. Ending the owner `$client` does the same. A later action still works with a connection that closes on release. - `server/src/index.ts`: shutdown closes the dedicated set next to the other private pools. - `doc/DATABASE.md`: documents the idle cap, the "not an admission limit" rule, and the transaction-scoped session state contract. - Session state audit: every current caller uses `pg_advisory_xact_lock` / `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or temporary tables. A reset round trip per call is therefore not added. ## Verification - New `packages/db/src/dedicated-connection-pool.test.ts` (embedded PostgreSQL, 10 tests). - Red on `origin/master`: 200 sequential calls used 200 distinct sessions (`expected 200 to be 1`). All 10 tests failed. - Green with this change, 3 runs in a row: 200 sequential calls use 1 session; 7 concurrent callers each hold a different session for the whole transaction, and afterwards at most 4 stay open and later calls reuse them; a failed transaction's connection is closed; a terminated socket during the action is not reused; an idle connection terminated by the server reconnects; a client the action ended is not reused; transaction-local settings do not leak; shutdown and owner `$client.end` close every retained connection; nested actions do not block. - Focused suites pass: packages/db `work-signals`, `prepared-queries-postgres`, `response-capture-postgres`, `checkpoint-pool`, `client`, `client-options`, `client-teardown-registry`, `postgres-connection-recovery`, `client-disconnect-replay` (132 tests). Server suites that use dedicated connections: `skill-package-store`, `status-card-evaluation-work` (updated: it now proves advisory locks are released, since sessions are reused), `agent-instruction-working-copies`, `workspace-recovery-retry-copyback`, `native-execution-lease-batch`, `company-skills-empty-runtime`, `decisions-service`, `legacy-controller-lease`, `server-startup-feedback-export` (467 tests). - `pnpm verify --explain` selects 2779 files for a `packages/db/src/client.ts` change. Per AGENTS.md I ran the focused checks above and let CI give the broad coverage. - Benchmark: 200 calls of `withDedicatedDbConnection(db, d => d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))` against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating before/after process pairs per mode. Main-thread time is `eventLoopUtilization().active`. | Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call (median, range) | | --- | --- | --- | --- | | TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) | | TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) | | Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) | | Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) | At the staging rate of about 21 calls per second, this removes roughly 100 ms of main-thread work per second (TLS figures; staging CPUs may be slower). ## Risks - A retained connection that the server or network drops while idle: postgres.js reconnects on the next query when it has seen the close (tested). If the drop and the reuse race, the first query of that action fails like a main-pool query would. Before this change, a dedicated action always started on a fresh connection. - Through a transaction-mode pooler, `SET application_name=...` is a server-side parameter switch, so its count may not drop as much as the client-side connection count. Measure the deployed effect with pooler client connections and main-thread CPU, not only that statement. - Concurrency above 4 opens extra connections exactly as before; they close on release. Total retained connections per owner grow by at most 4 idle clients, which still close after the idle timeout. - An action that sets session-level state would now leak it to the next action. The contract is documented on `withDedicatedDbConnection` and in `doc/DATABASE.md`; no current caller does this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c36b96ae00 |
fix(recovery): hold the start of queued automatic runs while a company's automatic-run brake is on (#21561)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The recovery system starts automatic runs: continuations, retries, and sweep replacements > - #21443 added a per-company automatic-run brake. It stops recovery from creating new automatic runs while a company's runs keep failing before they start > - The brake did not hold automatic runs that were already queued. The dispatcher still started them, so a burst of queued recovery runs kept failing in the same way > - This pull request adds the brake's start gate to the queued-run dispatch pass, and wakes a held company at its next try without a polling loop > - The benefit is that a failing company stops spending runs on automatic work, people's work still starts at once, and held work starts again on its own when the brake releases ## Linked Issues or Issue Description Refs #21443 Refs #21433 Refs #21427 Refs #21505 **What happened?** While the automatic-run brake was open for a company, its already-queued automatic runs (continuations and retries) still started. Each one failed before it started, for the same cause that opened the brake. **Expected behavior** A queued automatic run must stay queued while the brake is open. Only the brake's single probe, and then the half-open release rate (2, 4, 8 per window), can start one. A person's work (a Retry, or a continuation of a person's request) must never be held. Held runs must start again on their own, with nobody acting. **Steps to reproduce** 1. Make 20 automatic runs in one company fail before they start, so the brake opens. 2. Queue automatic continuations for that company. 3. Run a dispatch pass. Before this change, the continuations start. ## What Changed - `server/src/services/heartbeat/queue.ts`: the dispatch pass asks `reserveAutomaticRunStart` after the tried/backoff skip and the claim-attempt budget check, and before the budget increment and the full row read. A held run spends no claim attempt, does no row read, and does not enter the declined-claim backoff. If the brake grants the start (its probe or a half-open release start) but the claim then returns null, throws, or finds the row gone, the pass gives the start back, so another automatic run can use it in the same pass. The dispatcher's person-origin answer is passed as is, and it is `undefined` only when the dispatcher did not check origin (an agent with one queued run). Then the brake's own cached person check decides. - `server/src/services/heartbeat/queued-run-line.ts`: `queuedRunDispatchFacts` also returns `personOriginChecked`, so the queue knows when the dispatcher's negative answer is real. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch deadline worker adds companies whose next try is due to its scopes, and includes the next try time in its next wake. A held company wakes for its next try without a polling loop. If the brake's bounded wake list had to drop a company, the worker wakes all companies at the earliest dropped time. - `server/src/services/recovery/automatic-run-brake.ts`: when the brake holds a run, it records the company's next try time (probe time or the next release window) in a bounded map. A probe or half-open release grant returns a release that gives the start back with a compare-and-set on its own run or window. It exports `reserveAutomaticRunStart`, `takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and `isRestartClassFailureCode` from `continuation-cause.ts` directly. The rules are the same as before; the lazy loader keeps only the imports that would make an import cycle. - `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests through the real heartbeat queue): bounded queries for 250 held runs over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over 3 passes; a declined probe claim gives the probe back and another automatic run starts in the same pass, and the probe stays with that run on a later pass; an unused half-open start goes back to its window; with 276 held companies (past the 256-company wake list) every company is still woken; a work run behind 250 held runs starts in the same pass; a person's Retry is never held; the only queued run is not held when it continues a person's Retry; exactly one probe starts across two server instances; the release gives 2, 6, 14 cumulative starts over three windows; held runs are not in the declined-claim backoff after the pause; the dispatch worker wakes the company at the next try; and an end-to-end test that a held automatic run stays queued while the brake is open, then starts and succeeds on its own at the next try. - Two small test updates in `automatic-run-brake.test.ts` and `automatic-run-brake-import.test.ts` for the new cache size field and the moved rule. - `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`: one test for the sweep path from #21505 (continue a blocked task behind a run refused at dispatch). The sweep only records the continuation (no wake, no new run), and `deliverReconciledExecutions`, which the brake already gates, creates the run only after the pause. No second gate is needed on that path. ## Verification - `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11 passed. - Red before for the review fixes: on the first PR head, the 1,500-run cost test made 3,848 queries (bound 130), the declined-probe test started 0 runs (expected 1), and the 276-company wake test missed 20 companies. - Red before: with the `queue.ts` gate line removed, 5 of the 8 tests fail (bounded pass, single probe, release rate, backoff, end-to-end). - Red before for the person case: with the dispatcher's `false` passed through as final, "never holds the only queued run when it continues a person's Retry" fails (0 started). It passes after the change. - `automatic-run-brake.test.ts` 19 passed, `automatic-run-brake-import.test.ts` 1 passed, `automatic-run-brake-recovery-passes.test.ts` 28 passed, `failed-continuation-continues-again.test.ts` 32 passed (after merging master with #21505). - Nearby queue tests (on the final head): `heartbeat-run-dispatch-work` 15, `queued-run-dispatch-stacking` 6, `heartbeat-queued-run-claim-isolation` 10, `assigned-work-stalled-queued-run` 12, all passed. - `server` `tsc --noEmit` is clean. ## Risks - Behavior change only while a company's brake is open or half-open. When the brake is closed, the gate returns from its cache with no query. - The first time the gate sees a non-person automatic run while the brake is on, the brake does one person-origin query for that run. The result is cached per run (bounded to 1,024 runs). - No migration. No API or UI change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
84cb28f9b7 |
fix(connections): keep machine sign-in secrets and settings in one locked commit (#21566)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21473 added machine sign-in (MCP OAuth client credentials) for organization connections > - Auto-merge squashed #21473 from an earlier head, so its last review fixes did not reach master > - Those fixes close a P1 race and two smaller setup gaps that Greptile found on #21473 > - This pull request applies that last commit to current master, with its tests > - The benefit is that a client ID and its secret always stay together, and key switching and app selection work as intended ## Linked Issues or Issue Description Refs #21473 **What existing behavior does this improve?** Machine sign-in setup and rotation, added in #21473. **Problem** - A connection test wrote a replacement client secret before checking whether the connection changed. If the save was then refused, the secret stayed and could pair with another client ID, so the next token request failed. - Switching a client-secret connection to a Paperclip key always created an RS256 key, even when the sign-in service accepts only ES256. - Machine sign-in dropped the setup's app ID and created a separate app. **Proposed improvement** Write the secret and the settings in one locked transaction after the change check. Record the accepted signing algorithms and use them. Pass the app ID through. ## What Changed - `client-credentials-service.ts`: `saveConnection` locks the row, checks that the sign-in method and credential references are unchanged and that the connection is not removed, then writes any new credential and the settings in the same transaction. Test, secret rotation and key staging all use it. - Setup records `tokenEndpointAuthSigningAlgValuesSupported`. Switching from a secret to a key uses `chooseAssertionAlgorithm` on it. - UI: `MachineSignInSetup` takes `applicationId`, and the custom-server flow passes `prefill.applicationId`. - Tests: two new server tests and one UI expectation. ## Verification - `server/src/__tests__/tool-access-client-credentials.test.ts`: 25/25 pass. The 2 new tests fail on master's service code ("never leaves a refused test's secret behind with another client ID", "creates the kind of key the sign-in service accepts when switching from a secret"). - `MachineSignInSetup.test.tsx` and `GenericMcpConnectStep.machine.test.tsx`: 8/8. The app ID expectation fails on master's component. - Server `tsc --noEmit` clean. UI `tsc` clean except the pre-existing generated Storybook catalog imports. ## Risks Low. The change is limited to machine sign-in. A concurrent change now returns 409 `client_credentials_changed` instead of saving partly. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bbed32450c |
perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how many concurrent agents one server process can carry > - It assumed 20 text deltas/s and 1 other event/s per agent, and every other event was an `item.completed` > - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other events/s, with a mix in which 39% of the other events are server trace spans that take a cheaper path > - This pull request makes the recorded rates and mix the model's default, keeps the old rates as an explicit stress profile, adds a burst profile, and mirrors the delta batching of #21520 > - The benefit is a model that measures what production does, so the next optimizations target the real per-agent cost ## Linked Issues or Issue Description Refs #21418 (load model and plan). Refs #21520 (delta batching that the model now mirrors). **What existing behavior does this improve?** The load model's default rates, its event mix and its delta batching rule, and the plan document's measurements. **Subsystem affected** `scripts/benchmark-agent-scaling-load.ts`, its test `server/src/__tests__/agent-scaling-load-model.test.ts`, and `doc/plans/2026-11-13-thousand-concurrent-agents.md`. **Current behavior** The model defaults to 20 deltas/s and 1 other event/s per agent. Every other event is a native `item.completed` with a chat-progress wake and a checkpoint save. It cuts a delta batch on the timer even while the previous batch commits. **Proposed behavior** - `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute. `PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each rate can still be set on its own. - `EVENT_MIX=recorded` (default) sends other events in the recorded proportions. A `run.performance.span` is an ordinary append with no wake and no checkpoint, as the server's run trace writes it. `tool.execution.*` and `item.completed` are native appends with a chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one type only. - Delta batching follows #21520: deltas that arrive while a batch commits all join one next batch. - The report adds `profile` and `eventsIssuedByType`. **Reason and benefit** At the recorded rates the per-agent cost is the non-delta runner events and their checkpoint saves, not deltas. The plan and the next optimizations need that picture. **Breaking changes** None for the product. Anyone who runs the model with no options now gets the recorded profile; `PROFILE=stress` gives the earlier load. ## What Changed - Profiles, the recorded event mix (a 100-slot cycle per agent, offset by agent so the fleet sends the mix at every moment), the span append path, and the coalesced delta batches. - The model test now checks the profile in the report and that both server trace spans and runner events with a progress wake are in the mix, for the recorded profile and for the stress profile with one transaction per delta. - The plan has a new section "Load model at recorded rates": the recorded source (40 recent native runs on one hosted workspace, as aggregates), the profiles, the mix and its server paths, fresh runs at 500, 1,000 and 2,000 agents plus burst and stress, and what they show. It records the 3-connection checkpoint pool as a computed per-process limit on a hosted database, and that SQL building in Drizzle costs about as much main-thread time as result parsing on the hosted instance. A new section "Considered and not done" records why `run.performance.span` rows stay (about 3% of the statements per agent, with readers) and why checkpoint saves per critical event only are not done now. ## Verification - `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2 tests). The new assertions fail against the previous script: it reports no `profile` and no `eventsIssuedByType`. - `server/src/__tests__/agent-scaling-load-model.test.ts` also checks each profile's rates with no overrides, and that a rate set on its own wins. - Load model runs, recorded in the plan (60 s, main pool of 10, embedded PostgreSQL on the same machine, each run started below a 1-minute load average of 15): recorded rates at 1,000 agents (event-loop utilization 0.61) and 2,000 (0.85) finish every operation with no failure; 500 agents all in their busiest minute run at 0.95 with p99 lateness under 0.4 s. About 4.4 statements per agent per second at recorded rates, 35% of them on the checkpoint pool. - Earlier runs at load averages of 20–55 had shown one process falling behind at 500 burst agents and at 2,000 recorded; the quieter runs after the model fixes (event order, spans off the runner stream) do not, and the plan says so. The checkpoint pool as a per-process limit is now stated as a computed limit for a hosted round trip (about 400 agents in a busy minute at 1 ms), not as measured. - Greptile findings fixed: a waiting delta batch no longer takes deltas sent after the next runner event, takes every delta queued when it starts, and spans no longer join the runner event stream. ## Risks - Low: a benchmark script, its test and a plan document. No product code changes. - The machine that measured the plan's table was shared; the plan marks the rows that the machine distorted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6dd0349d52 |
fix(ui): name a queued turn's hold once, in the turn header (#21564)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's chat shows its run as a turn: a header line with the run's
state, then the turn body
> - #21405 and #21447 made a queued turn name why it waits: "Waiting for
Codie · …", "Waiting for PAP-5656 to finish", "Waiting to start"
> - On staging, the turn body repeated that label under the header
("Waiting for PAP-5656 to finish" twice). The tail used by other
adapters showed a fixed "Waiting to start..." even while the pill above
said "Working" or named a hold
> - #21447 removed the repeat only for the agent's waiting line
> - This pull request names every queued hold once, in the turn header
> - The benefit is one clear statement of why a task waits, with no
repeat and no contradicting line
## Linked Issues or Issue Description
Refs #21405, #21447
**What happened?**
On staging (revision
|
||
|
|
351b69941c |
perf(checkpoint): write a routine controller checkpoint save in one statement (#21553)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote runner's controller saves a checkpoint after each runner event it commits, on a 3-connection pool, before it acknowledges the event > - Each save is a transaction of 4 statements (5 where commit notifications are on): BEGIN, the owner lock with the transaction id, the write, COMMIT > - What forces the transaction is the live-run recovery wake that every save registers. Recovery does not react to a routine save > - This pull request writes a routine save (new or changed event, command and cursor records) with one statement that checks the owner and writes, and keeps today's transaction for every other save > - The benefit is one statement instead of four for most checkpoint saves, on the pool that limits one process in a busy minute ## Linked Issues or Issue Description Refs #21528 (owner lock returns the transaction id). Refs #21549 (load model at recorded rates; the plan records the checkpoint pool as a per-process limit). **What existing behavior does this improve?** `nativeRunnerCheckpointWriter` in `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Subsystem affected** Controller checkpoint persistence for remote native runs. **Current behavior** Every save locks the coordinator row (`native_run_finalizations` `for update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery queues), and upserts the changed parts, in one transaction. **Proposed behavior** A routine save is one statement: a CTE locks the coordinator row only while the lease owner, boot id, generation and an unexpired lease (`clock_timestamp()`) all match, and the upsert writes only through that row. It registers no recovery wake. If the owner does not match, it writes nothing and the save takes today's full path, which runs the complete owner check (successor rules included) and fails with `native_runner_checkpoint_owner_lost` when it should. A save is routine when it has changed parts, no reset, no removed parts, no header change, no predecessor, and the run is its own authority. Every other save (the first save of a controller, which is a reset, a save after a failed save, header or retirement changes, removals, source-run consumers) is unchanged. **Reason and benefit** At recorded rates about a third of one process's statements are checkpoint saves. In a busy minute the 3-connection pool cannot keep up at 500 agents (#21549). **Breaking changes** None. No schema change. ## What Changed - `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in `nativeRunnerCheckpointWriter`. ## Why a routine save needs no recovery wake I read the six consumers of `notifyLiveRunRecovery`: - `nativeProcessSessions`, `nativeQuestionCancellations`, `activeExecutionRecovery`, `settledNativeProcesses` (`live-recovery-work.ts`): each selects candidates from `heartbeat_runs` status, runtime mode, `native_run_finalizations` phase, lease and retry times, the context snapshot and process identity. None reads checkpoint rows to decide eligibility. - `strandedAgentStatus` (same file, `agents` rows) and `activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`, `heartbeat_runs.last_output_at`): neither reads checkpoint rows. - The checkpoint rows matter to recovery in two places: `listExpiredNativeRunnerAuthorities` requires that a checkpoint exists, and the remote bootstrap proof reads its parts. A checkpoint first appears with a reset save, which keeps its wake. A routine save only adds event and command parts, which can only take a run further from the bootstrap proof; recovery reads the current rows when its own deadline (lease expiry) fires. - The intent of the wake also fenced a recovery pass against a save in flight. The routine save is one atomic statement and holds the coordinator row lock while it writes, so a recovery claim that takes that row waits for it, and a reader sees the save whole or not at all. ## Verification - Red/green on embedded PostgreSQL (`native-runner-checkpoint-save.test.ts`); the new tests failed against `origin/master` first: - A routine save is one statement (`with owner as ... for update ... insert into native_runner_checkpoints`) and registers no recovery wake (before: 4 statements and a wake). - A reset save keeps BEGIN, the owner lock with the transaction id, and COMMIT, and still registers its recovery wake. - A routine save that waits while another connection takes over the lease (new owner and generation) falls back to the full path and fails with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged. - The same race where the new owner also resets the checkpoint (deletes every part and writes its own header): the save fails the same way and the checkpoint holds only the new owner's header. - A lease that expires while the routine save waits for the coordinator row lock (the holder does not change the row) writes nothing: the statement judges the lease after the lock, as `lockNativeRunnerCheckpointOwner` does. My first version checked the lease in the locking CTE itself, and PostgreSQL does not re-check a condition after a lock wait on an unchanged row; the existing `native-runner-restart-recovery.integration.test.ts` test "fences checkpoint delivery when ownership expires during an unchanged row-lock wait" failed on it, and so does the new test. Both pass now. - The existing test "refuses a save after the owner's lease expired and writes nothing" passes through the new fallback. - Mutation checks: with the owner conditions removed from the statement, both race tests and the expired-lease test fail; with the lease check moved back into the locking CTE, the lock-wait expiry test fails. - Other focused suites, one file at a time: `native-runner-restart-recovery.integration` (231 passed, 14 skipped), `native-runner-restart-checkpoint-cost.integration` (1; it greps for `insert into "native_runner_checkpoints"`, so the statement quotes the table name), `native-recovered-owner-followup` (97), `native-controller-recovery.integration` (30), `native-runner-checkpoint-authority.integration` (1), `native-restart-recovery-isolation` (1), the port suite with conformance (105), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51 tests pass; on this busy machine its teardown hook timed out, as it has before on master). `pnpm --filter @paperclipai/server run typecheck` passes. - Statements per routine save: 4 → 1 (5 → 1 where commit notifications are on). I did not rerun the load model for this change: the machine is too busy for a fair timing comparison today, and the statement count is exact. - CPU per operation (a local script, not committed: 600 routine saves of one event part and the cursor for one run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after (encryption included in both); wall time per save 0.45–0.73 ms before and 0.22–0.30 ms after. ## Risks - A routine save no longer wakes the six recovery queues. Their own deadlines and the 30 s reconstruction still run; see the reading above for why no decision depends on a routine save. - The owner check of the routine statement accepts only the exact active owner. Any other state goes to the unchanged full path, so a successor or a finished result keeps today's rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
67652580ee |
perf(tool-access): index enabled policies so each decision checks only candidates (#21563)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).
**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).
**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).
**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.
**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.
**Breaking changes**
None. The policy data model, API, and listing contract do not change.
## What Changed
- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.
## Verification
- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.
| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
| --- | --- | --- |
| Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
| Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
| Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
| Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
| Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
| Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
| Selector evaluations | 305,600 | 1,164 |
| Raw selector property reads | 6,155,830 | 9,490 (compile only) |
```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
```
- Not verified: the hosted staging profile after deploy.
## Risks
- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
82ce8e64ba |
feat(runner): let the session driver admit a turn that the provider started (#21453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Core reads Runner events through the Codex app-server driver, also for ACPX agents. The driver refuses a turn that the controller did not start, except a goal turn of Codex. > - This pull request lets the driver accept a turn that the provider started, only directly after a turn that the Runner marked as waiting. > - The benefit is that the next parts can use that turn. Nothing reaches this code until the last part sends the opt-in. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 1. Its base branch is `feat/adopt-claude-provider-started-turn`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the driver does with a turn that the provider started. **Subsystem affected** Paperclip Runner package, the controller side: `src/drivers/codex` (the driver that Core uses to read a native session). **Current behavior** - A `turn/started` that the controller did not ask for fails the session with `turn_binding_mismatch`, except for an active Codex goal. - An interrupt with no active turn fails with "already terminal". **Proposed behavior** - A turn terminal that the Runner marked with `awaitingProviderTurn` and that has no result puts the driver in a waiting state. The driver event `turn.completed` carries the marker. - In that state, a `turn/started` with `origin: "provider"` is admitted the same way as a goal turn, and the driver event carries `origin: "provider"`. In each other state it still fails closed. - In that state with no active turn, an interrupt gives an acknowledgement item with the status `provider_turn_wait_ended`. It sends nothing to the Runner. The driver also gives that acknowledgement when it gets the internal `paperclip/runTerminal` notification with a private marker; a notification without the marker does nothing. - An interrupt for a provider-started turn that is running is sent to the Runner only when it names that turn. - The driver saves the waiting state in the persisted session (`awaitingProviderTurn`, written only during a wait). After a controller restart, the restored session still admits the provider-started turn and still acknowledges an interrupt during the wait. The driver restores the state only when the session has no active turn, no stored result and at least one finished turn, so a new run cannot take the wait of the run before it. - When a restart finds that the saved active turn is finished, the turn read of the transport and `reconcile()` of the driver carry the same marker as the live `turn/completed` notification. `runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from the saved `turn.completed` event to the turn object of a read. - The driver ends the waiting state when the Runner reports the run terminal. **Reason and benefit** The consumer in the next part needs these three signals: a marked terminal, an admitted turn, and a clear end of the wait. ## What Changed - `codex-session-terminal.ts`, `codex-session-notifications.ts`, `codex-session-state.ts`, `codex-harness-session.ts`, `codex-driver-values.ts`: 37 lines added, 4 removed. - `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines). - `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`, `backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`: the optional field in the two persisted shapes and its copy in each direction. - `live/runnerd-codex-transport.ts`: the marker on a recovered turn terminal. The loop that builds the earlier terminals of a `thread/read` is now the function `runnerdPriorTurnTerminals`, with the same logic, so that a test can call it. ## State saved by the previous release - One new optional field in the persisted driver session: `awaitingProviderTurn`. A snapshot outside a wait has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so the driver does not wait. This is the behavior of today. - **State from this release, read by the previous release:** the previous reader copies the fields that it knows and ignores this one. No reader rejects it, and the integrity digest covers the stored object as written. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - 6 driver tests: a provider-started turn after a marked terminal is admitted and can report a result; a provider-started turn with no marked terminal fails closed; an interrupt during the wait is acknowledged; a running provider-started turn is not interrupted by the end of the wait; the marked notification ends the wait and a notification with no marker does not; a normal terminal is not changed. - Restart tests: a session restored from a snapshot with the waiting state admits the provider-started turn and settles its result; a snapshot outside a wait has no new field; a restored session with no waiting state refuses a provider-started turn as today; an interrupt during a restored wait is acknowledged. A saved active turn that the read reports finished with the marker gives a `turn.completed` with `awaitingProviderTurn: true`; with no marker it is an ordinary terminal. - `harness-driver-backend.test.ts`: the field goes through a serialized checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324 passed): the marker is on a recovered terminal only for a `turn.completed` with a boolean `true`. - `src/native-session-runtime.test.ts`: 297 passed, with the real `paperclip-runnerd`. - Mutation checks for the restart work, each made a named test fail: no restore of the state; no copy in the backend; no copy in the transport; no marker argument in `reconcile()`; no end of the wait at the run terminal. - Mutation checks: the acknowledgement branch removed, the admission removed, and the marker check removed each made a test fail. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** The Runner sends the marker only after the opt-in of part 4. - **The admission rule is shared with Codex goal turns.** The existing goal tests pass with no change. - **Known limit after a controller restart.** If the Runner already runs a provider-started turn and the saved active turn is the turn before it, recovery fails closed, as today for each unexpected active turn. - **A controller turn that starts during a wait** clears the waiting state when its `turn/started` arrives, not when it is sent. A snapshot in that short time can hold an active turn and the field together; the restore rule drops the field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
ca7d42ba71 |
fix(heartbeat): refresh queued dispatch when task drains end (#21562)
## Thinking Path > - Paperclip runs agent work for a company. > - A task drain holds new run starts while maintenance or idle handoff is in progress. > - A queued dispatcher can keep an unreconciled request and a 30-second retry while the drain is active. > - Ending the drain can resume domain work without refreshing that dispatcher's saved deadline. > - This change refreshes the pending queued owner when a drain ends or its expiry is observed. > - Normal admission checks and real retry deadlines still apply. ## Linked Issues or Issue Description Related: #20640. That change refreshes deferred receipt workers. This change refreshes the separate queued-run dispatcher. **What happened?** A queued-work notification received during a drain could leave a 30-second fallback in the dispatcher after the drain ended. The owner stayed unreconciled after useful work had settled. This delayed its next pass and the instance's idle state. **Expected behavior** Refresh an unreconciled queued owner when normal dispatch becomes allowed. Keep other drain holds, admission rules and scheduled retry deadlines. **Steps to reproduce** 1. Let the normal dispatcher complete its first pass. 2. Start an idle task drain and commit a company-scoped queued-work notification. 3. Stop the drain, or observe its expiry, before the saved fallback is due. 4. Check that the scoped queued pass runs before that fallback. The two regression cases fail on the unchanged owner and pass with the fix. **Paperclip version or commit** The main-specific patch is based on `f7b1651799d8c0c660a76266878c420a12f0beca`. The bug was found in assembled sleep acceptance. The affected queued-dispatch, drain and notification modules match main. ## What Changed - Listen for normal drain changes and observed expiry in the queued-run owner. - Wake that owner only when dispatch is allowed and its reconciliation is pending. - Forward the deadline worker's optional stop callback after its stop fence. - Remove both subscriptions when the owner or coordinator stops. - Add release, expiry, remaining-gate, shutdown, retry-deadline and idle controls. ## Verification Local checks used the integrated feature branch that exposed the issue. The main-specific port preserves main's existing `canRun` logic and excludes the private branch's extra cancellation hook. - Both new release and expiry cases failed at the intended scoped-resume assertion before the product change. - Six dispatch cases pass with the normal package import. - Fourteen existing deadline controls and nineteen queue-order controls pass. Their inputs did not change during the test-import correction. - The complete normal Server typecheck and its three affected-file program checks pass. - The first typecheck caught a test import outside the Server root. The import now uses the existing public package wildcard. The corrected six-case suite and normal typecheck pass. - Normal Source SDK input verification passes. - Exact-head CI at `fd623d3ba9345e5ec8f252c55932ff797cd57e8f` completed: 54 passing checks, four expected skips and one neutral result; Snyk passed. Greptile reviewed this exact head at 5/5 with no actionable findings. All review threads were read; none remain open. A full local repository build and suite were not run in this checkout. The existing admission-drain contract in `doc/execution-semantics.md` already requires prompt admission after drain release. This fix restores that contract. No configuration or command documentation changes are needed. ## Risks Drain events can request an earlier pass for a pending owner. The normal permission, reconciliation, coalescing and earliest-time checks remain active. The scheduled retry owner has no new listener. Shutdown must remove both listeners; the coordinator-stop control covers that path. This PR does not claim deployed suspension or complete transport sleep acceptance. ## Model Used OpenAI `gpt-6.1-sol` assisted with implementation and independent review. The Codex managing agent performed technical review and verification. The host does not expose its exact runtime model ID or context-window size. Tools used include source inspection, code edits and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (not required: the existing contract remains accurate) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
9824277f65 |
perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Most runner events that are not text deltas (`tool.execution.started/progressed/completed`, `item.completed`) wake the chat progress projection > - That wake is registered inside the append's transaction, so each such event takes 4 statements (BEGIN, the run lock read, the append, COMMIT), where an event without a wake takes 1 (#21511) > - The chat progress projection rebuilds itself from the stored events; the wake only tells it when to look > - This pull request writes an event whose only wake is chat progress with the one-statement append, and sends the wake after that statement commits > - The benefit is 1 statement instead of 4 for most non-delta runner events, with response, reassessment, retention and recovery wakes unchanged ## Linked Issues or Issue Description Refs #21511 (one-statement append). Refs #21528 (lock read returns the transaction id). Refs #21549 (load model at recorded rates). **What existing behavior does this improve?** `appendHeartbeatRunEvent` for native source events whose only wake is chat progress. **Subsystem affected** Server run event ingest (`heartbeat-run-events.ts`), the chat progress notification helper, and two additive exports in `packages/db/src/work-signals.ts`. **Current behavior** A progress event opens a transaction, reads the run lock with `runtime_mode`, the bound task and the transaction id, probes for a replay, registers the chat-progress wake as a tracked intent, appends, and commits. **Proposed behavior** When chat progress is the event's only wake, the append takes the one-statement path of #21511. That statement also returns the run's `runtime_mode` and bound task id from the row it locks, and, where commit notifications are on, calls `pg_notify` inside the same statement for runs that are native and bound to a task, so other server processes get the wake at commit. After the statement returns, this process publishes the wake as settled to its own listeners. A replay, a gap, a changed run or any other fallback takes the existing transaction, which registers the wake as before. **Reason and benefit** At recorded rates these events are about 61% of the non-delta events (#21549), so this cuts about 3 statements per such event. **Breaking changes** None. No schema change. ## What Changed - `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other wake clear) also qualifies for the one-statement path. - `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`, rendered once) from the locked row, plus a conditional `pg_notify`. - `chat-native-progress-notifications.ts`: `chatNativeProgressCommitNotification` and `publishChatNativeProgressSettled`. - `work-signals.ts` (additive): `databaseWorkCommitNotification(owner, topic)` returns the cross-process payload when notifications are on; `publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled wake to this process's listeners. ## Verification - Red/green on embedded PostgreSQL. Each new test failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "appends an event whose only wake is chat progress in one statement and wakes progress after it commits": 1 statement (before: a transaction), the wake is published once as settled with no intent, and a replay publishes nothing. - "keeps the transaction for the last events of a turn and a run, and one statement for progress-only events": `tool.execution.started/progressed/completed` and a non-final `item.completed` take one statement; a final `item.completed`, `run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled` and `turn.interrupted` keep the transaction. - "sends the progress wake to other server processes with the one statement's commit": with commit notifications on, another connection that listens on the work channel receives exactly one chat-progress notification, and a replay sends none. - `chat-native-progress-work.test.ts` "wakes the projection from a one-statement progress event after the statement commits": the real chat progress worker publishes the progress for an event appended in one statement. - Mutation check: without the settled publish after the statement, that test fails. - Lost-wake repair (condition b): an event row stored without any wake models a lost hint. - "repairs a lost progress wake with the run's next progress event": nothing is published for the stored event; the run's next progress event publishes the progress. - "repairs a lost progress wake with the startup reconcile": a worker that starts after the stored event publishes it. - The existing response-wake test now uses a final reply and still shows the intent before the write, and the race test of #21528 uses a final reply so it still covers the transaction path. - Other focused suites, one file at a time: the port suite with conformance (105), `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25), atomic-append (31), `chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - CPU per operation (a local script, not committed: 800 `tool.execution.progressed` appends to one native task run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms after in the warm rounds; wall time per event 0.84–1.04 ms before and 0.52–0.53 ms after. - Inside a caller's transaction: new test "keeps a progress-only event inside a caller's transaction on the locked path, and settles its wake only when that transaction ends". `publishDatabaseWorkSettled` publishes at once only outside a transaction; inside one it waits for the outermost commit and drops the wake on rollback (new `work-signals.test.ts` test, which failed with the unguarded publish). The one-statement path is only taken when the owner is not a transaction (`!is(db, PgTransaction)`, #21511), so the two guards agree. ## Risks - **Lost wake.** If the statement's outcome is unknown (the connection drops after the server ran it), this process sends no wake for that event. In a chat channel (Slack, Telegram) the progress line for that task can then stay one step behind (for example it does not move from "working" to the latest tool step) until the run's next runner event, a run milestone, or a server restart. A turn normally ends with an event that keeps the transaction (`turn.completed/failed/cancelled/interrupted`), and the run's completion notifies chat run milestones, which include chat progress (`notifyChatRunMilestones` in `run-completion.ts` and `native-run-finalizer.ts`). So progress cannot stay stale after a run ends; in the worst case it lags one step for the rest of a turn. The projection rebuilds from stored events, so nothing is lost or duplicated. - The chat-progress wake no longer publishes an intent before the write. Intents let the coordinator fence a pass against an in-flight write; the settled wake after the commit makes the projection run again after the row is visible. - `packages/db/src/work-signals.ts` gets two additive exports and no change to existing behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c0c7c6abb |
perf(ui): load the @task mention pool after the task page settles; add a cold-load perf spec (#21557)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People often open a task by its URL, for example from a link in a message or after a reload > - #20592 made a hard load of a task URL read the chat's first frame from one `chat-bootstrap` request > - The task page still sends a large read as it mounts: 100 full recently updated tasks for the `@task` mention picker. On staging this read returned 395 KB and took 3.9 s of server time > - The hosted database is CPU-bound, so reads that do not affect the first frame must not compete with the reads that do > - Also, no test measured a cold direct load under realistic latency or guarded the one-bootstrap request shape > - This pull request starts the mention pool after the task page settles. It also adds a cold-load perf spec with a delay on each API request and an always-on request-shape test > - The benefit is less concurrent server work during a cold task open, and a regression guard for the one-round-trip first frame ## Linked Issues or Issue Description Refs #20592 (hard-load chat bootstrap that this pull request measures and guards). Refs #20355 (the `chat-bootstrap` endpoints). **What existing behavior does this improve?** The task page loads the `@task` mention pool when it mounts. **Subsystem affected** Board UI task page (`ui/src/pages/IssueDetail.tsx`) and the issue-detail perf harness (`tests/perf/issue-detail`). **Current behavior** `GET /api/companies/:id/issues?limit=100&sortField=updated&sortDir=desc` starts with the chat's first-frame reads. It starts before the first chat frame in 14 of 14 local cold loads. The picker needs it only after the person types `@`. **Proposed behavior** The read starts after the page has painted and its own reads have finished. On a page that keeps reading, it starts after at most 5 s. **Reason and benefit** The read never affects the first frame. Deferring it removes a large concurrent read from the busiest part of a cold open. **Breaking changes** None. If the person types `@task` in the first moments after a cold load, the task suggestions can appear about half a second later. ## What Changed - `ui/src/hooks/usePageSettled.ts`: new hook. It returns `true` once `whenPageSettles` runs (the page has painted, the tab is visible, no query is fetching, and the browser is idle), or after `PAGE_SETTLED_MAX_WAIT_MS` (5 s). The cap keeps a long-running read from keeping the picker empty. - `ui/src/pages/IssueDetail.tsx`: the mention-pool query is enabled only when `usePageSettled()` is true. - `ui/src/hooks/usePageSettled.test.tsx`: tests that the hook waits for in-flight reads and that the cap applies. - `ui/src/pages/IssueDetail.test.tsx`: mocks the hook to `true`, so existing tests keep their behavior. - `tests/perf/issue-detail/cold-bootstrap.perf.spec.ts`: new spec. It delays every API response (300 ms by default) and opens the task URL in a new browser context, with the JavaScript in the HTTP cache and no app storage. The finish event is the first readable frame: the history is revealed, the latest saved message is visible and the composer is editable. - The request-shape test always runs. It requires one `chat-bootstrap` request and no per-endpoint first-frame read before the frame. Reads that start after the live socket is created are excluded, because they are its catch-up refresh. - The benchmark runs only when `PAPERCLIP_ISSUE_PERF_COLD_RUNS` is set. It writes the median, the spread, the request count before the frame, the last requests before the frame, the JavaScript bytes transferred and a request timeline. - The delay uses CDP `Fetch` interception instead of Playwright routing, because routing disables the HTTP cache. - `tests/perf/issue-detail/README.md`: documents the spec. - My earlier draft also added a deadline for a stalled bootstrap. Master already has one (`CHAT_BOOTSTRAP_SHARE_DEADLINE_MS`, 5 s), so this pull request does not include it. ## Verification Measurement setup: - Production UI builds of master `4a78124a3e` and of this branch, served by one local server (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`, embedded PostgreSQL). - A delay of 300 ms on each API request. - Cold direct task URL with 24 saved comments and a plan. - Master and the branch alternated: 2 rounds of 7 runs each. - The host was shared with other agent sessions. | Build | Runs | First readable frame, median | p25–p75 | Min–max | API requests before frame (median) | Mention pool started before frame | |---|---:|---:|---:|---:|---:|---:| | master | 14 | 1,422 ms | 1,336–1,525 | 1,284–7,398 | 50 | 14 / 14 | | this branch | 14 | 1,390 ms | 1,338–1,457 | 1,272–2,207 | 49 | 0 / 14 | - The spec adds the API delay through CDP request interception, so the HTTP cache stays on. The measured loads downloaded 0 KiB of JavaScript in every run. The spec records this as `jsTransferBytes`. Each build has one outlier (7,398 ms on master, 2,207 ms on this branch), from host contention. - With this branch, the mention pool starts after the frame in every run (350–400 ms after it in an earlier set). - The first-frame timing is neutral locally. The local database answers the pool read in milliseconds, so it does not compete with other reads. The intended benefit is lower concurrent server load on a busy database. This pull request does not show it as a local latency gain. - On master and on this branch, the hard load adopts the bootstrap and the frame is no longer held by task reads. The frame now comes right after the shell reads return: `auth/get-session` and `instance/settings/experimental`, then `companies` and `workspace-preferences/me`. Profiling marks confirm this: the chat thread mounts after `workspace-preferences/me` returns, and all readiness gates are ready within 30 ms. - The request-shape test passed 5 of 5 on master and 5 of 5 on this branch. - `layout-stability.spec.ts`, `plan-startup.spec.ts` and `cold-bootstrap.perf.spec.ts` against this branch's build: 18 passed. - `ui` Vitest for `usePageSettled.test.tsx` and `IssueDetail.test.tsx`: 278 passed. The `ui` typecheck passed. `pnpm check:token-gates` passed. - `pnpm verify` exits 2 because `tests/perf/issue-detail/README.md` is a global CI input. Required CI supplies the broad coverage. To reproduce the measurement: ```sh PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:3287 PAPERCLIP_ISSUE_PERF_COLD_RUNS=7 PAPERCLIP_ISSUE_PERF_LABEL=branch \ pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts cold-bootstrap -g benchmark ``` ## Risks - Low risk. The only product change is when one read starts. - If the page keeps a query in flight, the pool waits up to 5 s. During that time, `@task` suggestions list only agents, projects and members. - The request-shape test treats reads that start after the live socket is created as catch-up refreshes. If the socket connects before the task page mounts, such a test would not see a first-frame read that started later. The test still requires exactly one bootstrap request before the frame. - The perf spec is not in the default CI browser suites. It runs when the issue-detail harness runs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing, Playwright browser runs). It wrote the change, the tests and the measurements. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.2 |
||
|
|
54c3616ad9 |
fix(recovery): ignore the non-assignee comment wake marker in the failed-preparation proof (#21550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a Runner run fails in setup before any provider work, a person's Retry must start a new run > - Admission proves "no provider work" for a failed preparation by checking that the run record holds only known, harmless fields > - The run profile allowlist did not include `aiConnectionNonAssigneeCommentWake`, a boolean that dispatch writes when it admits a run from a non-assignee comment > - So a run with that marker never passed the proof. Its Retry waited on `previous_execution_unverified` forever, even after saved-work recovery finished and the workspace was handed on unchanged > - This pull request accepts the marker (boolean only) in the proof, and in the related workspace-wait parent proof > - The benefit is that these tasks get a working Retry, and a Retry that was saved earlier starts by itself once the proof passes ## Linked Issues or Issue Description Refs #21481, #21497. **What happened?** On a hosted staging instance, a Runner run failed in setup ("RPC call environmentExecute timed out") while it prepared. Saved-work recovery finished ("Saved work recovered."). The workspace was handed on unchanged for this run. A person pressed Retry two times. Both Retries waited on `previous_execution_unverified`, and no run started. **Expected behavior** A run that failed in setup before any provider work (`executionRecovery.providerWorkStarted: false`, no process, no execution events, stopped sandbox) gets a working Retry. A saved Retry starts by itself when its gate clears. **Steps to reproduce** 1. Let dispatch admit a Runner run from a non-assignee comment, so the run profile has `aiConnectionNonAssigneeCommentWake: true`. 2. Make the run fail in setup during preparation. 3. Press Retry. See that the request waits on `previous_execution_unverified`. **Paperclip version or commit** Observed on staging at `e82c4f0a31`; still present on `master` at `6b631dd477`. ## What Changed - `server/src/services/cancelled-native-startup.ts`: the failed-preparation proof accepts `aiConnectionNonAssigneeCommentWake` when its value is a boolean. The workspace-wait parent proof ignores the same key. - `server/src/services/legacy-preparation-retry-hold.test.ts`: staging-shape tests for a recovered setup failure (failed restore, a workspace that still names its location, the marker, all together), and a Retry that attaches to an earlier refused request and then starts. ## Verification - `cd server && npx vitest run src/services/legacy-preparation-retry-hold.test.ts`: 12 passed. Without the fix, the `comment_wake_marker`, `all` and `attached_after_refusal` cases fail (red), and the other cases pass. - Focused test file only (the machine was loaded). CI supplies typecheck, build and the full suite. ## Risks - Low risk. The marker only records why dispatch admitted the run. Every other proof clause (no process, no execution events, no native result, stopped leases, settled preparation evidence, workspace repair) is unchanged. A non-boolean value still fails the proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6b9e45787f |
fix(recovery): continue a task whose system retry was refused at claim behind a saved-work hold (#21505)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, automatic recovery continues the task, through the fresh-continuation fallback, in a new run that starts from the saved work > - A follow-up run (for example one started by a new comment or by finished subtasks) can fail before any provider work and leave a saved-work hold. The system then queues its own retry, and the dispatch gate cancels that retry at claim with `execution_reconciliation_required` because the hold is still in place > - The fallback takes over the hold when that happens, but it continued only "recent interruptions". That test required the failed run to be a recovery run itself or to fail with a restart code. A first follow-up run that failed with `adapter_failed` passed neither test, even though the system had already decided to retry it. So the fallback answered "a person must press Retry". The backlog pass used the same test, and the assigned-work sweep never listed the task. The task stayed blocked with no run, no pending path and no Retry button > - This pull request treats a system retry refused at dispatch as proof that the system continues the task. It also stops cancelled-at-dispatch attempts from counting against the hourly budget, lets the sweep continue a blocked task whose latest run was refused at dispatch, and keeps recovery notices from asking for a Retry that the task does not offer > - The benefit is that such a task continues by itself in exactly one fresh run from the saved work, whether the saved work was recovered before or after the refusal, and a person's own failed Retry keeps today's behaviour ## Linked Issues or Issue Description Refs #21394, #21348, #21363, #21478 **What happened?** A task's follow-up run, a legacy run that was not a recovery run, failed before any provider work with `adapter_failed`. Its files could not be restored, so it left a saved-work hold (`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure: restore_failed`). The system queued a retry of that run. The dispatch gate cancelled the retry at claim with `execution_reconciliation_required`, and the "older run hold" path handed the hold to the fresh-continuation fallback (`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback found the refused run and asked `interruptionEligibility` whether the interruption was recent. That predicate required `continuesAutomaticallySql(source)` (a recovery run, or a restart, deploy or bootstrap code), so it returned `person_owned` / `retry_needed`. The stranded reconciler then blocked the task with "Press Retry", but the saved-work hold reports that Retry is not available. This happened twice in a local restart-chaos wave. In one case the saved work was recovered two minutes before the claim. In the other it was recovered thirty seconds after the claim. **Expected behavior** A task never rests blocked with no live run, no pending path and no Retry. While the saved-work recovery runs, that recovery is the pending path. When it finishes, one continuation starts from the saved work through the fallback (isolation, the shared hourly cap). A run that was cancelled at dispatch before it started does not use up the automatic budget. A person's own failed Retry is still left for that person. **Steps to reproduce** 1. Fail a follow-up run (wake reason `issue_children_completed`) before provider work with `adapter_failed` and `workspaceRestoreFailure: restore_failed`. 2. Queue the system's retry of that run (`retryOfRunId`, wake reason `issue_continuation_needed`) and claim it. The dispatch gate cancels it with `execution_reconciliation_required`. 3. Before this change, no continuation is recorded, the stranded reconciler blocks the task, and nothing ever continues it. **Paperclip version or commit** master at |
||
|
|
f7b1651799 |
feat(acpx-sidecar): adopt Claude's own cycle as a provider-started turn when the Runner arms it (#21316)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The parts before this one give the adapter an ordered stream of those cycles (#21421), declare the contract (#21422), and make `paperclip-runnerd` wait for such a turn (#21423). > - This pull request is the last part in the Runner. The sidecar reads the stream, and gives a cycle to `paperclip-runnerd` as a provider-started turn only when `paperclip-runnerd` armed it. > - The benefit is that a later change in Core can let a run continue when background work finishes. Until Core sends the opt-in, a production run does the same as today. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request was first opened with the full change (49 files). It is now split. The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. This pull request: the sidecar adopts a cycle. Its base branch is `feat/runnerd-await-provider-turn` (#21423). The code at the head of this branch is the same as before the split; only the base changed. Review comments made before the split are about code that is now in the parts above. Refs #19912 (design). **What existing behavior does this improve?** What the ACPX sidecar does with a cycle that Claude starts by itself. **Subsystem affected** Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts`. Claude only. **Current behavior** - The sidecar does not read the cycle stream of the adapter (#21421), so each cycle is declined and the background state becomes unknown. - The sidecar does not know the command `session.provider_turn`, so `paperclip-runnerd` (#21423) never waits. **Proposed behavior** - The command `session.provider_turn` with `armed: true` arms the sidecar for one turn. The sidecar accepts it only for the terminal that reported a running background shell, with no active turn and a provider that is alive. One flag holds the arming. It is cleared at each turn start and turn end, on suspend and close, and when the provider is lost. - One loop, `consumeProviderCycles`, reads the stream. For each cycle: - if the sidecar is armed and no turn is active, the sidecar adopts the cycle: it emits `runtime.turn_started` with `{ "origin": "provider" }`, then the events, the usage and the terminal of the cycle, as for a normal turn; - if the cycle cannot be decided yet (the prompt turn is still settling, or `paperclip-runnerd` did not arm yet), the sidecar holds it for a maximum of 5 seconds, and then adopts or drops it; - in each other case the sidecar drops the cycle. The stream ends, and the background state becomes unknown, so `paperclip-runnerd` restarts the provider at the end of the turn as today. - A Paperclip tool call or a permission request of a held cycle waits for the decision of its own cycle. It is never bound to the prompt turn or to a different cycle. - If an adopted cycle proposes a plan (`ExitPlanMode`), the sidecar publishes the same `plan.proposed` event as for a controller turn, and the turn ends with the same terminal: completed, with no `backgroundTasks`, so `paperclip-runnerd` does not wait again. A controller turn is not changed. - If the provider exits while the sidecar is armed with no turn active, the sidecar sends a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`, and `paperclip-runnerd` ends the wait. - `turn.cancel` for a provider-started turn cancels the cycle and closes the provider. - With no arming, nothing is shown. No controller sends the opt-in yet, so `paperclip-runnerd` never arms. **Reason and benefit** - Only `paperclip-runnerd` decides that a cycle becomes a turn. The sidecar cannot start a turn by itself. - One reader of one ordered stream means that two cycles cannot overlap. ## What Changed - `src/cli/acpx-runtime-sidecar.ts` (127 lines added): the command, the arming flag, `consumeProviderCycles`, `adoptProviderCycle`, `pumpProviderTurn`, and the wait of `waitForTool` for a cycle that is not decided. - `src/cli/acpx-runtime-sidecar.provider-turn.test.ts` (new, 320 lines) and two fake hosts in `src/test-support` (227 lines). One host follows a script. The second runs the real cycle stream and the real background task tracker. - `src/cli/acpx-runtime-sidecar.test.ts` (2 lines): one more parameter for a test harness. No schema, no Rust and no persisted state changes in this part. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `pnpm run typecheck:typescript`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript`: no errors. The server check `node scripts/verify-runner-vendor-dependencies.mjs`, the cache guard and the contract tests: passed. - `npx vitest run src/cli src/drivers/acpx src/live/runnerd-acpx-provider-turn-wait.test.ts src/drivers/runner-tool-bridge.test.ts src/native-session-runtime.provider-turn.test.ts`: `128 passed (128)` files, `2147 passed`, 9 skipped. - `cargo test -p paperclip-runner-core --lib` and the `acpx_*` suites: all passed. No Rust changed in this part. - The sidecar tests of this part: a cycle that starts before the arming command, or while the prompt turn settles, is held and then adopted with its tool call; a cycle is never shown when `paperclip-runnerd` did not arm, disarmed, armed for a different terminal, armed for a quiet terminal, or armed after the provider exited; two cycles in sequence with `{ live: 2 }`, then `{ live: 1 }`, then an unknown state; cancel; a lost provider; Codex gets no change. - Mutation checks, each made the named tests fail: the sidecar arms with no matching terminal; no hold; the old hold condition that drops a cycle while the prompt turn settles; no wait of a tool call for the decision. - Live run with a real Claude account, not in CI, with the real ACP runtime and the real Anthropic API (model `claude-sonnet-5-5`), at the adapter level: one background shell gave `{ live: 1 }`, one cycle with its own usage, then an unknown state. Two shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown. All Paperclip tool calls were admitted, and none was refused. Not run: a live run through the real sidecar process, `paperclip-runnerd` and Core (that path needs an API key, not a local sign-in); Linux; `pnpm verify --full`. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in, so the sidecar is never armed. The new code that runs today is the loop that reads the stream: with no arming it drops each cycle, and the background state becomes unknown as before. - **A follow-up for a command that finishes while Claude reports on an earlier one is lost.** The Runner waits again only while a background command is still running. The run ends as it does today. The reason is in #21421. - **A cycle that starts in the last second of a prompt turn** can have its first messages in the prompt turn and the rest in the provider-started turn. Tool calls stay safe: the admission rule of #21278 refuses a call that the active turn did not announce. - **The 5 second hold** is not tested under load. A cycle that is held longer is dropped, and the run takes the path of today. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
87d363a5c0 |
perf(ui): reveal a returning Mine visit from the saved cache (#20594)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Mine inbox is the board's main list of work for one person; the
app saves its query cache in `localStorage` so a returning visit can
paint from it
> - On a hard load the saved cache gave no benefit: Mine restored its
rows but still held them until the first approvals, join-request and
dismissal reads answered, because those items sort in among task rows by
time and a late one would push rows down (#20407)
> - Those three reads were not saved, so every returning visit waited on
the network for items it showed a minute before
> - This pull request saves what Mine showed from those reads, as one
record per user and company, and lets it answer the wait on a returning
visit while every read refreshes in the background
> - The benefit is that a returning visit shows its rows with no API
response at all; the first visit keeps its bounded wait unchanged
## Linked Issues or Issue Description
**What happened?**
A returning hard load of `/:companyPrefix/issues?view=mine` with the
saved cache present (`paperclip:query-cache:v1:<user>:<company>`) showed
rows no sooner than with the cache cleared. On the hosted staging
instance, rows appeared about 40 ms after `GET
/companies/:id/join-requests?status=pending_approval` returned in 7 of
10 runs.
**Steps to reproduce**
1. Open Mine once so the app saves its query cache.
2. Delay API responses (for example 300 ms) and reload.
3. Rows appear only after the approvals and join-request reads answer.
**Expected behavior**
Rows show from the saved cache at once. The reads refresh in the
background.
Related: #20407 (kept approvals and join requests in the reveal gate),
#20408 and #20419 (Mine snapshot and its perf guard), #20591 (removes
the second wave of reads; merged), #20592 (task chat preload; merged).
## What Changed
- `ui/src/lib/inbox-mine-reveal-sources.ts` (new): the record `["inbox",
companyId, "mine-reveal-sources", userId]` holds the approvals visible
in Mine, the pending join requests, and the dismissals of those items
and of Mine's failed runs. A record over 256 KB is replaced by an empty
`incomplete` record that never answers the wait. `readMineRevealSources`
accepts a record only for the same user and company.
- `ui/src/lib/persisted-query-cache.ts`: the saved cache keeps that
record per user and company, only when its data names the same user and
company as its key.
- `ui/src/pages/Inbox.tsx`:
- Mine writes the record once approvals, join requests and dismissals
all have server answers.
- On a returning visit with a restored Mine snapshot, the saved
approvals, join requests and dismissals stand in while their reads are
pending, and after a transient failure. A refusal (401/403/404) or a
server answer replaces them. The first-attempt wait is skipped only when
saved values stand in; the first visit is unchanged.
- Selection fix: the selected row's key is recorded only when the
selection moves. Before, every list change recorded the row now at the
old index before the re-anchor ran, so a row inserted above the
selection (a late approval, or any live insert) moved the selection to
another row.
- `ui/src/hooks/useInboxBadge.ts`: `useInboxDismissals` takes an
optional saved `fallback` and returns `hasServerData`. Other callers are
unchanged.
- Tests:
- `Inbox.test.tsx`: a returning visit reveals rows and the saved
approval while the reads are pending, then keeps selection on the same
row when a newer approval arrives above it; a transient failure keeps
the saved approval and a 403 drops it; a record stamped for another user
is ignored and the page waits as a first visit does.
- `inbox-mine-reveal-sources.test.ts`: projection, size limit,
user/company checks, persistence scope.
- `persisted-query-cache.test.ts`: the record round-trips for the same
user, restores as invalidated, and another user's record is not saved.
- `useInboxDismissals.test.tsx`: fallback while pending, after a
transient failure, dropped after a refusal, and no fallback.
- `tests/perf/collections/mine-returning-visit.spec.ts` +
`returning-visit-fixture.ts`: production-build browser checks and the
timing case; README updated.
## Verification
- `cd ui && pnpm exec vitest run src/pages src/lib src/hooks
src/components/coordinator`: 453 files, 5574 tests pass.
- Browser checks on a disposable local instance (production UI build,
`local_trusted`, embedded PostgreSQL, Chromium headless, 1440×900):
`PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec
playwright test --config tests/perf/collections/playwright.config.ts
mine-returning-visit.spec.ts`
- With every read of the company held, task rows and saved approvals
show and no company read has answered. Fails on `master`, passes here.
- An approval dismissed after the cache was saved stands in, then leaves
when the reads answer. Fails on `master` (no saved approvals), passes
here.
- Another company's Mine never shows the saved company's rows. Passes on
both.
- Timing (5 fresh contexts per arm, alternating saved/cleared, every API
response +300 ms, same instance and fixture, host load average about
20). "Responses before rows" counts API responses that finished before
the first row:
| Build | Saved cache: median (range) | Responses before rows | Cleared
cache: median (range) |
| --- | --- | --- | --- |
| `master` (`99edcd28a7`) | 1028 ms (906–3011) | 16–19 | 1502 ms
(1423–1592) |
| This PR | 625 ms (589–672) | 0 | 1632 ms (1359–1692) |
| This PR + #20591 | 625 ms (604–671) | 0 | 1495 ms (1398–1585) |
Saved cache: difference of medians 403 ms. The remaining 625 ms is
JavaScript download and route-chunk loading in a fresh browser context;
no API response precedes the rows. Cleared cache: no change is claimed;
the spread overlaps.
- Existing collection specs on this build: `parent-context.spec.ts`
passes; `startup-regressions.spec.ts` has 4 failures (partial
failure/context error and two 429 cases) that fail the same way on the
`master` build. `mine-workspaces.spec.ts` was not run (needs a database
URL for its fixture).
- Review follow-up (
|
||
|
|
b7922570e1 |
fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.
Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.
Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on
|
||
|
|
4a78124a3e |
feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn that ended with no result while a background shell is running, this stops the shell and the run fails. > - #21278 binds each tool call to its turn, and #21293 reports how many background shells are running at the end of a turn. Part 4 declared the contract. > - This pull request makes `paperclip-runnerd` wait for a turn that Claude starts by itself, when the controller asks for it, and admit that turn into the same run. > - The benefit is that the Runner has one place that decides to wait and one place that ends a wait. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 5. Its base branch is `feat/prp-provider-turn-contract`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** What `paperclip-runnerd` does after a Claude turn ends with no result while a background shell is running. Nothing changes until a controller sends the opt-in. **Subsystem affected** Paperclip Runner: the ACPX provider backend in `runner-core` (Rust). Claude only. **Current behavior** - After each completed turn the Runner emits `run.terminal`. A turn with a running background shell gets the provider restart, and the shell stops. **Proposed behavior** Nothing changes unless the controller sends `providerStartedTurns: true` in the payload of `turn.start`. No controller sends it yet. With that field: - The Runner keeps the provider and waits when all of these are true: the agent is Claude; the turn completed; the run has no accepted `paperclip_finish` or `paperclip_block` result; the terminal event reported `backgroundTasks` with `live` above zero; each condition of #21293 for a kept process is true; the sidecar accepted `session.provider_turn` with `armed: true`. - `turn.completed` then has `awaitingProviderTurn: true`, and no `run.terminal` follows. The Runner holds that `run.terminal`. - When the sidecar reports a turn that the provider started, the Runner admits it only during a wait and publishes `turn.started` with `origin: "provider"`. In each other case that report fails the session closed. - When a wait ends with no turn active, the Runner publishes the held `run.terminal`. This applies when the provider process exits, the sidecar exits, the controller cancels or stops the run, the session is closed or suspended, and after a Runner restart. - The held `run.terminal` and the removal of the wait are saved in one write, and the controller gets the event from the next poll. A crash after a cancel or a stop cannot lose it, and it is never sent two times. - A controller `turn.start` during a wait replaces the provider first, as before, and the held event is dropped. - A `run.attach` from a different run during a wait ends the wait when the attach succeeds. The Runner then discards the held `run.terminal`, because that event is of the old run and must not end the new run. This is the same rule that a Runner restart applies. The wait is removed only in the state that the attach saves for the new run. If the attach fails, for example when the old session does not stop, the old run keeps the wait and gets its held `run.terminal` one time, from the next poll or from a cancel or a stop. A `run.attach` from the same run publishes the held event to that run. - The Runner waits at most 5 seconds for the sidecar to answer a request to arm or to disarm a provider-started turn. This is the limit that turn cancellation already uses. Before, the limit was the 30 seconds of an ordinary request, and a cancel, a stop or a shutdown had to wait behind it. No answer is handled as a lost sidecar: the session fails closed, and the Runner still publishes the held `run.terminal` one time. - The sidecar of this branch does not know `session.provider_turn`, so the arming request fails and the Runner does not wait. The behavior comes with part 6. **Reason and benefit** - The wait is one entry in the saved session state. One rule ends it at each save, so each new way to leave the open session ends the wait with no new code. - A wait must not outlive the provider or the sidecar, and a cancel must stop it. ## What Changed - `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests): `provider_turn_wait { turn_id, held_run_terminal }` in the saved state; the rule that ends a wait at save; `settle_turn_events`, which decides at one place to arm, to create the wait and to put `awaitingProviderTurn` on `turn.completed`. - `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its terminal, the arming request, and `admit_provider_turn`. - `acpx_provider_session.rs`, `acpx_provider_backend.rs`, `acpx_provider_state.rs`, `provider_events.rs`, `acpx_park_preparation.rs`: the opt-in from `turn.start`; the record that the run has a result; the provider-started turn in the state and in the PRP events; cancel and stop with no turn; no turn-end restart and no park during a wait. - `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no event also checks that the sidecar process is alive. A child process can keep the output stream of a dead sidecar open. - `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and failures for tests. - Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`, `tests/support/acpx_quiet_turn_restart.rs`, `src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon), `src/native-session-runtime.provider-turn.test.ts` (the controller of today). - `runner/DURABLE_TRANSPORT.md`: the rules. ## State saved by the previous release - One new optional field in the saved ACPX session state: `providerTurnWait`. It is written only during a wait. In each other state the saved file has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so there is no wait. A unit test covers this. - **State from this release, read by the previous release (a rollback):** outside a wait, the file loads as before. During a wait, the previous Runner refuses the file with its "ACPX provider state is malformed" error, because its loader refuses each unknown field. Only a run that waits at the moment of a rollback is in this case, and no run waits before a controller sends the opt-in. - **Runner restart during a wait:** the wait is restored. The provider is a new process, so the wait is over, and the controller receives the held `run.terminal` one time. A test with the real daemon covers this. - **Controller of today or of the previous release:** `turn.start` has no opt-in. A test runs a turn with a running shell and a turn with an unknown background state, and compares the events, the commands after the turn, the restart, and the park state before and after the restart. They are the same. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored. All 17 `acpx_*` suites passed. - `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends with exactly one `run.terminal` when the sidecar exits while a child process keeps its output open, when the provider is lost, when the controller cancels, and after a Runner restart; with no opt-in, `run.terminal` follows the turn and the sidecar is never armed. - Rust tests: wait and admit; no wait with a result; no wait with no opt-in; an active goal neither marks nor arms a wait; a sidecar that refuses to arm; a provider-started turn that the Runner did not wait for fails the session closed; cancel and stop during a wait; provider and sidecar loss; restart; the saved shape in both directions. - `src/native-session-runtime.provider-turn.test.ts`: the real `consumeTurn` of today gets the event sequence with no opt-in and finalizes it as `native_finalization_missing`, as today. - Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a different run attaches after the snapshot reports `warmAttachReady: true`; the new run gets no `run.terminal`, the saved state has no wait, and the old session is closed. A same-run attach gives exactly one `run.terminal`. An attach that fails when the old sidecar is dead leaves the wait with the old run, and the old run then gets exactly one `run.terminal` from a poll, `run.cancel`, `turn.interrupt` or `turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in the pending events of the new run; with the wait ended before the attach is complete, the old run gets no `run.terminal` after a failed attach. - No answer from the sidecar, in `tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not answer the disarm request; `run.cancel` and `turn.interrupt` each return `already_settled` in less than 10 seconds, exactly one `run.terminal` reaches the run, `session.close` completes in less than 2 seconds with no second `run.terminal`, and the provider process is gone. With no answer to the arm request, the turn ends with its `run.terminal` and with no wait. With the limit removed, each test fails at 30 seconds. - Mutation checks, each made the named tests fail: the rule that ends a wait at save; no `run.terminal` after a restart; arming with no opt-in; the marker set with no wait; the sidecar process check removed. Not run: a rollback with the binary of the previous release; a run with the opt-in and a sidecar that answers the arming request with an error (the code reads each error as not armed); Linux; the full suite. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in. - **A rollback of the Runner during a wait** makes the previous Runner refuse that saved session. See "State saved by the previous release". - **The wait has no time limit in the Runner.** The controller must close, suspend or cancel. Core will own that limit. - **Park is refused during a wait.** The controller must end the wait before it parks. - **A new run can attach during a wait.** The attach is not blocked, because the Runner has no time limit on the wait and a new run can send only `run.attach`. After an attach that succeeds, the old run gets no `run.terminal` from the Runner; the controller must not depend on that event. The request that disarms a sidecar that the attach keeps has no test, because no test can keep a session across an attach during a wait. - **After a sidecar that does not answer, `runner.suspend` returns an error at once** ("provider lifetime cleanup is not yet proven"), and `session.close` completes. This is the behavior of today for each lost sidecar; this PR does not change it. - **The shared code that changed** runs for each Claude turn: the record of a run result, and the settle step at each turn end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.1 |
||
|
|
689f3b190e |
fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Automatic recovery continues interrupted and failed tasks without a person, with a per-task limit of 6 automatic attempts per hour > - There is no company-level limit. When the sandbox provider refuses to create sandboxes, every task fails the same way, and recovery keeps creating new runs for each task > - On a hosted instance, one hour of a provider outage made 510 automatic runs, 587 failures and 0 successes across 289 tasks. No task had more than 4 runs, so the per-task limit did not engage. Each start asked a refusing provider for a sandbox and used server memory, which made the outage longer > - Recovery must still continue every task by itself. The answer is to wait while the infrastructure fails, not to give the task to a person > - This pull request adds a company-scoped pause for automatic work: it opens when recent automatic runs fail before the agent starts, the recovery passes stop creating new automatic runs while it is open, and it closes by itself when runs start normally again > - The benefit is that an infrastructure outage no longer multiplies itself through recovery, and recovery resumes with no person acting ## Linked Issues or Issue Description **What happened?** During a sandbox provider outage, automatic recovery created hundreds of new runs per hour across hundreds of tasks. All of them failed before the agent did any work. The per-task limit did not engage because no single task reached it. **Expected behavior** When most recent automatic runs of a company fail before they start, recovery waits instead of creating more runs. It tries again at a small rate. It resumes at full rate when runs start normally again. A person's work is never held. **Steps to reproduce** 1. Make the sandbox provider refuse lease creation (HTTP 503) for a company with many assigned tasks. 2. Let recovery continue the failed tasks. 3. Count automatic runs created per hour: it grows with the number of tasks, and every run fails at sandbox creation. Refs #21394 (continuation classification this change will reuse when it lands), #21433 (start-pass rewrite that the start hook will build on). ## What Changed - New table `automatic_run_brakes` (one row per company, cascade delete with the company; migration `0480`). Migration `0481` builds `heartbeat_runs_company_finished_idx (company_id, finished_at desc) where finished_at is not null` concurrently, so the signal reads the runs that finished in the window directly. It stores the state (closed, open, half_open), the reason with the window numbers, and the compare-and-set fields that two servers share. A company with no row is closed. - New module `server/src/services/recovery/automatic-run-brake.ts`: - Signal: on the database clock, the automatic runs of the company that finished in the last 10 minutes (and since the last state change), newest 500 by finish time, plus up to 200 running runs. A run counts as a failure only when it ended in its own start phase: no provider work (`executionRecovery.providerWorkStarted = false` or the native `nativePreDispatchFailure` record), a legacy run still preparing, or a transient setup code. A run interrupted by a server stop, restart or lease loss is neutral, so a release swap does not open the pause. A run that succeeded, or ended for a person-owned or task reason, counts as healthy. - Opens at 12 or more counted runs with 80% or more failures, or on 5 sandbox refusals (`environment_provider_unavailable`) in a row. - Each outcome counts once, at its own time: a finished run at its finish time, a running run that got past setup at its start time. Read time never orders evidence. - Closes in two steps: open moves to half_open after at least 2 minutes when the newest 2 counted runs are healthy or the newest one succeeded; half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64) and then closes. It reopens from half_open if the signal trips again. It closes by itself after 15 minutes with no automatic activity. - Reads are one indexed query per company, cached for 5 seconds in a bounded in-process map (256 companies). Each state change writes one structured log line (`automatic_run_brake.opened|half_open|closed|reopened`) with the numbers. - It also contains the start gate `holdAutomaticRunStart` (one probe at a time, at most one new probe per 60 s, release budget per window by compare-and-set, no I/O while the cached "held until" time has not passed, person origin taken as an argument with a cached fallback). This pull request does not call it from the start pass; see Risks. - Recovery passes defer creating runs while the pause is open. The gate is only where a run is created: - `deliverReconciledExecutions`: after the agent-room check and right before the wake, the delivery asks for a creation slot; when the pause holds it, the continuation stays pending. The held backlog (`continueHeldExecutions`) is not gated: recording a pending continuation creates no run and costs nothing, and its delivery passes this gate once. - Assigned-work sweep `act()`: after the process rate limit and before it claims a continuation, it asks for a creation slot; when held, it returns `deferred_rate_limit` with the next try time, so nothing is written to the per-task attempt ledger. New assignments with no earlier run are not deferred. - A reserved slot is given back (compare-and-set on its own timestamp) when no run is created: the wake is refused or throws, the task is capped, the claim changed, or the task went back to review. The give-back is conditional in SQL on no run for that task having been created since the reservation, so a wake that commits its run and then throws keeps the slot. - The sweep's overdue saved-work recovery path returns before the slot is asked for: it starts a recovery attempt, not a run, and is never held. - Person work is never deferred. The gate is called only at the delivery (`agentHasRoom`) and the sweep (`act()`), and each call passes the task and the run it continues. While the pause is open, the gate does not hold the record when any of these is true; it holds only when none is. With no task or no source run it does not hold (fails open). | Shape | How it is recognised | | --- | --- | | A Retry or "move forward" waiting for the task (`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an unstarted wake request from a user for the task | | A Retry folded (`coalesced`) into the continuation, or a Retry while the delivery is pending | a wake request from a user for the task, requested after the source run finished (any status) | | A person's message or a new assignment by a person after the last run | the same: a wake request from a user for the task after the source run finished | | A held task selected because a person pressed Retry (the refused Retry run) | a run for the task created after the source run finished whose wake is a person reason (`task_retry`, chat message), or that has `taskRetryRequestId` or `explicitUserContinuation` | | A continuation of a person's Retry (any depth) | the source run's chain reaches a person's run (`listPersonOriginSourceRunIds`, the dispatch rule) | | A Retry during pending delivery that becomes the continuation itself (`deliverContinuationAsRun`) | not gated: that run is created by the person's own wake, never by these passes | | The held backlog's saved-input path | not gated: recording or releasing a held continuation creates no run here | - To prevent a deadlock, a pass may still create one automatic run per minute per company when no automatic run of the company started in the last 2 minutes, however many runs wait (compare-and-set, shared by servers). - `GET /api/companies/:id/assigned-work-invariant` now returns `automaticWork` with the state, the time it paused, the window numbers and a plain sentence, for example "Automatic work is paused for this company since T because 47 of 50 recent automatic runs failed before starting." - `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is now exported, unchanged, so the gate uses the same person-origin rule as dispatch. - The outcome rules (person-owned codes, restart codes, transient setup codes, native pre-dispatch record) come from their owning modules through one memoised dynamic import. A static import of `heartbeat/retries.ts` made a module cycle at server start. ### What the pause consumes, and how waste is bounded | Token | Taken | Given back or bounded | | --- | --- | --- | | Creation slot (one per minute per company while open) | right before the run is created, after every other check | released only when no run for the task was committed after the reservation | | Probe slot (start gate) | when the start gate lets one automatic run through | another run may take it when the reserved run has not started within 10 seconds (claim declined, agent paused or full) or was cancelled, deleted or finished | | Gradual-release start (start gate) | one per run the start gate lets through | a declined claim spends one start of that window; the window ends within 60 seconds and the next one doubles | | Probe success count | once per probe run, by compare-and-set on its id | a probe run counts once; cancelled or deleted probes count as neither success nor failure | ### When the pause opens, stays open and closes | State | Leaves when | Bound | | --- | --- | --- | | closed | opens when, in the last 10 minutes since the last change, 12 or more counted automatic runs finished and 80% or more failed before starting, or the newest 5 counted runs were sandbox refusals | none needed | | open | moves to half_open when it has been open 2 minutes or more and the newest 2 counted runs are healthy, or the newest one succeeded | evidence arrives at least once a minute: when no automatic run started in the last 2 minutes, a pass may create one per minute | | open | closes when no automatic run started or finished for 15 minutes (queued runs do not count) | 15 minutes after the last activity | | open | stays open only while new automatic runs keep failing before starting | each new try is spaced by the existing passes and by one creation per minute when nothing starts | | half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6 minutes | | half_open | reopens when the signal trips again, counted only since half_open began | none needed | Restart-class endings (server stop, restart, lease loss, process loss, interrupted native session) are neutral in every state, so a release swap does not open the pause. ## Verification - `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests, pass): classification; opens on 20 failures before start; 50 queued automatic runs give exactly one probe and no failure or attempt; a person's Retry, a new assignment, a run a person joined and the continuation of a person's Retry are not held; mixed and person-owned outcomes stay closed; old data is ignored; 40 working runs interrupted by a restart stay closed, then the same continuations failing at sandbox acquisition open it; 5 refusals in a row open it; failed probe waits for the interval, two probes past setup move to half_open, the release gives exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen from half_open; probe slot frees after cancel, delete or no start; quiet close after 15 minutes; two instances on one database agree and start one probe; one creation per minute while open across servers; 250 held runs over 100 passes use at most 5 queries (at most 255 with the person-origin fallback, one per run, once); caches stay at 256 companies and 1024 runs. - `npx vitest run src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one test per person shape above at the delivery and the sweep, with the pause open and no creation slot, plus a control that still holds a continuation no person asked for): includes "no pause record: every pass behaves as before", "the waiting run cannot start: a pass continues a task on another agent, then the pause closes", "one held task spends one creation slot", and "a person's Retry or message is never paused at the delivery, the sweep or the held backlog".: sweep, delivery and held backlog create nothing while open, write no attempts, and continue by themselves after recovery; the sweep still continues one task a minute when nothing automatic waits. With the three deferral lines removed, all 4 fail (red), and they pass with them (green). - The pause reads as closed and logs once when its run rules cannot load (test in `automatic-run-brake.test.ts`). - `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the module loads after the heartbeat service, in the server's import order. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify`: see the comment below for the result. CI supplies the full suite. ## Risks - This pull request does not hold starts. Already queued runs and `transient_failure_retry` runs still start while the pause is open; they act as the natural tries. The start gate goes in as a follow-up on top of #21433, at a point before that pass counts a claim attempt and reads the full run row, so a held run costs no attempt and no read. - The sweep reuses the `deferred_rate_limit` outcome for a deferral, to keep its outcome type unchanged while #21394 edits that file. - The automatic-run predicate is built from master exports (`RECOVERY_WAKE_REASONS`, retry markers). It will switch to `isAutomaticContinuationRun` when #21394 lands. - Migration `0480` adds one small table. - Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`) runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement: it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in `packages/db/src/client.ts`, like 0470 and 0476, and the runner drops and rebuilds an invalid index after an interrupted build. The index is declared in the Drizzle schema, so `pnpm db:generate` reports no changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On a large `heartbeat_runs` table the build is online (no write lock) but takes time proportional to the table. After it exists, it is one more index on a hot table: each run that finishes writes one more index entry. - No user-visible text changes outside the operator report. ## Model Used - Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell, file edits, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
6826800425 |
feat(ci): report the required checks on merge-queue commits (#21545)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each merge to `master` publishes the standard image, and the time from merge to published image limits how fast a fix reaches users > - #21096 lets `docker.yml` push an image that the merge queue already built, but the merge queue is not on > - The `master` ruleset requires `ci / verify` and `ci / e2e`. Only `pr.yml` reports them, and only for pull requests > - A merge queue waits for the same two names on its merge-group commit. No workflow reports them there, so every queued merge would stall > - This pull request adds a workflow that reports both names on merge-group commits > - The benefit is that the owner can turn the merge queue on with one ruleset change ## Linked Issues or Issue Description Refs #21096 **What existing behavior does this improve?** The merge-queue image build from #21096. It cannot run until the merge queue is on, and the merge queue cannot be turned on safely. **Current behavior** `ci / verify` and `ci / e2e` come from `pr.yml`, which runs on `pull_request` only. With a merge queue, GitHub waits for both checks on the merge-group commit. They never arrive. Each queued pull request would time out and leave the queue. **Proposed behavior** `merge-queue-checks.yml` runs on `merge_group` and reports `ci / verify` and `ci / e2e`. It runs no tests. It does not check out the queued commit. It has no token permissions and no secrets. **Reason and benefit** A pull request enters the queue only after both checks pass on the pull request. A queued merge therefore has the same protection as a direct merge today. The exact-source gate still tests each `master` commit before publication. Real tests on the combined tree can replace these jobs later under the same names. ## What Changed - Added `.github/workflows/merge-queue-checks.yml`. It reports `ci / verify` and `ci / e2e` on merge-group commits. - Added `scripts/__tests__/merge-queue-checks-workflow.test.mjs`. It fails if the names differ from the names that `pr.yml` reports, or if the workflow gains a checkout, an action, a secret, a token permission or a self-hosted runner. - `pr-trusted.yml` now runs that test and `scripts/merge-group-image.test.mjs` in a step that skips test files an older checkout does not have. The second test came with #21096 but no CI step ran it. - `scripts/merge-group-image.mjs find` now waits up to 15 minutes for a merge-queue image build that is still queued or running. The queue can land a commit before its image build ends. Without the wait, the `master` push always builds from source. - `doc/RELEASING.md` explains the workflow and lists the ruleset settings that turn the queue on and off. ## Verification - `node --test scripts/__tests__/merge-queue-checks-workflow.test.mjs`: 2 pass. - `node --test scripts/merge-group-image.test.mjs`: 10 pass, with new cases for unfinished builds. - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: pass. - `actionlint` on both changed workflows: no findings. - Not verified: no `merge_group` event can fire before the queue is on. The first queued merge is the live test. `pr.yml` calls `pr-trusted.yml@master`, so the new CI step runs only after this merges. Owner steps after merge: 1. In the `master` ruleset, add **Require merge queue**: Squash, build concurrency 5, group size 1 to 5, status check timeout 60 minutes. 2. After the first queued merge, confirm that the `master` commit SHA equals the SHA of its merge-group runs. ## Risks - Low risk while the queue is off. The workflow has no trigger other than `merge_group`. - With the queue on, the queue does not test the combined tree. Two pull requests that pass alone can still break `master` together. This is the same as today. - A merge-queue image build that hangs delays the source build by up to 15 minutes. A build that fails does not. - If GitHub does not accept a job named `ci / verify` for the required context, queued merges time out. Rollback: remove the merge queue rule. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f6c5986b61 |
feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service
## Linked Issues or Issue Description
**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.
**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.
**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.
**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.
**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.
## What Changed
- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.
## Verification
- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.
## Risks
- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
6b631dd477 |
fix(recovery): check held retained orphans only for candidate leases (#21542)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciliation pass lists in-progress records of each
kind every 30 seconds, and settles the records whose owner is gone
> - One kind, the retained orphan sandbox, lists leases with a large SQL
filter that has correlated subqueries on `heartbeat_runs`, `issues`,
`issue_recovery_actions` and `environment_leases`
> - The planner ran those subqueries for every released, failed and
expired lease in history, before the cheap orphan predicates. The cost
grew with lease history, not with the number of orphans
> - On a busy hosted database one page of this listing ran for 17 to 138
seconds as one statement. It held an AccessShareLock on `heartbeat_runs`
for all of that time, so an online `ALTER TABLE heartbeat_runs ADD
COLUMN` with a 5 second lock timeout failed on every try
> - This pull request runs a cheap candidate filter first and evaluates
the held checks only for candidate rows
> - The benefit is that the listing cost now follows the number of
candidates. Schema changes on `heartbeat_runs` no longer wait behind it,
and the pass reads far fewer pages
## Linked Issues or Issue Description
No public issue exists. This describes the bug in the format of the bug
report template.
**What happened?**
A hosted instance could not apply a pending migration that adds a column
to `heartbeat_runs`. The deployer sets `lock_timeout = 5s` and tries
three times. All three tries timed out. The database showed one session
that ran `select "id", "company_id", "heartbeat_run_id", "issue_id",
"metadata", "updated_at" from "environment_leases" ...` in state
`active` for up to 138 seconds. Its transaction start and query start
differed by less than 0.1 seconds, so it was one statement. It held an
AccessShareLock on `heartbeat_runs`. That instance had about 18,600
lease rows: 13,240 expired, 4,695 released, 610 failed, 69 retained, 17
active and 9 pending cleanup. Each failed try also stalled all
statements on `heartbeat_runs` for up to 5 seconds.
**Expected behavior**
The ownership listing must finish in a short time that does not grow
with lease history. It must not keep a lock on `heartbeat_runs` long
enough to block an online schema change.
**Steps to reproduce**
1. Use a database with many terminal environment leases (expired,
released or failed) and few or no retained orphan sandboxes.
2. Run the `environment_retained_orphan_sandbox` listing
(`OWNABLE_KINDS`, `listInProgress`), or wait for the ownership
reconciliation pass.
3. Run `EXPLAIN (ANALYZE, BUFFERS)` on the listing. The `heartbeat_runs`
primary key subplans run once for each terminal lease, not only for
candidates.
4. While the listing runs, run `ALTER TABLE heartbeat_runs ADD COLUMN
probe jsonb` with `lock_timeout = '5s'`. The ALTER waits until the
listing ends.
**Paperclip version or commit**
`master` at
|
||
|
|
1dee321844 |
perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every native event that announces work to another part of the server (chat progress, a final response, an evidence outcome, recovery) registers a wake in its transaction before it writes the event > - Such an event takes six statements: BEGIN, the run lock read, a replay probe, a read of the transaction id for the wake, the append, and COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock, the transaction id, the write, COMMIT > - In the agent scaling load model these two paths are now the largest per-agent cost after #21511 and #21520, and other events fall behind first at 600 agents > - This pull request returns the transaction id from the lock read that each path already sends, and moves the replay probe into the run lock read when it is safe > - The benefit is four statements for such an event and four for a checkpoint save, with the same locks, the same wake order and the same replay rule ## Linked Issues or Issue Description Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The statement count of native events that register a wake (`appendNativeSourceEvent`) and of controller checkpoint saves (`nativeRunnerCheckpointWriter`). **Subsystem affected** `packages/db/src/work-signals.ts` (one new export), `server/src/services/heartbeat-run-events.ts`, `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Current behavior** `signalDatabaseWork` sends `select pg_current_xact_id()` for the first wake of each transaction. A native event with a wake reads the run lock, then probes for its replay in a separate statement, because a statement that waits for the lock keeps its earlier snapshot. **Proposed behavior** - The run lock read and the checkpoint owner lock read also return `pg_current_xact_id()::text`. Both already lock a row, so the transaction has an id at that point. `primeDatabaseWorkTransactionId` gives it to the transaction's wake state, and the wake sends no statement for it. - For an event that may register a progress, response, reassessment or retention wake, the run lock read also reads `next_event_seq` from its own snapshot and checks for a replay in that snapshot. Every event append advances `next_event_seq` in its transaction. So when the locked row has the same `next_event_seq` as the snapshot, no append committed in between, and the probe result is current. Otherwise, or when the probe finds a replay, the separate probe runs as before. **Reason and benefit** Fewer round trips while a pool connection is held in a transaction. In the load model these paths hold the main pool and the three-connection checkpoint pool longest per event. **Breaking changes** None. No schema change. ## What Changed - `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`, exported from `@paperclipai/db`): inside a transaction whose id is not yet known, it records the id. Outside a transaction it does nothing. An uncertain commit still resolves with `pg_xact_status` on the same id. - `appendNativeSourceEvent`: the lock read returns the transaction id when the event has any wake. For a probing wake it is one statement: a CTE that reads `next_event_seq` from the snapshot, a CTE that locks the run row (`for no key update`, unchanged), and `exists (...)` for the replay. The wake is still registered after the replay check and before the append. - `lockNativeRunnerCheckpointOwner` returns the transaction id with its owner lock and primes the wake state. Its return value and every caller are unchanged. ## Verification - Red/green on embedded PostgreSQL; each new assertion failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "registers a native event's wakes before its write and only when the event is new": four statements (before: six), the wake's intent and settlement are published, and a replay publishes no wake. - New: "finds a replay that another writer committed while the lock read waited, and registers no wake for it". Another connection holds the run row (the test waits until PostgreSQL reports the append waiting for that lock), stores the same event and moves `next_event_seq`; the waiting append returns `duplicate`, runs the separate probe, and publishes no wake. With the `next_event_seq` comparison removed, this test fails. - New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the owner lock with the transaction id, the write, COMMIT (before: five statements), and its recovery wake is published; a save after the lease expired writes nothing and fails with `native_runner_checkpoint_owner_lost`. - `work-signals.test.ts`: a primed transaction sends no id statement for its wakes; a primed transaction whose connection is killed after its write stays pending and then resolves through `reconcileDatabaseWork`, with no row stored. - Added after merging `origin/master` (with #21511, #21520 and #21436) into this branch: - "gives the wake the transaction id of the transaction that took the run lock": inside a caller's transaction, an event with a progress wake sends no separate `pg_current_xact_id()` statement; the transaction's connection is then killed, and `reconcileDatabaseWork` checks `pg_xact_status` with exactly the id that `pg_current_xact_id()` returns on that same transaction. Priming a wrong id makes it fail. - "primes nothing when the lock read fails, so the transaction is not left uncertain": a lock read that fails on `lock_timeout` leaves no pending work; after the same failure inside a savepoint, the next signal on that transaction still sends its own `pg_current_xact_id()`. Priming before the lock read makes it fail. - Both call sites pass only the value returned by the statement they just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)` then `primeDatabaseWorkTransactionId(tx, run.xid)`) and `lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then `primeDatabaseWorkTransactionId(tx, xid)`). - Retest on the combined tree, one file at a time: atomic-append 28 (including the equivalence test and both race tests), the port suite with conformance 105, checkpoint-save 2, `live-recovery-work` 51, `work-signals` 25, runner `native-session-runtime` 297; server typecheck passes. No code change was needed for the combination. - Other focused suites pass, one file at a time: the port suite with its conformance suite (100), `chat-native-progress-work` (35), `native-observer-replay` (1), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-recovered-owner-followup` (97), `native-restart-recovery-isolation` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25). - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). Stacked: this pull request merged locally with #21511, and the model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model applies the port's batching rules itself; that mode is not in any pull request). Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. "Before" rows are `origin/master` from the same session. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 / 12,000 | 1 / 70 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 / 24,000 | 2 / 1,274 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 / 72,000 | 1,136 / 4,980 | | 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 | 71,958 / 72,000 | 1,319 / 5,712 | | 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 / 48,000 | 1,974 / 3,835 | With the three changes, one process finishes 99.9% of the offered work at 600 agents, but its event-loop utilization is 0.90–0.93, above the 0.75 target, and p99 lateness is about 5 s. It saturates between 600 and 800 agents. - Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5 where commit-time notifications are on). ## Risks - The transaction id is read in a statement that already assigns one (a row lock), so no transaction gets an id it did not have before. - The probe in the lock read is used only when `next_event_seq` did not change between the snapshot and the lock. Every writer of `heartbeat_run_events` moves `next_event_seq` in the same transaction (native and ordinary appends, the run coordinator store, the retry queue). A future writer that inserts events without it would make this probe miss a concurrent replay; the append statement then still finds the replay and stores nothing, but the wake would already be registered (a spurious wake, not a lost or duplicate event). - This pull request and #21511 both edit `heartbeat-run-events.ts`, in different functions. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dc018f9bc0 |
Keep agent-owned connections healthy during catalog refresh (#21539)
Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization. Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation. Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
833689a3fc |
fix(queue): say a long-queued turn is waiting to start, and name the routine execution that holds it (#21447)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21405 made a task that waits for its agent's concurrency limit show "Waiting for <agent> · N slots max · P of Q in line · Configure" > - On hosted staging, after that deploy, a routine task stayed queued for 8 minutes while its agent had 19 free slots. Its chat said "Working for 7m · Thinking", and the server label was a plain "Queued" > - The real cause was the routine's one-execution-at-a-time rule: the previous execution of the same routine was still open with its run. That is correct scheduling, but the UI did not say it > - A queued turn also kept the startup feedback ("Working") for as long as it stayed queued, with no limit > - This pull request names the routine hold and limits the startup feedback to 30 seconds > - The benefit is that a queued task always says that it waits, and why when the server knows ## Linked Issues or Issue Description Refs #21405 **What happened?** On staging, one routine task's continuation run was queued at 17:39:03Z and claimed at 17:47:10Z. The agent ran 1 of 20 runs. The chat showed "Working for 7m · Thinking". The run's projection label was "Queued" with no cause. **Why did it happen?** 1. The routine allows one open execution with a run at a time (unique index `issues_open_routine_execution_uq`). The previous execution of the same routine was still open and held its run until 17:47:10.172Z. The waiting run was claimed 345 ms after that task closed. The claim was refused with "Another execution for this routine is already in progress" until then. This is the routine's intended policy, not a scheduling bug. 2. The execution projection did not name that hold, so the server label was "Queued". 3. The task chat keeps the startup feedback ("Working" and "Thinking") for a queued turn with no named hold (a DESIGN.md rule). That rule had no time limit. **Expected behavior** A routine task that waits for the routine's open execution says "Waiting for <task> to finish". A queued turn that has no named hold says "Waiting to start" after a short grace period. ## What Changed - Server: `executionProjectionsForRuns` reads, in one query per batch, whether another open task of the same routine holds the routine's execution. For a never-started queued run that waits on it, the projection sets the cause `routine_execution_in_progress`, the label "Waiting for <task identifier> to finish", and a next action. This check comes before the agent-limit check, because a free slot would not start the run. - UI: a new hook, `useWaitingToStart`, keeps the startup feedback for 30 seconds. After that, a queued turn with no named hold shows "Waiting to start": an hourglass, no spinner, no shimmer. The task chat's runner header, its current-activity tail and the live run pill use it. The hook uses one timer at the threshold and no per-second ticks. - DESIGN.md: the startup-feedback rule now has the 30-second limit and the routine hold. - Storybook: `WaitingToStart` and `WaitingForRoutine` stories, with play checks. **Why 30 seconds:** DESIGN.md keeps "Working" from message submission through admission, so that a normal start does not flicker. On staging, runs that were not held started 3 to 8 seconds after they were queued (from the company's run list between 17:30 and 17:48Z). 30 seconds is several times that, and shorter than any real wait the user should be told about. ## Verification - Red/green server test (embedded Postgres), `execution-projection-agent-queue.test.ts`: a routine task waits while another open task of the same routine holds a run. The test checks the label "Waiting for Q-3093 to finish" and the cause. It checks that the holder is not marked as waiting on itself, and that the label returns to plain "Queued" when the holder closes. It fails without the server change. - Red/green UI tests: `TaskChatRunnerTurn.test.tsx` (startup feedback until the threshold, then "Waiting to start" in the header and the tail; a running turn is never called waiting) and `TaskChatLiveRunPill.test.tsx` (the label changes, with no spinner or shimmer). Both fail without the UI change. - `vitest run` on the four `execution-projection-*` server suites (36 tests) and all `task-chat` and `TaskChatThread` UI suites (1,414 tests). UI and server typecheck. - The new stories render in headless Chromium. - On staging, before this change, the #21405 waiting line was confirmed with a temporary 1-slot agent: "Waiting for Queue Check (temp) · 1 slot max · 2 of 2 in line · Configure". The temporary tasks and agent were cancelled and terminated. - After deploy: confirm the new labels on staging. ## Risks - Low risk. There is no schema change. - One extra query per projection batch that includes a never-started queued run on a routine task. It is a join on the company's routine-execution tasks through the existing `issues_company_origin_idx` index. - A queued turn that is admitted after 30 seconds, for a reason the server does not name, now says "Waiting to start" instead of "Working". This is the intended change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Chrome for staging checks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a82e235828 |
fix(heartbeat): keep recovery behind people's work, guarantee a start to tasks people wait on, and admit recovery a few runs at a time (#21427)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs when a slot is free. #21234, #21277, #21349 and #21391 order them: person work first, a recovery share, aging, and a share for the longest-waiting run > - On hosted staging, one agent with 20 slots had 295 queued runs. 139 of them were system recovery wakes that entered the queue within one hour. Runs that people asked for waited 7 hours > - Two rules let this happen. Recovery took every fourth start even when a person's run waited, and an aged recovery run moved into the agent-work tier ahead of todo tasks. The continuation producer also admitted any continuation held past 5 minutes, however deep the queue was > - This pull request keeps recovery behind people's work, gives a person's long-waiting run a bounded start, re-checks a recovery run before it starts, and admits recovery to an agent's queue a few runs at a time > - The benefit is that a task a person asked for starts within a bounded number of dispatches, and a recovery burst cannot flood an agent's queue ## Linked Issues or Issue Description Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502 **What happened?** On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and all 20 slots held hour-long runs. 295 runs were queued. 139 had wake reason `issue_recovery_action_restored` (source `execution.reconciled`), enqueued between about 10:46Z and 11:47Z. Two tasks that people created were stalled by a restart at 07:25Z. Their next runs were queued at 07:29Z and were still queued 7 hours later. Both were system wakes: one `issue_monitor_due`, one `issue_continuation_needed` from the assigned-work sweep. Recent starts had waited from 1 to 290 minutes. **Why did it happen?** 1. The recovery share gave every fourth start to system recovery, also while a person's run waited. 2. A recovery run that waited one hour aged into the agent-work tier. Recovery tasks are usually in progress, so they sorted ahead of todo tasks in that tier. 3. The oldest-run share served the oldest aged run of any class. Older agent work and older recovery came before a person's run. 4. Only a wake that a person requested counted as person work. A system wake on a task that a person created and waits on (a monitor check, a continuation after a restart) competed as ordinary work or recovery. 5. `deliverReconciledExecutions` admitted a continuation held past `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue depth. At one per agent per 15-second pass, one agent got about 139 continuations in an hour. **Expected behavior** A run that a person created or asked for never waits behind system recovery. Waiting never fails or cancels it. A long wait moves it forward, so it starts within a bounded number of dispatches. Recovery enters the queue a few runs at a time. A recovery run that is no longer needed is dropped quietly. **Steps to reproduce** 1. Give an agent 20 slots, all busy. 2. Queue 60 agent assignments, then two system wakes on tasks people created (a monitor check and a continuation), then 30 later person runs on in-progress tasks, then a burst of 139 recovery wakes. 3. Free one slot at a time while people and agents keep adding work. 4. See that the monitor check starts only at dispatch 244. ## What Changed - `server/src/services/heartbeat/queued-run-dispatch-order.ts`: - System recovery has its own tiers below agent work (fresh, then backlog). Aging moves old backlog only into the fresh recovery tier, never ahead of agent work. - The recovery share applies only when no ready person run waits. It comes out of agent work, never out of a person's turn. Within the share, the longest-waiting aged recovery run goes first. - The oldest-run share serves the longest-waiting aged run that a person asked for or whose task a person created (`issues.created_by_user_id`), first. This includes a system continuation of such a task. Then it serves the longest-waiting aged agent-work run. It never serves other recovery. - New dispatch fact `personTask`, read with the queued tasks' status and priority (same query). - `queuedRunDispatchLine` (the O(n log n) place-in-line order from #21405) follows the same rules. `queued-run-line.test.ts` still proves it equals the dispatcher's order over 400 random queues, now with `personTask` in them. - `server/src/services/heartbeat/queued-run-line.ts`: the start-history counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so dispatch, the place-in-line view and the tests share one rule. - New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and `dispatchCountersFromStarts`. The queue and the tests use the same counter rule (which includes the #21391 rule that a person's recovery restart does not use up the recovery share). - `server/src/services/heartbeat/queue.ts`: - At claim, a system recovery run (`issue_recovery_action_restored`, `issue_continuation_needed`, not requested by a person) is re-checked. If the task is reassigned, or another run continues it (started after it was queued, running, a queued non-recovery run, or a newer queued recovery run), the run is cancelled before it starts with `recovery_no_longer_needed` and its wake is skipped. This is the same quiet path as the existing closed-task check: no failure, no provider work. A person's undelivered message on the run still keeps it. - `server/src/services/recovery-queue-admission.ts` (new): `QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check. - `server/src/services/execution-recovery-resolution.ts`: a continuation is admitted only while its agent has fewer than 3 queued recovery runs, fresh or backlog. The 5-minute hold bound still orders held continuations, but no longer overrides the cap. The rest stays pending in its recovery action. The existing continuation pass (every 15 s while any continuation is pending) admits the next one after a queued recovery run starts or is dropped. No new polling. - `server/src/services/recovery/legacy-workspace-rehome.ts`: the pending-wake delivery for rehomed and released-writer tasks uses the same cap. A held wake waits 60 s for its next turn and does not count as a refusal. - The oldest-run share counts as served only by a start that people wait on while its candidate is one, so older agent work never uses up a person's turn (`servesOldestShare`, used by the dispatcher, the place-in-line order and the start history, which now records `personWaiting`). - `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts as the task's pending path while its assignee has queued recovery runs. - `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's stalled-queued-run replacement): a start of work people wait on does not count as passing over a queued run that people do not wait on. Waiting behind people's work is intended, so the sweep does not cancel such a run and replace it with a recovery continuation. Test: `assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run that waits behind work on tasks people created". It fails without the rule. - `server/src/services/recovery/assigned-work-sweep.ts`: a pending continuation still counts as the task's wake path past `PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery runs, so the sweep does not treat a held task as stranded. **Bounds** - A run a person asked for, or any run on a task a person created: once it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within `OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1) dispatches. Before that, runs a person requested already go first. There is no time limit, and waiting never cancels it. - Recovery run, proposed bounded behavior: at most 3 queued per agent. When no person run waits, an aged recovery run starts within `RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1) dispatches. At start it is re-validated against the current task, and it is dropped quietly if stale. While people keep the agent busy, recovery waits. It is held durably in its recovery action, not lost and not failed. ## Verification Regression test that reproduces today's shape (`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`, "deep per-agent queue: a recovery burst behind 20 busy slots"). It uses 20 busy slots, one dispatch per freed slot, 60 older agent assignments, the two 07:29 system wakes on tasks people created (monitor check and continuation), 30 later person runs, a 139-run recovery burst, and new person and assignment arrivals on every step. The same model, run against master's dispatcher and this one over 600 dispatches: | | Before (master) | After | | --- | --- | --- | | Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20 starts/h) | 4 | | Dispatch at which the 07:29 continuation starts | 5 | 8 | | Recovery starts while a person-requested run waited | 89 | 0 | With person-requested runs in the 07:29 slots instead (the first version of the test), master starts them at dispatch 244 and 248, and this change at 4 and 8. - Red on master: the new dispatch tests fail (`expected -1 to be greater than or equal to 0`, and recovery started ahead of agent assignments). Green with this change. - Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`, "queues at most a few recovery runs per agent..."): with the cap removed, 12 of 12 continuations enter one agent's queue in 12 passes. With the cap, 3 enter, the rest stay pending, and all 12 are delivered as queued runs leave. The queue never holds more than 3 recovery runs. - Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older, higher-priority assignments and no person arrivals, an aged `issue_monitor_due` run on a person-created task starts within `OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With the old reset rule it does not start in 8 dispatches. - Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the agent's recovery queue full, the wake stays pending with no failure counted, the sweep reads the task as `pending_intent`, and the wake is delivered after a queued recovery run leaves. - Start-time re-check, embedded Postgres, real heartbeat queue (`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose task was reassigned, or that a newer run already continued, is cancelled with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter is never called. A recovery run that the task still needs starts. - Updated tests that encoded the old rules: recovery's share and aging now come out of agent work, not out of a person's turn. Sweep liveness while held is covered. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes. ## Risks - Behavior change: a system run on a task a person created now takes the oldest-run share ahead of older agent work. On an agent whose queue is mostly such tasks, agent-made subtasks get that share less often. They still start in normal order. - Behavior change: while people keep an agent fully busy, system recovery for that agent waits. This is intended. Recovery is held durably and starts once no person run waits. - Behavior change: a queued system recovery run can be cancelled at start as no longer needed. It is cancelled before any provider work, with the same quiet outcome as a continuation whose task closed. - A continuation held behind the cap stays pending longer than 7 minutes. The sweep now treats it as live while its agent has queued recovery runs. If such queued runs can never start (for example dependency-blocked), the next continuations wait behind them. - One extra indexed read per recovery claim, on the existing `(company_id, context_snapshot->>'issueId', created_at)` index. No schema change. Mixed versions are safe: an old server keeps its old order until it is replaced. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a726dce4f0 |
feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools from connected apps through the Paperclip MCP gateway > - Some tool calls take a long time: exports, builds, large searches > - The MCP progress utility lets an app report how far a call is, but Paperclip ignored it > - So the task chat showed only a spinner until the call finished, with no sign of life > - This pull request reads the app's progress and shows it on the tool's row while the call runs > - The benefit is that people can see that a long call is working, how far it is, and how long it has run ## Linked Issues or Issue Description **Subsystem affected** Connected apps (MCP gateway) and the task chat. **Problem or motivation** A long tool call from a connected MCP app shows only a spinner in the task chat. Paperclip does not send a `progressToken`, and it drops `notifications/progress` from the app. **Proposed solution** Send a `progressToken` on each `tools/call` that an agent run makes. Read the app's progress notifications from the call's response stream. Show a progress bar (or an indeterminate sweep), the latest message, and elapsed time on the running tool row. Keep the latest state so that a reload mid-call shows it. **Alternatives considered** A new transport for progress. The existing company live-events socket already carries per-run events with run read checks, so this change uses it. **Additional context** Spec: https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress ## What Changed - `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`. `readMcpHttpResponse` gives each notification on the response stream to an observer before the result. The observer cannot fail or delay the call. New `mcpProgressNotificationOf` matches a notification to its token. - `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an agent run send a per-call token and forward progress to a reporter. Calls without a run send no token. - New `server/src/services/tool-call-progress.ts`: checks each update (finite, increasing, plain single-line text of at most 200 characters). It publishes a `heartbeat.run.tool_progress` live event at most every 250 ms (latest wins) and saves the latest update on the invocation at most every 1 s. Nothing is awaited by the call. A final `done` event goes out when the call ends. - DB: nullable `tool_invocations.progress` jsonb column (migration `0479`, `ADD COLUMN` only). A save never writes onto a finished call. - Live delivery: the new event follows its run like transcript output. A client gets it only for runs it follows, after the run read check, in the run's company only. - New route `GET /api/issues/:issueId/tool-call-progress` (task read check) returns the latest progress of the task's running calls, for a reload mid-call. Documented in `docs/api/issues.md`. - UI: `ToolCallProgressProvider` (one per chat) keeps the latest update per call. A running tool row subscribes and shows a bar or sweep, the message, the percent and elapsed time (design tokens, reduced motion honored). Finished rows drop it and never fetch or re-render for progress. `LiveUpdatesProvider` ignores the event for query invalidation. - `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool (`steps`, `stepMs`, `withTotal`) in its own file. - Storybook: `Tasks/Tool progress` stories (running with total, running without total, done). - Docs: new `doc/connections/MCP-PROGRESS.md`; `MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported. ## Verification - Red/green: on the original source, 5 of the new server tests fail (token in `_meta`, SSE with interleaved progress and result, notification matching, run-interest delivery, gateway end-to-end). With this change they pass. - `server/src/__tests__/mcp-http.test.ts` (13), `tool-call-progress.test.ts` (9: throttling, latest wins, bounded text, out-of-order drop, finish, failures never throw, company/run/tool scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres: company and task scoping, no write onto a finished call), `realtime/live-event-delivery.test.ts` (run interest), `tool-gateway.test.ts` end-to-end against the real SDK kv demo server over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7 throttled events, all for the right company, run, task and tool, the last one `done`; no event reaches another company; the invocation keeps the progress; the result is unchanged. - UI: `tool-call-progress.test.tsx` (10: name matching, row states, plain-text rendering, drop on complete, restore after reload, no fetch for finished rows). All 67 task-chat test files pass. - `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP with and without a total. - Storybook: checked the three stories in a browser. The bar fills to 40%, the sweep animates, the elapsed time ticks, and the done row shows no progress. - `pnpm verify`: see the CI checks for the final revision. ## Risks - Low. The migration adds one nullable column. - An SSE response is now bounded per message, not per stream, so handled notifications (such as progress) do not use up the result's size limit. Each message is still bounded. - Progress code runs only while a run's tool call is in flight. It adds a few bytes to the request and throttled, fire-and-forget work. Agent boot and time to first token do not change. - A row finds its call by run and tool name. If one run calls the same tool twice at the same time, both rows show the newer call's progress. - Progress from a call that became an MCP Task is not shown. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5d0b27acca |
fix(run-events): watch both late appends for rejection as soon as they start (#21536)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores each run event through `appendHeartbeatRunEvent` and `appendHeartbeatRunEventBatch` > - Pull request #21511 added a test that deletes a run while two appends wait for its row lock > - Both appends must fail, but the test attached the rejection check of the second append only after it awaited the first > - When the second append failed first, its rejection had no handler, and Vitest failed the whole shard with one unhandled error > - This pull request attaches both rejection checks when the appends start > - The benefit is that server test shards on master and on unrelated pull requests pass again ## Linked Issues or Issue Description Refs #21511 **What happened?** Server test shards failed on master and on unrelated pull requests. Every test passed. Vitest reported one unhandled error: `Error: heartbeat_run_event_binding_mismatch`, from `server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`. **Expected behavior** The test awaits both expected failures. No rejection stays without a handler. **Steps to reproduce** Run the general server test shard that contains `heartbeat-run-event-atomic-append.test.ts` on master at `cefef8f11d`. Examples: master "verify / General tests (server (1/10))", pull request #21443 "ci / General tests (server (9/12))", pull request #21505 "ci / General tests (server (1/12))". ## What Changed - In the test "waits for Stop on the run row and appends after it, and fails without rows after a run deletion", the two `expect(...).rejects` checks are now created immediately after the two appends start. The test awaits them after it releases the deletion. - No product code changed. The product code rejects correctly; the test left one rejection without a handler for a short time. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-run-event-atomic-append.test.ts`: 25 of 25 tests pass, three runs in a row, no unhandled error. - The failure before this change is the master shard named above. It depends on which append rejects first, so I did not get a local reproduction on demand. ## Risks Low risk. The change is in one test. The assertions are the same. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
81d06cfc97 |
feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools in connected apps through the tool gateway, which speaks MCP to each app > - MCP Tasks (#21263) lets an app answer a long tool call with a durable task. It shipped behind the experimental setting `enableMcpTasks`, off by default > - No public app returns tasks without credentials, so nobody had seen the feature work against a real server, and it was hard to understand > - This pull request adds a small demo app that uses Tasks, tests the gateway against it over HTTP, and records a live end-to-end run > - The live run found one bug: every task stop sent the call's audit record, with a server stack trace, to the agent. This pull request fixes it > - The benefit is a repeatable way to try and test Tasks, plain-language docs, and evidence for the decision to turn the setting on ## Linked Issues or Issue Description Refs #21389 Refs #21263 ## What Changed - Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin `paperclip-tasks-demo-mcp-server`; run `node packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is stateless MCP 2026-07-28 with `server/discover`, and it declares `io.modelcontextprotocol/tasks`. It has six tools: `slow_report`, `needs_approval` (form question), `will_fail`, `cancellable`, `expires`, and `quick_echo`. It saves task state to a file, so it can restart mid-task. It needs no credentials. It has a status page at `/` and JSON at `/api/state`. - I put it in the existing demo package, so there is no new workspace package and no `pnpm-lock.yaml` change. - Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the real gateway against the real demo server over HTTP, with one test per tool plus a restart. - Fix: `server/src/routes/tool-gateway.ts` no longer sends `details.execution` to callers. That record has request IDs and the server-side error stack. The audit still keeps it. Add `server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the fix, green after). - Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a person sees", "Try it yourself", the live-run results, and a recommendation. The package README documents the demo server. - `enableMcpTasks` stays off by default. ## Verification - `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13 new demo-server tests). `tsc -p tsconfig.test.json` is clean. - `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass. `tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2 before the fix. - Existing suites that cover the changed route and Tasks: `mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results in a comment below). - `pnpm verify` exits 2, because the doc is a global CI input. Required CI gives the broad coverage. - Live end-to-end run on a local `test-drive` instance with `enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The demo was connected through "Connect your own MCP server": | Behaviour | Result | `tools/call` count at the app | | --- | --- | --- | | Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 | | Question (`needs_approval`) | The card was answered in the UI. `tasks/update` was sent, and the task finished | 1 | | Failure (`will_fail`) | `provider_task_failed` with the app's message | 1 | | Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 | | Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 | | Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s. The result was delivered once, and the agent was woken once | 1 | After the run, all 6 `tool_mcp_tasks` rows were settled and none had a next poll. ## Risks - Low risk. The demo server is new code in a demo package, and the bin is additive. - The route change removes `execution` from tool-gateway error responses. No UI code or test reads that field from these responses. The audit and tool-call events still record it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1f388ed385 |
Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm sandbox keeps an agent's Runner alive between runs, so the next run on the task starts fast > - When a run ends, the server closes the idle warm Runner in-process. A separate SQL worker stops the sandbox when its warm window ends (#21410, merged today) > - Both steps start at the same moment. The sandbox stop wins, so the idle close fails. The failed owner stays registered, and the next run throws `native_instruction_materialization_handoff_unavailable` before any provider work > - Recovery must just work: a follow-up run must never fail because a predecessor's warm copy is gone > - This pull request lets the idle close finish before the sandbox stops, prepares instructions fresh when the warm copy cannot be handed off, and releases an owner whose sandbox the host already gave up > - The benefit is that automatic follow-up runs start normally after a warm window ends ## Linked Issues or Issue Description Refs #21472 (base of this stacked PR), #21410 (introduced the warm-window stop). **What happened?** In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three automatic follow-up runs failed before any provider work with `native_instruction_materialization_handoff_unavailable`: run `5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a release swap. In each case, the predecessor's idle close had failed first with `provider_state_unreadable` / "Daytona sandbox lease … is no longer active". The cause: the warm-window stop from #21410 stopped the sandbox at the same moment that the idle close started. For the first run's predecessor, the close was requested at 20:30:34 and the sandbox got SIGTERM at 20:30:37.8. **What did you expect?** The follow-up run prepares its instructions fresh and continues. The warm-window stop does not interrupt the host's own idle close. **Steps to reproduce** 1. Run a warm Daytona task until it ends, so its warm window starts. 2. Let the warm window end while the idle timer closes the Runner. 3. Start a follow-up run on the same task. It fails with `native_instruction_materialization_handoff_unavailable`. Two of the three tasks then ended stranded. That downstream part (the claim-gate cancellation with `execution_reconciliation_required` and the stranded notice) is owned and fixed by another session. It is **not** in this PR. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts` `claimWarmNativeInstructionCopy`: after it retires a stale warm owner whose copy cannot be adopted, it returns `null`. The caller then prepares instructions fresh. Before, it threw `native_instruction_materialization_handoff_unavailable`. - `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window. The in-process idle close (close grace plus the 30 s proof read) finishes first. - `native-session-executor.ts` idle timer: when an idle close fails, it reads the lease rows for that sandbox. If every row is no longer `active` or `retained` (the host gave the sandbox up), it releases the in-process owner. The next run then prepares fresh. - `doc/execution-semantics.md`: one paragraph about the stop margin and fresh preparation. ## Verification Red → green (each test fails on the base `35ba76d0ef` and passes on this branch): - `native-session-executor.test.ts` › "hands off collection before probing or retiring a warm owner: unavailable" — red: throws `handoff_unavailable`; green: returns `null` (fresh preparation). - `native-session-executor.test.ts` › "forgets a warm owner whose idle close failed only once the host gave up its sandbox: pending_cleanup / released / retained" — red on all three; green: the owner is released for `pending_cleanup` and `released`, and kept (retired by the successor) for `retained`. - `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the warm runner time to finish before stopping its sandbox" — red: the sandbox stops 3 s after the window; green: it stops only after the margin. Two existing tests ("keeps a sandbox still inside its warm window, and wakes when the window ends", "reconciles a warm sandbox left running across a restart") now expect the deadline with the margin. Local runs on this branch: - `server`: `tsc --noEmit` clean. - `vitest run src/__tests__/warm-sandbox-expiry.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed. - `vitest run src/services/native-runtime/native-session-executor.test.ts -t "hands off collection before probing or retiring a warm owner|forgets a warm owner whose idle close failed"`: 10/10 passed. Not run: a Daytona wave with this change. ## Risks - Warm sandboxes now stop up to 2 minutes later than the configured window. This is a small cost increase per warm sandbox. I chose a fixed margin over an in-process "is a close running?" check because the stop is pure SQL. The margin works when two servers run at the same time (the server that runs the stop may not be the server that owns the Runner), and it needs no busy-loop or cross-server signal. - The margin does not cover a re-armed idle timer or a close that takes longer than 2 minutes. Fresh preparation (fix 1) and the owner release (fix 3) cover those cases. - Not run on Daytona. The evidence is from the wave above and the unit tests. - In the wave, a `restore_failed` copy-back onto the just-restarted sandbox was also seen. I did not find its root cause. It is not changed here. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a4c570aede |
feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use connected MCP apps through the tool gateway. > - An MCP app can ask the person a question in the middle of a tool call (MCP elicitation). There are two kinds: a form, or a link to open. > - Today a form question is a dead end: > - A 2026-07-28 server's form `input_required` stops with "needs information Paperclip can't collect yet". > - An older server's `elicitation/create` makes a card, but the answer is never sent back, and the server waits on an open stream. > - Paperclip does not declare form support, so compliant servers never ask. > - This pull request shows the question as a card in the task chat. The card names the app, and only a person can answer it. When the person answers, the agent wakes. The agent's repeated call sends the answer to the app exactly once, for both protocol eras. > - The benefit: apps that need a choice or a detail from the person now finish their work. Restarts lose nothing, and no call is left waiting. ## Linked Issues or Issue Description Refs #21140 (stateless MCP client; this PR closes its "form elicitation" follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs #21389 (MCP Tasks follow-ups: the card answer validation item). **Is your feature request related to a problem?** When a connected MCP app asks the person a form question during a tool call, the call stops and cannot continue. A 2026-07-28 server gets a "needs input" error. An older server gets a card whose answer never reaches it. **Describe the solution you'd like** Show the question in the task chat. Let a person answer, skip, or leave it. Wake the agent when the question is resolved. Send the answer on the agent's repeated call, once, with the server's exact state. **Describe alternatives you've considered** Holding the tool call open until the person answers. We rejected this: it holds a stream and a run slot, and it does not survive deploys or restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records nine design decisions with their alternatives. ## What Changed - **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec, the inventory, the decisions, the journey, tests, and risks. - **New module `server/src/services/mcp-elicitation.ts`.** It is the single mapping for direct calls and MCP Tasks: - It turns each spec form field into a real card field: text with defaults, number, yes/no, single choice (`enum` and `oneOf`), and multiple choice (`items.enum` and `items.anyOf`). - It turns card answers back into typed `content`. - Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or expired sends `cancel`. - It refuses forms that ask for passwords, keys, tokens, card numbers, or one-time codes. - It keeps app text plain, so links in it are not clickable. - It seals `requestState` with AES-256-GCM, bound to the company and connection. - **New table `tool_mcp_elicitations`** (migration 0479). It is company-scoped, with one row per question round. It holds a keyed call identity, the sealed state, the questions and their cards, a claim time, and a 24-hour expiry. - **Gateway, stateless servers:** - Paperclip declares `elicitation: {form, url}` on task tool calls. - A form-only `input_required` saves the round, makes the cards, and stops with a wait message. - A repeated call before the person answers stops again without contacting the app. - After the answer, the repeated call claims the round and sends `inputResponses` and the exact `requestState`, once. - A failed send releases the claim. A crash mid-send lets the claim lapse. - **Gateway, older servers:** - Paperclip declares form support at `initialize`. - The first question gets `cancel` at once, so the server never waits. - When the same question (same words and schema) arrives on the repeated call, Paperclip answers it from the card. - **Gateway, mixed rounds and retries:** - A form beside a URL step shows the link first. - A repeated call that stopped for a question runs again in the same run. It is no longer replayed by the per-run idempotency key. - **Cards:** the title is "*App* needs your input". The resolver policy is `human_only`. "Send to *App*" sends the answer. A user comment does not supersede the card. - **Activity log:** new events `tool_gateway.elicitation_sent`, `tool_gateway.elicitation_refused`, and `tool_gateway.elicitation_expired`. They record the action only, never the values. - **MCP Tasks:** task cards use the same mapping and gain the app name, real fields, and `human_only`. A secret or an unsupported form stays unsupported. - **Storybook:** `Tasks and conversations/Questions and approvals/MCP App Questions` has five states: Asking, Answered, Skipped, Expired, and TwoQuestions. The fixtures are generated from the server mapper's output. - **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are updated. ## Verification - `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass on the branch. - **Red/green:** 9 of the 9 gateway tests fail with master's `tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new module. - The tests use a fake 2026-07-28 server and a fake 2025-06-18 server that asks over the open stream and waits for the reply. - Covered: answer, skip (`decline`), withdraw (`cancel`), two questions at once, a restart while waiting, a crash mid-send, another company with the same app and arguments, expiry, a refused password field, an agent that tries to answer (rejected), and capabilities. - Existing suites: `mcp-tasks`, `mcp-protocol-versions`, `tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See the results comment below. - Storybook: the five stories render in headless Chromium, and the play tests pass. The only console errors are the 404s from the backend-less API calls that every story makes. - `tsc --noEmit` is clean for `server` and `ui`. `node scripts/storybook-catalog.mjs` passes. ## Risks - **Older servers may start asking questions.** Declaring form support at `initialize` can make older servers ask where they did not before. Mitigation: the first question always gets `cancel` at once, so no call hangs, and the agent gets a clear next step. - **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one indexed lookup for a waiting round. Nothing is added before the first token or to agent boot. - **Migration.** One additive table with no backfill, so it is safe to apply online. - **Changed Tasks behavior.** A task card that is withdrawn or closed now sends `cancel` instead of `decline`, as the spec defines. A skipped card still sends `decline`. - **Stale cards.** Expiry is checked only when the agent calls again. A card can sit past 24 hours until the next call. Then it is withdrawn and the app asks again. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use (shell, file edits, headless browser), with extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4879a146fb |
fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paid model calls (company decisions, fast responses, browser use, prompt-injection checks) pass a budget admission before they start > - Admission runs in a REPEATABLE READ transaction and share-locks the credential's secret row, so a key rotation cannot slip between admission and the call > - Every credential read also wrote "Last used" (`last_resolved_at`, `updated_at`) to that same row > - When two calls ran at once, one call's credential read aborted the other's admission with a serialization failure (40001). The retry then saw the first call's new budget hold and denied the work as `budget_blocked` > - This is the flaky `decision-models.test.ts` failure that held Cloud readiness, and so releases, on 2026-11-13 > - This pull request stops credential reads from writing the secret row. "Last used" now comes from the access event that every read already records > - The benefit is that concurrent paid calls are no longer wrongly denied or retried, and the release gate no longer fails on this test ## Linked Issues or Issue Description No public issue exists. The bug is described here. **What happened?** Cloud readiness `verify / General tests (server (4/10))` failed on master commits |
||
|
|
e67f3bd63a |
test(environments): add an opt-in live Daytona proof for warm sandbox expiry (#21463)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, such as Daytona, through reusable environment leases > - #21410 made Paperclip stop a warm sandbox when its warm window ends > - Its tests use a real database but a mocked provider stop, and staging had no warm reusable sandboxes to observe > - This pull request adds an opt-in live test that proves the behavior against real Daytona > - The benefit is a repeatable live proof of warm expiry, resume, and recovery after a restart ## Linked Issues or Issue Description Refs #21410 **What happened?** #21410 has no live provider proof. Staging creates no warm reusable sandboxes today, so the deployed change cannot be observed there. **Expected behavior** A repeatable live check shows that a warm sandbox stays started inside its window and stops after it. It also shows that the next turn resumes the same sandbox, and that a restarted cleanup worker still stops the sandbox on time. **Steps to reproduce** 1. Set `PAPERCLIP_LIVE_WARM_EXPIRY=1` and `DAYTONA_API_KEY`. 2. Run `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.live.test.ts` in `server`. **Additional context** The test makes no model turn. It drives the production environment runtime and the cleanup worker against real Daytona sandboxes. ## What Changed - New `server/src/__tests__/warm-sandbox-expiry.live.test.ts`. The test is skipped unless `PAPERCLIP_LIVE_WARM_EXPIRY=1` is set. It uses: - the real Daytona plugin and real sandboxes; - `environmentRuntime.acquireRunLease` with a 2 minute warm window; - `releaseRunLeases(..., "keep_running")`; - the production `registerEnvironmentCleanupWork` on a real `createDeliveryWorkCoordinator`, which must wake itself at the deadline. - The test checks the following: - The sandbox is started 20 s before the window ends. - After the window, the worker stops the sandbox. The sandbox is stopped, not deleted, and the lease is released as stopped and kept. - The second turn resumes the same sandbox (`acquisition: resumed`). - A new database client, a freshly loaded plugin module, and a new cleanup worker start in the middle of the window. That worker stops the sandbox after the window ends, not before. - At the end, the test deletes only the sandboxes whose labels match its own company and environment. ## Verification - Live run on 2026-11-13 against the Daytona org, window 120 s, sandbox `170bb215…`: - 18:39:32.624Z turn 1 lease retained; window ends 18:41:32.624Z - 18:41:13Z sandbox `started` (20 s before the window ends) - 18:41:36Z sandbox `stopped`, auto-delete -1: the worker stopped it 4 s after the window ended - 18:41:39Z turn 2 resumed the same sandbox (`resumed`, `started`); window ends 18:43:39.549Z - 18:42:10Z restart: a new worker booted; sandbox still `started` - 18:43:45Z sandbox `stopped`, 5 s after the window ended - 18:43:45Z fixture sandbox deleted; Daytona returns 404 - Provider calls: one acquire, one resume, two stops, no destroy before cleanup - Without the env flag the test is skipped (1 skipped). ## Risks - Low risk. The change adds a test file only, and the test is skipped by default. It makes real provider calls only when someone opts in. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, Daytona API). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e6df84a992 |
fix(ci): count a step cancelled as "A task was canceled." as runner loss when the log shows the shutdown (#21514)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs on AWS spot runners, and `ci-runner-loss-rerun.yml` re-runs a PR's first attempt once when every failed job lost its runner (#21291) > - On 2026-11-13 a watcher was added to the runner image that stops the GitHub runner about 10 seconds after AWS announces a spot reclaim, so the lost job ends in about 28 seconds instead of up to 10 minutes > - When the runner is stopped this way, the cancelled step's check annotation is usually "The operation was canceled.", but a step that was still starting up gets "A task was canceled." instead > - The classifier only recognised the first text, so it counted such a job as a real failure and refused the re-run, even though the job log has the shutdown-signal line > - This pull request recognises both cancellation texts, still only together with shutdown evidence in the log > - The benefit is that lost jobs stopped early in a step are re-run automatically too ## Linked Issues or Issue Description Refs #21291 **What happened?** In the first hour after the spot-notice watcher went live, a `ci / Build` job was stopped 13 s after it started. Its job log has `##[error]The runner has received a shutdown signal...`, but its only failure annotation was `A task was canceled.`. `classifyJob` treated that annotation as a real failure, so the whole run was not re-run. **Expected behavior** A job whose log shows the shutdown signal is counted as runner loss, whichever cancellation text the step was given. **Actual behavior** It is counted as a real failure, and the run stays red until a person re-runs it. ## What Changed - `.github/scripts/runner-loss-rerun.mjs`: the cancellation annotation check accepts "The operation was canceled." and "A task was canceled.". The log check still decides. A cancellation with no shutdown or lost-communication line in the log is still a real failure, and its evidence now quotes the actual annotation text. - `.github/scripts/tests/runner-loss-rerun.test.mjs`: a test for "A task was canceled." with and without shutdown evidence. - `doc/DEVELOPING.md`: the runner-loss paragraph names both texts. ## Verification - Red: the new test fails against the unchanged script (25 pass, 1 fail). - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 26 pass. - `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's command): 808 pass. - The workflow runs from `master`, so this PR's own CI cannot exercise the new behaviour; the tests are the evidence. ## Risks - A job cancelled as "A task was canceled." for another reason is still treated as a real failure unless its log has the shutdown or lost-communication line, so a person cancelling a run still blocks the automatic re-run. - No change to triggers, permissions, the once-only rule or the stale-head checks. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
01f764fe61 |
test(evals): pin the case clock in the original-deadline rejection tests (#21501)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite tests the chat catalog observers that grade
agent work
> - One observer test fails at random on unrelated pull requests
> - The test simulates a late poll result with a fixed clock jump from
the fixture time. The observer computes its deadline from a later clock
read. If 2 ms or more pass between the two reads, the jump does not pass
the deadline.
> - This pull request pins the clock to the fixture time for the
deadline rejection cases, as the sibling deadline tests do
> - The benefit is a deterministic test that still fails when the
late-deadline guard is removed
## Linked Issues or Issue Description
**What happened?**
`tests/runner-e2e/chat-catalog-domains/identity-plan-decision-observer.test.ts`
> "rejects late await original deadline without resetting it" failed in
the "ci / Runner support suite" job of PR #21418 (run 43503383519,
2026-11-13 20:38Z). The error was: `expected [Function] to throw error
including 'CONTROL_SOURCE_NOT_SETTLED' but got
'CONTROL_AFTER_SOURCE_SETTLED'`. PR #21418 changes only a plan document
and a benchmark script.
**Expected behavior**
The test passes every time. The result must not depend on how fast the
test worker runs.
**Steps to reproduce**
1. Add `await new Promise(r => setTimeout(r, 2));` before the `act` call
in the "late await" case.
2. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts
chat-catalog-domains/identity-plan-decision-observer.test.ts`.
3. The test fails with the CI error. With a gap of 0 or 1 ms it passes.
**Paperclip version or commit**
`master` at
|
||
|
|
ce5f10d534 |
fix(wake-queue): start a person's saved message after a legacy run is continued from saved work (#21518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run fails and cannot resume exactly, Paperclip continues the task in a fresh run from its last saved work > - If a person sent a message while the task was blocked, the fresh-continuation fallback lets that saved message start the next run (`delivery: saved_input`) and does not queue its own run > - The release step that promotes the saved message skips the "needs reconciliation" hold for a native run that the fallback superseded, but not for a legacy run > - So for a legacy run the task moves to To do, the person's message stays saved, and no run starts until a watchdog finds the task idle > - This pull request applies the same superseded-run rule to legacy runs in the release step > - The benefit is that a person's message after a failed run starts the next run at once, for legacy runs as for native runs ## Linked Issues or Issue Description **What happened?** A person sent a message on a task whose last legacy run failed. Paperclip continued the task from saved work and moved it to To do. The message stayed saved. No run started until the assigned-work watchdog queued one later. **Steps to reproduce** 1. Let a legacy run fail with an active recovery hold, so the task is Blocked. 2. Post a comment as a person. The wake is saved as `deferred_issue_execution`. 3. Let the fresh-continuation fallback continue the task. It records `delivery: saved_input`. 4. Run the deferred receipt and stranded-queue passes. **Expected behavior** One run starts and carries the person's message. **Actual behavior** No run starts. The release step returns `released` because `legacyExecutionNeedsReconciliation` is true for the superseded legacy run. ## What Changed - `server/src/modules/wake-queue/adapters/postgres.ts`: the release pre-drain check does not ask for legacy reconciliation when the run is superseded by a fresh continuation. This matches the existing native rule in the same function. - `server/src/services/fresh-continuation-fallback.test.ts`: a red/green test for native and legacy runs. A person's comment goes through the real wake path while the task is blocked, the task continues from saved work, and exactly one run must start with the comment. ## Verification - Red: before the fix, the legacy case fails. The saved wake stays `deferred_issue_execution` and no run starts. The native case passes. - Green: after the fix, both cases pass. - `pnpm verify` for the affected selection. ## Risks - Low risk. The change applies only when the fresh-continuation fallback has already marked the run superseded. That decision already records that the old run is not replayed, and the successor runs as a fresh turn. Other callers of `legacyExecutionNeedsReconciliationWithEvidence` (retry scheduling, recovery) are not changed, so a superseded run still cannot be retried automatically. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f610e6a880 |
fix(recovery): replace a queued run the dispatcher keeps passing over instead of counting it as a pending path (#21502)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The assigned-work sweep is the backstop that makes sure every open agent task has a live run, a pending path, or a named wait on a person > - It counts a task with a queued run as healthy. A queued run that the dispatcher declines on every pass is never started, so the task waits forever and the sweep never looks again > - On a hosted instance one agent had more than 30 queued continuations that the claim declined on every pass. The sweep saw each task as healthy > - This pull request makes the sweep recognize a queued run that the dispatcher keeps passing over, report it, and replace it once with a new continuation when it is automatic work > - The benefit is that an automatic task cannot stay stuck behind a run that never starts, while a person's work is never cancelled ## Linked Issues or Issue Description Refs #21391, #21349, #21433, #21394, #21443, #21450 **What happened?** An agent had more than 30 queued automatic continuations (`issue_continuation_needed`) that never started. The claim declined each one on every dispatch pass. #21391 fixed that specific cause in the chat-control gate. But the sweep still trusted any queued run as a pending path. A run that the claim declines for any persistent reason keeps the task "healthy" forever. **Expected behavior** A queued run that the dispatcher keeps passing over is not a healthy pending path. The sweep reports it. For automatic work it cancels the run through the run lifecycle and queues one new continuation in its place. For a person's work it reports the run and leaves it. **Steps to reproduce** 1. Queue an automatic run for a task and make the claim decline it persistently (for example, a chat-control admission marker that does not verify). 2. Let the run wait for more than one hour while its agent starts other runs. 3. Run the assigned-work sweep for the task. Before this change, it reports the task as `live_run` and does nothing. ## What Changed - New violation `queued_run_stalled` in `recovery/assigned-work-invariant.ts`. A queued run is stalled when all of these are true (read from the database, on the database clock): - it has been queued for longer than `QUEUED_RUN_AGING_MS` (60 minutes, from `queued-run-dispatch-order.ts`); - it is the oldest queued run of its agent; - its agent started more than `OLDEST_DISPATCH_SHARE` other runs after the run became its oldest queued run: after the later of the run's aging time and the moment the last older run of the agent started or finished. - A stalled run is not a live run, and it does not let the task's interruption projection ("working") count as a pending path. Its own wake is not counted as a pending wake. A stalled run on a blocked task still counts as live, so the block decides first. - `listSuspects` lets a task through when its only active run is a stalled queued run. - `act()` handles the violation after all existing early returns: - person work (the #21349 source-chain rule through `listPersonOriginSourceRunIds`, which this change exports, or an explicit user continuation): reported, never cancelled; - a replacement that stalls again (`replacesStalledRunId` in its context): reported, not cancelled again; - both log one warning per run, with the run, agent and how long it waited, and look again after `CONTINUATION_CAP_WINDOW_MS` (one hour); - automatic work: the existing claim under the task row lock re-checks the facts and the shared hourly cap and spacing. Inside that claim, before the person check, it reads the run's row version (`xmin`) and the number of wakes attached to it. Then the run is cancelled through the run lifecycle (`setRunStatusFromLive` from `queued`, with that version and wake count as its condition; wake cancelled, run event, `releaseIssueExecutionAndPromote` with `suppressImmediateRecovery`). A person's wake that joins the run after the check, or a claim that starts it, changes one of them, so the cancel does nothing. Then one continuation wake is queued with `previousRunId` and `replacesStalledRunId` set to the cancelled run. If the cancel does nothing, the unpublished claim entry is removed and nothing is queued. - New `heartbeat/stalled-queued-run.ts` builds that cancel from the run lifecycle functions. `heartbeat.ts` and `recovery/service.ts` pass it to the sweep. - Plain activity text: "Continued automatically: this task had waited over an hour to start while its agent started other work, so Paperclip queued it again." ### Why this signal - #21433 keeps refused claims and claim back-off in memory only (`declinedAgedClaimAt`, `claimBackoffs` in `heartbeat/queue.ts`). Another server cannot read them. Run rows can be read from any server, on the database clock. - After #21349, an aged run ranks no lower than ordinary work, and the oldest-run share starts the oldest ready queued run at least once in every `OLDEST_DISPATCH_SHARE` starts. So when a run has been its agent's oldest queued run since after the aging bound, and its agent started more runs in that time than the share allows, the dispatcher offered the run a slot and the claim declined it. Starts that happened while older runs were still waiting do not count: those older runs were getting their turns. - A run behind a full agent sees no such starts and is left alone. Under a slow backlog only the oldest queued run can match. So the younger queued runs keep their place in line. - Starts are counted only after the run reached 60 minutes of age, not after the current server's boot. 60 minutes is longer than a release swap overlap. So a run that only waits for a new server to reattach to its Runners does not match. #21450 will add a reattach-phase gate before this path. ### Why this does not set `retryOfRunId` An earlier plan added `retryOfRunId` to the sweep's continuation so the chat-control gate could follow it. An audit of its readers found three changes in behavior: the admission-time chat-control check would start to refuse sweep wakes; the claim would wait on the source run's cleanup (a possible wait forever); and the agent prompt would call the run "a Retry" that "re-delivers the wake". #21391 already makes these runs claimable. So this change leaves `retryOfRunId` unset, as today. The replacement links its source with `previousRunId`, as the sweep already does. ### Questions from review, answered from the code - **How often is the invariant pass triggered, and can two passes overlap?** The `assigned-work-invariant` delivery queue (`recovery/work.ts`, `scoped-deadline-work.ts`) runs on database-work notifications, on each task's own deadline (`retryAt`), and on a 30 s reconstruction of the suspect list. Passes are at least 1 s apart (`minimumPassIntervalMs` default). In one process, passes of the queue never overlap (`running` guard in `delivery-work-coordinator.ts`). Two servers can sweep at the same time. Each repair is serialized by the claim under the task row lock (`FOR UPDATE` in `claimContinuation`), which re-reads the facts and the spacing. This change does not add a new loop. - **Can two runs or no run exist for one task?** Two runs: no. The claim's spacing makes a second server back off, and the cancel happens before the new wake (a queued run on the task would otherwise absorb the wake). No run: there is a short gap between the cancel and the new wake, because wake admission locks the task row in its own transaction. If the server stops in that gap, the task has nothing pending, and the next pass repairs it as `no_pending_path` under the same cap. ## Verification - `server/src/__tests__/assigned-work-stalled-queued-run.test.ts` (new, embedded Postgres, real `createHeartbeatQueue` admission and `claimQueuedRun`): - an automatic run that the real claim declines persistently, whose agent started runs after it aged: reported, cancelled, exactly one replacement, and the real claim starts the replacement; - a run behind a full agent: left alone, not a suspect; - a person's declined Retry: reported, not cancelled, one warning; - a replacement that stalls again: not cancelled on two checks, one warning, next look one hour later; - two sweep instances on the same task at the same time: one replacement; - three aged runs queued in order, the first two get their turns slowly: the third is not stalled when it becomes oldest, and is stalled only after more than four further starts pass it over; - a person's message joins the run (real wake admission) between the claim and the cancel: the run is not cancelled, there is no replacement, and the person's wake stays attached to the run; - the run starts between the claim and the cancel: no cancel, no replacement. - First round, red before the fix: 4 failed, 1 passed (the full-agent case passes before and after). Green: 5 passed. - Review round: the three new tests on the previous head: 2 failed (the backlog case and the person-join case), 1 passed (the started-run case, which the `queued` check already covered). Green: 8 passed. - Existing files run one at a time: `assigned-work-invariant.test.ts` 31 passed, `assigned-work-sweep-round-trips.test.ts` 2 passed, `failed-continuation-continues-again.test.ts` 28 passed, `recovery-wake-herd.test.ts` 35 passed, `saved-work-recovery-overdue.test.ts` 4 passed. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: clean. - `pnpm verify`: not completed locally (the machine is heavily loaded and the run was stopped when the code changed). Required CI supplies the broad coverage. ## Risks - A task's oldest queued automatic run is cancelled and queued again at most once per run. Its replacement starts at the back of the line. Only a run that the dispatcher already passed over can match. - A replacement that stalls again is only reported, so a persistent claim refusal cannot cause repeated cancellations. - The new subqueries run only for queued runs older than one hour. They use the existing `(company_id, agent_id, started_at)` index. Healthy runs and the path before `turn.start` do not change. - #21443 (automatic-run brake) and #21450 (reattach first) touch `assigned-work-sweep.ts`. The new path comes after every early return that `act()` has today, so their gates can go before it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). A subagent on the same model audited the `retryOfRunId` readers. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a96f972e1 |
Connect Tailscale networks through device sign-in (#21526)
## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - SSH environments can reach machines through a company Tailscale connection. > - The initial connector requires a managed OAuth app that Tailscale limits to its own tailnet. > - Customers need to connect their own tailnets without registering an app or supplying a key. > - This pull request uses Tailscale's normal device sign-in for new connections. > - Customers approve the Paperclip device and then select their machines in SSH environments. ## Linked Issues or Issue Description **Problem or motivation** The current Tailscale enrollment path cannot serve independent customer tailnets through the documented self-service OAuth app model. It also requires Cloud broker configuration and pilot admission before a customer can start. **Proposed solution** Start a private Tailscale device in the tenant and open its normal sign-in link. Save its encrypted device state before reporting a successful connection. Retain the existing SSH machine selection and agent account behavior. **Alternatives considered** Auth-key entry would require customers to copy a credential. A public OAuth integration still needs a confirmed registration path. Device sign-in uses a documented Tailscale flow without either requirement. See [device authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication) and [OAuth app limitations](https://tailscale.com/docs/features/oauth-apps). **Related PR search** Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for `tailscale`. This follows the merged network connector in [#20617](https://github.com/paperclipai/paperclip/pull/20617) and preserves the SSH agent-account support in [#21248](https://github.com/paperclipai/paperclip/pull/21248). The open [#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes storing an OAuth client and checking its health; this change instead enrolls a device through interactive sign-in so customers do not need an OAuth app. ## What Changed - New connections use device sign-in. Existing connected devices and pending legacy OAuth callbacks remain supported. - Pending sign-ins are bound to their company and initiating user. They have a fixed ten-minute deadline. Device state uses server-owned vault secrets with trusted version and digest checks. - The active Tailscale CLI waits for consent and device approval. Cancellation, expiry, secret deletion, and stale completion cannot restore a cancelled connection. - The connector stays open during sign-in. It provides a link when popups are blocked and shows device approval separately. Progress checks stop on completion, failure, expiry, or unmount. - Shared response types, OpenAPI contracts, SDK source pins, and the API/setup documentation describe the new flow. No database migration or Cloud provider configuration is required. ## Verification - 109 focused tests cover network processes, the real database and encrypted vault, routes, UI, and legacy connection behavior. - Server and UI typechecks pass. - Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure locally. All 70 OpenAPI and Tailscale route tests and server typecheck pass after synchronizing the API contract. - Follow-up `79de05f38b9`: the transient-recovery regression failed before the fix; all 20 connector UI tests and UI typecheck pass after it. The query-error policy check reports no findings. - The public SDK input registry check passes. - [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2) passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57 successful checks, no failures or pending checks. The interrupted canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile is 5/5 on this head with no unresolved review threads. - `pnpm verify --explain` exits 2 because the documentation change selects broad CI coverage. Focused local checks passed; broad CI passed on `29cf296a597e470d432b84767050bc539caa280e`. - Live local qualification on `29cf296a597e470d432b84767050bc539caa280e`: native device sign-in completed with interactive user approval. An isolated Ubuntu 24.04 ARM64 EC2 machine appeared in discovery and passed SSH pairing and workspace probes. The assigned Claude Code agent used a Paperclip-managed subscription, created `hello_world.txt` containing exactly `Hello, world!` plus a newline, posted a completion comment, and marked the task done. An independent read through the private connector verified the bytes, remote user, and hostname. The remote machine had no stored Claude login and no leftover temporary credential directories. - Restarted the local Paperclip server and verified the same Tailscale device identity, online peer discovery, successful SSH probe, and unchanged file without new consent. Temporary public SSH ingress was removed. - Local qualification covers one user tailnet and the final Core revision. Browser UI acceptance, two unrelated tailnets, a hosted restart, and an agent task on the final staging revision are not yet verified. ## Risks - The enrollment method changes. It must pass the live qualification above before broad rollout claims. - Existing encrypted device state and the vault master key must survive a restart or redeploy. Device expiry or removal still requires reconnecting. - The existing one-serving-process-per-instance constraint remains. Concurrent replicas need durable network ownership. - Pending setup uses bounded progress requests and an event-driven CLI with a deadline. It adds no idle database scan. ## Model Used OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed by the runtime. No subagents were used. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
cefef8f11d |
perf(workspaces): inspect only hinted workspaces after a scoped commit (#21519)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up execution workspaces whose task tree is done. A
background sweep (`terminal-workspaces`) finds these workspaces.
> - Every workspace write and every task status change sends a hint to
this sweep. Each hint starts a full rotation over every open candidate
workspace, and each candidate costs a recursive read of its task tree.
> - Runs write workspaces all the time, so with live runs the sweep
never stops, and its cost grows with the number of open workspaces.
> - On a hosted instance with 5 live runs, the task tree read ran 11.6
times per second. On a local copy with run churn it ran 81 to 101 times
per second.
> - This pull request inspects only the workspaces that a hint names or
reaches. Full rotations stay for startup, saved deadlines, failures and
hints without a scope.
> - The benefit is that a workspace change costs a few statements, not
one tree read per open workspace.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The terminal workspace sweep (`registerTerminalWorkspaceWork` in
`server/src/services/terminal-workspace-work.ts` and
`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).
**Current behavior**
Producers send hints such as `workspace:<id>` and `issue:<id>`. The
worker ignores the scope: each settled hint wakes the queue, and the run
starts or continues a full keyset rotation over all candidate
workspaces, 50 per page with 1 second between pages. A hint that arrives
during a rotation also requests a rescan, so a new rotation starts after
the current one. Every candidate with an open task tree costs one
recursive tree read and is skipped.
**Proposed behavior**
The worker keeps the workspace ids and task ids from scoped hints. When
the queue runs only because of such hints, it calls
`sweepTerminalWorkspaces` with `targets`. The sweep then reads only
these candidates:
- the workspaces named by `workspace:` hints;
- the workspaces whose source task is the hinted task or one of its
ancestors (the tree of a workspace is its source task and its
descendants);
- the execution workspace of each hinted task.
A targeted inspection does not move the rotation cursor or its boundary
and does not clear saved deadlines. The per-workspace checks are the
same code as in a rotation.
A full rotation still runs at startup, when a saved deadline is due,
when a hint has no scope or an unknown scope, while a rotation is in
progress, and after a failed targeted inspection.
**Reason and benefit**
Background work must not grow with the backlog of open workspaces. With
4 candidate workspaces, a hint for one task tree now reads 1 tree
instead of 4. A hint during a rotation no longer starts another full
rotation.
## What Changed
- `server/src/services/terminal-workspace-work.ts`: parse scoped hints
into targets. Run a targeted inspection when only scoped hints are
pending. Keep the full rotation for startup, due deadlines, unscoped
hints, continuation pages and failures.
- `server/src/services/terminal-workspace-work.ts` also falls back to a
full rotation when more than 50 workspaces wait for inspection. It keeps
the 1-second pause between rotation pages and runs targeted inspections
between them.
- `server/src/services/issues.ts`: moving a task also hints its former
parent. Removing a task also hints its parent and its execution
workspace. Without this, a former parent's workspace could never be
reached.
- `server/src/services/execution-workspaces.ts`:
`sweepTerminalWorkspaces` accepts `targets`. A targeted page that fills
its limit requests a full rescan. and filters the candidates to the
targeted workspaces. A targeted inspection leaves the rotation state
alone. `terminalWorkspaceWorkSettled` takes `{ targeted: true }` from
the worker, and then a scoped hint does not request a full rescan.
Direct callers keep the old rescan behavior.
- `server/src/index.ts`: pass the new option through the existing
wrapper.
- Tests in `server/src/__tests__/terminal-workspace-work.test.ts` and
`server/src/__tests__/execution-workspaces-service.test.ts`.
## Verification
- New test `inspects only the workspaces a hint reaches, however many
trees stay open`: 3 workspaces with open trees and 1 done tree whose
child task is hinted. Before the change the targeted call checks 4
workspaces. After the change it checks 1, reads 1 tree, and archives the
done workspace.
- New test `inspects only hinted workspaces after a commit, and keeps
unscoped commits and deadlines on the full rotation`: before the change
the worker never passes targets. After the change a scoped commit runs a
targeted inspection, and an unscoped commit and a due deadline still run
a full rotation.
- `npx vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/git-workspace-expiration.test.ts` (in `server/`) pass.
- New tests `falls back to the full rotation when more workspaces are
hinted than one inspection reads`, `keeps the pause between rotation
pages while hinted workspaces wait` and `hints the former parent and
workspace when a task moves or is removed`. Each fails on the first
commit and passes on the head.
- `pnpm check:module-boundaries` passes.
Local benchmark. The real server ran against PostgreSQL with
`log_statement=all`. The fixture had 1 company, 60 agents, and open
tasks scaled from 150 to 5,000. Recovery re-dispatched runs to agents
that fail at once, and every run creates or updates an execution
workspace, so the hints keep coming. Each window is 180 seconds and
starts 150 seconds after the server is ready. The machine was shared.
The table gives the rate of the task tree read (`select id, status,
completed_at, cancelled_at, updated_at from issues ... WITH RECURSIVE
issue_tree`):
| Open tasks | Before (per s) | After (per s) |
| --- | --- | --- |
| 150 | 17.56 | 0.93 |
| 1,500 | 109.83 | 5.16 |
| 5,000 | 113.82 | 4.40 |
Total statements and transactions per second depend mostly on how fast
the failing runs cycle in the fixture, so they are noisy. Before →
after: 343 → 287 at 150, 1,647 → 1,562 at 1,500, 1,530 → 1,678 at 5,000
statements/s. In that window the 5,000-task run after the change also
had 10% more run events. With no activity, the server issued 0
statements before and after the change.
## Risks
- A workspace that waits on something without a hint is no longer
inspected as a side effect of unrelated activity. The known cases keep
their path: a workspace held by an active run keeps its past-due
deadline, so the 5-minute deadline rotation inspects it again; cooldown,
compaction, Git idle and retry deadlines still start full rotations. A
delivery that changes outside Paperclip without a work product change
was already only found by a later rotation.
- A failed targeted inspection requests a full rotation, so a lost
target is recovered by the rotation.
- Low migration risk. No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
3cd671575d |
perf(attention): scope native finalization retention hints to the run's task graph (#21521)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server archives old attention items (reviews, blocked tasks,
decisions, failed runs) in a background queue, `attention-retention`.
> - A hint for a whole company makes this queue reload every attention
candidate of the company, 100 per page. Each page also reads review
attention, blocker graphs and source evidence.
> - Each native run finalization sent such a company hint, although a
run changes only the attention items of its own task graph.
> - On a hosted instance with about 1,400 open tasks and live native
runs, the decision retention read ran 26.7 times per second. Native run
finalization is one of the producers that sent a company hint on every
run.
> - The same finalization already sends the issue-scoped hint through
`notifyNativeStatusReassessment` when the run has a task or a run id.
This pull request sends the company hint only when the run has neither.
> - The benefit is that a finished run reloads only the candidates of
its task, its ancestors and the tasks it blocks, not every candidate of
the company.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The attention retention hint that native run finalization sends
(`notifyNativeFinalizationWork` in
`server/src/services/native-finalization-notifications.ts`).
**Current behavior**
`notifyNativeFinalizationWork` calls `notifyAttentionRetentionWork(tx, {
companyId })`. The `attention-retention` queue has no reconstruction
interval, so each finalization reloads all eight candidate kinds of the
company, page by page, with the batch eligibility reads for each page.
**Proposed behavior**
`notifyNativeFinalizationWork` sends the company hint only when the
scope has no `issueId` and no `runId`. Otherwise
`notifyNativeStatusReassessment`, which the same function already calls,
sends `notifyIssueAttentionRetentionWork(tx, { companyId, issueId, runId
})`. That helper resolves the task of the run, adds its parents and the
tasks it blocks (at most 100), and sends a hint with these task ids.
When the run has no task, or the graph has more than 100 tasks, the
helper sends the company hint itself.
**Reason and benefit**
Background work must not grow with the backlog of open tasks. The
attention items that a run can change are the failed run, the review,
the recovery actions, the decisions and the blocker items of its task
graph. The issue-scoped hint covers all of them, and the ordinary task
write path uses the same helper.
## What Changed
- `server/src/services/native-finalization-notifications.ts`: send the
company attention retention hint only for a finalization without a task
or run.
- `server/src/__tests__/attention-retention-work.test.ts`: new test
`scopes native run finalization to the task graph of the run`.
## Verification
- The new test sends the finalization hint once with a run id and once
with a task id. Before the change both hints are `{ companyId }`. After
the change both hints name the task and its parent.
- `npx vitest run src/__tests__/attention-retention-work.test.ts` and
the native work tests (`native-result-commit-work`,
`native-environment-settlement-work`, `native-run-projection-work`,
`native-recovery-notice-work`, `retained-native-session-cleanup-work`,
`native-status-reassessment-work`, `muse-terminal-settlement`,
`ownership-reconciliation-report`) pass (in `server/`).
- `pnpm check:module-boundaries` passes.
## Risks
- No new reads. The issue-scoped hint was already computed in the same
transaction by `notifyNativeStatusReassessment`.
- An attention item outside the task graph of the run is not reloaded by
this hint. Agent status, budget and company changes keep their own
hints.
- Low risk. No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a82fa96540 |
perf(run-events): collect deltas that arrive while a batch commits into one next batch (#21520)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane port commits streamed text deltas of a run in batches (#21460): the first text at once, later deltas after at most 250 ms > - Each batch is cut when its timer fires, even when the previous batch of the same run has not committed yet > - When the database is slow, every run then queues one small batch per 250 ms behind its slow commit. The number of statements stays the same exactly when the database has the least room for them, and the queue grows > - This pull request cuts a batch only when the previous batch of the run has committed. Deltas that arrive while a batch commits all join the next batch > - The benefit is fewer, larger statements under load, with no change when the database keeps up, and no longer wait for any delta ## Linked Issues or Issue Description Refs #21460 (batched deltas). Refs #21511 (one statement per delta batch). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** How `PaperclipControlPlanePort` groups queued text deltas into batches when a batch commit is slower than the batch time. **Subsystem affected** Server run event ingest: `server/src/services/native-runtime/paperclip-control-plane-port.ts`. **Current behavior** `#flushStreamingEvents` takes the queued deltas when it runs (on the 250 ms timer, at 100 deltas, or before any other event) and chains their commit behind the previous batch. If one commit takes two seconds, the run queues about eight batches, each one statement, behind it. **Proposed behavior** `#flushStreamingEvents` chains one cut behind the previous batch and takes the queued deltas when that cut runs. While a cut waits, later flush requests join it. A run has at most one batch committing and one cut waiting. **Reason and benefit** Under load, the number of delta statements per run falls to about one per commit time instead of one per 250 ms. When commits are faster than 250 ms, nothing changes. **Breaking changes** None. Rows, receipts, order and the first-text rule are the same. ## What Changed - `#flushStreamingEvents` queues one cut on the run's commit chain (`#streamingFlushQueued`). The cut takes every delta queued at that time, sets the first-text flag, commits the batch, and then releases the pending count of that batch. - A flush while a cut waits returns the same chain, so a caller that waits for it (every other event, `checkpointSession`, `completeRun`, `replayEvents`, `provesNoRunProviderWork`) still waits for every delta queued before it. - A batch can now hold more than 100 deltas when commits are slow. The 100-delta rule still asks for a flush; the batch is one statement with a fixed number of bind parameters (#21460, #21511). ## Verification - Red/green: new port test "collects the deltas that arrive while a batch commits into one next batch, before the next other event" (`paperclip-control-plane-port.test.ts`). A transaction holds the run row, so the first batch waits. Six more deltas arrive 15 ms apart with a 5 ms batch time, then a non-streaming event. Before: 8 inserts (one per timer). After: 3 inserts (the waiting batch, one batch of six, the other event). Rows are stored 1–9 in order and every receipt resolves. - The whole port suite with its conformance suite passes: 101 tests. Other suites, one file at a time: `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `chat-native-progress-work` (35). I ran the four streaming port tests two more times; they pass each time. - `pnpm --filter @paperclipai/server run typecheck` passes. `pnpm check:module-boundaries` passes. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). The model applies the port's batching rules itself, so for these runs I changed my local copy only (not in this pull request): with `DELTA_COALESCE=1`, `flushDeltas` queues one cut on the agent's append chain and takes the queued deltas when the cut runs, as the port now does. All rows run with #21511. Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. | Agents | Batching | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | this change | 60 s | 9.7 | 0.51 | 3,832 | 151 / 3,087 | 12,000 / 12,000 | 3 / 3,994 | | 400 | cut on timer (before) | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 / 24,000 | 5,721 / 15,238 | | 400 | this change | 60 s | 12.5 | 0.78 | 7,010 | 892 / 3,131 | 23,993 / 24,000 | 1,896 / 4,193 | | 600 | cut on timer (before) | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 / 36,000 | 2,280 / 30,985 | | 600 | this change | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | this change (repeat) | 120 s | 13.6 | 0.74 | 6,498 | 1,225 / 6,057 | 58,984 / 72,000 | 11,465 / 25,361 | | 800 | this change | 60 s | 11.5 | 0.80 | 6,228 | 2,054 / 7,186 | 24,877 / 48,000 | 24,544 / 31,331 | The process does more work per second with this change (more other events finish), so statements per second rise; delta lateness falls. 600 agents are still not carried reliably: other events fall behind. #21528 removes two statements from each such event and one from each checkpoint save; its description has the stacked results. ## Risks - A delta never waits longer than before: it already waited for the previous batch of its run. When the database keeps up, batches are the same as before. - Under load, one batch can be large. The live view then gets one larger step instead of several small steps behind the same wait. - A failed batch now rejects more deltas at once (every delta that joined it). The runner resends them, as before; each resend is idempotent. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0bba04675c |
perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - After #21460, text deltas commit in batches, but each batch still takes five statements: BEGIN, the run lock, a replay probe, one insert, and COMMIT. A single event without a wake takes four > - In the agent scaling load model, the process issues about 36 statements per agent per second and saturates its one thread between 200 and 400 agents > - This pull request writes a batch of in-order native events, or one such event, with one statement that commits on its own > - The benefit is about half the statements per agent, less CPU per agent on the main thread, and less time on a pool connection, with the same rows, receipts, cursor and locks ## Linked Issues or Issue Description Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The append of native source events that continue their source's stored cursor: batched text deltas, the first delta after another event, and every other native event that registers no wake. **Subsystem affected** Server run event ingest: `server/src/services/heartbeat-run-events.ts`, and the port comment in `paperclip-control-plane-port.ts`. **Current behavior** A delta batch takes BEGIN, a run lock read (`for no key update`), a replay probe, one insert, and COMMIT: five statements. One delta that commits at once takes four. In the load model at 200 agents the process sends about 6,700 statements per second at event-loop utilization 0.72. **Proposed behavior** The same append is one statement. Its update of the run row takes the same row lock. It requires the stored cursor to end just before the batch, requires the `next_event_seq` that its own snapshot read, and finds no replayed event. Then it moves the sequence and the cursor and inserts every row. If any check fails, it writes nothing and the append takes the existing locked transaction. **Reason and benefit** Fewer statements free the main thread and the shared pool for other agents. In the load model at 200 agents, statements per second fall from about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to 0.56. **Breaking changes** None. No schema change. Rows, receipts and the cursor are the same. Servers from before this change keep working on the same runs. ## What Changed - `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one statement: a CTE reads `next_event_seq` from the statement snapshot, an `update heartbeat_runs ... returning` locks the run row and moves `next_event_seq` and the source cursor, and an `insert ... select from jsonb_array_elements($9)` writes every row. The text is fixed, with nine bind parameters for any batch size. - **Why the `next_event_seq` check.** A statement that waits for the run row lock keeps the snapshot it took before the wait. Its replay probe cannot see events that the lock holder committed. Every event append (native, ordinary, the run coordinator store, the retry queue) moves `next_event_seq` in its own transaction. So the update requires the value that its snapshot read. PostgreSQL re-checks that condition on the newest row version after the wait, and any append committed meanwhile makes the statement write nothing. The unique indexes on `(run_id, source_event_id)` and `(run_id, source_instance_id, source_seq)` are a second guard: a unique violation from this statement also falls back. - A source without a cursor starts at 0. The probe still finds any stored event of that source, so a run that an older server wrote falls back and recounts. - `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the one statement first for native events without a wake, outside a caller's transaction. Every other case (a replay, a gap, an event with a wake, a cursor that lags, a run changed meanwhile, a binding mismatch, a run deleted meanwhile, a caller's transaction) takes the existing transaction, unchanged. - Inside a caller's transaction the append keeps its SAVEPOINT path. A failed statement there would abort the caller's transaction, and the fallback could not run. - `assertWritable` runs before the statement is sent and after it returns (see Risks). ## Verification - Red/green on embedded PostgreSQL. I ran the new tests against the previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests in the file pass with the change (`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`): - One statement for an in-order event and for a batch (before: 4 and 5). - A writer revoked before the send sends nothing; a statement that fails on its third row stores no row and no sequence; the resend commits. - A writer revoked while the statement runs: the rows stay, the receipt is rejected, and the resend is a replay with the same rows. - A batch waits for Stop on the run row and appends after it; a batch and a single event that wait for a run deletion fail with `heartbeat_run_event_binding_mismatch` and store nothing. - An older server stores events 2 and 3 while it holds the run row, without the cursor: the waiting batch 2–4 falls back, returns duplicate, duplicate, committed, and the cursor ends at 4. - Two connections append the same 200-event stream in batches of 10: each event is stored once, in source order, sequences 1–200 without a gap, one `committed` and one `duplicate` receipt per event. - A 7,000-event batch (more than any bind parameter limit at 10 parameters per row) is one statement. Before: it fell back to one statement per event and timed out at 15 s. - A run without a cursor starts at its first event; a run with events from an older server replays and then continues. - Equivalence: 12 random streams of 30 steps (in-order batches, gaps, replays of earlier ranges, fills of gaps, ordinary events, conflicting replays). Each step goes to two runs, one through the new path and one forced through the one-by-one locked path. Receipts, errors, rows, `next_event_seq` and cursors are equal at every step. - Mutation check: with the `next_event_seq` check and the unique-violation fallback removed, the older-server race test fails. - The port suite (`paperclip-control-plane-port.test.ts`, with the package conformance suite) passes: 125 tests. Two tests changed: the streamed-event test now expects one statement and the new revocation behavior, and the batch test checks that the five deltas are one statement sent before the boundary event. - Other focused suites pass, one file at a time: `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `chat-native-progress-work` (19), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51). In 2 of 3 runs with this change, all 51 tests passed but the file's `afterAll` (closing the side pools and the embedded database) hit its 30 s timeout on this busy machine; the third run passed cleanly, and one run on `origin/master` passed. The native events in that file all register a wake, so they do not take the new path. - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in #21418 (per-source counts stop when the window closes). I did not change its batching rules for this pull request. Model rates: 20 deltas/s, 1 other event/s, 0.3 API requests/s per agent; default pool of 10 connections; embedded PostgreSQL on the same, shared machine. I ran "before" (`origin/master`) and "after" one after the other, with a pause between runs, and I give the load average at the start of each run. The machine was busy with other work (load 10–15 on 18 cores), so compare rows that ran close together. No run had a failed operation, a failed checkpoint save or a lost lease. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 / 24,000 | 5,721 / 15,238 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 / 36,000 | 2,280 / 30,985 | | 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135 / 52,222 | 36,000 / 36,000 | 698 / 3,422 | At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization and 7,051 → 3,210 statements/s. One process with this change alone still does not carry 600 agents: at 400 and above the 10-connection pool and the shared database are the limit (event-loop utilization stays below 0.75 while other events fall behind). #21520 and a third pull request for the remaining statements of other events and checkpoint saves continue from here; their description has the stacked results. - Statements: a delta batch 5 → 1; one delta or one native event without a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save (5). ## Risks - **`assertWritable` window (read first).** Before, the writer check ran inside the transaction, before the first statement and before COMMIT. A revocation while the statements ran rolled the append back. Now a revocation while the one statement runs does not roll it back: the rows commit, and the caller gets the error instead of the receipt, so no observer runs and the runner gets no ack. This check guards a controller that hands its run to a successor at restart (`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the earlier check was not atomic with COMMIT either, so a revocation during the COMMIT round trip already committed; the window grows from one COMMIT round trip to one statement round trip plus its wait for a pool connection; (2) the append is idempotent by source event id and payload hash, so the successor sees the committed row as a replay of the event that the runner resends; (3) the old controller never acts on the rejected receipt. A new test covers exactly this. - A crash loses nothing new: the statement commits or not as a whole, and a resend after an unknown outcome is a replay. - The check on `next_event_seq` makes the statement write nothing whenever any other event of the run committed after its snapshot. Under heavy contention on one run, more appends take the fallback. The fallback is the previous code path. - The statement text is fixed SQL sent through the driver (`$client.unsafe`), like the rest of the event stream statements. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2178c3df88 |
fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a recovery read to a remote Runner goes unanswered, the next attempt of the same run carries `replaceUnansweredRunner` > - That attempt verified the Runner was alive and then refused to adopt it anyway. On a resumed sandbox lease the reuse check then failed with `resumed_not_suspended` and left the Runner running. Otherwise a second Runner was launched, and its launch fence killed the busy Runner, which lost the provider turn. > - After a late reconnect a Runner is often slow, not dead: it is still replaying the events it saved with no controller attached > - This pull request makes the attempt check progress first. It adopts a live Runner that is making progress and replaces only a Runner that made no progress. > - The benefit is that a slow but working Runner keeps its turn, and the retry no longer causes the failure it was meant to fix. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13. The retry after a first-attempt timeout failed with these codes (counts from the traced sample of 136 failures): - `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended` (part of 40 `ingress` failures) - `runner_remote_process_identity_unavailable` (25): the Runner was gone by the time the retry checked, because the launch fence or a previous retry had killed it - "restored the conversation after process loss, but…" / `provider_turn_lost_on_restore` (10): the replacement Runner restored from the old state and the provider turn was gone Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9 failed. Over 420 s: 14 of 15 failed. The process ID of the Runner changed inside the same sandbox in the failed runs, which shows the replacement. Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit` verifies the Runner is alive and then returns `undefined` because of that flag, so `adoptedProcess` is `undefined`. **Expected behavior** A live Runner that is making progress is adopted. A Runner that made no progress is stopped with proof and then restored from its own journal. **Steps to reproduce** 1. Reattach a remote run whose previous attempt left a recovery read unanswered (`replaceUnansweredRunner: true`), with the Runner alive and its saved event sequence past the controller's last acknowledgement. 2. On master the attempt launches a second Runner, or fails with `resumed_not_suspended` on a resumed sandbox lease. 3. The new tests in `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` do this. **Paperclip version or commit** `master` at |
||
|
|
47a2c82d35 |
fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server stops and the incoming server takes over the live runs on remote Runners > - The incoming server reached those runs late: it waited up to 60 s for the outgoing server's controller lease to lapse, re-verified each run's whole log from object storage, and deferred a reattach when the agent directory was busy. Queued starts competed with the reattaches for the same resources. > - On hosted staging, the later a live run was reached, the more likely it was to fail. Runs reached in under 150 s never failed; runs reached after 420 s almost always failed. > - This pull request hands live runs over at the start of shutdown, makes a reattach skip work that the same run already did, and holds queued starts while handed-over runs are being reached, with a 180 s cap. > - The benefit is that live runs are reached within seconds of a swap instead of minutes, so they continue instead of failing. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each run that crossed a swap we measured the gap from the last Runner output to the moment the new server recorded the Runner's process identity: - gap under 150 s: 0 of 9 runs failed - gap over 420 s: 14 of 15 runs failed - median gap: 180 s for runs that resumed, 470 s for runs that failed In one traced run the incoming server waited on `remote_controller_lease_live` three times (09:14:21, 09:15:18, 09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the run's whole log from object storage. A control run spent 23 s in `task.prepare`. Queued starts used the same preparation path, the same 8 object-I/O slots and the same database pool, and nothing gave a reattach priority. The first-attempt failures were `PRP command session.snapshot timed out` and `codex_history_read_failed`. The retry then failed with `runner_harness_state_mismatch: resumed_not_suspended`, `runner_remote_process_identity_unavailable` or `provider_turn_lost_on_restore`. **Expected behavior** Every live run is reached within about 30 s of the swap. Queued work starts after that, or after a bounded wait. **Steps to reproduce** 1. Run a two-server release swap with live remote runs and queued runs. 2. Observe `native.recovery.transition` "waiting" events with `remote_controller_lease_live`, then the claim, then a long `heartbeat.prepare.run_output` span. 3. The new tests in `reattach-phase.test.ts` and `heartbeat-native-early-hand-off.test.ts` model this. **Paperclip version or commit** `master` at |
||
|
|
493519fc88 |
fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (runnerd) relays provider events to the server and waits for cumulative ACKs before it polls the provider again. > - After a control frame, runnerd uses a 1 ms queued read only when the executor reports a live provider poll. Otherwise it waits up to the 250 ms runtime read timeout, and any frame that arrives in that time skips the poll. > - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex through ACPX and others) always wait the full 250 ms after the last ACK of every batch, and steady control traffic starves their provider polls. > - This pull request lets an ACPX executor report a live poll while its restored session has an active turn. > - The benefit is that ACPX output is relayed without the 250 ms idle wait, and tool-heavy turns no longer starve provider polls. ## Linked Issues or Issue Description **What happened?** A routine run started several subagents that made about 190 tool calls in about 10 minutes. On the server, consecutive runner events arrived at least 576 ms apart (median about 1.5 seconds), one event at a time, while the time from each event's `emittedAt` to its insert was 0 to 0.1 seconds. The runner fell several minutes behind the provider. The backlog then overflowed the transport's event buffer (fixed separately in #21524) and the turn failed. **Expected behavior** A live ACPX turn is polled again as soon as its outstanding events are acknowledged and no control frame is queued, the same as a Codex facade turn. **Steps to reproduce** 1. Run an ACPX turn that produces events while the controller sends commands and ACKs. 2. After each ACK, runnerd calls `receive_json`, which waits up to 250 ms before it polls the provider. A frame that arrives in that time skips the poll. **Relevant logs or output** ``` min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched insert time - emittedAt: 0.0 to 0.1 s ``` Related: #21524 bounds the receive buffer that this backlog overflowed. ## What Changed - `acpx_provider_backend.rs`: `AcpxCommandExecutor::provider_poll_is_live` returns true only when the session was restored, has no persistence error, does not wait for a next-turn generation, and its state is `turn_starting` or `turn_active` with an active turn, no unconfirmed provider exit, and the run it owns. - `native_provider_backend.rs`: `SelectedExecutor` forwards `provider_poll_is_live` to the ACPX executor. - `tests/acpx_warm_attachment.rs`: new test through `NativeProviderCommandExecutor`. The session is not live before `run.prepare` or after `session.open`, is live after `turn.start`, and a new executor that has not restored its session is not live. What does not change: - runnerd still polls only when the read returns no frame, so a queued stop or suspend still runs before the next provider poll. - The ACK gate (no poll while sent events are not acknowledged) does not change. - The 1 ms read is used only while the source sequence advanced since the last quiet poll, as for Codex today, so an idle turn does not busy-loop. ## Verification - `cargo test --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test acpx_warm_attachment` (in `packages/paperclip-runner`): 7 passed. The new test fails without the new method, because the default returns false. - `cargo fmt --all -- --check` and `cargo check --locked --workspace` pass. - `cargo test --release --locked -p paperclip-runner-core --no-fail-fast`: 1,127 passed, 3 failed. The failures are `codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames` ("failed to signal owned process group: Operation not permitted") and `stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement` and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay exit timing). They also fail without this change in the macOS sandbox where I ran the suite, and none of them uses the ACPX executor. CI covers them on Linux. - The existing `durable::tests::ack_release_poll` cases cover the runner side of the live read: an ACK releases the next poll without the read timeout, and a command queued behind the ACK runs first. - I did not run a live provider. After the deploy, the gap between runner events of an ACPX run with queued output should drop below 250 ms. ## Risks - Low risk. The change only selects the existing 1 ms queued read for ACPX during an active turn. - A turn that produces events continuously now polls more often. Each poll still writes the same durable state as before, so disk writes per event do not change. - Analysis also found that runnerd waits for every sent event to be acknowledged before it polls again. A bounded in-flight window would raise throughput further, but it changes the tested rule that control queued behind an ACK runs before the next poll. That is left out of this pull request on purpose. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
254744f77c |
fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.
## Linked Issues or Issue Description
**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.
**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.
**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.
**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```
No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.
## What Changed
- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.
## Verification
- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.
## Risks
- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
|
089748fbf9 |
fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.
## Linked Issues or Issue Description
**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.
**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.
**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.
**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
guidance: "Reconcile the existing operation before repeating it. ..." }
```
No related public PR exists. I searched open and closed PRs for this
fence.
## What Changed
- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
- A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.
## Verification
- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.
## Risks
- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
|
e82c4f0a31 |
fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**
This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it
## Linked Issues or Issue Description
**What happened?**
Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.
- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.
**Expected behavior**
A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.
**Steps to reproduce**
1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.
**Paperclip version or commit**
`master` at
|
||
|
|
1332ab987f |
feat(prp): declare the contract for provider-started turns (#21422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The Runner, its sidecar and the controller speak two protocols with schemas: PRP between the controller and the Runner, and the sidecar protocol between `paperclip-runnerd` and the ACPX sidecar. > - A turn that the provider starts needs new fields and one new command in those protocols. > - This pull request declares them and generates the readers again. It changes no behavior. > - The benefit is that the next two pull requests show behavior only, with the contract already in review. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 4. Its base branch is `feat/claude-provider-cycle-stream`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production. The protocol schemas. **Subsystem affected** Paperclip Runner: `protocol/` and the generated TypeScript and Rust contract code. **Current behavior** - PRP has no way to say that a turn was started by the provider, or that the Runner waits for such a turn. - The sidecar protocol has no command to arm the sidecar and no event for a turn that the provider starts. **Proposed behavior** - PRP (`schemas/provider-turn.schema.json`, new): `providerStartedTurns` (boolean) in the payload of `turn.start`, `origin: "provider"` on `turn.started`, and `awaitingProviderTurn: true` on `turn.completed`. `command-v3` and `event-v3` refer to it. - Sidecar protocol (`provider-schemas/acpx-sidecar.schema.json`): the command `session.provider_turn` with `{ turnId, armed }`, the event `runtime.turn_started` with `{ origin: "provider" }`, and the payload of the diagnostic for a lost provider. The schema enforces that payload for a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`. Diagnostics with other codes keep their open payload. - `paperclip-runnerd` decodes `runtime.turn_started` and refuses it. No sidecar sends it yet. The Runner of today refuses an unknown event too. - No controller, Runner or sidecar uses the new fields in this pull request. **Reason and benefit** The repository rule for Runner contracts says that new fields must be in the schema with generated readers. A separate pull request lets a reviewer read the contract alone. ## What Changed - `protocol/schemas/provider-turn.schema.json` (new), `command-v3.schema.json`, `event-v3.schema.json`, `protocol/provider-schemas/acpx-sidecar.schema.json`, `protocol/manifest.json`. - `scripts/generate-acpx-sidecar-contract.mjs` and `scripts/generate-protocol-schema-module.mjs`. - Generated: `src/drivers/acpx/generated-sidecar-contract.ts`, `src/protocol/generated/schema-bundle.ts`, `src/protocol/generated/standalone-validators.ts`, `runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs`. - `runner/crates/runner-core/src/acpx_event_payload.rs` (3 lines): the Rust `match` on the event type must name the new event to compile. - `test/protocol-contract.test.mjs`. - The contract digests of the pack change (`prp.acpx-sidecar`, `prp.command-v3`, `prp.event-v3`, `rust`, and the new `prp.provider-turn`). No stored file refers to these digests. No provider `commandDigest` and no profile version changes. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. The contract tests: 32 passed. - The sidecar suite and the tracker tests: 141 passed. Not run: the full suite. CI supplies that coverage. ## Risks - **The pack contract digest changes.** Older image packs are not used again. - **The opt-in field is accepted by the schema but has no effect yet.** A controller that sends it before part 5 gets the behavior of today. - **State saved by the previous release:** this pull request changes no persisted shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |