mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
f452b449fd1893a0d46e6a63c5d54ccba6ef2e6e
638
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6b6c94a4ea |
perf(chat): read the GitHub response run by issue run indexes (#21577)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel keeps one response comment for each GitHub request. Two background sweeps (acknowledgements and terminal responses) and the acknowledgement step read the latest run of the assigned agent for the request's issue with `readGitHubResponseRun` > - That function sends two statements. Both match runs with `coalesce(native_issue_id::text, context_snapshot->>'issueId') = <issue id>`. No index has that expression, so each statement reads every run of the agent and decompresses the large `context_snapshot` of each legacy run > - On a hosted test database, these two statements were the two most expensive statements by database time. In a 701 s window, each ran 6.7 times a second, for 203 s and 137 s of database time (about 30 ms and 20 ms for each call). Together they kept about half of one database session busy > - This pull request reads the latest run with two index lookups (native binding, and legacy context without a native binding), the same way #20671 changed the terminal-response sweep. It decides request ownership on that one run in the same statement > - The benefit is that each call reads a few index entries in one statement instead of the agent's whole run history in two statements. The results do not change, and no migration is necessary ## Linked Issues or Issue Description Refs #20671 (the same rewrite for the terminal-response discovery page; its Risks section names this function as the remaining case). **What existing behavior does this improve?** `readGitHubResponseRun` in `server/src/services/chat-channels/github/response-run-state.ts`. The GitHub acknowledgement sweep calls it once for each candidate request. The terminal-response sweep and the acknowledgement step call it through `githubResponseExecution`. **Subsystem affected** GitHub chat channel: request acknowledgement and terminal-response recovery. **Current behavior** The function sends two statements: 1. The latest run of the agent for the issue that woke for the request: `heartbeat_runs` joined to the inbound `chat_message_links` row of the delivery on `wakeCommentId` or `wakeCommentIds`, ordered by `created_at desc`, limit 1. 2. The latest run of the agent for the issue, with the `responseRunColumns` projection, ordered by `created_at desc, id desc`, limit 1. Both statements filter on `coalesce(native_issue_id::text, context_snapshot->>'issueId') = $issueId`. The planner can only use `heartbeat_runs_company_agent_started_idx` or `heartbeat_runs_company_created_at_desc_idx`. It reads all runs of the agent and evaluates the `coalesce` on each row, which decompresses the context of each legacy run. On the hosted test database, `heartbeat_runs` has about 31,000 rows and 1.96 GB, because `context_snapshot` is large. **Proposed behavior** One statement: - The latest run ID comes from two disjoint branches, each `order by created_at desc, id desc limit 1`: runs with `native_issue_id = $issueId` (through `heartbeat_runs_company_native_issue_contract_id_uq`), and runs with `native_issue_id is null and context_snapshot->>'issueId' = $issueId` (through `heartbeat_runs_company_ctx_issue_created_idx`). Together the branches are exactly the rows where the old `coalesce` equals the issue ID. - The statement reads that one run by primary key with the same `responseRunColumns` projection. - An `exists` subquery on the inbound `chat_message_links` row of the delivery decides whether that run woke for the request. It uses the same `wakeCommentId` / `wakeCommentIds` match as before, but on one run only. - When the issue ID is null, the function returns no run without a query. The old statements could not match a null issue ID either. **Reason and benefit** These were the two most expensive statements on the hosted test database. On a local fixture with the hosted shape, the median call goes from 111.6 ms to 1.03 ms (about 108 times faster), and the buffers for one call go from about 73,000 to 98. **Breaking changes** None. The function signature, the returned run, and the ownership result do not change. ## What Changed - `readGitHubResponseRun` reads the latest run of the agent for the issue through the two index-backed branches, and decides request ownership on that run in the same statement. Two statements become one. - Add `response-run-state.test.ts`. It runs the old two-statement function (kept in the test as the oracle) and the new function on the same inputs, and it requires equal results. It also requires a plan bound: one call must read fewer than 150 `heartbeat_runs` rows while the agent has 1,500 unrelated runs. - Add `scripts/benchmark-github-response-run-state.ts`. It loads the baseline module from a given commit, compares the complete output of both versions for every input, measures both, and prints the plans. ## Verification Why the old statements were slow, and the thesis: each statement filters the agent's whole run history with an expression that no index has, and detoasts the context of every legacy run. If each precedence branch uses its own issue index and ownership is checked on the one latest run, the cost no longer depends on the agent's run count, and the results stay identical. Red/green regression test: - `cd server && npx vitest run src/services/chat-channels/github/response-run-state.test.ts` - Red (test file on the old `response-run-state.ts`): the plan bound fails with `expected 3597 to be less than 150`. The equivalence test passes, because the oracle is the old code. - Green (this change): 2 of 2 pass. The equivalence fixture has two companies, two agents each, 60 issues each, and one or two GitHub requests for each issue. Runs cover these cases: native binding, legacy context without a native binding, a native binding to another issue while the legacy context names this issue, another agent, another company, `wakeCommentId`, `wakeCommentIds` as an array, `wakeCommentIds` as a JSON string, a link to the outbound message of the delivery, a link to another company's comment, a link to the sibling request of the same issue, a request with no inbound comment, ties on `created_at`, and a null issue ID. It also calls with another company's endpoint. The test requires more than 20 results in each outcome (owned, latest run not owned, no run) and both binding kinds. Local benchmark with the hosted shape: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-response-run-state.ts --baseline-ref=3c3e6c2d75 ``` Fixture (synthetic): one company, 4 agents with about 7,750 runs each (31,000 runs), 2,000 issues, 300 GitHub requests. Each run has about 60 KiB of incompressible context, and `heartbeat_runs` is 1,932 MB. Runs are native or legacy at random, and about half link to the request comment. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted database is Neon, so absolute times are local measurements only. The benchmark compares the complete output for all 600 inputs (262 owned, 329 latest run not owned, 9 no run): equal. It then times 20 inputs with runs (two agents), with 2 warmups and 7 alternating samples. | | Statements per call | Median ms per call (range) | Buffers for one call | | --- | --- | --- | --- | | Before | 2 | 111.6 (100.9–240.5) | 18,358 hit + 26,498 read, then 14,997 hit + 13,489 read | | After | 1 | 1.03 (0.81–1.39) | 80 hit + 18 read | Plans for one call. Before: statement 1 is a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that removes 7,674 rows by filter (75.7 ms). Statement 2 is an `Index Scan` on `heartbeat_runs_company_created_at_desc_idx` that removes 19,801 rows by filter (60.2 ms). After: one `Index Scan` on `heartbeat_runs_pkey`. Its `InitPlan` is a `Merge Append` of two `Limit` index scans on `heartbeat_runs_company_native_issue_contract_id_uq` and `heartbeat_runs_company_ctx_issue_created_idx`, and its `SubPlan` is one `Index Scan` on `chat_message_links_delivery_idx` (0.128 ms). Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed (all GitHub chat-channel integration tests, including the acknowledgement and terminal-response sweeps). - `cd server && npx tsc --noEmit`: passed. - `pnpm verify` exited 2 because a file under `scripts/` changed, which requires the broad CI coverage. Required CI covers the rest. Why the function runs 6.7 times a second. The acknowledgement sweep and the terminal-response sweep rebuild their candidate set when a hint arrives, and they call this function once for each candidate on each rebuild. A run milestone for a run with no issue (for example, every status change of a run without a task in `setRunStatus`, or a comment merged into an active run in the wake queue) sends a company-wide hint to both sweeps. So in a busy company, both sweeps rebuild about once a second (their minimum pass interval), and each rebuild reads every parked candidate again, although nothing about those candidates changed. In a local reproduction with 8 parked candidates (4 for each sweep) and the real coordinator and workers, 10 company-wide milestone hints a second gave 8.75 calls a second; 1 hint a second gave 7.98 calls a second; 10 hints a second for an unrelated issue gave 0 calls; and no hints gave 0 calls. So the hosted rate is consistent with about 7 parked GitHub requests in one company and a steady stream of company-wide run milestones. This pull request makes each call cheap. A separate pull request will address the repeated rebuilds. ## Risks - Low risk. The change is one function. The run match is the same set of runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id = <uuid>` for the canonical UUID text that callers pass (the issue IDs come from UUID columns), and a null `native_issue_id` makes the old `coalesce` use the legacy context. - Ownership: the old code compared the latest run that woke for the request (ordered by `created_at` only) with the latest run (ordered by `created_at, id`). That is true exactly when the latest run woke for the request, except when two runs that both woke for the request tie on `created_at` with the latest run. In that case the old result depended on the physical row order. The new code always returns the deterministic answer (the latest run woke for the request). - The number of calls does not change in this pull request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3c15acd3fc |
perf(recovery): look up held runs by key so the fresh-continuation deadline stops scanning run history (#21568)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off, the fresh-continuation fallback continues the
task in a new run. The execution-disposition worker schedules its next
pass with `nextFreshContinuationCheckAt` and lists due work with
`heldExecutionsAwaitingFreshContinuation`
> - Both queries share two predicates (`heldExecutionWhere` and
`recentInterruptionSourceSql`). These predicates matched the run as
`source.id::text = <evidence text>` and the task's runs as
`native_issue_id::text = <issue id>::text`. A cast on an indexed column
prevents an index lookup
> - So for each held record, PostgreSQL read the company's whole run
history (two times), and it decoded each run's wide context JSON. On
staging, the deadline query took about 13.5 s of database time in each
27 s worker cycle. It was the largest statement by time
> - This pull request compares the uuid columns directly. It turns the
recorded run id into a uuid only when the text is the canonical
lowercase form, which is exactly when `id::text = text` can be true
> - The benefit is that each held record costs a few index lookups. The
same records are selected, and the worker cycle no longer keeps a
database session busy
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The fresh-continuation scheduler queries: `nextFreshContinuationCheckAt`
(the disposition worker's deadline),
`heldExecutionsAwaitingFreshContinuation` (the backlog listing, also
probed by the schedule), and the "left alone" count. They are in
`server/src/services/fresh-continuation-fallback.ts`.
**Subsystem affected**
Server: recovery and the execution-disposition worker
(`executionControlSchedule().disposition` in
`execution-control-work.ts`).
**Current behavior**
Neon statement statistics on staging (window of 160 s on 2026-11-14):
the `select
min((issue_recovery_actions.evidence#>>'{freshContinuationCheck,nextCheckAt}')::bigint)
…` statement ran 6 times at about 13.5 s each. That is 81 s of database
time in 160 s, about half of one busy session. All other statements in
the window used less than 35 s together. Staging has 31,399 runs (1.96
GB, wide `context_snapshot`), 75,285 recovery records, and 282 records
with a check deadline.
The local plan shows the cause. For each candidate record:
- `source.id::text = coalesce(evidence->>'runId',
evidence->>'sourceRunId')` cannot use the primary key. PostgreSQL
filters the company's failed runs (6,767 rows removed in each loop, and
6,798 again in the outer semi-join).
- In the recent-interruption check, `max(held.finished_at)` matches runs
of the task with `native_issue_id::text = source_issue_id::text or
issue_id::text = … or context_snapshot->>'issueId' = …`. The casts
cannot use the task-link indexes, so PostgreSQL filters the company's
run history and decodes each run's context JSON (29,937 rows removed in
each loop).
**Proposed behavior**
The queries look up the source run by its primary key and the task's
runs by the existing `(company_id, native_issue_id, …)`, `(company_id,
issue_id, …)` and `(company_id, context_snapshot->>'issueId', …)`
indexes. The results do not change.
**Reason and benefit**
On a staging-shaped local fixture, the deadline query goes from 2,330 ms
to 27 ms (87x), and the backlog listing and count go from 3.7-5.7 s to
19-27 ms. Buffer accesses go from 2.15 M to 28 k.
**Breaking changes**
None. No schema change and no migration.
## What Changed
- `server/src/services/fresh-continuation-fallback.ts`:
- Add `sourceRunKey`. It compares the recorded run id with
`heartbeat_runs.id` through the existing `canonicalExecutionUuid` guard
(cast only when the text matches `^[0-9a-f]{8}-…-[0-9a-f]{12}$`).
`uuid::text` always gives that lowercase form, so this matches exactly
the rows `id::text = text` matched. Uppercase, braced, padded, numeric
and invalid ids still match nothing, and no cast error is possible.
- Use it in `heldExecutionWhere`, `interruptionEligibility` and the
listing's order expression.
- `runOfTask` compares `native_issue_id` and `issue_id` with the issue
id as a uuid. Both callers pass a uuid: the `source_issue_id` column, or
the id of a task row read in the same transaction.
- Export `heldExecutionWhere` and `recentInterruptionSourceSql` so the
regression test can compare them with the previous predicates row by
row.
- Add `server/src/services/fresh-continuation-schedule-queries.test.ts`
(regression test, see Verification).
- Add `scripts/benchmark-fresh-continuation-deadline.ts`. It seeds a
staging-shaped fixture, captures the SQL that the baseline revision and
the working tree send, compares complete positional rows, and reports
timings and plans.
## Verification
- Red/green: `fresh-continuation-schedule-queries.test.ts` "finds the
deadline by run-key and task-link lookups, not a scan of run history per
record" counts the `heartbeat_runs` rows that the deadline query's plan
reads (loops x (rows + rows removed)). With the previous predicates it
fails: 549,166 rows read against a bound of 9,442 (two times the run
count). With this change it passes.
- Equivalence: the same file seeds 900 held records with a seeded random
generator. The records cover these cases: `runId`, `sourceRunId`, both
keys, uppercase, braced, padded, numeric, invalid and unknown ids; a
source run in another company; source statuses (failed, timed_out,
interrupted, cancelled, succeeded, running) with and without
`finished_at`; the assignee and other agents; newer runs linked by
native id, legacy issue id, or context only; unrelated runs;
cancelled-at-dispatch runs; live (queued, running, scheduled_retry)
runs; earlier and later creation times; retries of the source; person
Retry wakes; old and recent finish times; restart, person-owned and
sandbox-resume error codes; and issue statuses and assignees. For every
record it compares `heldExecutionWhere()`, `heldExecutionWhere({
includeActive: true })` and `recentInterruptionSourceSql(now)` with
verbatim copies of the previous predicates. It also compares
`nextFreshContinuationCheckAt` and the full ordered
`heldExecutionsAwaitingFreshContinuation` rows with the previous
queries. The test checks that each predicate is true and false for more
than 5 records, so the comparison is meaningful.
- Focused suites pass locally: `fresh-continuation-fallback.test.ts` and
`fresh-continuation-schedule-queries.test.ts` (82 tests). Related
recovery suites pass: `fresh-continuation-retry-delivery`, `task-retry`,
`fresh-continuation-move-forward-offer`,
`execution-recovery-dead-controller`,
`automatic-run-brake-recovery-passes`, `execution-control-work`,
`recovery-wake-herd` (with the new test: 8 files, 252 tests). `node
server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a global CI input),
so CI supplies the broad coverage.
- Benchmark: local embedded PostgreSQL 18.1, `jit=on`,
`shared_buffers=128MB`, macOS arm64, a shared machine with other work.
Baseline `67652580ee`. Fixture: 5,669 issues, 31,351 runs with 60 KiB
incompressible context JSON (1,962 MB), 75,285 recovery records (282
with a check deadline, a third overdue). One warm-up and 5 alternating
timed pairs. Both versions return identical positional rows for every
query.
| Query | Baseline median (samples, ms) | This PR median (samples, ms) |
Speedup | Shared buffers |
| --- | --- | --- | --- | --- |
| `nextFreshContinuationCheckAt` | 2,330 (1,962-2,421) | 27 (21-28) |
87x | 2,149,802 -> 28,227 |
| `heldExecutionsAwaitingFreshContinuation` limit 5 | 3,692
(3,296-4,014) | 19 (18-24) | 194x | 2,831,539 -> 23,970 |
| same, limit 1 (schedule probe) | 3,554 (3,210-4,861) | 20 (18-24) |
179x | 2,831,539 -> 23,970 |
| left-alone count | 5,746 (5,090-5,992) | 27 (25-28) | 213x | 5,435,614
-> 40,953 |
With 1 KiB contexts (rows stored inline) the deadline query goes from
4,096 ms to 19 ms. Staging measured 13.5 s for the deadline query. The
local fixture reproduces the same plan shape, but the staging data
distribution and Neon storage are not reproduced exactly, so the staging
gain is not verified yet.
```sh
pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-fresh-continuation-deadline.ts \
--baseline-ref=67652580ee --context-kib=60 --samples=5 --warmups=1
--print-plans
```
- Plan after the change: the source run is an `Index Scan using
heartbeat_runs_pkey` (0 rows removed), and each task-run probe is a
`BitmapOr` over the native-id, issue-id and context-issue indexes. The
newer-run `not exists` already used that `BitmapOr` before. It does not
change.
## Risks
- Low risk. The SQL change is limited to how two keys are compared. The
regression test compares the old and new predicates for every record of
a varied fixture.
- `runOfTask` now casts its issue id to uuid. A caller that passes a
non-uuid text would get an error instead of no match. The two callers
pass a uuid column and a task row's id.
- Follow-up, not changed here: the disposition schedule calls
`deadline()`, which returns "retry in 15 s" when the earliest check is
already due. An active hold that the disposition pass does not pick up
(for example, one that records its run only as `sourceRunId`, which
`unresolvedExecutionDispositions` does not read) keeps the deadline
overdue, so the worker runs again every 15 s with nothing to do. A local
probe confirmed this. This explains a fixed cadence of 15 s plus the
query time (about 27 s on staging). After this change each such pass
costs milliseconds, not seconds.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL benchmarks and
tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
behavior or command change; no doc update needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
bbed32450c |
perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how many concurrent agents one server process can carry > - It assumed 20 text deltas/s and 1 other event/s per agent, and every other event was an `item.completed` > - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other events/s, with a mix in which 39% of the other events are server trace spans that take a cheaper path > - This pull request makes the recorded rates and mix the model's default, keeps the old rates as an explicit stress profile, adds a burst profile, and mirrors the delta batching of #21520 > - The benefit is a model that measures what production does, so the next optimizations target the real per-agent cost ## Linked Issues or Issue Description Refs #21418 (load model and plan). Refs #21520 (delta batching that the model now mirrors). **What existing behavior does this improve?** The load model's default rates, its event mix and its delta batching rule, and the plan document's measurements. **Subsystem affected** `scripts/benchmark-agent-scaling-load.ts`, its test `server/src/__tests__/agent-scaling-load-model.test.ts`, and `doc/plans/2026-11-13-thousand-concurrent-agents.md`. **Current behavior** The model defaults to 20 deltas/s and 1 other event/s per agent. Every other event is a native `item.completed` with a chat-progress wake and a checkpoint save. It cuts a delta batch on the timer even while the previous batch commits. **Proposed behavior** - `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute. `PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each rate can still be set on its own. - `EVENT_MIX=recorded` (default) sends other events in the recorded proportions. A `run.performance.span` is an ordinary append with no wake and no checkpoint, as the server's run trace writes it. `tool.execution.*` and `item.completed` are native appends with a chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one type only. - Delta batching follows #21520: deltas that arrive while a batch commits all join one next batch. - The report adds `profile` and `eventsIssuedByType`. **Reason and benefit** At the recorded rates the per-agent cost is the non-delta runner events and their checkpoint saves, not deltas. The plan and the next optimizations need that picture. **Breaking changes** None for the product. Anyone who runs the model with no options now gets the recorded profile; `PROFILE=stress` gives the earlier load. ## What Changed - Profiles, the recorded event mix (a 100-slot cycle per agent, offset by agent so the fleet sends the mix at every moment), the span append path, and the coalesced delta batches. - The model test now checks the profile in the report and that both server trace spans and runner events with a progress wake are in the mix, for the recorded profile and for the stress profile with one transaction per delta. - The plan has a new section "Load model at recorded rates": the recorded source (40 recent native runs on one hosted workspace, as aggregates), the profiles, the mix and its server paths, fresh runs at 500, 1,000 and 2,000 agents plus burst and stress, and what they show. It records the 3-connection checkpoint pool as a computed per-process limit on a hosted database, and that SQL building in Drizzle costs about as much main-thread time as result parsing on the hosted instance. A new section "Considered and not done" records why `run.performance.span` rows stay (about 3% of the statements per agent, with readers) and why checkpoint saves per critical event only are not done now. ## Verification - `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2 tests). The new assertions fail against the previous script: it reports no `profile` and no `eventsIssuedByType`. - `server/src/__tests__/agent-scaling-load-model.test.ts` also checks each profile's rates with no overrides, and that a rate set on its own wins. - Load model runs, recorded in the plan (60 s, main pool of 10, embedded PostgreSQL on the same machine, each run started below a 1-minute load average of 15): recorded rates at 1,000 agents (event-loop utilization 0.61) and 2,000 (0.85) finish every operation with no failure; 500 agents all in their busiest minute run at 0.95 with p99 lateness under 0.4 s. About 4.4 statements per agent per second at recorded rates, 35% of them on the checkpoint pool. - Earlier runs at load averages of 20–55 had shown one process falling behind at 500 burst agents and at 2,000 recorded; the quieter runs after the model fixes (event order, spans off the runner stream) do not, and the plan says so. The checkpoint pool as a per-process limit is now stated as a computed limit for a hosted round trip (about 400 agents in a busy minute at 1 ms), not as measured. - Greptile findings fixed: a waiting delta batch no longer takes deltas sent after the next runner event, takes every delta queued when it starts, and spans no longer join the runner event stream. ## Risks - Low: a benchmark script, its test and a plan document. No product code changes. - The machine that measured the plan's table was shared; the plan marks the rows that the machine distorted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6826800425 |
feat(ci): report the required checks on merge-queue commits (#21545)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each merge to `master` publishes the standard image, and the time from merge to published image limits how fast a fix reaches users > - #21096 lets `docker.yml` push an image that the merge queue already built, but the merge queue is not on > - The `master` ruleset requires `ci / verify` and `ci / e2e`. Only `pr.yml` reports them, and only for pull requests > - A merge queue waits for the same two names on its merge-group commit. No workflow reports them there, so every queued merge would stall > - This pull request adds a workflow that reports both names on merge-group commits > - The benefit is that the owner can turn the merge queue on with one ruleset change ## Linked Issues or Issue Description Refs #21096 **What existing behavior does this improve?** The merge-queue image build from #21096. It cannot run until the merge queue is on, and the merge queue cannot be turned on safely. **Current behavior** `ci / verify` and `ci / e2e` come from `pr.yml`, which runs on `pull_request` only. With a merge queue, GitHub waits for both checks on the merge-group commit. They never arrive. Each queued pull request would time out and leave the queue. **Proposed behavior** `merge-queue-checks.yml` runs on `merge_group` and reports `ci / verify` and `ci / e2e`. It runs no tests. It does not check out the queued commit. It has no token permissions and no secrets. **Reason and benefit** A pull request enters the queue only after both checks pass on the pull request. A queued merge therefore has the same protection as a direct merge today. The exact-source gate still tests each `master` commit before publication. Real tests on the combined tree can replace these jobs later under the same names. ## What Changed - Added `.github/workflows/merge-queue-checks.yml`. It reports `ci / verify` and `ci / e2e` on merge-group commits. - Added `scripts/__tests__/merge-queue-checks-workflow.test.mjs`. It fails if the names differ from the names that `pr.yml` reports, or if the workflow gains a checkout, an action, a secret, a token permission or a self-hosted runner. - `pr-trusted.yml` now runs that test and `scripts/merge-group-image.test.mjs` in a step that skips test files an older checkout does not have. The second test came with #21096 but no CI step ran it. - `scripts/merge-group-image.mjs find` now waits up to 15 minutes for a merge-queue image build that is still queued or running. The queue can land a commit before its image build ends. Without the wait, the `master` push always builds from source. - `doc/RELEASING.md` explains the workflow and lists the ruleset settings that turn the queue on and off. ## Verification - `node --test scripts/__tests__/merge-queue-checks-workflow.test.mjs`: 2 pass. - `node --test scripts/merge-group-image.test.mjs`: 10 pass, with new cases for unfinished builds. - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: pass. - `actionlint` on both changed workflows: no findings. - Not verified: no `merge_group` event can fire before the queue is on. The first queued merge is the live test. `pr.yml` calls `pr-trusted.yml@master`, so the new CI step runs only after this merges. Owner steps after merge: 1. In the `master` ruleset, add **Require merge queue**: Squash, build concurrency 5, group size 1 to 5, status check timeout 60 minutes. 2. After the first queued merge, confirm that the `master` commit SHA equals the SHA of its merge-group runs. ## Risks - Low risk while the queue is off. The workflow has no trigger other than `merge_group`. - With the queue on, the queue does not test the combined tree. Two pull requests that pass alone can still break `master` together. This is the same as today. - A merge-queue image build that hangs delays the source build by up to 15 minutes. A build that fails does not. - If GitHub does not accept a job named `ci / verify` for the required context, queued merges time out. Rollback: remove the merge queue rule. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
21bf7ebcee |
feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path > - Paperclip manages work for AI agents. > - SSH environments run tasks on computers that users own. > - A Paperclip server needs network access before it can reach a private computer. > - A reusable Tailscale connection belongs in Connectors. > - Connected machines should become choices in the existing SSH environment form. > - This change connects a tailnet and resolves selected machines through a private network client. > - Agents retain their choice of harness and AI account. ## Linked Issues or Issue Description **Subsystem affected** Connectors, environment settings, SSH execution, company secret storage, and runtime packaging. **Problem or motivation** Entering a Tailscale address in the SSH form does not connect the Paperclip server to that tailnet. Requiring a separate device key for every environment also duplicates network setup. **Proposed solution** Add a Tailscale Connector with **Connect Tailscale Network**. Use brokered OAuth consent to enroll one private network client per connection. Let SSH environments select a connected network and stable machine identity. Preserve the enrolled node state through the company vault so a restart does not reuse the one-time token. **Alternatives considered** A separate Tailscale environment type would duplicate SSH behavior and mix network access with harness selection. This change uses the existing `ssh` driver. Related #15648 adds OAuth client credentials and a health check; this change uses user consent and supplies outbound task execution. Related #11743 concerns HTTPS brokering. **Roadmap alignment** This extends the existing Connected Apps, environment, and Personal & Shared AI Accounts capabilities. This PR is stacked on #21248, which provides SSH account selection independently of Tailscale. The hosted OAuth broker is a companion control-plane change. ## What Changed - Preserve Cloud reauthentication in the current browser tab. Explain canceled or failed consent in the connector and offer a retry of the same authorized saved connection. - Record Cloud enrollment starts initiated through the Tailscale connector in the company activity log. - Check install-preparation descendant liveness through the kernel process table in the Runner support test. Keep the direct child’s strict reaping assertion and reject any non-zombie descendant. - Align the application image stage contract with the Tailscale client stage after the upstream Runner-image refactor. Keep the exact stage list and check every application stage, including `tailscale-client`, for prohibited Runner compiler commands. - Group Tailscale consent, discovery, daemon lifecycle, pairing, and deletion under `server/src/services/connectors/tailscale/`, with their service tests. Update existing route, runtime, environment, vault, and company-deletion imports. Keep the existing contracts, authorization, audit, and transaction owners. - Handle early stdin closure in the Kubernetes real-shell test helper. Stop the input stream on `EPIPE`, preserve nonzero child exits, and reject a zero exit when input was lost. Add two real-process regression cases. Production transfer code and confinement checks are unchanged. - Read configured Tailscale executable paths when a client starts, after supported `.env` loading. Keep explicit command overrides available. - Discard pending pairing results after switching to Direct SSH or closing the form. Retain the existing company, connection, machine, and port checks. - Store network credentials with an immutable server-owned purpose from their first vault row. Exclude them from generic catalogs, bindings, environment projection, and value reads. Use a dedicated audited server consumer for pinned connector reads. Block unclassified legacy serialized node state at the same value boundary. Keep SSH private-key binding separate from connector credentials. - Gate network queries and setup on verified local-implicit or instance-admin access. Hide Tailscale catalog rows for other members, show an admin requirement on direct setup navigation, and avoid device discovery before management access is known. - Keep failed-client cleanup in the network manager. A failed request cannot disconnect a newer retry for the same identity; explicit connector disconnect and enrollment retirement still stop their networks. - Check for readable Ed25519, ECDSA, or RSA host public keys before pairing edits `authorized_keys`. Print a known-hosts entry for every available key, or fail before changing authorized keys if none are available. - Return self-hosted Cloud enrollment to the Tailscale connector. Save the unprefixed setup path and let the shared callback add the verified company prefix. Keep external and unrelated destinations rejected. - Hide pairing instructions after an SSH port change. Ignore late key-selection results for the old port. Operators prepare again for matching known-hosts output. - Retire connector references before deleting a company or its vault credential. Stop company daemons after the database commit. Keep failed secret retirement retryable and prevent late node-state writes from restoring deleted connections. Discard retired state without blocking behind the vault deletion lock. - Reload a mismatched credential before replacing a ready daemon. Recheck the active entry and manager generation after the read. An older request cannot stop a newer healthy socket or cross shutdown. - Pin connector credentials to the exact vault version and SHA-256 digest recorded by trusted enrollment and node-state writes. Ignore generic secret rotations. Advance both fields with a compare-and-set guard. Reject unpinned or tampered credentials before network startup. - Save an explicit Direct SSH selection and clear old Tailscale fields after config merging. Preserve the operator's entered direct host. - Route all Tailscale SSH runtime access through the connection service after SSH-key loading. Pin access to the environment's saved connector identity and recheck after device discovery. Remove standalone key-based daemon startup and reject raw enrollment keys or credential-only network configurations. Align the shared config type and SDK pins. - Correct two E2E assumptions exposed during connector qualification. Keep the two-window chat fixture stale until its own mutation. Validate the original failed run and its permitted write-revocation timestamp in the legacy recovery browser test. - Add the Tailscale Connector page, catalog entry, OAuth return flow, connected-network management, device discovery, and disconnect. Document all six routes in OpenAPI and the API reference, including their instance-admin requirement. - Add company-scoped connection storage in `0480_lazy_lethal_legion`, trusted credential version/digest fields in `0481_round_pet_avengers`, and server-owned secret purpose/backfill in `0482_aberrant_rumiko_fujikawa`. Generate all three migrations through the maintained command and retain the five newest snapshots. - Add machine pairing that generates a fresh SSH key per request, stores its private half in the company vault, and uses the generated public half in the install command. Do not trust editable saved metadata. Preserve existing authorized keys, including a file without a final newline. Keep SSH username, workspace, and trusted host keys explicit. - Use the existing SSH driver. Resolve the machine's current address from its stable device ID. Reject disconnected, removed, offline, and wrong-company selections. - Hide Network when Direct is the only option. Select a sole connected tailnet automatically and hide its selector. Remove the requested setup helper copy. Do not restrict the harness to Claude. - Run a private userspace Tailscale daemon with a separate socket. Persist its node state before execution. Restore it after restart, bound startup, serialize reconnect cleanup, and close it during server shutdown. - Guard OAuth completion with a durable attempt marker. Disconnect and newer authorization invalidate older callbacks. Close replaced daemons and remove cancelled enrollment secrets without changing newer connection state. - Preserve SSH host verification and tailnet-specific session identity. Honor the configured Tailscale executable path, including spaces and OpenSSH percent escaping. - Package checksum-pinned Tailscale clients and their license in the standard runtime image. Exercise the client build stage in PR CI and include it in image smoke checks. ## Verification - Merge status: all CI and image checks pass, and current-head Apex is 5/5 with no unresolved comments. GitHub rejected a direct merge because the active master ruleset requires code-owner approval for `.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3) applies; eligible maintainers already have review requests. Auto-merge is enabled with the squash method for the verified head. The PR remains open pending that required approval; no branch policy was bypassed. - Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI verification passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320). All required checks pass, all applicable test/build gates are green, and the branch is mergeable against master. SSH prerequisite #21248 and Cloud broker paperclipai/paperclip-cloud#1314 are already merged. - [Apex reviewed this exact revision at 5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437). There are no unresolved review threads. The PR is classified as Critical impact. - Final image qualification passes for the [application Runner path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267), [amd64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198), and [arm64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156). Typechecks, Runner support, native/Rust tests, provider tests, serialized-server suites, browser shards, and the remaining applicable CI checks pass. This is source/image CI qualification, not a production rollout. - The last master conflict affected only Docker stage declarations and their boundary test. Both the checksum-pinned Tailscale stage and upstream `runner-lockfile` guard are retained, including the upstream `deps` inheritance. The Tailscale client stage is byte-identical. All 180 cases across five Docker/image contract suites pass locally, along with server and UI typechecks. The maintained 543-input SDK check has no drift. Runtime and lock pins are unchanged. Secret/PII and whitespace scans pass. - Seven OAuth regression cases failed before the fixes. The current-tab test uses the real handoff helper, checks that the session is stored before reauthentication navigation, mounts the shared return page, resumes consent, and verifies handoff cleanup. Callback tests check trusted saved-connection context, fixed safe outcomes, failure audit metadata, denied membership and expired-state rejection, JSON API errors, and no provider text in browser returns. UI retries require a connection in the current company's catalog. All 80 OAuth route/service/UI/shared-handoff cases pass; server and UI source/test typechecks pass. - [Server shard 4/12 on the prior refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436) passed 2,784 cases and skipped 21, then failed a reassignment fixture's `TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its automatic retry found the surviving `REA` prefix. The unchanged test passes all three cases locally. A controlled reader/cleanup overlap reproduces the same deadlock, but the exact competing CI query was not logged. Each case now has a distinct company prefix and identifier; the owned database is removed at suite end. The three test bodies and assertions are byte-identical. Eight local reassignment/lock-release cases pass, and the [final server shard 4/12 passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392). Production reassignment logic, deadlines, and retries are unchanged. - The initial master integration passed 330 cases across 17 suites. Coverage includes connector state, device discovery, network lifecycle, pairing, routes, SSH binding and exports, environment persistence, and connector UI. All three unpublished migrations were regenerated after master's `0479_short_darkstar`: `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL body is byte-identical to its earlier unpublished migration. Generated snapshots preserve existing master tables and add only the intended Tailscale table and secret-purpose column. Exactly five snapshots remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift checks pass. The later Docker refresh did not alter this migration sequence. - Earlier attempts remain part of the verification record. On `1c020a0`, Rust cache restoration and server shard 5/12 were interrupted by shutdown signals, Runner shard 2/3 lost communication with no retrievable log, and an amd64 provider download failed with npm `ECONNRESET`. Same-head retries recovered, without establishing the cause of the communication loss. Later superseded runs also had cancellations with incomplete logs. These interrupted attempts are not counted as passes. The final revision has fresh successful CI. - Local platform limits remain recorded. The Kubernetes suite on macOS passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and skipped four Linux-only cases. Unchanged source reproduced the same 12 failures. An initial run also lacked provider dependencies until the maintained preparation command installed them. Those local attempts are not green qualification. Final Linux provider CI passes. Local Docker qualification was unavailable; final image checks passed in CI. - Before PR separation, a live tailnet/SSH Claude Code task completed on Ubuntu ARM64 without provider login for the remote user. The selected Paperclip subscription ran it, wrote the expected file, posted completion, and marked the task done. File contents and private runtime cleanup were checked. This is prior live evidence; the final revision was not live-retested. Other providers have automated coverage only. Hosted registration for unrelated tailnets and production concurrency remain qualification limits below. - The Superagent security and supply-chain reports remain neutral and inconclusive. Earlier attempts to restart the failed security scan returned HTTP 404. Supply-chain intelligence lacked coverage for extracted pinned dependencies and the shared workflow reference. These reports do not establish successful security qualification. The required CI and review gates pass; this external coverage gap remains disclosed. ## Risks - The Superagent security and supply-chain reports are inconclusive. The security scan re-run request returned HTTP 404, and the supply-chain scanner lacks coverage for the extracted dependencies and shared workflow reference. These neutral results are not successful security qualification. - The server-owned secret purpose is internal database metadata, not an HTTP-writable field. The generic secret service owns enforcement so direct API reads and runtime projection share the same restriction. Previously copied node state remains valid outside Paperclip until the enrolled device is revoked in Tailscale; a local disconnect cannot stop an external copy. - Tailscale network configurations now require a connected Connector and selected machine. The unreleased key-only path is removed; any saved configuration from that path needs Connector setup. - The connector remains a pilot. The local test used an OAuth app registered in the consenting user's tailnet. Qualify hosted app registration for unrelated customer tailnets before general availability. - This implementation supports one serving process per instance. Concurrent replicas need exclusive ownership of the persisted network identity before production rollout. Preserve the database and company vault master key across restarts. - Each machine-pairing preparation creates a new vault secret. Unused generated keys follow the existing secret deletion lifecycle. A saved SSH key remains available as an explicit choice. - Apply additive migrations `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before application rollout. The new credential version and digest columns are nullable. Existing unpinned connections require fresh OAuth consent; the migration deliberately does not trust or backfill a generic secret value. Regenerate or renumber migrations if master advances the sequence before merge. - The target needs an SSH server, its chosen harness, and allowed tailnet access. A reachable network does not grant SSH or provider authorization. Pairing reads standard `/etc/ssh` host-public-key files; custom host-key locations require manual known-hosts configuration. - Roll back the application revision to remove new entry points. Existing connection secrets and registered Tailscale nodes retain their separate lifecycle. Do not drop the table or delete node state as part of an application rollback. - Production qualification still needs denied-policy, revocation, cancellation, and deployment-overlap checks. Local Docker and a production deployment were not performed. ## Model Used OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and this extends existing connector and environment features - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the feature request fields - [x] I have not referenced internal issues, local URLs, or private machine identifiers - [x] My branch name describes the change - [x] I have run focused tests locally and they pass - [x] I have added tests for new behavior and failure paths - [x] I have updated documentation - [x] I have documented migration and runtime risks - [x] All Paperclip CI gates are green - [ ] Required code-owner approval received for the workflow change - [x] Greptile Apex is 5/5 on the final head with no unresolved comments - [x] I will address CI and review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
647641a99b |
feat(ci): let docker.yml copy the published Runner image, behind a repository variable (#21340)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as an attested image, once for each set of its inputs > - Since #21332 the application image takes its Runner only from the `runner-artifacts` stage, and a build can replace that stage with the published image > - No workflow does that yet, so every application image still builds its own Runner, and nothing on an application image says which Runner it has > - A build that copies the published Runner must also build the lockfile that Runner was built from. The workflow refreshes the lockfile before every build, and the server build then refuses the published Runner > - This pull request lets `docker.yml` copy the published Runner image and label the application image with it, behind a repository variable that is off, and makes that build use the committed lockfile > - The benefit is one Runner for all images of the same inputs, named on the image, with a switch that goes back to today's builds without a revert ## Linked Issues or Issue Description Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver), #21307 (the report-only job), #21308 (the server reads a pack from its Runner image), and #21331, #21332, #21333 (the application image takes its Runner from `runner-artifacts`). No issue exists for this change. **Problem or motivation** `docker.yml` builds the application image on `master` and on tags. Each build makes its own Runner. The report-only job that runs the resolver on `master` feeds no build. Downstream image composers cannot tell from an application image which Runner it has. There is a second problem, which only shows when a build copies the published Runner. The Runner's build stamp covers the whole `pnpm-lock.yaml`. The step that vendors the Runner into the server fails when the stamp does not match the tree. `docker.yml` refreshes `pnpm-lock.yaml` on the runner before every build. The refresh rewrites the file whenever the manifest of any package is ahead of it, also a package the Runner does not read. On the tree of this pull request it rewrites the file even for a pristine checkout: the registry changed the deprecation text of one package. The published Runner was built from the committed file, so the image build would fail. **Proposed solution** One job resolves the Runner once for the run. Both architecture builds need it and add its answer to their build: a build context that replaces the `runner-artifacts` stage, two labels, and the build argument with which the build checks the Runner. A build that copies the published Runner does not refresh the lockfile: it builds the committed tree, the tree the Runner was published for. The job runs only when the repository variable `RUNNER_IMAGE_CONSUME_ENABLED` is exactly `true`. This pull request merges with the variable unset, so no image changes when it merges. **Alternatives considered** - Resolve in each architecture build. The two builds could then disagree, and one image index would hold two Runners. - Write the labels in the `Dockerfile`. A `LABEL` cannot be left out for an inline build. - For the lockfile, copy the Runner image's `/pnpm-lock.yaml` over `/app/pnpm-lock.yaml` in the `build` stage. Nothing after that step reads the file: the two vendor checks are its only readers in `build`, `production` copies it and takes `node_modules` from `deps`, both `cloud-*` stages install with `--ignore-workspace --no-lockfile`, and the server reads only a plugin's own lockfile and the pack's own lockfile at run time. But the image would then ship a lockfile that its `node_modules` did not come from, the copy changes that file's layer for every inline build, and each `Dockerfile` edit makes a new Runner image. - For the lockfile, bind-mount the Runner image's lockfile over the file for the two steps that run the vendor check. This also edits the `Dockerfile`, and it makes the lockfile part of the check always true for a published Runner. - For the lockfile, let the resolver refresh and compare. The answer would depend on the registry at two different moments. - Skipping the refresh needs no `Dockerfile` edit, keeps the stamp check as strict as it is, and builds exactly the tree the inputs digest was computed from. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request is built on `runner/app-image-consume`. Merge in this order: 1. #21331, #21332, #21333: the application image takes its Runner from the `runner-artifacts` stage, checks the inputs digest a build names, and stops early on a stale lockfile. 2. `runner/app-image-consume`: the wait and the `unsupported` answer of the resolver. 3. This pull request. The same change for the preview cloud image build in `release.yml` is deferred. It has no pull request yet. ## What Changed - `.github/workflows/docker.yml`: the job `runner_image` replaces `runner_image_report`. - It runs only when `vars.RUNNER_IMAGE_CONSUME_ENABLED == 'true'`, in this repository. A fork never runs it. - Steps: `inputs` with no credential; `resolve`; then `gh attestation verify` of the full index against the signer `runner-image.yml` on `refs/heads/master`. - For a `master` push, `resolve` runs with `--wait` and the job token. For a tag or a manual run it does not wait and receives no token. - Only the verification step writes the job's outputs. An image that does not verify fails that step with a warning and writes no output. The job is `continue-on-error`: the job is red, the run is not, and the builds are inline. - Permissions: `contents: read`, `actions: read`, `attestations: read`. No registry login, no write. - Limits: 5 minutes for `inputs`, 55 for the lookup, 5 for the verification, 70 for the job. The wait of 40 minutes ends inside them. - `build-and-push` needs `runner_image` and adds its outputs to the build: `build-contexts`, one more line of `build-args` (`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`), two more lines of `labels`. With empty outputs these are the values the build had before. - The step "Refresh lockfile for Docker build context" runs only when the answer is not `published`. - The step that looks for a merge-queue image of the commit runs only when the answer is not `published`. The two steps after it already depend on its result. - Labels on an image that copied the published Runner: `io.paperclip.runner.image=ghcr.io/paperclipai/paperclip-runner@sha256:<full index digest>` and `io.paperclip.runner.inputs-digest=<digest>`. An image without them made its own Runner. - The `Dockerfile` does not change. The Runner image digest does not move. - Tests that pin lines this changes follow: `scripts/merge-group-image.test.mjs` (the condition of the merge-queue lookup), `scripts/reuse-canary-image.test.mjs` (the `needs` of the build job), and `docker-build-stamp.test.ts` (no workflow names a build context; `docker.yml` now names exactly one). - `server/src/__tests__/docker-runner-workflows.test.ts`: the report-only tests give way to tests of the new job. - `server/scripts/prepare-runner-vendor.test.mjs`: one fixture test of the lockfile case. - `doc/runner-image-contract.md` and `doc/DOCKER.md`: how an application image gets its Runner, the labels and what their absence means, the variable, the wait, the verification, the lockfile rule, the merge-queue rule and the rollback. ## Verification ### Real builds `docker buildx build --target build --platform linux/arm64 --output type=cacheonly`, with a `docker-container` builder made for this test. The Runner image is the publication form (`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1`, `--target runner-artifacts`) built from a clean export of this branch, exported as an OCI layout: inputs `c4f9d089…34f0`, image `sha256:bef7e9cb…ffef`. The application builds receive `--build-context runner-artifacts=oci-layout://…@sha256:bef7e9cb…` and `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=c4f9d089…`. The refresh command is the one in the workflow, run with pnpm 9.15.4 in the export: `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. | Build context | Result | | --- | --- | | The committed tree | Passes in 78 s. Both vendor checks: `reusing the current Runner build (inputs unchanged …; 1333 files and 1860 outputs checked)` | | `ui/package.json`: `clsx` `^2.1.1` → `^2.1.0`, then the refresh command (what the workflow did before this change) | Fails at `RUN node server/scripts/prepare-runner-vendor.mjs`: `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 forbids the Runner rebuild this step needs (inputs changed since the recorded build; 1333 files checked)` | | The committed tree, then only the refresh command | Fails the same way. The refresh changed only the `deprecated` text of one package | | The same `ui/package.json` change with no refresh (what the workflow does now for a `published` answer) | Passes in 66 s. `deps` resolves the manifest inside the image; both vendor checks reuse the Runner | | One line added to `packages/paperclip-runner/src/index.ts` | Fails at the vendor step, as before: a Runner of other inputs is still refused | Lockfile behind the Runner's own manifests (`packages/paperclip-runner/package.json`: `smol-toml` `^1.9.1` → `^1.9.0`), no build context: | Step | Result | | --- | --- | | `runner-image-resolve.mjs inputs`, then `resolve` | `lock: behind`, no digest; `inline (lock-behind)` | | Build without the refresh | Fails in 4 s at the `runner-lockfile` stage: `ERROR: pnpm-lock.yaml is behind the manifests the Runner is installed from …` | | Build after the refresh command (what the workflow does for an `inline` answer) | Passes in 193 s: the `runner-lockfile` stage passes, the build makes its own Runner with its pack, and both vendor checks reuse it | ### Tests macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64, with `jq` and `zstd` added), on this branch: - `pnpm exec vitest run` on `docker-runner-workflows`, `runner-image-resolve`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse`, `cloud-image-sentry`, `runner-lockfile-guards` and `runner-image-identity` in `server/src/__tests__`, and `server/scripts/prepare-runner-vendor.test.mjs`: 225 tests in 11 files pass on Linux; on macOS 224 pass and 1 is skipped (it needs bash 4.4). - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on both. - The changed test files typecheck with the options of `server/src/__tests__/tsconfig.json`. What the workflow tests hold: - Variable off: the job's condition is the variable and the repository, with no alternative. With empty outputs, the build step has exactly the 3 build arguments, 0 build contexts and 1 label entry it had before, and the refresh step runs. - Published: both architecture builds are one job that reads only the outputs of `runner_image`, with no architecture in those lines. - The verification step runs as real shell against a stub `gh`. When `gh` succeeds it writes the context, the labels and the build argument. When `gh` fails the step exits 1, writes no output and prints the warning. The step has no `continue-on-error`; the job has. The condition of the build job does not name the resolver job and has no `success()`. - The lookup step runs as real shell against a stub `node`. It passes `--wait` only when the event is a `master` push. - Credentials: only the lookup and the verification steps have a token. The steps that build have no `env`. - The refresh step has the condition `needs.runner_image.outputs.mode != 'published'`, and it is the only `pnpm install` of the build job. - The fixture test gives the vendor step a Runner that another tree built for the committed lockfile. The step reuses it. After a change of one specifier of another package in the lockfile the step fails and names the lockfile. With the committed lockfile back it reuses the Runner again. - The merge-queue workflow and the pull-request check do not name the resolver, the variable, a build context or a Runner label. Read-only checks against the real services: - `gh attestation verify oci://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…1bf7 --repo paperclipai/paperclip --signer-workflow paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref refs/heads/master`: exit 0 for the full index, and exit 0 for the slim index `sha256:69ab2001…69b3`. - The same command with `--signer-workflow …/docker.yml`: exit 1 for both. With `--source-ref refs/heads/not-master`: exit 1. With `--repo cli/cli`: exit 1. For a digest that does not exist: exit 1. With no token: exit 4. - The parsed YAML of `docker.yml` differs from its base only in the resolver job, the `needs` of the build job, the conditions of the refresh step and of the merge-queue lookup, and the three build inputs. Never run: - `docker.yml`. Only GitHub can run it, and it has no pull-request trigger. - A build with the variable on. The first one is the first run after a maintainer sets the variable. - A build that pulls the published Runner from the registry as `docker-image://`. The real builds used the same image form from a local OCI layout. - An `amd64` build. - The `production` and `cloud` targets. The real builds stopped at `build`, the last stage that checks the Runner. ## Risks - With the variable on, an application build on `master` starts after the resolver job. That is 2 to 3 minutes when the image exists, and up to 40 minutes from the push when the commit's own Runner publication is in progress. `docker.yml` holds one concurrency slot for each ref, so a long wait can make more `master` commits lose their image build to a newer push. - A build that copies the published Runner ships the committed `pnpm-lock.yaml` in `/app`, while `deps` resolves newer manifests of other packages inside the image. A local `docker build` has always done this; a CI image did not. Nothing in the image reads that file. - Any other build that gives the `Dockerfile` a published Runner and a refreshed lockfile fails at the vendor step. That is the intended answer to a tree the Runner was not built for, and the message names the lockfile. - The pack of an image that copied the published Runner names the commit that published the Runner (`runnerSourceRevision`), which can be older than the commit of the image. A consumer that compares the two must accept that before the variable is set. - A failed verification makes the resolver job red on a green run. That is on purpose: it means a published tag whose attestation does not verify. - `scripts/merge-group-image.mjs` refuses to promote a merge-queue image for a push that changes `docker.yml`. The push that lands this change is such a push. The merge queue is not enabled. - With the variable off there is no report of the resolver on `master` any more: the job that replaced the report-only job does not run. - Rollback: set the variable to another value or delete it. The next run builds inline, with the refresh. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds, read-only calls to the GitHub API and the container registry). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
e824c5cc16 |
feat(ci): let the Runner image resolver wait for the publication of a master push (#21339)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, once for each set of its inputs > - The application image will copy that Runner instead of making its own. `scripts/runner-image-resolve.mjs` tells a build whether the Runner of its inputs is published > - On a `master` push, the commit's own run of `runner-image.yml` is often still publishing when the application build asks. The resolver answers once, so that build is told `absent` for an image that exists some minutes later > - The resolver also cannot say that a tree is not safe to give a published Runner to. A `Dockerfile` that does not check the Runner it receives builds with any Runner > - This pull request adds an opt-in wait for the commit's own publication run, and an `unsupported` answer for a `Dockerfile` that cannot check its Runner > - The benefit is that the next pull request can let `docker.yml` copy the published Runner without a blind timeout and without a label that is not true ## Linked Issues or Issue Description Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver), #21307 (the report-only job), and #21331, #21332, #21333 (the application image takes its Runner from `runner-artifacts`). No issue exists for this change. **Problem or motivation** A `master` push starts `runner-image.yml` and `docker.yml` at the same time. Publication takes about 30 minutes from the push, and several of those minutes are runner queue time. An application build that asks the resolver at once gets `absent` for every commit that changes the Runner's inputs, and for commits that follow one closely. A fixed poll of the tag has the opposite fault: when no image will ever come (the lockfile is behind, or the run failed), every build waits the full timeout. **Proposed solution** `resolve --wait` waits for a run, not for a time. It finds the runs of `runner-image.yml` for the push of the commit to `master`, and it looks up the image again when they end. No caller uses the flag in this pull request, so no workflow changes behavior. **Alternatives considered** - Poll the tag until a timeout. Every build with no image to come waits the full timeout. - Call `runner-image.yml` from `docker.yml` as a reusable workflow. Publication would then share the one concurrency slot of `docker.yml`. - Wait until the run completes and only then look. The run goes on after publication (it writes the build cache), so the wait would last longer than needed. The command looks up the image while a run is in progress. - Verify the attestation in the resolver. The workflow that uses the answer does it, in a step of its own, in the next pull request. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request is built on #21333. Merge in this order: 1. #21331, #21332, #21333: the application image takes its Runner from the `runner-artifacts` stage, checks the inputs digest a build names, and stops early on a stale lockfile. 2. This pull request: the wait and the `unsupported` answer of the resolver. 3. `runner/app-image-consume-docker`: `docker.yml` copies the published Runner, behind a repository variable. The same change for the preview cloud image build in `release.yml` is deferred. It has no pull request yet. ## What Changed - `scripts/runner-image-resolve.mjs` - `resolve --dir <directory> --wait`. When the first answer is `absent` or `incomplete`, the command lists the runs of `runner-image.yml` for the commit (Actions API, `GITHUB_SHA`, `GITHUB_TOKEN`) and looks again when they end. - Only a run of the `push` event on `master` in this repository counts. A pull-request, scheduled or manual run of the same commit is never waited for. The request filters by event and branch, and the script checks each run again. - A run that fails, is cancelled or publishes nothing ends the wait when it ends. With several runs for one commit, the command waits for all of them. - While a run is in progress, the command looks up the image every 30 seconds, so the wait ends at publication. - A run that is not listed yet is looked for during 2 minutes. - The limit is 40 minutes from the push: `RUNNER_IMAGE_WAIT_MINUTES`, the one constant. The push time is the creation time of the oldest run of the commit, so a build that starts late waits less. - Three failed API requests in a row end the wait. No token or no full commit SHA means no wait. - After every ending the command looks up the image one last time. No ending is an error. Without an image the answer is `inline`, and the command prints a notice with the reason. - `inputs` reads the `Dockerfile` first. Without a line `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` it runs no build, and `resolve` answers `inline` with the reason `unsupported`. - `resolve` reports `build_args`: the value of that argument for a `published` answer. - The wait and the list of runs take the clock, the sleep and `fetch` as arguments, so the tests use no network and do not sleep. - `server/src/__tests__/runner-image-resolve.test.ts`: every outcome of the wait and the answer that follows it, the run filter, the limit measured from the push, the API failures, and the command line. - `doc/runner-image-contract.md`: a "Waiting for a publication" section, `build_args`, and the `unsupported` answer. ## Verification macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64, with `jq` and `zstd` added), on this branch on top of #21333: - `pnpm exec vitest run` on `runner-image-resolve`, `docker-runner-workflows`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse`, `cloud-image-sentry`, `runner-lockfile-guards` and `runner-image-identity` in `server/src/__tests__`, and `server/scripts/prepare-runner-vendor.test.mjs`: 218 tests in 11 files pass on Linux; on macOS 217 pass and 1 is skipped (it needs bash 4.4). - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on both. - The changed test file typechecks with the options of `server/src/__tests__/tsconfig.json`. Read-only checks against the real services: - The lookup, with no credential, answered `published` for the inputs `d4df566f…af00` and `0e6662ad…d1ca`, with the index digests that `master` published, and `absent` for a digest that does not exist. - The real answer of the Actions API for the `runner-image.yml` runs of four `master` commits went through the wait, with a clock that does not sleep. Three completed runs ended it at once. A queued run ended it at the limit of 40 minutes from its creation time. A commit with no run ended it after the 2 minutes for a run that is not listed. - Of the 60 newest runs of `runner-image.yml`, the filter accepted the 19 `push` runs on `master` and none of the 41 pull-request runs. - `runner-image-resolve.mjs inputs` ran on a clean export of the tree of the next pull request, with a `docker-container` builder. The answer was `lock: current` and the inputs digest `c4f9d089…34f0`. The `Dockerfile` of #21332 declares the argument, so the answer was not `unsupported`. The same command on a tree with one changed range in `packages/paperclip-runner/package.json` answered `lock: behind`, and `resolve` then answered `inline (lock-behind)`. Never run: - `resolve --wait` in a workflow. No workflow calls it in this pull request. - The full test suite and the full typecheck. CI must supply them. ## Risks - `resolve` now prints a notice for every `inline` answer. The report-only job in `docker.yml` therefore shows one notice on each `master` push whose Runner image is not published yet, until the next pull request replaces the job. Nothing reads that job. - `inputs` answers `unsupported` for a tree whose `Dockerfile` does not declare `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` at the start of a line. #21332 adds that line; a later edit that moves or renames it turns every answer to `inline`. - The wait is bounded by time from the push, not by its own duration. A build that starts more than 40 minutes after the push looks once and does not wait. - A registry that does not answer adds at most about 3 minutes to one lookup, because the registry client retries. A caller must give the step more than 40 minutes. The next pull request gives it 55. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only calls to the GitHub API and the container registry). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
999709b0a6 |
fix(docker): stop a stale-lockfile build early and keep beta and stable releases buildable (#21333)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The application image now takes its Runner from the `runner-artifacts` stages, which install with a frozen lockfile > - A contributor cannot commit `pnpm-lock.yaml`. After a change to a manifest of the Runner, the lockfile on `master` is behind until the refresh merges > - In that time a plain `docker build` fails late, inside a long step, with a message that is easy to miss. And a beta or a stable release can be cut from such a commit, because the release workflows refresh the lockfile for their own build > - This pull request adds an early check with a clear instruction to the image build, and the same check to the beta and the stable release > - The benefit is a failure in seconds that says what to run, and beta and stable tags that always build from source ## Linked Issues or Issue Description This is the third part of a chain. The first part is #21331. The second part takes the Runner of the application image from the `runner-artifacts` stage. Refs #21204 and #21306 (the Runner image and its lockfile rule). No issue exists for this change. **What happened?** A tree with a lockfile behind the manifests of the Runner fails `docker build` after the daemon compile (169 s in the test). The error is `ERR_PNPM_OUTDATED_LOCKFILE` in the middle of the largest `RUN` of the `Dockerfile`. **Expected behavior** The build fails within seconds of the check running. The message names the files and the command to run. The application's dependency install starts from the check, so it cannot start before the check passes. Stages that do not depend on it (the Runner's toolchain download and daemon compile, and the tool layer of `production`) can already be running, and the failed check cancels them. A beta or a stable release from such a commit does not start. How often this happens: of 72 nightly tags since 2026-08-24, 9 are on a commit whose lockfile is behind the manifests of the Runner (2026-09-01, 09-18 through 09-23, 09-28 and 10-26). Of 12 beta tags, none is. A beta is the promotion of a nightly, so a beta can be cut from such a commit. **Steps to reproduce** 1. Change a dependency range in `packages/paperclip-runner/package.json`. Do not refresh `pnpm-lock.yaml`. 2. Run `docker build .`. **Paperclip version or commit** `master` with the previous pull request of this chain. **Deployment mode** Not applicable. This is the image build and the release tooling. ## What Changed - `Dockerfile`: a `runner-lockfile` stage copies only the manifests the Runner build installs from, the lockfile and the patches. It runs the frozen, filtered, offline lockfile check, with the arguments of `runnerLockState`. `deps` starts from it. For a stale lockfile it prints the files and `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`. For any other failure it prints the output of pnpm and says so. - `scripts/check-release-runner-lockfile.mjs`: exports the Runner inputs of a commit with `git archive` and gives them to `runnerLockState` (`scripts/runner-image-publish.mjs`). Exit 0 for current, 1 for behind, 2 for an error. - `scripts/release.sh` and `scripts/release-lib.sh`: `require_runner_lockfile_current` runs for the `beta` and the `stable` channel, after `--print-version` exits and before any build. It runs in a dry run too. It does not run for `canary` or `nightly`. - Both beta paths end in the `publish_beta` job: the promotion of a nightly, and a `candidate/beta-*` branch. That job checks out the commit that `select_beta` chose and runs `release.sh beta` there. So the check reads the committed files of the commit the beta is built from. A beta is refused before its soak starts. The stable that promotes it is checked again. - Tests: `server/src/__tests__/runner-lockfile-guards.test.ts` (18 tests). Three of them pin the checkout of `publish_beta`, `preview_stable` and `publish_stable` to the selected commit. Two tests plus one stub in `scripts/__tests__/release-dry-run-notes.test.mjs` cover the beta, the candidate beta and the stable path. - Docs: `doc/DOCKER.md`, `doc/RELEASING.md`, `doc/runner-image-contract.md`. ## Verification Builds: `linux/arm64`, local. They ran before the rebase onto `master`, except the last one. `master` changed neither the `Dockerfile` nor a manifest in between. - A tree with `ajv` changed from `^8.20.0` to `^8.19.0` in the manifest of the Runner: `docker buildx build --target build` fails after 6 s at `runner-lockfile`, with the message. - The same tree after `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile` on the host, which is what the image workflows run: `--target build` builds. The check passes, the Runner stages install, and the vendor step logs `reusing the current Runner build` in both of its runs. - After the rebase: `build` and `runner-provider-pack` built from this branch without a build commit. The check passed. - The release check with the real pnpm: `behind` for the first tree, `current` for the second, exit 0 for this checkout. - `vitest run server/src/__tests__/runner-lockfile-guards.test.ts` with the other changed server tests, on macOS and in `node:24.22.3-bookworm` (249 pass in 11 files on Linux). The test file typechecks. - `node --test scripts/__tests__/release-dry-run-notes.test.mjs`: 14 pass. `scripts/release-lib.test.mjs`: 15 pass on Linux; on macOS 4 of its tests fail on `master` too. `release-verify-workflow.test.mjs` and `acpx-patch-packaging.test.mjs`, which read the release scripts: pass. - `docker buildx build --check`: no warning. - The Docker builds above ran before the second part of the chain got its expectation step and before the beta guard. The expectation step was built on the second part alone. This branch as it is now was not built again; its tests ran. - Not run: the release workflow, any `linux/amd64` build, the full test suite. Only CI and a release dry run can prove the release path. ## Risks - **A beta can be refused.** When the newest nightly is on a commit whose lockfile is behind, `release.sh beta` stops, in the dry run and in the real run. The operator must promote a nightly whose commit includes the lockfile refresh, or cut a candidate branch that includes it. `node scripts/check-release-runner-lockfile.mjs --commit <ref>` gives the answer before the dispatch. - **A stable can be refused** only when its beta was cut before this change, or with release tooling that does not have the check. Promotions run the release tooling of the source commit. - The check runs in `release.sh`. In the beta workflow it runs in `publish_beta`, after the approval of the `npm-beta` environment and after the candidate verification. In the stable workflow it runs after verification, not in the `preflight_stable` job. The selection jobs have no pnpm. - `release.sh stable` now needs `git`, `tar` and `pnpm` on `PATH` before its first build step. The workflow has them. - `scripts/__tests__/release-dry-run-notes.test.mjs` is not in a CI command today. The Vitest file carries the enforced tests. - The image build has one more small stage. It reads only manifests, so a source change does not run it again. - The digest of the Runner image changes again, because its identity covers the whole `Dockerfile`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
05fef9f27b |
perf(server): add a load model of per-agent server cost and the plan for 1,000 concurrent agents (#21418)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every agent run of a workspace > - A hosted workspace slows down with about 50 agents running at the same time > - The target is 1,000 agents at the same time, served by several server processes > - This pull request adds the plan and a local load model that measures what each running agent costs the server > - The benefit is a measured ranking of the causes, and a repeatable before/after number for each fix ## Linked Issues or Issue Description **What existing behavior does this improve?** The server controls agent runs. With about 50 runs active, requests wait and run events fall behind. **Current behavior** Each text delta from an agent is one database transaction. In the load model, 50 agents at 20 deltas per second issue about 3,900 statements per second, and the process completes about half of the offered deltas. Agent authentication slows from 7 ms to 464 ms because it shares the connection pool. **Proposed behavior** The plan gives three phases: measure, decrease the cost per agent, then let several server processes own runs. Each fix is one pull request with a test and a before/after number from the load model. **Reason and benefit** The ranking was an estimate from code. The load model replaces the estimate with a measurement and gives each fix a number to improve. ## What Changed - Added `doc/plans/2026-11-13-thousand-concurrent-agents.md`: finish line, evidence, load model result, phases, and owner decisions. It now also has a "Load model after Phase 1" section. That section has a table at 50, 100, 200 and 400 agents and a row at 200 agents with deltas and leases as before Phase 1. The 400-agent delta count was measured before the count fix and is marked as not reliable. - Added `scripts/benchmark-agent-scaling-load.ts`: N simulated agents call the real event append, controller checkpoint save, lease renewal, and agent authentication code against an embedded PostgreSQL. It reports statements for each database pool, event-loop utilization, and lateness for each source. - Deltas: by default the model commits deltas as the server port does after #21460. The first text after another event commits at once. Later deltas commit in one transaction (`appendHeartbeatRunEventBatch`) per 250 ms or 100 deltas, and the stream does not wait for the receipts. `DELTA_BATCH_MS=0` keeps one transaction per delta, as before batching. - Leases: by default the model renews every lease through the shared schedule from #21444 (`startNativeSessionExecutionLeaseRenewal`, `renewNativeExecutionLeasesNow`). `LEASE_MODE=single` keeps one transaction per run, as before. - The commit counter is read before the checkpoint drain, so commits that come after the window do not count in it. - Per-source counts freeze when the window closes. An operation that settles later, while queued work drains, is not done. - Batch lease mode needs `LEASE_S` of at most 20. The shared schedule renews a remote lease on its own 20 s after its last renewal, and the model does not count that work, so the script rejects a longer interval. - Added `server/src/__tests__/agent-scaling-load-model.test.ts`. It runs the model end to end in both modes: the default (batched deltas and shared lease renewal) and `DELTA_BATCH_MS=0 LEASE_MODE=single`. It checks that every source finishes without failures and that each pool is counted. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-scaling-load-model.test.ts` on a tree that contains #21460 and #21444: 2 of 2 pass (both modes). The test also checks that no source reports more done than offered. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-scaling-load.ts` with `AGENTS=50 SECONDS=20` and higher agent counts. The results are in the plan. - `pnpm check:module-boundaries` passes. - No product code changed. ## Risks Low risk. The pull request adds a document and a script that no product code imports. The load model runs on loopback PostgreSQL, so its absolute numbers are not a prediction for a hosted database. The plan states this limit. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. Subagents on the same model read the code paths. - The two-mode load model test and the merge with master: Claude Opus 5.5 (`claude-opus-5-5`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
0d832a02d3 |
perf(ci): shorten the general server and chat test lanes (#21393)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `PR` CI workflow before it can merge > - The `General tests (server (N/12))` and `General tests (chat (N/3))` jobs finish last, so they set the wall-clock time of CI > - Before this change, server shards took 9.5 to 11.7 minutes and chat shards took 6.4 to 9.6 minutes; the target is 7 minutes or less > - Measurement shows that the chat lane was unbalanced and that the server lane is limited by CPU on the 4-vCPU runners > - This pull request balances the chat lane by recorded test time, runs three chat partitions per job, and takes general server weights from PR runs > - The benefit is a chat lane under 7 minutes and measured evidence for the shard count that the server lane needs ## Linked Issues or Issue Description **What existing behavior does this improve?** The PR CI general server and chat test jobs. They are the slowest jobs of every pull request. **Subsystem affected** CI test scheduling: `scripts/run-vitest-stable.mjs`, `scripts/test-line-shard.mjs` and `scripts/ci/refresh-test-durations.mjs`. **Current behavior** Baseline: green `PR` runs [43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012) and [43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316), all on m7i-flex.xlarge (4 vCPU, 16 GB). - Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step, effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports `tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums 1,410 to 1,765 s of file time. The longest file is 303 s (`deferred-execution-receipt-work.test.ts`), so the floor is total work divided by workers, not one file. - Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each job ran two partitions of `chat-channels.integration.test.ts`. The partitions were balanced by collected-case count, but one `it.each` line can run many cases. One of six partitions recorded 532 s of test time against a 377 s mean. **Proposed behavior** - Chat partitions keep each source line whole and balance by recorded time per collected entry. - Each chat job runs three partitions in place of two. - General server weights come from the server jobs of merged PR runs, which run files with the same concurrency as the lane they place. **Reason and benefit** The slowest job sets the CI wall-clock time for every pull request. The chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes. ## What Changed - `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional recorded durations. With durations, it balances source lines by recorded time; entries without a record take the median. Without durations, the behavior is unchanged. - `scripts/chat-shard-durations.json` (new): recorded time per collected chat entry, from the CI profile records of PR run [43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171). 926 of 928 collected entries are measured; the other two are the bogus collection entries that the launcher already drops. - `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that manifest from downloaded `ci-profile-*` artifacts. Each executed case maps to the collected `it`/`it.each` entry that registers it. A case without an entry fails the refresh. - `scripts/run-vitest-stable.mjs`: a sharded chat job runs three partitions in place of two, and the chat launcher passes the recorded durations. A chat job collects the suite once with `vitest list` in place of once per partition, and the per-partition filter checks run at the same time. Before, each partition ran two blocking collections of the 86,000-line file one after another. The server worker count stays at three. - `scripts/ci/refresh-test-durations.mjs` and `scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general server weights from the two newest merged-PR runs. Each run must have the full server shard count from `.github/ci-task-matrix.json` and cover at least 90% of the lane. Serialized weights still come from master Cloud readiness runs. - `scripts/general-server-shard-durations.json`: refreshed with the new source (PR runs 43439145752 and 43439279537). - Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`, `scripts/ci/test-selection-pr-timings.test.mjs` and `server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted partition, the entry mapping, the manifest format, the PR-run collection rules and the three-partition chat launcher. - `doc/DEVELOPING.md`: describes the three chat partitions, the chat weights and how to refresh them, and the new source of the server weights. ## Verification Experiments ran in this PR's own CI with a temporary resource sampler (removed before review). | Setting (run) | Server step vs. same shard's baseline | CPU busy | Max memory used | Result | |---|---|---|---|---| | 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) | 62-73% | 6.8 GB | 1 flaky retry | | 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean | | 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard failure: `decision-models.test.ts` timed out at 15 s twice | | 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB | 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at 30 s on m5a | The server lane is CPU bound: at four workers the CPU is 82-89% busy and wall time improves by only about 10%. Five workers fail tests. Memory is not the limit (no swap, no memory pressure, at least 6 GB free). Node's compile cache and Postgres `fsync=off` showed no attributable gain (import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to 0.2-0.7% of CPU time), so neither is in this PR. Final configuration (server: 3 workers with refreshed weights; chat: 3 weighted partitions per job), job minutes / test-step seconds / effective concurrency: | Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503 attempt 1 | B: attempt 2 | C: attempt 3 | |---|---|---|---|---|---| | server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 / 2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 | | server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85 | 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 | | server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81 | 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 | | server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 / 2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 | | server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85 | 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 | | server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69 | 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 | | server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 / 2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 | | server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83 | 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 | | server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68 | 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 | | server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84 | 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 | | server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 / 2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 | | server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 / 2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 | | chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 | 5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 | | chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 | 6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 | | chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 | 5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 | Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B / C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server worker setting is unchanged, so its run-to-run spread is mostly runner type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min). Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five server shards on m5a.xlarge summed 2,545-3,037 s of file time against about 1,700 s elsewhere; three reached the 20-minute job limit and two failed tests on 15 s timeouts. The server lane settings in this PR are the same as master's, so this is a fleet finding: m5a.xlarge cannot run a server shard inside the job limit (follow-up for the fleet configuration, outside this PR). Flakiness of the final setting: the nine chat jobs in A, B and C had no failure and no retried test. Server shards in A and B had no failure and no retried test. Before the last commit (single chat collection), three runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server retries there (2 in one run, 1 in another) were on the unchanged three-worker server setting. Local checks (correctness only): - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs`: 68 passed. - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/vitest-chat-shards.test.ts`: passed. - `pnpm verify` exits 2 (global CI input changed), so the required CI checks supply the broad coverage. ## Risks - Three chat partitions per job use more memory: the peak was 11.4 GB of 16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63% busy. - Stale chat weights only change balance, never which tests run. Shard membership is still checked with `vitest list` before each run. - The weekly refresh now downloads general server logs from merged-PR runs. Runs that selected a subset of files are skipped before download because their shard count is smaller. - The server lane is still above 7 minutes. See the remaining floor below. **Remaining floor and shard-count recommendation (for a follow-up under `.github/`)** A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs give at most about 2.8 effective concurrency, so the test step is about 560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test step of about 360 s, so about 1,000 s of file time per shard. The current 19,200 s of general server file time therefore needs about 19-20 shards of 4 vCPUs at 3 workers. More workers on the same runners do not help, because the CPU is already the limit. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, GitHub CLI) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c6301c92ff |
fix(ci): keep the typecheck job working on stacked PRs and stop its CLI guidance test timing out (#21451)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the required `Typecheck + Release Registry` job from the trusted workflow on `master` > - #21404 replaced that job's six sequential steps with one script, `scripts/ci-typecheck-lane.mjs`, that runs independent checks in parallel > - PR runs take the workflow from `master` but check out the PR's merge commit. A stacked PR's merge commit is built on its base branch, so it can lack a script that only `master` has > - The lane also ran the CLI guidance test beside the typechecks at a lower CPU priority, and one of its cases is slow enough to cross its 15 second timeout there > - This pull request makes the job fall back to the earlier commands when the script is absent, and runs the CLI guidance test before the parallel checks > - The benefit is that stacked PRs typecheck again and the job stops failing on a timeout that has nothing to do with the PR ## Linked Issues or Issue Description Refs #21404 **What happened?** In the first eight PR runs after #21404 merged (2026-11-13 18:06Z), the typecheck job failed three times for reasons unrelated to the PRs: - Twice on #20617, a PR stacked on `feat/ssh-agent-ai-accounts`: `Error: Cannot find module '/home/runner/_work/paperclip/paperclip/scripts/ci-typecheck-lane.mjs'` (jobs 132076887259 and 132079570039). The job failed 0 seconds into the step. - Once on run 43480469363 (job 132075678943): `CLI guidance safety failed with exit code 1`, from `Error: Test timed out in 15000ms.` in `allows only exact-allowlist pnpm paperclipai commands on every guidance surface`, which took 17059 ms. The same case passes in about 5 seconds when the test runs alone. **Expected behavior** The typecheck job passes or fails on the PR's own code, including for stacked PRs. **Actual behavior** Stacked PRs always fail the job. Other PRs fail it intermittently (1 of 6 runs that reached the lane). **Steps to reproduce** Open a PR whose base branch does not contain `scripts/ci-typecheck-lane.mjs`, or run the lane on a loaded 4 vCPU runner. ## What Changed - `.github/workflows/pr-trusted.yml`: the lane step runs only when `scripts/ci-typecheck-lane.mjs` exists in the checkout. A new step runs the six earlier commands, in their earlier order, when it does not. Only the `typecheck_release_registry` job changed. - `scripts/ci-typecheck-lane.mjs`: the CLI guidance test moved from the parallel checks to the end of the sequential preparation steps, at normal priority. - Tests: `scripts/__tests__/ci-typecheck-lane.test.mjs` and `.github/scripts/tests/ci-plan.test.mjs` assert the two conditions, that the earlier commands appear only in the fallback step, and that the CLI guidance test is a preparation step. - `doc/DEVELOPING.md`: the "Typecheck lane in CI" section describes both. ## Verification - `node --test scripts/__tests__/ci-typecheck-lane.test.mjs`: 8 pass. - `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's command): 807 pass. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: 11 pass. - Not exercised by this PR's own CI: PR runs use the workflow from `master`, so the new conditions first run after merge. The script change does run here, because the script is read from the PR's checkout. - Not measured: the lane's wall time with the CLI guidance test moved. The test takes 2 to 5 seconds alone, so the expected cost is that much. ## Risks - The fallback step repeats six commands in the workflow. It is needed only until open stacked PRs are based on a branch that has the script, and can be removed later. - `hashFiles` returns an empty string when no file matches, which is what the two conditions rely on. - If a future PR deletes or renames the script, the job silently uses the slower commands instead of failing. The wiring tests would fail in that PR. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), via Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
5859e7cfa6 |
perf(apps): avoid repeated writes during permission saves (#21033)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps give agents access to tools through saved profiles and approval rules. > - A save replaces the full set of action entries and access bindings. > - It also updates each Ask first rule and each retired rule in a separate query. > - Adding one agent can therefore cause hundreds of database round trips. > - This pull request keeps unchanged rows and batches the required changes. > - The benefit is faster saves with the same action permissions. ## Linked Issues or Issue Description **What happened?** An app permission save repeats database work for actions and agents that did not change. With 100 actions and 500 retired rules, one agent addition makes 636 queries. Of these, 599 update approval policies. Related boundary coverage: Refs #19973. This change keeps that issue's Off and agent-selection protections. **Expected behavior** Adding an agent applies the saved Allowed, Ask first, and Off choices. The save changes only the rows that need a change. Independent approval rules remain in force. **Steps to reproduce** 1. Create an app with 100 active actions and 99 Ask first rules. 2. Add 500 disabled app approval rules. 3. Save access for three agents. Then add a fourth agent. 4. Count database queries during the save. **Paperclip version or commit** Baseline: `51d77648bc62`. **Deployment mode** Built from source. The reproduction uses disposable local PostgreSQL. ## What Changed - Keep action entries and access bindings when their saved values match. - Insert new bindings in one query. Remove only bindings and entries that no longer apply. - Read managed approval rules for the selected app. Skip unchanged rules and rules that are already disabled. - Batch new or changed Ask first rules and disable stale enabled rules in one query. - Keep existing policy conditions, priority, author, and creation time when a rule changes. - Add real PostgreSQL regression tests and a repeatable benchmark. Document independent approval rules. ## Verification - Red: the original regression made 2,995 policy writes across five saves and failed the zero-write check. - Green: 636 focused service and runtime tests passed after the rebase. CI supplies the broad checks required by `pnpm verify` (exit 2). - Benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-app-permission-save.ts --baseline-ref=51d77648bc62 --output=/tmp/app-permission-save.json`. - Local benchmark: two warmups and six samples in alternating order. Restore three agents outside each timed sample, then add the fourth agent. Median save time fell from 458.51 ms to 38.05 ms (12.0 times faster). Query count fell from 636 to 32. Unchanged policy writes fell from 599 to zero. - Compare all saved profile, entry, binding, and policy values after mapping fixture IDs and excluding generated row IDs and write times. Check Allowed and Ask first decisions for the added agent. Tests also cover Off, an unassigned agent, additive access, removal, independent policies, company boundaries, and re-enabled rules. - CI passes on `069e2c76252a19ec3c11a061de0506eccb0463d1`: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/43162600023). This includes typecheck, build, server tests, and all eight browser shards. - [Greptile](https://github.com/paperclipai/paperclip/pull/21033#issuecomment-6815608829) gives the final head 5/5. Its one finding is resolved. The separate Paperclip Review check is still queued. Its review task stopped after a Runner failure. - Hosted request timing and the specific reported approval prompt are not verified. No live permissions changed. ## Risks - The save now retains unchanged row IDs and timestamps. Consumers must use permission values to determine access. - Existing transaction locks and permission checks remain in place. The runtime concurrency tests exercise these paths. - The benchmark measures a local service with synthetic data. It does not establish hosted endpoint latency. - No migration or startup change is required. ## Model Used OpenAI Codex, GPT-6. Exact provider model ID is not exposed in this session. Used reasoning, tool calls, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
64ec273845 |
perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the trusted PR workflow, and the `Typecheck + Release Registry` job is one of its required checks > - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner (run 43344818387), against a target of 7 minutes or less for every job > - The job runs six steps one after another, and the first step type-checks the whole server program a second time, at about 13.5 GiB, which makes the runner swap > - Typecheck results are the check that authors most want back quickly, so this job is on the critical path of most pull requests > - This pull request removes the duplicate server type check and runs the independent checks in parallel through one small script, inside the same memory budget > - The benefit is about 39% less wall time for the job's checks on the same machine, with peak memory down from about 14 GiB (plus swap) to about 8.5 GiB ## Linked Issues or Issue Description **What existing behavior does this improve?** The `Typecheck + Release Registry` job in `.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The target for every job in the workflow is 7 minutes or less. **Subsystem affected** CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under `scripts/` and `server/scripts/`. **Current behavior** After about 75 s of setup, the job runs six steps in sequence on a 4 vCPU runner, about 470 s in all (run 43344818387): `typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s, test projects 38 s, CLI guidance test 5 s, release registry tests 95 s. Most of these steps use one thread. The `typecheck:build-gaps` step builds the server with a full type check of the whole server program. The server layer step then checks the same files again, and the `Build` job checks them a third time. That full program peaks at about 13.5 GiB, so the job peaks at about 14 GiB of 16 GiB and uses swap. **Proposed behavior** The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker image does) and still checks the runtime assets. It runs the steps that write shared files first, one at a time. Then it runs the remaining checks in parallel, one per CPU, with the server declaration layers first, because the test layers and test projects read them. The checks that are not on that path run at a lower CPU priority. **Reason and benefit** On the same runner, the job's checks take 61% of the time and about 60% of the memory (see Verification). The job should take about 6 minutes on the fleet instead of 9.2. **Breaking changes** None. The same files are type-checked with the same options. Local commands do not change. `pnpm run typecheck:build-gaps` still works as before. ## What Changed - `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It writes shared outputs first, in sequence: links, plugin SDK, server build without type checks, the runtime asset check, the sandbox provider installs, and the Runner typecheck (it rebuilds the eval kernel, which other checks read). Then it runs these checks in parallel: the server declaration layers, each server test layer, each test project, the test programs, the other build-gap workspaces, the CLI guidance test and the release registry tests. A passing check prints its output in a collapsed log group. A failing check prints its full output between `=== FAILED` and `=== end of` lines, with an error annotation. The script ends with a table of each check's start, duration and result, and exits non-zero if any check fails. - `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option that checks one layer against the configs and declarations of an earlier `--declarations-only` run, and an exported `listLayers`. - `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and `--skip-prepare` options, so the installs happen before the parallel checks start. - `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option, so the script still finds the build-gap workspaces and still fails when one has no `typecheck` script. - `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry` job runs the script instead of the six steps. No other job changes. - Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to `test:release-registry`), new cases in `server/scripts/typecheck-layers.test.mjs` and `scripts/typecheck-test-programs.test.mjs`, and the workflow wiring assertions in `.github/scripts/tests/ci-plan.test.mjs` and `scripts/__tests__/runner-release-crate-cache.test.mjs`. - `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because `package.json` changed. - `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new options. ## Verification This PR's own CI ran the old job definition, because PR runs use the workflow from master. So I measured the commands on a stand-in machine: a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`, now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon 6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs [43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017), [43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958), [43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105), [43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837) and [43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493). Each step ran under `/usr/bin/time -v`, and a sampler read `/proc/meminfo` every 0.25 s. **Old steps, one at a time (3 runs, run 43445393958):** | Step | Wall s | CPU s | Peak RSS of one process | System memory peak | | --- | --- | --- | --- | --- | | `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7 GiB | | ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB | 13.7–14.7 GiB, swap up to 738 MiB | | ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB | 3.9 GiB | | Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB | | Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB | | Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB | | CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB | | Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB | CPU seconds are about twice the wall seconds for the `tsc` steps even on one checker thread. So each step uses about two of the four vCPUs. **Old sequence and new lane on the same runner (4 runs, run 43450786493).** Each job copied one installed checkout, then ran both, in alternating order: | Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak | New memory peak | | --- | --- | --- | --- | --- | --- | | 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB | | 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB | | 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB | | 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB | Hosted runners differ in speed by up to 1.7 times, so only the same-runner ratio is meaningful. Both arms include a cold compile of the Runner release crate (45–80 s) that the fleet restores from cache. The fleet ran the old steps in 470 s; at the measured ratio, the new lane should take about 290 s, and the job about 6 minutes with setup. The new lane peaks at about 55% of memory, with no swap. Local checks (on a loaded Mac, for correctness only): - `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass. - `node --test .github/scripts/tests/*.test.mjs` — 807 pass. - `node --test scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass. - `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/` — the new cases pass. ## Risks - This changes a required job that every PR depends on, and its own CI cannot prove it. Please watch the first `Typecheck + Release Registry` runs on master after merge: the wall time (expect about 6 minutes), the summary table at the end of the step, and any `terminated by` message. - A type error now shows in that check's `=== FAILED` block, not in a step of its own. Passing checks are folded, so the failure is the only expanded block. - When the server declaration layers fail, the test layers and test projects are reported as skipped. Before, the test layers still ran after a source error. Fix the source errors first; the next run checks the tests. - The Honeycomb CI time board and any step-name queries see one step, `Typecheck, CLI guidance safety and release registry tests`, instead of six. The script prints the duration of each check. - The release registry tests run beside the type checks at normal priority. Their slowest test took 46–78 s here, against a 180 s timeout, and 114 s on the slowest stand-in runner. - Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB, far below the old 14 GiB. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use (shell, GitHub CLI, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
84a4d345b3 |
refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by the `runner-artifacts` stages of the `Dockerfile` > - The application image does not use those stages. It has five stages of its own that compile the daemon, it builds the Runner again inside the server build, and it builds a second provider pack > - So one `Dockerfile` holds two recipes for the Runner, and the two results differ: another base image and another daemon binary > - This pull request removes the second recipe. The application image copies the Runner out of the `runner-artifacts` stage, which the same build runs once > - The benefit is one Runner recipe, and an application image whose Runner is the Runner the contract describes. A later pull request can replace the stage with the published image ## Linked Issues or Issue Description This is the second part of a chain. The first part is #21331, which removes the cache check that cannot run on this `Dockerfile`. Refs #21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode of the vendor step), #21307 (the pull-request check of the application image) and #21312. No issue exists for this change. **Problem or motivation** The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps` and `runner-build` compile the daemon on Debian trixie. The server build then runs the full build of the Runner package. The `runner-provider-pack` stage builds the provider pack. The `runner-artifacts` stages do all of that again for the Runner image, on Debian bookworm, with a stripped daemon. **Proposed solution** Read only the `runner-artifacts` stage from the application stages. Build it inline in every build. Do not pass a published image yet. **Alternatives considered** - Pass the published Runner image to the build in this pull request. That needs the resolver, labels and a rule for commits without an image. It is the next step, and this one must be safe alone. - Keep the old stages as a fallback. Two recipes is the problem. ## What Changed - `Dockerfile`: remove the five stages. `build` starts from `base`. It copies `/runner-package/dist` and the build stamp from `runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and runs the vendor check before the first application build. - `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN` with a bind mount of `runner-artifacts` copies the pack, the receipt and the archives, and moves the large files to their own tree. `build` and `runner-provider-pack` now come after the stages they read. - `Dockerfile`: the three `COPY` instructions of `production` that take the pack are byte-identical. A test pins all three. - `Dockerfile`: `build` declares the build argument `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step runs the new `scripts/check-runner-image-expectation.mjs` on `/runner-image.json` of `runner-artifacts`. **This pull request declares the argument. No build sets it. The next pull request, which gives a published Runner image to the build, is its first user.** With the argument set, the build fails unless `runner-artifacts` is the published image of exactly those inputs. `docker-app-check.yml` also runs when the script changes. - Comments: the `Dockerfile` has 220 comment lines, down from 247. The comments of the two moved stages moved with them, unchanged, so the diff shows them as added lines. Eight comments that named a removed stage or a step that moved get the smallest correction. Two comments in `prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the `Dockerfile` relies on the standalone rebuild; that clause is removed. - Tests: replace the two cargo-chef tests of `docker-build-stamp.test.ts` with three tests of the new shape. Update the pack test, `docker-image-layer-reuse.test.ts`, `vite-build-commit.test.ts`, the stage rules of `runner-image-identity.test.ts`, and the include-layout test of `runner-release-crate-cache.test.mjs`, which now reads `runner-artifacts-runnerd`. - Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The application image" in `doc/runner-image-contract.md`. ## Verification All builds: `linux/arm64`, local, `--build-arg PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The builds ran before the rebase onto `master`, on the head of #21310 as it was before it merged. `master` changed neither the `Dockerfile` nor an input of the Runner in between. - `docker buildx build --target production`, cold, before and after: 506 s and 410 s. Compressed size: 2,971 MB and 2,900 MB. - The image config and the text of all 33 history rows are identical. The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ # buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly once. - A comparison of every file of the two images (path, type, size, mode, link target, and SHA-256 under `/app` and `/opt`). The differences are in "Risks". All other files are byte-identical, the root `node_modules` included. - From the image after: the four `/app/packages/paperclip-runner/...` paths and the contents of `/opt/paperclip-runner/` exist. `paperclip-runnerd --build-metadata` runs. `provider-pack.json` has `runnerSourceRevision` equal to the build commit and no `runnerInputsDigest`. The entry points `index.js`, `live/index.js` and `testing.js` of the Runner import. - The vendor step logged `reusing the current Runner build` in both of its runs. Every `Compiling` line of the log is in `runner-artifacts-runnerd`. - `--target build`, `--target runner-provider-pack` and `--target cloud` built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or `pkg-config`. - `--target runner-artifacts` and `runner-artifacts-slim`, before and after: `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon, `provider-pack.json` and the two stable layers have the same digests before and after. - Without a build commit: `build` and `runner-provider-pack` built, and the pack was skipped with the same message. This run was on the next branch of this chain, after the rebase. The last steps of `production` were not run for this case. - The expectation argument, on this `Dockerfile`, `--target build --output type=cacheonly`, after the rebase: - No argument, inline Runner: the build passes (233 s). The step passes. - Argument, inline Runner: the build fails at the step. The message says that the `runner-artifacts` of the build are not a published Runner image. - Argument `d4df566f…` and the published image of those inputs as `--build-context runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`: the step passes, and the build runs no Runner stage. The vendor step then fails with `inputs changed since the recorded build`, because this tree differs from the tree of that image by one comment line in `build-inputs.mjs`. With that line put back, the same build completes, and the vendor step reuses the published build. - Another digest with the same context: the build fails at the step. The message names the inputs the image has. - A tree whose lockfile is behind the manifests of the Runner fails at the frozen install of the Runner stages, after 169 s, with the message that stage already had. - `docker buildx build --check`: no warning, before and after. `scripts/check-docker-deps-stage.mjs` and `scripts/check-node-version-policy.mjs` pass. - Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the changed server test files with the other tests that read the `Dockerfile` or the image workflows (231 pass in 10 files on Linux), `vite-build-commit.test.ts` (10 pass), `runner-release-crate-cache.test.mjs` (11 pass), `docker-deps-postinstall.test.mjs`. The changed server test files typecheck. - Not run: the full test suite, any `linux/amd64` build, `podman`, the workflows and the release pipeline. Only CI can prove amd64, `docker.yml`, `docker-merge-group.yml`, `release.yml`, `docker-app-check.yml` and `runner-image.yml`. ## Risks - **The digest of the Runner image changes.** Its identity covers the whole `Dockerfile`. The next `master` push publishes a new `inputs-<digest>`. The contents do not change: see the comparison above. - **A build that is given a Runner of other inputs fails; it does not ship that Runner.** With the argument, it fails at the expectation step. Without the argument, it fails at the vendor step when the build inputs of the Runner differ. One comment line in `build-inputs.mjs` changes those build inputs in this pull request, so the image published before this pull request cannot be vendored into a tree after it. - **What a consumer of the application image sees** (measured on `linux/arm64`): - The daemon is the stripped binary compiled on Debian bookworm: 23.2 MB, was 36.4 MB, at `/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the vendored copy in `server/dist`. Its hash changes. It has no symbols for a crash backtrace. - `/app/packages/paperclip-eval-kernel/dist` is absent. Only `dist/eval/workflow-harness.js` of the Runner imports that package. Only `eval/index.js` exports the harness again, and no module imports that file. The server runtime bundle does not name the package. - The provider pack: `provider-pack.json` is byte-identical. Two provider executables and one licence file are hard links to their other copy in the pack, where they were separate files. One empty directory is absent. The pack is 195 MB smaller, and the layer of its large files gets a new digest once. - The transport archives have other bytes, because the Runner build gives large files a fixed timestamp. The receipt names the same files with the same digests. - Paths, modes, owners, the image config and the layer order do not change. - **What a contributor and a maintainer of the image build see:** - **The image build does not run the drift checks of the Runner.** Before, the server build ran the full `build` script of the Runner, so a failed drift check failed the image build on `master`. Now the Runner stages run only the output steps of that script. Repository CI runs the checks on every pull request. `.github/docker-context-checks.Dockerfile` and three exceptions in `.dockerignore` exist for those checks in the image build; a separate follow-up removes them, and this chain does not edit them. - **The dependency-caching layer of the daemon is gone.** Before, cargo-chef compiled the Rust dependencies in a layer of their own, and a change to the Rust source compiled only the Runner's own crates. Now such a change compiles the dependencies again (146 s here for the whole daemon). A change to the server, the UI or the Runner's TypeScript still reuses the compiled daemon. - **A lockfile behind the manifests of the Runner fails the build.** Before, `docker build` resolved it inside the image. The image workflows refresh the lockfile first. A local build prints the command to run. The next pull request makes this failure early and clear. - **The build needs more disk**: about 25 GB of builder storage here, was about 20 GB, with the image export. - Rollback: revert this pull request. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
594f9d3ec2 |
perf(runner): keep at most one checkpoint waiting behind a slow write (#21411)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee
## Linked Issues or Issue Description
**What happened?**
A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.
**Expected behavior**
Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.
**Steps to reproduce**
1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.
`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.
**Paperclip version or commit**
`master` at
|
||
|
|
bef2aac905 |
chore(ci): remove the Docker dependency cache check of the Runner stages (#21331)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. The application image contains it > - Today the application image compiles the Runner in its own stages, with a cargo-chef dependency layer > - A pull-request job proves that a fresh builder reuses that layer. The job runs a script that appends to the `Dockerfile` and names those stages > - The next pull request removes those stages. The application image then takes its Runner from the `runner-artifacts` stages. The job cannot run against that `Dockerfile` > - This pull request removes the job and its script > - The benefit is that the next pull request changes only the image build, and its checks stay green ## Linked Issues or Issue Description Refs #21204, #21229, #21230, #21306 and #21307 (the Runner image and the preparation for this change). No issue exists for this change. **Problem or motivation** `.github/workflows/docker-runner-check.yml` has a pull-request job, `Compile isolated native Runner`. It runs `scripts/check-docker-runner-cache.sh`. The script adds stages to a copy of the `Dockerfile` that start from `runner-build` and copy from `runner-plan`. It reads the build log for the `runner-deps` step. The job runs on every pull request that changes the `Dockerfile`. The pull request that removes those stages would fail this job. **Proposed solution** Remove the job, its trigger and its script first, in a pull request of its own. **Alternatives considered** - Remove the job in the same pull request as the stages. That pull request is already large. - Rewrite the script for the `runner-artifacts` stages. Those stages have no dependency layer, so the property does not exist. `runner-image.yml` already builds them on a pull request. This is the first of three pull requests that move the application image onto the `runner-artifacts` stages. It has to merge first: the removed job runs on every `Dockerfile` change and builds stages that the second pull request deletes. The other two follow on top of it. ## What Changed - Delete `scripts/check-docker-runner-cache.sh`. - Remove the `runner` job and the `pull_request` trigger from `docker-runner-check.yml`. Only the manual jobs remain. The concurrency group no longer reads a pull request number. - Remove `Docker Runner check` from the workflow list of `pr-merge-review.yml`. That notifier acts only on pull-request runs, and the workflow has none now. - Remove the paragraph of `doc/DOCKER.md` that describes the check. ## Verification - `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs`: 22 pass. - `vitest run server/src/__tests__/runner-image-workflow.test.ts`: pass (1 test is skipped on macOS, where bash is 3.2). - `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/workflow-target.test.ts`: 28 pass on macOS. These files read `docker-runner-check.yml`. - The same commands in a `node:24.22.3-bookworm` container: all pass, except one test of `workflow-target.test.ts`. It fails with `jq: command not found`. It reads another workflow, and the container has no `jq`. - Not run: the full test suite, and the workflows themselves. Only CI can run the workflows. ## Risks - Until the next pull request merges, a pull request that changes only the native inputs of the Runner (`packages/paperclip-runner/runner/**`, `protocol/**`, `rust-toolchain.toml`) does not build the `runner-build` stage of the application image. `runner-image.yml` builds the daemon from the same files in the `runner-artifacts` stages on such a pull request. `docker-app-check.yml` builds `runner-build` when the `Dockerfile` or `.dockerignore` changes. - No check proves the cargo-chef layer reuse after this change. The next pull request removes that layer. - The job was not a required check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
2659ff1d22 |
fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every push to `master` publishes a canary, and the canary's image reaches staging > - Each canary waits for `Cloud source verified v1` from the Cloud readiness run of its exact SHA > - Cloud readiness runs on AWS Spot workers. When a worker is reclaimed, the gate failed and the image watcher cancelled the readiness run > - Nothing retried that attempt. On a busy master the newest commit's release failed, and merged work did not reach staging until a person re-ran the run > - This pull request re-runs the failed jobs of a readiness attempt whose every failure is a lost runner. A real check failure still fails closed > - The benefit is that a Spot reclaim no longer holds a release ## Linked Issues or Issue Description **What happened?** On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained it published for more than an hour. The release workflow keeps one running and one pending canary. A newer push replaces the pending one, so the newest commit always runs next. That part works as designed. The failures came from the readiness gate: | Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause | | --- | --- | --- | --- | | `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` | Spot workers reclaimed at 13:49:56Z. The run was then cancelled by `github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by hand | | `8af889f612` | 43434587048 / 2 | Release superseded while pending; readiness failed in the same reclaim | Spot reclaim | | `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` | Spot reclaim | | `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` | Flaky test `decision-models.test.ts` (concurrent budget admission) | | `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` | Flaky test `fresh-continuation-fallback.test.ts` (truncate) | | `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` | Both flaky tests | Over the last 200 master pushes, release runs ended 14 success, 33 failure, and 150 cancelled (superseded while pending, which is expected). **Expected behavior** A lost runner is not a source failure. The gate retries the lost jobs and publishes the newest commit that passes. **Steps to reproduce** See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master simulation". It replays the 13:49:56Z reclaim through both waiters. ## What Changed - `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed readiness attempt with the existing PR classifier (`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when every causing job (`failure`, `timed_out`, `startup_failure`) lost its runner. Cancelled siblings are fallout. A run that is still going is cancelled first, then `rerun-failed-jobs` runs on the next poll. The ceiling is readiness attempt 4. Writes go only to `cancel` or `rerun-failed-jobs` of a canonical run, after an identity check. - `scripts/cloud-source-verification.mjs` (canary gate in `release.yml`): on `SourceVerificationFailed`, ask the retry first. The source wait goes from 60 to 90 minutes. - `scripts/watch-release-source.mjs` (master image watcher in `docker.yml`): on a runner loss, keep the image attempt alive and wait for the retried proof. Do not cancel it. - `.github/workflows/release.yml`: `verify_canary` timeout goes from 70 to 100 minutes. - `.github/workflows/pr-trusted.yml`: run the new test file. - `doc/RELEASING.md`: new "Busy master and lost runners" section. ## Verification - `node --test scripts/cloud-readiness-runner-loss.test.mjs scripts/cloud-source-verification.test.mjs .github/scripts/tests/*.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: all pass. `release-verify-workflow.test.mjs` passes 51 of 51 but does not exit locally. This also happens on `origin/master` without this change. - Red/green: with the two waiter files reverted, 3 of the 6 simulation tests fail (the canary gate, two reclaims in a row, and the image watcher). With the change, all 20 pass. - Dry run against real runs (read-only, writes captured): 43435593630/2, 43434587048/2, and 43423674511/2 give `rerun-failed-jobs`. 43439942063/1, 43421480302/1, and 43422042261/2 (real test failures) give `failed` with no write. - `pnpm verify` exits 2 because a workflow file changed. Required CI supplies broad coverage. ## Risks - A real failure classified as a runner loss would be retried. The classifier needs the runner's own shutdown or lost-communication `##[error]` line in the job log. It is the same classifier PR CI uses, and unreadable evidence fails closed. - On a reclaim, the waiter cancels the readiness run to retry sooner. Jobs that were still healthy run again. This costs runner time, not correctness. - Flaky tests still fail a canary. This change does not retry them. They need their own fixes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f79e2c5d84 |
feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - `master` now publishes the Runner alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner inputs (#21204, #21229, #21230) > - The application image still compiles its own Runner on every build. The next steps make it copy the published Runner instead > - Before a build can do that, it must answer one question the same way as every other build: "does a published Runner exist for exactly this tree?" It must also fail clearly when the Runner it was given does not match the tree > - This pull request adds the script that answers the question, and a mode of the Runner vendor step that fails instead of rebuilding > - The benefit is that the later change to the image build is small, and that a wrong or unreachable answer can only mean "build the Runner yourself" ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** Three builders make images that contain the Runner: `docker.yml`, the merge-queue build and the release preview build. Each one would have to repeat three steps to find the published Runner of its tree: check the lockfile, compute the inputs digest in the build, and look up `inputs-<digest>`. Three copies can decide differently. A build that copies a Runner built elsewhere also has no Rust toolchain. If the copied Runner does not match the tree, `prepare-runner-vendor.mjs` starts a rebuild, and the build fails late at `cargo` with a message that hides the cause. **Proposed solution** Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to `prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets the variable in this pull request, so no workflow job changes. **Alternatives considered** - One command that builds and looks up. Two commands keep a registry credential, when a caller has one, out of the step that builds. - A wait for the publication of the commit. The decision whether the application build waits is open, so the resolver has no wait mode. It reports `absent` and stops. - A build argument that names the expected inputs. No `Dockerfile` declares one yet. The change that makes the application image copy the Runner will add both ends together. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 1 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode (this pull request) 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands that share a directory: - `inputs --dir <dir>` exports the `runner-artifacts-context` target, runs the lockfile check on that export, and builds the `runner-artifacts-identity` target when the lockfile is current. It reads no credential. A lockfile that is behind stops there: no image is published for such a tree. - `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`, `reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and `labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to `$GITHUB_STEP_SUMMARY`. - `mode` is `published` only for a complete publication whose full index names these inputs and both platforms. Both images are pinned by index digest. The slim digest is the one the full index names in `io.paperclip.runner.slim-digest`; the slim tag is never read on its own. - Every other answer is `inline`, with the last four values empty: | `reason` | Meaning | | --- | --- | | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests | | `absent` | no tag exists | | `incomplete`, `broken` | a publication stopped, or its two tags do not belong together | | `mismatch` | the full index names other inputs, or lacks a platform | | `unknown` | the registry refused, failed or could not be reached, or the lookup threw | - The resolver reuses `runnerLockState`, `readRunnerImageManifest`, `runnerPublicationState` and `runnerImageTags` from `scripts/runner-image-publish.mjs`. That file now exports `environmentCredentials`; it has no other change. - `server/scripts/prepare-runner-vendor.mjs`: with `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current build as before. When the build is not current, it prints the reason, starts no rebuild, leaves the stamp in place and exits with status 1. It also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same time. Without the variable the step is the same as before. - `doc/runner-image-contract.md` has a new section, "Choosing the Runner for an image build". `doc/DEVELOPING.md` describes the new variable. - The resolver is not an input of the Runner image, so the inputs digest does not change. ## Verification macOS (Node 26.12.0), in `server/`: - `node node_modules/vitest/vitest.mjs run scripts/prepare-runner-vendor.test.mjs src/__tests__/runner-image-resolve.test.ts src/__tests__/runner-image-publish.test.ts src/__tests__/runner-image-identity.test.ts src/__tests__/runner-image-workflow.test.ts src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6 files. 27 of them are new resolver tests and 3 are new tests of the vendor step, whose end-to-end test also has a new case. - `tsc` on the new test file, with the settings of `server/src/__tests__/tsconfig.json`: no error. Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted read-only): the same files, 179 tests pass. Real runs, on clean exports of the tree (`git archive`), with no credential, on a `linux/arm64` Docker host: - On `master` at `62d673cc3f`: `inputs` gave `lock=current` and `inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`. `resolve` gave `mode=published`, `image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162` and `slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`. These are the two indexes that `master` published for those inputs. - The same tree with one line added to a Runner source file: another digest, `mode=inline`, `reason=absent`. - On `master` at `bc71c23e0a`, which changed the Runner's inputs, before its image was published: digest `88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`, `mode=inline`, `reason=absent`. - Four deliberate faults in the resolver each make a test fail: no index check, only `absent` is inline, the identity is built for a lockfile that is behind, and the slim image is taken by tag. Never run: - A lookup with a registry credential. The package is public, so no caller needs one now. - Any workflow. No workflow calls the resolver in this pull request. - `linux/amd64`. - The full test suite, the full typecheck and `pnpm build`. `pnpm verify --explain` reports exit 2, "broader verification required", so CI must supply them. ## Risks - Low for existing behaviour: nothing calls the resolver, and the vendor step is the same unless the new variable is set. - The resolver does not verify the attestation of the image. The workflow that consumes the image must do that from the digest the resolver reports. - The resolver reports both build contexts, `runner-artifacts` and `runner-artifacts-slim`. A build that reads only one stage gets one context it does not use. - A `broken` publication is `inline`, not a failure. The Runner image workflow fails for that state. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
db3d7223e0 |
fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.
## Linked Issues or Issue Description
**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.
**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:
- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.
**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.
**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.
## What Changed
- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".
No assertion is weakened. No pin is changed. No file under `.github/` is
changed.
## Verification
- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.
## Risks
- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
25e9a9c2e9 |
feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Parts 2 and 3 of this stack build the Runner alone as an image with two variants, and check it on pull requests. Nothing publishes it > - Downstream sandbox and VM images can copy the Runner only from a registry, and they must be able to verify where it came from > - A published image must be exactly the bytes that were checked, and its name must never point at something else later > - This pull request publishes both variants from `master` as `ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set of inputs, and attests them > - Nothing consumes the image in this pull request. The application image build does not change > - The benefit is one verified, attested Runner that later changes can copy by digest ## Linked Issues or Issue Description Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image build). No issue exists for this change. **Problem or motivation** The Runner image exists only inside a pull-request check. A consumer cannot pull it. There is no rule for which commit publishes an image, no protection for its tags, and no way to verify a copied Runner. **Proposed solution** Publish from a `master` push in this repository, and from nothing else. Build only when no image has the digest of the commit's inputs. Let the build legs hold no write access. Push the bytes that the legs checked by digest, attest each multi-platform index, and write the tags last. Write the full tag after the slim tag, so that the full tag means "both exist". **Alternatives considered** - Publish from `docker.yml`. That workflow holds one concurrency slot for each ref, and the Runner image needs its own signer identity, so that its attestation cannot stand in for the application image's. - Resolve the lockfile in the workflow before a `master` build, as `docker.yml` does. Then the published Runner depends on the state of the npm registry at run time, and the digest cannot be computed again from a commit. - Hand each variant to the publish job as its own OCI archive. The two archives hold the two shared layers two times: 1.27 GB for each architecture in place of 0.85 GB. - Write the full tag first. Then a consumer can see a full image whose slim image does not exist yet. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 4 of 4. Each part is on top of the one before it: 1. #21228 (merged): an optional field in the provider pack manifest 2. #21204: build and check the Runner image, with no publication 3. #21229: layers by change rate, and the slim variant 4. #21230: publish both variants from `master` (this pull request) ## What Changed - `.github/workflows/runner-image.yml`: adds the trigger `push` to `master` and the job `publish`. - `identity` checks the committed lockfile against the Runner's manifests, computes the digest in the build context, and asks the registry whether these inputs are published. - `build` is the job of part 3. It also records what it built, on every event. On a canonical `master` push it uploads that. It cannot write to the registry on any event. The only token in the job can read packages, and only the lookup step receives it. - `publish` runs only for a `master` push in this repository. It builds nothing and runs no dependency code. - Lockfile rule. A `master` push builds from the committed `pnpm-lock.yaml` and resolves nothing, so the digest is a function of the commit. When the committed lockfile is behind the manifests that the Runner build reads, the run writes a notice, publishes nothing, and stays green. The commit that merges the lockfile refresh publishes the image. Pull requests and manual runs resolve the lockfile first, always build, and never publish. - Hand-off. Each leg joins its two OCI archives into one OCI layout that names the images `slim` and `full`. The layout stores the two shared layers one time. `record` then reads the layout back, each image by its name, and checks every blob against its digest. The leg's record binds the layout to both image digests, the architecture, the inputs digest, and the commit. Those digests are the digests of the archives that part 3 unpacks and checks, so the tree that is checked and the image that is pushed are the same bytes. The publish job accepts only records for its own inputs and commit, and only a layout that still has the recorded bytes. - `record` runs on pull requests too. Only the upload is for a `master` push. A pull request therefore runs the hand-off on real build output. - Publication order. The publish job pushes the four images by digest (`skopeo copy --preserve-digests`). It makes a candidate index for the slim variant, then one for the full variant that names the slim index in the annotation `io.paperclip.runner.slim-digest`. It verifies that each index names exactly the recorded images, attests both, and then writes `inputs-<digest>-slim` and, last, `inputs-<digest>`. - The full tag is the completion marker. Every "is it published" lookup reads the full tag. `publication-state` reports one of: | State | Registry | What a run does | | --- | --- | --- | | `absent` | Neither tag | Builds and publishes | | `incomplete` | The slim tag alone | Builds and publishes. It replaces the slim tag with its own image, then writes the full tag | | `published` | The full tag, and the slim tag at the digest that the full index names | Nothing | | broken | The full tag, and a slim tag that is missing or points at another image | Fails, with the steps for an operator | | `unknown` | No clear answer | Builds. The publish job fails before it writes a final tag | - Recovery rule. A run that stops between the two tags leaves the state `incomplete`. It does not block those inputs. The next run never keeps the earlier slim image: it writes its own slim tag immediately before its full tag, verifies both against its own index digests, and then reads the pair back as every later run will see it. The broken state cannot be repaired by a run, because no run moves a full tag. An operator must delete the two package versions and re-run the workflow run of a `master` push with those inputs. - Pairing rule for consumers, in the contract: resolve the full tag first, take the slim digest from the annotation of the full index, and pull the slim image by that digest. - `verify-index` accepts `linux/arm64` with the variant `v8`, which some BuildKit versions record, and no other variant. - `scripts/runner-image-publish.mjs` (533 lines): the commands `lock-state`, `record`, `verify-legs`, `publication-state` and `verify-index`. It reads archives with the reader of `scripts/runner-image-oci.mjs` (607 lines here), which can now read one named image of a layout. `lock-state` refuses a checkout, because pnpm can rewrite the lockfile of the tree that it checks. The workflow gives it an export that no build reads. - The workflow test pins the build arguments of the image build. Build arguments are not part of the image identity. - `doc/runner-image-contract.md` (197 lines): the section "Publication": the lockfile rule, the order, the pairing rule, the recovery rule, the create-only tags, the internal `candidate-` tags, the command that verifies provenance, and the step to make the package public. `doc/DOCKER.md` says where the image is published. ## Verification Tests, and the build jobs of the workflow on GitHub. Nothing in this pull request ran against a registry, and no publish job has run. - Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and zstd): `runner-image-publish.test.ts` (75), `runner-image-workflow.test.ts` (12), the three other Runner image test files, the provider pack archive tests and the existing Dockerfile tests: 372 tests pass in 10 files. The same files pass on macOS. - The tests of the leg record failed on Linux in the first runs of this pull request: their fixtures named the archive members `./index.json`, which bsdtar finds and GNU tar does not. The fixtures now name the members as buildx does, and the publish script uses the one reader of part 3. - Leg records: both variants are accepted for this run's inputs and commit; other inputs, another commit, a changed layout, a record without a variant, an archive that lacks a blob, a full image that does not start with the slim image's layers, and a build that is not a publication build are refused. - Publication state: all five states, and the half-finished publication end to end. A slim tag alone lets the next run publish. A full image that is left with another run's slim tag is reported as broken. - Index: exactly the recorded images of one variant; the full index must name its slim index, and the slim index must name none; `v8` is accepted on arm64 only. - Registry lookups: a missing tag only when the registry says so; every refused, failed, or unreachable request is `unknown`. - Workflow: job permissions; the publish job and the upload are behind the canonical `master` push; the build steps receive no credential; the full tag is written last, after a strict lookup and both attestations; one export and a time limit for each build command; the two build arguments; actions pinned by commit; no expression inside a shell script. - `record` on the two OCI archives of a real `linux/arm64` build from part 3: the joined layout is 850,149,376 bytes, and both images read back by name with every blob under its digest. The two archives together are 1,273,646,592 bytes. The image digests in the record are the digests that `unpack` reports for the trees that were checked. - `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts: 33 pass. `docker buildx build --check` reports nothing. The image identity is the same as in part 3 (`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`), because this pull request changes no input of the image. GitHub, at this head: - The `Runner image` workflow passes with these changes: `Build and check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each leg builds both variants, checks them, and writes the leg record. `Publish and attest` is skipped, as it must be on a pull request. Never run: - The publish job. It runs only for a `master` push. - The publish path: the registry lookups with a workflow token, `skopeo copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx imagetools create` with index annotations, and the tag writes. - The attestations, and the `gh attestation verify` command in the document. - `actionlint`. It is not installed on the local host. - The full test suite and the full typecheck. CI must supply them. ## Risks - **Manual step.** GitHub creates the `paperclip-runner` package private on the first publication. A maintainer must make it public one time. Until then, consumers cannot pull the image without credentials. - The first `master` runs of the workflow are the first test of the publish path. A failure before the tags publishes nothing. A failure between the two tags leaves the `incomplete` state, and the next run with those inputs repairs it. - A slim tag can move while its publication is incomplete. A consumer must resolve the full tag first and take the slim digest from the annotation of the full index. It must never resolve `inputs-<digest>-slim` by tag on its own. The contract says so. - `candidate-` tags are written before the attestations and can name an index that is not attested. They are internal. - The broken state needs an operator, and it fails every `master` run with those inputs until then. It can arise only when someone changes or deletes a tag outside the workflow. - **Known residual.** Two `master` runs that both start before a new digest is published can each build the image. Each build leg and the publish job look again before they work, and the publish job runs one at a time for each digest. The cost is runner time, not correctness. - A commit whose lockfile is behind the Runner's manifests has no published Runner image until the lockfile refresh merges. A lockfile that the refresh cannot fix leaves every `master` run green with only a notice, for as long as it lasts. - The tags are create-only because the workflow never moves a tag of a complete publication. The registry does not enforce that. Consumers must pin the digest and verify the attestation. - `payload.runnerSourceRevision` in a pack from this image is the commit that published the image. Code that reads it as the application's commit must change before it consumes the image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
9b613562b7 |
feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Part 2 of this stack builds the Runner alone as the image `runner-artifacts`, in one layer of about 850 MB > - Most of those bytes are provider binaries and Node. They change only with a provider version or the Node pin, but one layer means that every Runner source change makes all 850 MB new > - About half of the image is the pack's transport archives. A consumer that copies the pack as files does not need them > - This pull request splits the image into layers by how often their content changes, and adds a slim variant without the transport archives > - The benefit is that a new image shares about 93% of its bytes with the one before it, and a consumer that needs no transport archives pulls half as much ## Linked Issues or Issue Description Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No issue exists for this change. **Problem or motivation** The image from part 2 is one layer. A registry and a builder store and move layers, not files. A change to one Runner source file therefore makes a new 850 MB layer, although 790 MB of its content is the same as before. The image also holds the provider pack two times: as files, and as transport archives of those files. Some consumers need only the files. **Proposed solution** Write the image as four layers: the pack's large dependency files, everything else, the transport archives of those large files, and the other transport content. Make the two large layers byte-identical from build to build. Add the target `runner-artifacts-slim`, which is the first two layers. The full image is the slim image plus the two transport layers, as the same layer blobs. Check all of this on the built images. **Alternatives considered** - Two unrelated images. Then the registry stores the shared content two times unless the layers happen to match, and nothing ties the two together. - Split by path only, with all files of 1 MiB or more in the first layer. Six of those files are the Runner's own bundles under `dist`. They change with the Runner's source, so the first layer would change with it too. - Leave the transport archives as they were written. Their bytes then depended on timing (see "What Changed"), and the archive layer never repeated. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 3 of 4. Each part is on top of the one before it: 1. #21228 (merged): an optional field in the provider pack manifest 2. #21204: build and check the Runner image, with no publication 3. #21229: layers by change rate, and the slim variant (this pull request) 4. #21230: publish both variants from `master` ## What Changed - `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY --link` layers. `runner-artifacts` starts from it and adds two more. The build stage sorts its output into the four layers. Every path is where it was in part 2: consumers copy by path and see no difference. | Layer | Content | Changes with | | --- | --- | --- | | 1 | The pack's dependency files of 1 MiB or more: provider binaries and Node | A provider version or the Node pin | | 2 | Everything else, with `runner-image.json` | Every Runner source change | | 3 (full only) | The transport archive of each file in layer 1 | The same as layer 1 | | 4 (full only) | The other transport archives, the overlay archive and the receipt | The same as layer 2 | - Stable bytes. The files of layers 1 and 3 get timestamp 0, and the large dependency files get it before their transport archives are written. A file under the pack's `dist` is the Runner's own output, so it and its archive are in layers 2 and 4. - `scripts/create-provider-pack-archive.mjs`, two separate commits. **This script also writes the application image's archives.** - The same file now gives the same archive every time. The script piped tar into Node's zstd compressor. When the pipe ended while the compressor still had its last chunk queued, Node finished the zstd stream two times, and the archive got an empty frame of nine bytes at its end. That depended on timing. In the pinned Linux base, 37 of 60 runs of the old code had the frame. The compressor is now ended only after it has taken every chunk. - Archive N holds file N of the receipt, and the receipt listed the files in path order. `dist` sorts before `node_modules`, so one more large file in the Runner's own output gave every dependency file another archive name, and layer 3 became a new blob with the same contents. The receipt now lists the files outside `dist` first. The readers (`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair file N with `N.tar.zst` and do not use the path order, so they need no change. For the application image, the archive numbers change one time, and the bytes of each archive do not change. - `runner-image.json` is the same file in both variants. Its new field `fullImageOnly` names the two paths that only the full image has. `paths` did not change. - `scripts/runner-image-layout.mjs check` takes `--variant slim|full`. It is required and has no default. A slim tree that holds a transport path fails. A full tree without them fails. - `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of the image, because the image build does not run it. - `unpack <oci archive> --out <tree>` writes the tree of an image from the layers of its archive, in order. It checks every blob against its digest. It refuses a whiteout, an absolute path, a path with `..`, a path that is written two times, an entry behind a link, and every entry that is not a file, a directory or a link. - `check-layers --slim <oci archive> --full <oci archive>` checks the table above on the built images. It reads the receipt to decide which archive belongs in which layer. It fails when the full image does not start with the slim image's layer blobs in the same order, when a path that is not a directory is in two layers, and when a shared layer holds transport content under any name or directory. - The script reads the archive and its layers itself. It does not use the system's `tar`, so every platform reads them the same way. It finds a member only under the name that buildx gives it. - `.github/workflows/runner-image.yml`: each leg builds each variant one time, as an OCI archive and nothing else. It unpacks both archives, runs `check` on each tree with its variant, runs `check-layers`, and runs the smoke matrix on the slim tree. The tree that is checked is therefore the bytes of the image, not a second export of the same build. Every step that builds has a time limit. The workflow still has no registry access. - Why one export: in the first runs of this pull request, the slim build stalled on all four legs. Each build command had two exports, a tree and an archive. The slim command, with every stage from the cache, wrote its archive and then made no progress on the tree for 50 minutes. A test now holds the rule of one export and a time limit for each build command. - The smoke matrix runs on the slim tree. The daemon is in a layer that both variants share, so one run covers both. - `server/src/__tests__/runner-image-identity.test.ts`: the test that holds the rules for the Runner stages now finds a stage wherever BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with another flag, and a `COPY` that continues on the next line. Nine samples show each case. - Two comments are corrected. The comment on `PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that no check reads the field. The `Dockerfile` comment on the publication argument named a workflow "that builds the image for publication". - `doc/runner-image-contract.md` (159 lines): the two variants, the section "Layers", how the checks get their tree, and the new checks. `doc/DOCKER.md` names the slim target. Script sizes: `runner-image-layout.mjs` 969 lines, `runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166, `create-provider-pack-archive.mjs` 114. ## Verification Image identity at this commit: `a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`. Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and zstd, the worktree mounted read-only): - `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts` (100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts` (8), `provider-pack-archive.test.ts` (47, none skipped), `retained-provider-pack-source.test.ts` (19), `remote-provider-pack-verification.test.ts`, `docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293 tests pass in 9 files. - The same files pass on macOS. The tests of `check-layers` failed on Linux in the first runs of this pull request: their fixtures named the archive members `./index.json`, which bsdtar finds and GNU tar does not. The fixtures now name the members as buildx does. `linux/arm64`, on a local Docker host, with the new export path: - Each variant builds as an OCI archive only: the full target in 156 seconds without a cache, the slim target in 5 seconds from the cache. - `unpack` writes the full image (10,519 entries that are not directories) in 4 seconds, and the slim image in 4 seconds. - `check <tree> --variant <variant> --expect-inputs-digest <digest> --expect-source-revision <sha> --expect-architecture arm64` passes on each unpacked tree. `runner-image.json` is the same file in both. - `check-layers --slim <slim> --full <full>` passes. - The smoke matrix passes on all five base images, from the unpacked slim tree. | Layer | Digest | Bytes | In | | --- | --- | --- | --- | | 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full | | 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full | | 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full | | 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full | - The slim image is 423,516,545 bytes in two layers. The full image is 850,111,053 bytes in four. Stability, with three builds that share no cache for the build step: | Build | Source revision | Runner build output | Layer 1 | Layer 2 | Layer 3 | Layer 4 | | --- | --- | --- | --- | --- | --- | --- | | A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` | `3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` | | B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` | `4e6c492d…` | | D | `dddd…` | one new source file that adds two large files under `dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` | - Layers 1 and 3 are byte-identical in all three: 790,547,848 of 850,111,053 bytes, 93%. Build D has two more large files under `dist`, eight in place of six, and another image identity. Its dependency files keep the archive numbers 0 to 20. With the earlier numbering, those numbers moved by two, and layer 3 was a new blob with the same contents. - Layer 1 has the same digest as before this round. Layer 3 has a new digest one time, because the archives have new numbers. Other checks: - `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts: 33 pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and `docker buildx build --check` report nothing. GitHub, at this head: - The `Runner image` workflow passes on both architectures with the new export path: `Build and check (amd64)` in 14m41s and `Build and check (arm64)` in 8m41s. Each leg builds both variants, checks both layouts and the layer rule, and runs the smoke matrix. The first runs of this pull request stalled in the slim export before any check. Never run: - `native-session-executor.test.ts`. It does not load on this host, with or without this change, because `@paperclipai/plugin-sdk` is not built here. It imports the reader of the retained pack and no archive number. - The full test suite and the full typecheck. CI must supply them. ## Risks - The two changes to `create-provider-pack-archive.mjs` change the application image's transport archives: about half of them lose nine bytes, and all of them get new numbers one time. An archive's content does not change. A reader pairs file N of a receipt with archive N of the same build. - The large dependency files in the pack, and in their transport archives, now have the modification time 0 (1970). Before, they had the build time. Nothing in this repository reads those times. - `unpack` is a reader of tar and gzip that this repository owns. It reads the layers that BuildKit, GNU tar and bsdtar write (tested with each), and it fails for a header it does not know. - The check on the built images, not the recipe, is what proves the layer rule; it runs in each leg of the workflow. - `check` now needs `--variant`. A caller that does not pass it fails with a clear message. No caller exists outside this repository yet. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
5d3fa712aa |
feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Today the application image build compiles the Runner. Other images that need the Runner build it again, or pull the whole application image to copy it out > - Each of those builds can produce a slightly different Runner, and each one costs time > - The Runner should be built one time for each change to its inputs, and every image should copy the same bytes > - This pull request adds the build target `runner-artifacts`, a contract for its layout, and a workflow that builds and checks it. It publishes nothing > - Nothing consumes the target in this pull request. The application image build does not change > - The benefit is one checked Runner build that later changes can publish and copy by digest ## Linked Issues or Issue Description Refs #21096 (the merge-queue image build) and #21167 (provider pack layers). Those are earlier work on the same build. No issue exists for this change. **Problem or motivation** The Runner is compiled inside the application image build. The daemon comes from the `runner-build` stage. The provider pack is built on top of the full server and UI build. Downstream sandbox and VM images need the same Runner. They rebuild it from source, or they pull the application image and copy paths out of it. There is no contract for those paths and no single identity for a Runner build. **Proposed solution** Build the Runner alone, from its own inputs, as a plain file tree. Give each build an identity: a digest of its inputs. Document the layout as a contract, and check every built tree against it. Keep every check that proves the Runner is good in this one build. **Alternatives considered** - Keep the current builds and only document the paths in the application image. This does not remove the duplicate builds, and consumers still need the full application image. - Build the Runner in `docker.yml`. That workflow has no pull-request trigger, so a change to the Runner build would have no check before it merges. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 2 of 4. Each part is on top of the one before it: 1. `runner/pack-inputs-digest`: an optional field in the provider pack manifest 2. `runner/runner-image-build` (#21204): build and check the Runner image, with no publication (this pull request) 3. `runner/runner-image-layers`: layers by change rate, and the slim variant 4. `runner/runner-image-publish`: publish both variants from `master` This pull request had one larger form before. Publication, the image layers and the second variant are now parts 3 and 4. ## What Changed - `Dockerfile`: eight new stages that end in the target `runner-artifacts` (`FROM scratch`). They install with a filtered, frozen lockfile from a context that holds only the Runner's inputs. The base is a Node image on Debian bookworm, pinned by digest. No existing stage changed. `cloud` is still the default target. - The tree holds: the stripped daemon, the Runner package's complete `dist` with the build stamp that its own build records, the provider pack with its transport archives, the Grok provisioning inputs, the transport contract source and a bundled transport module, the lockfile, and `runner-image.json`. - `scripts/runner-image-identity.mjs`: computes the image identity. It hashes every file under the paths that the build copies from the build context (path, executable bit, content; symbolic links by target), the whole `Dockerfile`, and the two platforms. The build runs it on the files that the build context supplies after `.dockerignore`. - `scripts/runner-image-layout.mjs`: the layout contract. `write` assembles the tree in the build. `check` verifies a built tree: every documented path, the build stamp, every pack artifact and tree digest in `provider-pack.json`, and every transport archive. It also extracts the transport archives the way a sandbox does and compares the result with the pack. No checked path may resolve outside the tree. - Publication form. The build argument `PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it does the pack manifest carry `payload.runnerInputsDigest` (part 1), and `runner-image.json` records `publication: true`. Another image build that runs the stage inline builds the same Runner, but its pack names no image. `check` requires the field in a publication build and refuses it in any other. `--expect-inputs-digest` passes only for a publication build. - `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd --build-metadata` on each base image that a consumer uses. Each base is pinned by digest. - `.github/workflows/runner-image.yml`: two jobs, on pull requests that touch an input and on manual dispatch. `identity` resolves the lockfile and computes the digest in the build context. `build` builds each architecture on a native runner, checks the layout, and runs the smoke matrix. The workflow has `contents: read` only. It holds no credential, reads no registry, and publishes nothing. It builds the publication form, so the check can bind the tree to the digest that the identity job resolved. The tree stays on the runner. - `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`: the tree digest function moves here from `build-provider-pack.mjs` without a change in behavior, so the layout check can use it. A test pins its value. - `.github/workflows/pr-merge-review.yml`: lists the new workflow, as `doc/PR-MERGE-REVIEW-WEBHOOK.md` requires. - `doc/runner-image-contract.md` (127 lines): the contract for consumers. `doc/DOCKER.md` links to it. Changes from the earlier form of this pull request: - The identity hashes the whole `Dockerfile`. The script no longer parses stages, and it no longer refuses recipes. `scripts/runner-image-identity.mjs` went from 397 lines to 166. A short test keeps the three rules that make the hash enough: the Runner stages start only from pinned images, only `runner-artifacts-context` reads the build context, and it copies only the paths that the identity hashes. - The contract document went from 384 lines to 127. - The workflow tests keep only the security properties: job permissions, no credential and no registry access, triggers, actions pinned by commit, and no expression inside a shell script. They no longer pin step names, order, or wording. ## Verification Verified on `linux/arm64` only, on a local Docker host: - `docker buildx build --target runner-artifacts --platform linux/arm64 --build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .` builds in 166 seconds without a cache. The image is one layer of 850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files. - `node scripts/runner-image-layout.mjs check <tree> --expect-inputs-digest <digest> --expect-source-revision <sha> --expect-architecture arm64` passes on the exported tree. - `docker buildx build --file docker/runner-image/Dockerfile.smoke --build-context runner-artifacts=<tree> docker/runner-image` passes on all five base images. - The digest from `--target runner-artifacts-identity` equals the digest in the built `runner-image.json`: `d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`. - The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication argument builds. Its pack has `runnerSourceRevision` and no `runnerInputsDigest`. `check` passes on it, and `check --expect-inputs-digest` fails with "this is not a publication build". - An edit to `ui/package.json` and to `server/src` leaves the digest unchanged. A comment added to an application stage of the `Dockerfile` changes it. - The two points above ran one commit state earlier. The only difference to this commit is one type annotation in a comment of `scripts/runner-image-layout.mjs`. - A named build context replaces the `runner-artifacts` stage, and no Runner stage runs. A build with the application build's view of the context accepts the tree's `dist` and build stamp as current (`assessRunnerBuild`). - The daemon has the same bytes in the five cold builds of this stack that were compared. - Tests: `cd server && node node_modules/vitest/vitest.mjs run` on `runner-image-identity.test.ts`, `runner-image-layout.test.ts` and `runner-image-workflow.test.ts` (132 tests), with `remote-provider-pack-verification.test.ts` and the existing Dockerfile tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33 pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and `docker buildx build --check` report nothing. Never run: - The `linux/amd64` build. It is the leg that downloads the Cursor and Copilot assets for the pack. - The workflow. No job has run on GitHub. - The transport archive round trip with GNU tar outside the build. The build runs it with GNU tar. The local host ran it with bsdtar. - `actionlint` and `hadolint`. They are not installed on the local host. - The full test suite and the full typecheck. CI must supply them. ## Risks - Low risk to the application image. The change adds stages and files. No existing target reaches the new stages. - The first run of the workflow is the first test of the amd64 build. This pull request triggers it. - The `Dockerfile` is one file for all images. Any edit to it, also to an application stage, changes the Runner image identity. That is the price of not parsing it. - The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag, not a digest. A new frontend release under that tag does not change the identity. - The image does not run the drift checks of the Runner's `build` script. They need files outside the Runner's inputs, and repository CI runs them. A test fails when the `build` script gains a step. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
9a9cd20d91 |
feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, predictable CI feedback. > - PR #16378 added a Honeycomb trace of the main PR workflow, but only for PRs with the `profile-ci` label. > - Ordinary CI time was therefore never recorded. All of the last 300 `CI profile` runs were skipped. > - Reviews and scans outside the `PR` workflow, such as Greptile, were not in the trace. > - This pull request exports one CODEOWNER PR run in ten without a label, and adds outside checks to every trace. > - The benefit is a continuous record of total CI time and of which job or check was slowest. The Honeycomb key stays out of reach of outside contributors. ## Linked Issues or Issue Description Refs #16378 **What existing behavior does this improve?** The Honeycomb CI trace that the `CI profile` workflow exports. **Current behavior** - The exporter runs only for a PR with the `profile-ci` label. The label also forces the full suite. - The trace has the jobs of the `PR` workflow only. Greptile, security scans, commit statuses and other workflows are absent. - The only total is the `PR` workflow time. **Proposed behavior** - One `PR` run in ten is exported without a label. The run number must end in 0. The trace has every job and step. Each span has a sample rate of 10. - Each trace has one span for each outside check and each other workflow on the head commit. - The root span has `ci.workflow_ms`, `ci.all_checks_ms`, `ci.last_to_finish` and `ci.slowest_check`. **Reason and benefit** Maintainers can see how long CI takes and what is slow, for example a long Greptile review, without a label on each PR. Sampling keeps the volume near 1.7 million events each month. Export of every run is near 17 million. ## What Changed - `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without the label). - `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for outside checks. Add the sample rate and the new root fields. Sampled runs do not warn about absent test artifacts. - `scripts/ci-profile/export.mjs`: select the mode. Read check runs, commit statuses and workflow runs for the head commit. Wait a maximum of 15 minutes for outside apps that are not complete. Sampled runs read no artifacts. - `.github/workflows/ci-profile.yml`: no new job. The job condition now requires a branch of this repository, an actor and rerun actor on the `AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add `checks: read` and `statuses: read`. Increase the timeout to 30 minutes for the bounded wait. - `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs, outside checks, the totals and the job condition. ## Verification - `node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail. - `actionlint .github/workflows/ci-profile.yml`: no findings. - Dry runs against real runs with the real CODEOWNERS check (`node scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`): - A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate 10, 5 outside checks and 1 other workflow. - A sampled run from an outside contributor (run number 64800): "No CODEOWNERS-authorized PR at this revision; no data exported." - A CODEOWNER run that is not in the sample: "This attempt was not selected for profiling; no data exported." - New tests cover: the sample rule, fork and outsider PRs, a bot rerun actor, a stale head, outside-check spans and totals, the bounded wait, and the job condition. - Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs installed dependencies, and this change does not touch the reporters. CI runs it. - No data was sent to Honeycomb before merge. The workflow runs from `master`, so the first real export occurs after merge. ## Risks - The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If that variable is empty, nothing is exported. All four CI CODEOWNERS are on the roster today. - The roster is only a first filter. The exporter still checks the PR author and both actors against CODEOWNERS on `master` before it reads or sends data. The key stays in the `ci-profiling` environment, which permits only `master`. - The export job can now run for a maximum of 30 minutes on a GitHub-hosted runner while it waits for an outside check. - GitHub gives no completion time for a workflow run. The span for another workflow ends at the last update of that run. - Honeycomb volume increases by about 1.7 million events each month. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use, code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
8ff45c0fa9 |
feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.
## Linked Issues or Issue Description
**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.
**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.
**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.
**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.
**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).
**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.
## What Changed
- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
- App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
- the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
- warnings in the sign-in step;
- `doc/connections/CIRCLEBACK.md`.
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.
## Verification
- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
- `ui_meta` is stored;
- an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
- a call without a view records nothing;
- the routes serve the view's HTML and CSP;
- the view's tool call works and a model-only tool is refused;
- another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
- Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
- Discovery stored the real view declarations:
- `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
- `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
- A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
- The 1 MB tool-result limit blocked the view document.
- Views were hidden inside the folded tool history.
## Risks
- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1d153f62b6 |
feat(docker): build the image in the merge queue and promote it on master (#21096)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds, probes, gates and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after that image is published. > - Each master push rebuilds the image from scratch: about 10.6 min for amd64 and 7.8 min for arm64. > - The owner approved a merge queue for Core. A merge queue tests the exact commit that master will become, before the merge. > - This pull request builds and probes the image for that exact commit in the queue, with no registry credentials. The later master push pushes those exact bytes instead of rebuilding. When no matching verified build exists, master builds from source exactly as today. > - The benefit is that the image legs on master drop from about 8–11 min to about 3–4 min, and nothing changes until the owner turns the queue on. ## Linked Issues or Issue Description Refs #21013 (it measured the image path and proposed this plan). **What existing behavior does this improve?** Every master push rebuilds the standard image, even when the same commit could already have been built while it waited to merge. **Subsystem affected** `.github/workflows/docker.yml` (`build-and-push` only), new `.github/workflows/docker-merge-group.yml`, new `scripts/merge-group-image.mjs`. **Current behavior** `build-and-push` always runs the full BuildKit build on master: 639 s (amd64) and 468 s (arm64) in run 43133035209. **Proposed behavior** On a `merge_group` event, build both architectures for the merge-group commit, probe PID 1, and upload an OCI archive plus a record of commit, tree, architecture and digest. On the master push for the same commit, each leg pushes that archive by digest instead of building. Everything after the leg (probe, exact-source gate, signed-index check, attestation) is unchanged. **Reason and benefit** The build moves before the merge, where it runs in parallel with the queue's other checks. The master leg only downloads and pushes. **Breaking changes** None. Without `merge_group` events, `find` returns no run and the leg builds as before. **Additional context** This PR does not touch the exact-source gate jobs (`source_verification`, `publication_verification`, `merge-and-push`), `scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another change owns them. ## What Changed - **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs only for this repository's master queue. Workflow permissions are `{}` and the job has `contents: read` only. There are no secrets, no registry login, no `cache-to`, no `push: true`, no environments, and checkout uses `persist-credentials: false`. It runs on GitHub-hosted runners, not the paid AWS fleets. Steps: build `target: production` with the same build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA, which becomes the master SHA). Read the public registry cache. Write `type=oci` to a tar. `docker load` it and run `scripts/assert-orphan-reaping.sh`. Only then write `record.json` and upload `merge-group-image-<arch>` (3-day retention). - **`scripts/merge-group-image.mjs` (new).** - `find` (master push only): selects a completed, successful run of this workflow, with event `merge_group`, the same `head_sha`, repository and head repository both `1170821064`, and a `gh-readonly-queue/master/` head branch. It needs exactly one unexpired artifact for the architecture. - `record` (merge group only): reads the archive's `index.json`, checks each blob's sha256 and that the image is `linux/<arch>`, and writes `{sha, tree, arch, digest}`. - `verify` (master push only): re-reads the archive the same way and requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse HEAD^{tree}` of the master checkout, same architecture, and archive digest = recorded digest. - **`docker.yml` `build-and-push`.** - The leg gets `actions: read` so it can download the artifact. - New steps run after the registry login: find, download, then promote (`verify`, then `skopeo copy --all --preserve-digests oci-archive:… docker://ghcr.io/…@<digest>`). The step sets its `digest` output only after the push succeeds. - All three new steps are `continue-on-error`. Buildx, labels, metadata and "Build and push by digest" run when `steps.promote.outputs.digest == ''`, so any failure builds from source. - "Select image digest" feeds the existing PID 1 probe and digest export. - **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):** - Run selection rejects another commit, another event, another workflow, a fork head, another repository, another queue base, and failed, cancelled or running builds. - Artifact selection. - Archive digest with and without provenance, a tampered blob, the wrong architecture, and a real tar read through the system `tar`. - Record checks for commit, tree, architecture and digest mismatch. - The CLI refuses `record` outside `merge_group` and `find`/`verify` outside a canonical master push. - Static checks of the merge-group workflow: credentials, triggers, permissions and runners. - Static checks of the `docker.yml` wiring: promotion is gated and falls back, the output is set only after the push, and every build step has the fallback condition. ### How promotion proves it publishes the verified tree 1. `find` accepts only a run whose `head_sha` equals `github.sha` of the push, meaning the queue landed the commit it tested (unverified: see "Merge method" in Risks). 2. `verify` compares the record's tree with the tree of the master checkout, and its commit with `GITHUB_SHA`. 3. The archive is content-addressed. The digest is recomputed from the archive blobs and must equal the recorded digest. `skopeo --preserve-digests` to that digest reference fails rather than produce other bytes. 4. The leg then runs the PID 1 probe on that registry digest. `merge-and-push` still signs only an index whose Linux images are the probed digests (#21013), after the unchanged exact-source gate. 5. The pipeline itself is trusted only as master already ran it. `find` and `verify` run the copy of `scripts/merge-group-image.mjs` from the push's `before` commit (`git show "$BEFORE:…"`), not the copy the push brings. `verify` refuses promotion if anything in `before..HEAD` (every commit of a multi-PR group, not only the tip) changes `docker-merge-group.yml`, `docker.yml`, the promotion script or the PID 1 probe. Such a push builds from source. So does the first push after this PR merges, because its `before` has no verifier. ## Verification - `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds a real git history in which a two-PR push changes the pipeline in its first commit only, and checks that the range check catches it. - `node --test` on the workflow and contract suites (cloud-readiness, docker-canary-promotion, post-merge-runner-routing, lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image, standard-image-contract, cloud-source-verification, merge-group-image): 492 pass. - `vitest run server/src/__tests__/docker-build-stamp.test.ts server/src/__tests__/container-init-reaping.test.ts`: 22 pass. - `actionlint` on both workflows: no new findings (the 6 remaining are pre-existing `ls` and word-splitting notes in `docker.yml`). - Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI layouts kept the index digest. It accepted a `docker://…@sha256:` destination and failed only at the network step. Cannot be exercised before the queue exists, because no `merge_group` event fires: - the merge-group run itself: the hosted-runner build time, disk use with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a ~5 GB artifact upload; - the cross-run artifact download on master; - whether `skopeo` exists on the runs-on AMI and accepts digest destinations in its installed version. Each master-side failure falls back to the full build. A merge-group failure only blocks the queue if its check is required, which is why that check is required last in the steps below. ## Risks - **Inert until enabled.** No `merge_group` events fire before the owner adds a merge queue. On master, `find` lists runs (one API call per leg, about 1 s), finds none, and the leg builds as today. The only always-on change is the leg's `actions: read` permission. - **Build cache freshness.** A promoted master push writes no `buildcache-<arch>`, and merge-group builds read but never write it. With every push promoted, the cache ages, so dependency or weekly tool-layer changes rebuild in each merge-group run until a fallback build refreshes the cache. A follow-up can refresh the cache from master without blocking publication. - **Publication time is still gate-bound.** The exact-source gate (about 13–14 min after push) still decides when the image publishes. See "Expected timing". - **Merge method (unverified).** Promotion needs the master commit to be exactly the commit the queue tested, because the image stamps the commit SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack revision). Matching only the tree would publish an image stamped with another SHA, so this PR does not relax the check. I expect the queue to fast-forward master to its tested commit. GitHub's merge-queue page does not state this, and it cannot be tested before the queue exists. If the queue lands a different SHA, promotion never matches and every push builds from source. That is safe but inert. Step 4 below checks this. - **skopeo on the AWS fleet.** The promote step checks for `skopeo` first and logs "Merge-queue promotion unavailable" if it is missing, then builds from source. If that notice appears, add skopeo to the fleet image. - **Hosted runner capacity.** amd64 merge-group builds run on 4-core hosted runners, colder and slower than the AWS fleet. That adds queue latency, not master latency. ### Settings the owner must enable, in order 1. **Prerequisite, a code change, not a setting.** The `master` ruleset (id 13619726) requires `ci / verify` and `ci / e2e`. Those come from `pr.yml`, which triggers only on `pull_request`. A merge queue waits for required checks on the `merge_group` event too, so every entry would stall until its timeout. First decide, and merge in a separate PR, how `ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on the merge-group commit, or a fast pass that relies on the PR-head result. Today the ruleset does not require up-to-date branches either. 2. Merge this PR. Nothing changes yet. 3. In Settings → Rules → Rulesets → `master` (13619726), add **Require merge queue**: - merge method Squash; - build concurrency 5, minimum group size 1, maximum group size 5, wait 0 min; - "Require all queue entries to pass required checks"; - status check timeout 60 min. Do not add the image build to the required checks yet. Promotion is then opportunistic: if the merge-group build has not finished when master is pushed, master builds from source. 4. After the first queued merge, confirm promotion can match. The master commit SHA must equal the `head_sha` of its `Docker merge-queue build` run, and the next master `Docker` run's "Promote the merge-queue image" step must succeed. If the SHAs differ, promotion stays inert under this merge method. Stop here and report it. 5. After about a week of green `Docker merge-queue build` runs, add `build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64, ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's required status checks. Then every queued commit has a verified image ready to promote. 6. The existing bypass actors (org admins, repository admins, integration 5534676) can still merge outside the queue. Those pushes have no merge-group run and build from source. ### Rollback - Remove the image checks from the required status checks (step 5), or remove **Require merge queue** (step 3). Master pushes then find no merge-group run and build from source. Nothing to migrate. - Code: revert this PR. Old merge-group artifacts expire after 3 days. Promoted images are ordinary digests under `ghcr.io/paperclipai/paperclip` and stay valid. ### Expected timing - Image leg on master: about 3–4 min (download about 5 GB, push the new layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min (arm64). - Push to published: still bounded by the exact-source gate on master, about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is the same bound, but with the legs as a second 11-min path and the serialized-run queue in front. - If the gate later accepts proof produced in the merge queue (owned by the gate change), push to published would be about 4–5 min (leg promotion plus `merge-and-push`). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only GitHub API and registry reads). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ca62ca9e2a |
fix(release): keep the Runner chaos evals out of the Core image publication gate (#21088)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after "Cloud source verified v1"
passes for the same commit in `cloud-readiness.yml`
> - That job needs `verify`, which is the whole `release-verify.yml`
call, and that call included the Runner chaos evals
> - The chaos eval job ("Restart, replay, trace, and recovery faults")
took 10.2-11.9 min, with a 7-9 min server finalization step. It is one
of the two slowest jobs on the gate
> - The owner decided on 2026-11-12 that the Runner chaos evals must not
be a publication prerequisite
> - This pull request runs the chaos evals as a separate job in the same
Cloud readiness run. They still run and report, but the source proof
does not wait for them
> - The benefit is an earlier image publication on every master commit.
A chaos failure stays visible as a failed check
## Linked Issues or Issue Description
**What happened?**
The Core image gate waited for the Runner chaos evals. In master runs
43133035381, 43134251278 and 43145252393 the chaos job took 10.2, 11.9
and 11.0 min. The gate opened at +13.5, +13.2 and +13.9 min.
**Proposed change**
Owner decision (2026-11-12): keep the chaos evals running on every
admitted master commit, but remove them from the set of jobs that the
image and Cloud discovery wait for.
**Why**
Every minute on the gate is a minute on every staging release.
## What Changed
- `.github/workflows/release-verify.yml`: new boolean input
`runner_chaos_evals`, default `true`. The `runner_chaos_evals` job runs
only when it is true. Release callers (`release.yml`,
`release-smoke.yml`) do not set it, so they stay gated on the evals.
- `.github/workflows/cloud-readiness.yml`:
- `verify` passes `runner_chaos_evals: false`.
- New job `runner_chaos_evals`, named "Runner chaos evals (not a
publication gate)". It needs `admission`, has the same admission
condition as `verify`, and calls `runner-chaos-evals.yml` with `ref:
github.sha`. It has no `continue-on-error`.
- `source_verified` ("Cloud source verified v1") still has `needs:
[verify]`. A comment names the owner decision and says not to add the
chaos job.
- `scripts/cloud-source-verification.mjs`: new `runnerChaosEvalsJob`
constant. The fail-fast check ignores jobs named `"<that name> / ..."`.
A chaos failure therefore does not raise `SourceVerificationFailed`. So
`watch-release-source.mjs` no longer cancels the readiness run and the
image run when only the chaos evals fail. Every other failed job still
fails fast and cancels as before.
- Tests:
- `scripts/cloud-source-verification.test.mjs`: a chaos failure does not
cancel the image or block the proof; a chaos failure while the proof is
pending keeps waiting; a required job whose name only resembles the
chaos job still fails fast.
- `.github/scripts/tests/cloud-readiness.test.mjs`: pins the job set
(`admission`, `verify`, `runner_chaos_evals`, `source_verified`),
`source_verified` `needs: [verify]` exactly, `verify` passes
`runner_chaos_evals: false`, the chaos job calls the eval workflow, and
release callers keep the default. Putting the chaos job back into
`source_verified`'s `needs` fails this test (checked locally: 2
failures).
Publication gate, before: `Cloud source verified v1` waited for
typecheck, build, general tests (17 jobs), serialized tests (5), Runner
workflow evals, Verify Paperclip Runner (3 lanes), and the Runner chaos
evals. After: the same list without the Runner chaos evals.
How a chaos failure stays visible:
- Its own check fails on the master commit.
- The Cloud readiness run concludes `failure`.
- The chaos eval bundle is still uploaded (`if: always()`).
- Nothing reports the chaos evals as passed. They are just not a
prerequisite.
Consumers:
- `docker.yml` reads the job "Cloud source verified v1" through
`scripts/watch-release-source.mjs`. No change to `docker.yml` is needed.
- Paperclip Cloud discovery (`src/fleet/cloud-readiness.ts`) runs with
`sourceOnly: true` in `cloud-distribution-policy.ts` and
`release-resolver.ts`. In that mode it needs the workflow path
`.github/workflows/cloud-readiness.yml` and one successful job named
"Cloud source verified v1" in the latest attempt. It does not need the
run conclusion. Both are unchanged, so Cloud needs no change.
- The non-`sourceOnly` mode reads the run conclusion and a job "Cloud
deployable v1". That job no longer exists in this workflow.
## Verification
- `node --test scripts/cloud-source-verification.test.mjs
.github/scripts/tests/cloud-readiness.test.mjs
.github/scripts/tests/post-merge-runner-routing.test.mjs`: 395/395 pass.
- Red/green:
- Before the fail-fast change, the two new chaos tests failed: the
watcher cancelled the image.
- With the chaos job added back to `needs`, the workflow test failed.
- `actionlint .github/workflows/cloud-readiness.yml
.github/workflows/release-verify.yml`: clean.
- Not run: a real Cloud readiness run of this change. GitHub shows the
result after merge.
- Expected gate: the slowest remaining gating job sets it. After #21016
that is a general-server shard at about 10.5 min, then chat (8.8-10.5
min) and serialized (9.2-9.5 min). Expected gate open is about +11.5 to
+12 min, down from +13.2 to +13.9.
## Risks
- A chaos-eval regression no longer blocks the staging image. This is
the owner decision. The failure stays visible as a red check and a
failed Cloud readiness run.
- npm canary publication reuses the same source proof, so it is also no
longer gated on the chaos evals. Stable and preview releases
(`release.yml`) still run them as a gate.
- The non-gating rule depends on the job name prefix. A test pins the
name to the constant in `cloud-source-verification.mjs`.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI). Standard reasoning mode.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Related: #21016 (server shard rebalance), #21013 (image publish time;
owns `docker.yml`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b853cc28ae |
perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server process starts, recovers saved work, and then reports ready on `/api/ready`. The deploy health gate waits for that signal. > - On a busy staging workspace, one boot took 85 s to bind the listener and 4.4 minutes to report ready. Small, idle stacks take 20–90 s. > - Two parts of that time do not need to block readiness. The server published several hundred UI assets one at a time before it bound the listener. After the listener bound, it waited for the first pass of each periodic recovery queue, and those passes grow with the backlog. > - The volume-free data gate, which can refuse a start, ran last. By then the server had already dispatched queued runs that a refusal would abandon. > - This pull request moves the refusal gate to the start, publishes the assets 8 at a time with replayable bodies, and runs the backlog-sized first passes after readiness, in their original order. > - The benefit is a faster time from container start to ready, and no work admitted before a gate that can refuse to start. ## Linked Issues or Issue Description No public issue exists. This is the problem description (enhancement template). **Is your enhancement related to a problem? Please describe.** Server boot time grows with the saved backlog. These timestamps come from the log of one staging boot (release |
||
|
|
bed5f70278 |
perf(ci): rebalance server test shards and repair the weekly weight refresh (#21016)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Core image is published only after the release verification of the same commit passes (`cloud-readiness.yml` calls `release-verify.yml`) > - The image is ready about 3 minutes before that gate opens, so the slowest verification job sets the staging release time > - The slowest job is a general-server test shard: 11.8-12.7 min, while other server shards take 8-9 min > - The shards are balanced by a duration manifest, but the manifest has not changed since 2026-10-31. The weekly refresh that updates it failed on 2026-11-02 and 2026-11-09 > - About 40 newer suites per shard use the 1 s median fallback weight, but several of them run for 90-200 s, so they pile up on one shard > - This pull request repairs the weekly refresh and refreshes the server weights from three current master verifications > - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8 min of test time) on every master verification and every PR run ## Linked Issues or Issue Description **What happened?** The Core image publication waits about 13.5 min for source verification. The critical path is one general-server shard. In three master verifications (runs 43133035381, 43134251278, 43145252393) the slowest server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and 8.4 min. **Root cause** `scripts/general-server-shard-durations.json` came from one run on 2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice: - 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A server shard log is 30-65 MB. One download took 59 s locally. Three parallel downloads overran the 120 s `execFile` timeout. - 2026-11-09 (run 42580964686): the refresh's own check failed. Merged-PR sources for the Runner lane have a leading `pull` key, and the test accepted only `run, attempt, sha, url`. **Expected behavior** The weekly refresh finishes and keeps the shards balanced as suites are added. ## What Changed - `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a 512 MB buffer. JSON calls keep 120 s. A failed call now gives a short error and does not print the partial 60 MB log. - `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`. It accepts the optional leading `pull` key that `collectMergedPrSamples` writes. The checked-in Runner manifest test and the collector test both use it, so the collector format stays covered. - `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 -> 45 min (the server lanes alone fetch about 30 large logs). - `scripts/general-server-shard-durations.json` and `scripts/serialized-shard-durations.json`: refreshed with `node scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381 43134251278 43145252393` (median of three successful protected-master Cloud readiness runs; 1757/1815 general files and 173/173 serialized files measured). - No test is added, removed, skipped or reassigned to a different lane. No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check are not touched. ## Verification Measured job timings (minutes from the first job start; three master verifications): | Job | 43133035381 | 43134251278 | 43145252393 | | --- | --- | --- | --- | | Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7 | | Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 | | Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 | | Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 | | Chat (slowest of 3) | 8.8 | 10.5 | 9.4 | | Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 | | Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 | | Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 | - Queue to runner start is 0.5-1.1 min for every job. In a server shard, setup (checkout, pnpm, Node, install) is about 50 s and the preflight plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8 min). Setup is not worth caching. - Shard prediction, using the refreshed medians as the cost of each file and the 3 fork workers per shard: - general-server, 10 shards: old weights max 10.3 / min 8.1 min; new weights 8.8 / 8.8 min. - serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights 6.6 / 6.6 min. - The old-weight prediction matches what CI saw: predicted 10.3 and 9.9 min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in run 43133035381. - Red/green: - `node --test --test-name-pattern="gh api" scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix (`log timeout 120000`) and passes after it. - With a `pull` key in the Runner manifest sources, the old "package lane manifest" test failed with the same assertion as run 42580964686. The new test passes. - The refresh script, which died before the fix, completed locally in 8 min after it. - `node --test scripts/ci/test-selection-timings.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the complete, non-overlapping shard cover checks. - `actionlint .github/workflows/ci-test-durations.yml`: clean. - Not run: the release-verify workflow itself, the scheduled refresh workflow, and the Runner/e2e lane refresh. The shard times above are predictions from logged per-file timings, not a measured CI run on this branch. ## Risks - Low risk. Weights only move files between shards of the same lane. The shard tests prove every file still runs exactly once. - The predicted gain is about 1.5 min on the slowest server shard (12-12.7 -> about 10.5 min job time). The gate then waits for "Pre-release Runner chaos evals / Restart, replay, trace, and recovery faults" (10.2-11.9 min). So the expected gate saving is about 1 min (13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a shard is about ±1 min. - A larger log timeout can make a stuck download wait up to 15 min. The 45 min job timeout still bounds the run. Candidates for the owner (not changed here): - The chaos-eval job is now the bound. Its "server finalization" step takes 7-9 min. Decide if it must gate image publication, or split it. - Server shard logs are up to 811k lines, mostly repeated stack traces from `server/src/services/execution-preparation-timing.ts`. That slows log upload and this refresh. - 9 test files run over 180 s (largest: `chat-channels.integration.test.ts` 429 s, `heartbeat-process-recovery.test.ts` 344 s). One file cannot be split across shards. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, GitHub CLI read-only). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Related: #19650 (earlier repair of the weekly refresh), #21013 (Core image publish time; separate files). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f8e1fb0278 |
perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after this image is published and attested. > - A master commit takes 29 min (p50) and 56 min (p90) to reach a published image. Every minute here delays a staging fix. > - The run logs show where the time goes. The exact-source gate (about 14 min) is the critical path today. The build legs finish about 3 min before it. After the gate, `merge-and-push` spends about 95 s to pull the 4.9 GB amd64 image for one probe. > - This pull request moves that probe into the build legs, which run before the gate. It also removes a duplicate 187 MB layer from the amd64 image. > - The benefit is about 95–100 s less from gate to signed image on every master commit, with the same proof. arm64 is now also probed. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time from a commit landing on master to the signed standard image (`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation). **Subsystem affected** `.github/workflows/docker.yml`, `Dockerfile`, `scripts/standard-image-contract.mjs`. **Current behavior** Measured on run 43133035209 (sha |
||
|
|
8818402e7a |
feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects requests that need a human decision. > - Teams need to know who owns each request and who may act on it. > - A shared close operation must differ from hiding a card for one person. > - This pull request adds personal routing and durable outcomes to the existing attention service. > - People can claim or transfer work and inspect what another person did for them. > - Domain permissions still decide who may resolve each request. ## Linked Issues or Issue Description **Subsystem affected** Server, shared contracts, and database. This is the routing foundation for a stacked inbox UI change. **Problem or motivation** A company-wide attention feed does not give teams a clear personal queue. Archive, withdrawal, transfer, and decisions by another person need distinct scopes and records. **Proposed solution** Resolve a request audience from its addressee, task participants, and configured owners. Provide personal, unassigned, company, all-assignee, and archived views. Add claim, transfer, personal archive and snooze, shared withdrawal, receipts, and personal history. **Alternatives considered** A single shared inbox leaves ownership and personal hiding ambiguous. The service keeps one canonical request and projects its audience instead of copying requests for each person. **Roadmap alignment** This extends the existing Multiple Human Users and Agent Reviews and Approvals capabilities in ROADMAP.md. **Additional context** Refs #19363 and #19520 for the current inbox navigation and confirmation replacement rules. Refs #20268 for related actionable-request work. That PR filters actionability; this stack adds audience routing and personal outcomes. This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this branch. ## What Changed - Add nullable company and project decision owners and a company budget owner. - Add audience views and personal badge counts to the canonical attention projections. - Add claim, transfer, withdrawal, personal archive, snooze, and restore operations with permission checks and activity records. Lock addressed transfers to the current addressee or routing administrators, and validate the recipient against canonical resolution policies. - Add receipts and scoped history for work handled by another person. - Keep closed-task requests and replaced confirmations out of current cards. - Add an idempotent migration and a concurrent receipt-history index. - Update typed fixtures so the existing UI remains compatible with the new contracts. ## Verification - Server layered typechecking passes, including all three test-tree partitions. - Seven routing and interaction-resolution suites pass all 249 tests against a dedicated external PostgreSQL test database on `4caecc21d8`. Six new regressions failed before the repair; they cover takeover followed by approval, legitimate delegation, excluded creators/review requesters, and unauthorized claims. - Prior focused verification covers private approval history and receipts, transfer visibility, audience escalation, withdrawal governance, hidden-card counts, per-recipient confirmation replacement, and SDK source contracts. - A synthetic three-person company passed 37 routing and lifecycle acceptance checks. The UI stack adds browser acceptance for the shared controls. - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331) passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with no open findings. The fresh security scan passes after explicitly rechecking the addressee-transfer repair. - `pnpm verify` reports exit 2 because a CI-global script changed; full CI supplies that broader coverage. Embedded PostgreSQL could not start on this Mac because shared-memory slots were exhausted, so local database checks use a separate external test database. Linux CI covers the embedded setup. ## Risks - Personal badge counts change to the viewer's assigned requests. Accessible feed totals and task membership keep their existing scope. - The migration adds nullable owner fields and a dismissal fingerprint. Its index runs outside a transaction. The migration is classified and rerunnable. - Routing changes the addressee used by resolution authorization. The current addressee or a routing administrator must authorize an addressed transfer; the recipient must satisfy writer access, canonical resolver exclusions, and task review policy under row locks. - The shared validator export barrel crosses 1,000 lines because it exports the new attention schemas. Schema implementation stays in the dedicated attention validator module. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (OpenAPI contracts and docs/api/attention.md) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
9dfdeaad86 |
Refine agent workspaces and organization charts (#20973)
Refine agent details, identity editing, responsive controls, and organization charts with three card tiers, clearer save feedback, and stable filtering. Guard instruction-file renames and root configuration updates in one transaction through the agent lifecycle integration, preserving conflict checks, rollback, and configuration history. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
e8e548e856 |
feat(db): classify migrations for application under the running release (#20913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time
## Linked Issues or Issue Description
**Subsystem affected**
Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.
**Problem or motivation**
Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.
**Proposed solution**
Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.
**Alternatives considered**
- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.
**Roadmap alignment**
This is release infrastructure. It adds no product feature.
## What Changed
- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".
Published shape:
```json
"migrationCompatibility": {
"version": 1,
"migrations": [
{
"idx": 468,
"tag": "0468_busy_wendell_rand",
"sqlSha256": "56a6e272…",
"compatibility": "online_compatible",
"classified": "online_compatible",
"override": false,
"reasons": [
{ "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
]
}
]
}
```
`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).
### Distribution over the existing migrations
No existing migration is rewritten or annotated.
| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |
Migrations per blocking reason (one migration can have several):
| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.
## Risks
- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
8ab955776d |
perf(github): check the requester provenance of an issue list's review waits in one batch (#20851)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can review a GitHub pull request. After the agent publishes the review, the task waits in review for the next GitHub reply. A later run of the same task can inherit this wait. > - Issue lists, issue summaries and inbox attention show these waits. For each waiting task, the server first checks the requester provenance of the waiting run (`chatRunSourceMessageId`). This check is an authorization check: it walks the identity-context lineage of the run and rejects a changed requester, another task, another company or another provider. > - #20673 batched the review reads and the publication reads of one list request. It kept the provenance check as one call for each waiting run, inside `Promise.all`. > - Thus a request with N waits still built and sent at least 2N statements at the same moment. The database is fast for each statement, but the server builds each statement on its main thread. > - On a hosted instance, a CPU profile of the server showed the main thread at 96% of one core. SQL text building, per-statement timers, socket writes and garbage collection had the largest shares. Connection samples showed 235 to 292 pool connections whose last statement was the source-message link read of this check, and 53 to 112 whose last statement was its identity-context read. > - This pull request checks the provenance of all the waiting runs of one request in one batch. The runs that need the same kind of read share one statement. Each run gets the same decision as a separate call, from the same walk and the same reads. > - The benefit is that a list request with 250 waits sends 17 statements, not 658, and uses about one quarter of the CPU time. The statement count no longer follows the number of waits. ## Linked Issues or Issue Description Refs #20673 (batched the review reads and the publication reads of the same request; this change does the same for the provenance reads that #20673 kept for each run). Refs #20551 (authorizes the waits of one issue page with one read). **What happened** `readInheritedGitHubReviewWaits` called `chatRunSourceMessageId` one time for each waiting run, inside `Promise.all`. Each call sent an identity-context read and a source-message link read. A run that continues an earlier run also sent a parent-context read and a source-run read for each continuation in its lineage. `listExternalReviewWaits` calls this function for the issue list, the issue summaries and the inbox attention of a company, so each of those requests repeated the same reads for every waiting task. Evidence from a hosted instance (sanitized): - CPU profile of the server process, 30 s, 5 ms samples: main thread at 96% of one core. Building SQL text for select statements: 17.7% inclusive. The idle-timer restart that the database client does for each statement: 7.6%. Socket writes: 6.2%. Garbage collection: 13%. - Three samples of the pool connections (about 494 connections). 235 to 292 connections had the source-message link read of `chatRunSourceMessageId` as their last statement. 53 to 112 connections had its identity-context read as their last statement. - The `DISTINCT ON` review read that #20673 added ran about 29 times each minute. It runs one time for each list request, so each request started hundreds of provenance reads. **Expected behavior** One list request sends a number of provenance statements that does not follow the number of waiting tasks. The decision for each run does not change. **Steps to reproduce** 1. Put many tasks of one company in review, each with an inherited GitHub review wait. 2. Open the issue list or the inbox. 3. Count the statements that the server sends. The identity-context read and the source-message link read run one time for each waiting task. The benchmark below does these steps on a local PostgreSQL. ## What Changed - `server/src/services/chat-channels/run-origin.ts`: - The provenance decision is now one walk (`sourceMessageWalk`). The walk asks for one read at a time and decides on the returned rows only. Its checks, their order and its error messages are the checks of the previous `chatRunSourceMessageId`. - The five reads of the walk (active identity context, source-message link, board message, parent context, source run) are defined one time in `provenanceReads`. Each definition takes its values either as literals or as SQL columns. - `chatRunSourceMessageId` (single run) drives the walk with one statement for each read, as before. The joins, the company scope and the filters are the same. The identity-context reads now select the seven columns that the walk uses instead of every column. - New `chatRunSourceMessageIds(db, runs, options)` (batch). It drives one walk for each run. In each round, the runs that wait for the same kind of read share one statement: `unnest(...)` of the request values, `CROSS JOIN LATERAL` the same single-row read with the request columns in place of the literals. Thus each run gets the row that its own single read returns, with `LIMIT 1` and the company scope on each join. The result for each run is `{ messageId }` or `{ forbidden }` (the 403 that the single call throws). A database error rejects the whole batch, as `Promise.all` did. - `server/src/services/chat-channels/github/review-wait.ts`: `readInheritedGitHubReviewWaits` checks the provenance of its runs with the batch. A 403 still gives `revoked`. The single-run `readInheritedGitHubReviewWait` does not change. - `server/src/services/chat-channels/run-origin.test.ts`: two new tests (see Verification). - `server/src/__tests__/github-review-wait.test.ts`: one new regression test for the statement count. - `server/scripts/benchmark-review-wait-provenance.ts`: the reproduction and the before/after measurement. It loads the pre-fix modules from `--baseline-ref` and fails unless both versions return the same waits. - `scripts/benchmark-github-review-wait.ts`: removed an assertion that `run-origin.ts` is byte-identical to its baseline. Both sides of that benchmark import the current helper, so the comparison stays fair. Without this change, the documented command of that benchmark fails. - `doc/DATABASE.md`: describes the batch provenance check and the new benchmark command. Statement count of one batch: the walk of one run does at most 3 + 4L reads, where L is the number of continuations in its requester lineage. The batch does the same number of rounds for its longest lineage. Each round sends at most one statement for each kind of read (5 kinds), for any number of runs. A request whose waiting runs accepted their GitHub comment directly sends 2 provenance statements. ## Verification Red/green regression test (`server/src/__tests__/github-review-wait.test.ts`): - `reads the waits of forty runs with the statements of four runs`. Before the fix: `expected { few: 10, many: 82 } to deeply equal { few: 4, many: 4 }`. After the fix: passes. Equivalence tests (`server/src/services/chat-channels/run-origin.test.ts`), on real PostgreSQL: - `gives every run of a batch the decision of its own check, in every provenance state`. 33 runs, each in a different provenance state, are checked with the single function and with the batch for five option sets (no options, legacy wake allowed, GitHub, GitHub with legacy wake, Slack). The test compares the decision of each run (message, `null`, or the 403 and its message) for: the full batch, the reversed batch, and a batch of one run. The states include the negative cases: a run of another company, a message that is linked only in another company, a message that is linked to another task, a conversation that moved to another task, a continuation on another task, a changed requester (run, identity context, continuation, board message), a link of another provider, a missing link, an outbound link only, a deleted message, an agent message, a rejected or pending identity context, a missing parent, another agent and a lineage cycle. The test also states the expected GitHub decision for each state, so the comparison cannot pass on two wrong answers. - `checks the runs of a batch with statements that follow the lineage, not the number of runs`. 4 runs and 40 runs each send 2 statements. 40 separate checks send 80. One continuation in each lineage gives 4 statements for the batch. - A mutation check: when the batch gives a row to the wrong request, both tests fail. Commands: - `pnpm vitest run server/src/services/chat-channels/run-origin.test.ts server/src/__tests__/github-review-wait.test.ts` - Focused run of 15 test files that import the changed modules, their callers or the shared seed helper, including `external-chat-wait.integration.test.ts`, `chat-github-tools.test.ts`, `slack-authority.test.ts` and `issue-list-review-wait-authorization.integration.test.ts`: 482 tests passed. In `idle-completed-history.test.ts`, `afterEach` hooks (a `TRUNCATE` of the test database) timed out at 30 s for 1 test in this run and for 3 tests in a second run of the file, while other agents loaded the machine. No assertion failed. The three tests of that file that reach the changed code pass when they run alone. CI runs the full file. - `server/src/__tests__/chat-channels.integration.test.ts` also uses the single check. It was not run locally (very large file). CI runs it. - Typecheck of the changed files and of the callers of `chatRunSourceMessageId` with a focused `tsc` program: no errors in these files. The full server typecheck was not run locally (shared machine). CI supplies it. - The first CI run of this pull request failed `ci / policy`: the module boundary check rejects an insert into the agents table outside the agent lifecycle module, and the benchmark seeded its fixture agent that way. The second commit seeds that agent with SQL. `pnpm check:module-boundaries` passes locally, and the benchmark gives the same statement counts. - `pnpm verify` returns exit 2 (broader verification required) because the change adds a file under `server/scripts/`. CI supplies the broad coverage. Before/after measurement: ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-wait-provenance.ts --baseline-ref=e2cf9a670fc5d5f5058fd0972beb71650f77c554 ``` One call of `listExternalReviewWaits` (the read of one issue list request). Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding. One company, one agent, one GitHub repository. Every wait has a published review. Every tenth waiting run continues three earlier runs. 3 warm-up calls, then 15 pairs in alternating order. Both versions return the same waits and the same inherited wait for each run. | Waits | Version | Statements | SQL text bytes | Wall ms, median (min–max) | CPU ms, median (min–max) | | --- | --- | --- | --- | --- | --- | | 20 | before | 60 | 31,084 | 14.1 (12.8–19.6) | 13.7 (8.7–39.3) | | 20 | after | 17 | 15,847 | 12.5 (9.4–22.7) | 8.8 (4.1–87.1) | | 250 | before | 658 | 300,611 | 63.3 (58.1–89.6) | 78.1 (64.8–103.3) | | 250 | after | 17 | 17,171 | 20.8 (18.9–29.2) | 18.7 (14.1–27.3) | | 1000 | before | 2,608 | 1,179,710 | 256.0 (221.0–363.2) | 303.3 (262.2–507.8) | | 1000 | after | 17 | 21,695 | 54.9 (46.7–81.7) | 51.0 (41.6–72.2) | Of the 658 statements before the fix at 250 waits, 650 are provenance reads: 325 identity-context reads, 250 source-message link reads and 75 source-run reads. After the fix, 9 statements do the same reads. The other 8 statements of the request do not change. Sizes are set with `WAITS=20` and `WAITS=1000`. Other agents used the same machine during the measurement, so the maximum values have noise. - The batch statements use index scans only (`run_identity_contexts_pkey`, `run_identity_contexts_company_run_idx`, `chat_message_links_company_comment_idx`, `chat_conversations_pkey`, `chat_endpoints_company_id_uq`, `heartbeat_runs_pkey`). The 9 batch statements of a request with 250 waits take about 1 ms of server execution in total. - Wall time and CPU time are almost equal before the fix. This shows that the cost is in the server process, not in the database. Not verified: - The effect on the hosted instance. This pull request does not deploy. The share of the main-thread CPU that this path uses there is not known; the profile gives the share of all statement building. ## Risks - The provenance check is an authorization check. The risk is a batch decision that differs from the single decision. Mitigation: both paths run the same walk and build each read from the same definition. The batch read is the single read inside `CROSS JOIN LATERAL`, for each request row. The equivalence test compares both paths in every state. - A database error (for example an invalid UUID in a legacy task binding) rejects the whole batch. The previous `Promise.all` also rejected the whole request. The callers of the batch pass the native task of each run. - The statement count of the batch follows the longest lineage of its runs. A batch with one very long lineage sends as many statements as that run sent before, in sequence. It does not send more. - The batch sends its statements in sequence on one connection. Before, the reads of N runs used up to N pool connections at the same moment. - The single-run callers (GitHub tools, review publication, response comments, Slack authority, the finalizer, the status committer) keep one statement for each read. Their identity-context reads select fewer columns. - No migration. No new index. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
ff8ec41e6a |
feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
34f2bb78ee |
feat(connections): connect Sentry with a long-lived user auth token (#20823)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use app connections to call provider tools, such as the
Sentry MCP server
> - The Sentry connection has only one method: browser sign-in through
the hosted MCP OAuth server
> - That grant expires. An unattended triage agent loses Sentry access
until a person signs in again
> - During that time, a scheduled triage routine completes each run but
reads no Sentry data
> - This pull request adds a second Sentry method that uses a user auth
token
> - The benefit is a Sentry connection that stays valid until a person
revokes the token
## Linked Issues or Issue Description
**Subsystem affected**
App connections: the Sentry entry in the connection catalog.
**Problem or motivation**
A Sentry connection that was created with browser sign-in on 2026-10-11
changed to "OAuth authorization expired. Reconnect this app to
continue." at 04:01 UTC on 2026-11-10. The agent that used it had no
Sentry tools after that time. Each hourly triage run reported the
blocker but could not continue. Unattended agents need a Sentry
connection that does not need a person to sign in again each month.
**Proposed solution**
The hosted Sentry MCP server accepts a Sentry user auth token in the
`Authorization: Sentry-Bearer <token>` header. It reserves `Bearer` for
its own OAuth tokens. Paperclip API-key methods already support a custom
header prefix. Add a Sentry API-key method that uses this header.
**Alternatives considered**
Reconnect with browser sign-in again. That restores access only until
the next grant expiry.
## What Changed
- Add the `mcp-user-token` method ("Use a user auth token") to Sentry in
`scripts/ingest-app-definitions.mjs`. It uses the same server URL and
sends the token as `Authorization: Sentry-Bearer <token>`.
- Regenerate `packages/shared/src/app-definitions/sentry.json`. I did
not include unrelated output that the content corpus drift caused for
other providers.
- Add a permission review entry for the new method in
`doc/connections/tool-method-permission-reviews.json`.
- Add a test for the new method and its header prefix.
- Document the token method, its required scopes, and its ownership risk
in `doc/connections/SENTRY.md`.
## Verification
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts`: 40
passed. The new assertion fails on `master` because Sentry has only one
method there.
- `pnpm exec vitest run src/__tests__/tool-access-service.test.ts -t
Sentry` (server): 2 passed.
- `pnpm exec vitest run ui/src/pages/apps/app-connect-policy.test.ts
server/src/__tests__/satisfied-connection-intents.test.ts`: 64 passed.
- `pnpm verify` returned exit 2 (a doc is a global CI input). Required
CI supplies the broad coverage.
- Live probe of `https://mcp.sentry.dev/mcp` with an `initialize`
request: `Authorization: Sentry-Bearer <invalid token>` starts an MCP
session (the token is checked on tool calls). `Authorization: Bearer
<same token>` returns `401 invalid_token`. This confirms the server
accepts the header scheme.
- Not done yet: a real agent tool call with a real token after
deployment.
## Risks
- Low risk. This adds a method. The existing OAuth method and saved
connections do not change.
- A user auth token acts as its creator and lasts until revoked. The
method guidance and docs say this. Rotate the token when that person
leaves.
- If Sentry changes the `Sentry-Bearer` scheme, connections that use
this method fail on tool calls. The OAuth method still works.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, web search).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
c444da7d05 |
perf(run-events): stop blocking checkpoint FK checks during event appends (#20670)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run sends a stream of runner events. The server appends each event to the run log. The runner controller then saves a checkpoint for recovery. > - Each checkpoint save inserts the new `event:N` row and updates the `cursor` row in `native_runner_checkpoints`. The new row has a foreign key to the run row in `heartbeat_runs`. > - The event append locks the same run row `FOR UPDATE`. That lock blocks the `FOR KEY SHARE` lock that the foreign key check takes. So each checkpoint save waits for the event append that is in progress. > - On a hosted database, the two-row checkpoint upsert had the largest total SQL time. The one-row (cursor-only) upsert, which has no foreign key check, was much cheaper. > - This pull request changes the append lock to `FOR NO KEY UPDATE`. Appenders still serialize. Child foreign key checks no longer wait. > - The benefit is shorter checkpoint saves, shorter waits before each runner ACK, and less lock wait on the database. ## Linked Issues or Issue Description **What existing behavior does this improve?** The controller checkpoint save for native runs (`nativeRunnerCheckpointWriter`), and the native run-event append (`appendHeartbeatRunEvent`). **Subsystem affected** Server run-event log and native runtime checkpoint persistence. **Current behavior** On one hosted database, Neon query performance showed this statement as the largest total: `insert into native_runner_checkpoints ... values (...), (...) on conflict (run_id, part) do update`. It ran 6,582 times at 62.6 ms (412 s) in the first 38 minutes after a compute restart, and 9,435 times at 31.8 ms (300 s) about two hours later. An earlier day showed 100,650 calls at 102 ms. The one-row form of the same upsert ran 965 times at 13.4 ms. The runner already writes only changed records (`captureCheckpoint` compares plaintext), so unchanged material is not rewritten. The two-row form is a new event row plus the cursor. Only the new row fires the foreign key check. `appendHeartbeatRunEvent` locks the run row `FOR UPDATE` for each native event, in a transaction with several statements. `FOR UPDATE` conflicts with `FOR KEY SHARE`, so the check waits for that transaction. **Proposed behavior** `appendHeartbeatRunEvent` locks the run row `FOR NO KEY UPDATE`. This mode still conflicts with other appenders, `FOR SHARE`, `FOR UPDATE`, deletes and updates. It does not conflict with `FOR KEY SHARE`. The append never changes key columns: `runUpdate` is limited to process fields, and the native and retry-exhaustion paths reject it. The fast path already uses a plain `UPDATE`, which takes the same lock mode. The execution-admission adapter already locks this row `FOR NO KEY UPDATE`. **Reason and benefit** The wait goes in both directions. A checkpoint save waits for an append. An append also waits for an open checkpoint transaction that holds `FOR KEY SHARE`. Both waits stop. The controller sends each ACK only after its checkpoint commits, so shorter saves also shorten the event loop. ## What Changed - `server/src/services/heartbeat-run-events.ts`: lock the run row `FOR NO KEY UPDATE` instead of `FOR UPDATE` in the transactional append path. Add a comment that names the contract. - `heartbeat-run-event-atomic-append.test.ts`: add a regression test. An append transaction stays open. A checkpoint row insert for the same run must finish under a 2 s `lock_timeout`. A second appender must still wait. After release, the sequence is 1, 2 with no gap and no duplicate. The existing test with 40 concurrent appenders on two connections still proves contiguous sequence numbers. Update the existing lock-statement assertion to the new clause. - Add `scripts/benchmark-run-event-checkpoint-lock.ts`. It replays a native event stream with the real append and the real checkpoint writer on separate connections. It times each checkpoint save for a baseline append module and the current one. ## Verification - Red: with the old `FOR UPDATE` clause, the new test fails with `canceling statement due to lock timeout`. With the change it passes. - `npx vitest run src/__tests__/heartbeat-run-event-atomic-append.test.ts src/__tests__/heartbeat-run-event-sequencing.test.ts src/services/native-runtime/native-runner-checkpoint-authority.integration.test.ts src/__tests__/native-recovery-notice-work.test.ts` (server): 4 files, 42 tests passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - `pnpm verify` exits 2, because a new script under `scripts/` is a global CI input. Required CI supplies the broad coverage. Local benchmark. Embedded PostgreSQL 18.1 on a laptop. 400 events per pass, 2 KiB event payload, 6 KiB header, real encryption. Order: warm-up, baseline, current, baseline, current. `HOLD_MS` keeps each append transaction open longer. It stands in for hosted round trips. | Hold | Pass | Median save | p90 | Max | Saves > 2 ms | Total save time | | --- | --- | --- | --- | --- | --- | --- | | 5 ms | Baseline (1) | 8.16 ms | 9.50 ms | 129.48 ms | 396 / 400 | 3,551 ms | | 5 ms | Current (2) | 1.44 ms | 2.08 ms | 12.66 ms | 44 / 400 | 677 ms | | 5 ms | Baseline (3) | 8.12 ms | 9.64 ms | 22.08 ms | 398 / 400 | 3,453 ms | | 5 ms | Current (4) | 1.50 ms | 2.37 ms | 10.54 ms | 80 / 400 | 700 ms | | 0 ms | Baseline (1) | 1.71 ms | 2.50 ms | 11.05 ms | 105 / 400 | 780 ms | | 0 ms | Current (2) | 0.97 ms | 1.42 ms | 10.28 ms | 22 / 400 | 458 ms | | 0 ms | Baseline (3) | 1.49 ms | 2.02 ms | 6.45 ms | 41 / 400 | 667 ms | | 0 ms | Current (4) | 0.91 ms | 1.24 ms | 9.80 ms | 19 / 400 | 420 ms | - Event order: the controller frame chain awaits each event's append and its local commit, but not its checkpoint save. The ACK goes through `#sendJson`, which queues the save and does not block the next frame. Only semantic inputs and command results await the save. So the save for event N overlaps the append of event N+1. Streaming events (`item.delta`) also go through the same locked append, with no checkpoint save, so a save can also overlap them. The benchmark uses this order. - Control without overlap (`SERIAL=1`, each save finishes before the next append, 5 ms hold): baseline median 1.15 / 1.07 ms, current 1.03 / 1.00 ms. There is no material difference. The gain comes only from the removed lock wait. - Results are the same in every pass. The script asserts that all event rows, the header and the cursor are stored, and that the run-log sequence is contiguous. - Command: see the script header. It needs a baseline copy of `heartbeat-run-events.ts` from the base revision. ## Risks - Low risk. The SQL change is one lock clause. Appenders still serialize with each other, with `FOR UPDATE` and `FOR SHARE` lockers, and with updates and deletes of the run row. Run deletion and key changes take `FOR UPDATE`, so they still wait for an append, and an append still waits for them. - The only new concurrency is with `FOR KEY SHARE` lockers. In this code base those are foreign key checks of child inserts. The one explicit `FOR KEY SHARE` read is on `issues`, not on `heartbeat_runs`. A foreign key check only needs the run row and its key to exist. The append does not delete the row or change its key. - The hosted cause is inferred. Neon shows calls and mean time only. We did not read lock waits on the hosted database. The local proof shows the mechanism and the size of the effect. A new query-performance window after deployment must confirm the result. - Persisted run state: no saved run, session, event or checkpoint shape changes. State saved by the previous release carries forward unchanged. The lock mode applies only inside one transaction, so a deploy during a run needs no special case. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81c52e3666 |
perf(chat): find terminal GitHub responses by issue run indexes (#20671)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel posts a terminal response when an agent run for a GitHub request stops. A background sweep finds acknowledged response comments (`processed`, not final) whose issue is blocked or whose latest run is terminal > - For each candidate, the sweep reads the status of the latest run of the assigned agent for the issue. It matches runs with `coalesce(native_issue_id::text, context_snapshot->>'issueId')` > - No index has that expression. So PostgreSQL reads every run of the agent for each candidate, and it decompresses the large legacy `context_snapshot` of each run > - On a hosted test database, this statement ran 13 times at a mean of 11 s in a 38-minute window (149 s in total), with single runs of 16 s and 41 s > - This pull request splits the run match into its two disjoint branches. Each branch uses an existing issue index > - The benefit is that each candidate reads a few index entries instead of the agent's full run history. The returned rows do not change, and no migration is necessary ## Linked Issues or Issue Description **What existing behavior does this improve?** The discovery page query in `registerGitHubTerminalResponseWork` in `server/src/services/chat-channels/github/terminal-response-work.ts`. The query runs when the sweep reconstructs its candidates, and once or twice for each candidate check. **Subsystem affected** GitHub chat channel terminal-response recovery. **Current behavior** The `WHERE` clause has `issues.status = 'blocked' or (select r.status from heartbeat_runs r where r.company_id = ... and r.agent_id = ... and coalesce(r.native_issue_id::text, r.context_snapshot->>'issueId') = <issue id> order by r.created_at desc, r.id desc limit 1) in (<terminal statuses>)`. The planner can only use `heartbeat_runs_company_agent_started_idx`. It reads all runs of the agent, then evaluates the `coalesce` on each row. That decompresses the context of each legacy run. The cost is the number of candidates times the number of agent runs. **Proposed behavior** The subquery takes the latest run from two branches and then the latest of the two: - runs with `native_issue_id = <issue id>`, through `heartbeat_runs_company_native_issue_contract_id_uq` - runs with `native_issue_id is null and context_snapshot->>'issueId' = <issue id>`, through `heartbeat_runs_company_ctx_issue_created_idx` The two branches are disjoint, and together they are exactly the rows where the `coalesce` is equal to the issue ID. Both branches and the outer query use the same order (`created_at desc, id desc`). So the selected run is the same, including ties. **Reason and benefit** This statement was the largest single-statement cost in the hosted window after the earlier database fixes. A local reproduction is about 490x faster on the default fixture, and the gain grows with the agent's run history. **Breaking changes** None. The API, the candidate rules and the results do not change. ## What Changed - Rewrite the latest-run status subquery in the GitHub terminal-response discovery page as two index-backed branches (native binding, and legacy context without a native binding). - Add two integration tests for the sweep. In the first, newer runs of the same agent that do not bind the issue (one with a native binding to another issue, but a legacy context that names this issue) must not hide the stopped run. In the second, a newer active run with a native binding to the issue keeps the request waiting. - Add `scripts/benchmark-github-terminal-response-sweep.ts`. It captures the SQL that the real worker generates at a baseline commit and at `HEAD`, compares all rows, and measures both. ## Verification Cause and thesis. The cost comes from the `coalesce` run match, which has no index and makes each candidate scan the agent's run history. Thesis: if each precedence branch uses its own issue index, the cost of the page no longer depends on the agent's run count, and the rows stay identical. The benchmark confirms the thesis. Local SQL benchmark: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-terminal-response-sweep.ts --baseline-ref=7b33384446 ``` Fixture (synthetic): two companies, each with one GitHub endpoint and one assigned agent. Each issue has one GitHub response comment. The first N comments are acknowledged and not final; the others are final. Every tenth issue is blocked. Runs alternate between native binding and legacy context, and each run has a 4 KiB incompressible context. The fixture also has these edge cases: newer active runs with a native binding to another issue but a legacy context that names this issue, active runs that tie on `created_at` with the latest run (between native and legacy runs), and newer active runs of another agent. The benchmark compares every column of every candidate row (the page limit is lifted for this check). Embedded PostgreSQL 18.1 on an Apple laptop. The hosted database is Neon with different compute, so absolute times are local SQL measurements only. Each size has 2 warmups and then alternates the baseline and the treatment. | Size | Runs (both companies) | Candidate rows equal | Baseline median (range) | Treatment median (range) | Speedup | | --- | --- | --- | --- | --- | --- | | Small: 200 issues, 3 runs each, 20 candidates, 7 samples | 1,372 | 26 of 26 | 18.7 ms (17.7–20.8) | 3.8 ms (3.3–4.3) | 4.9x | | Default: 1,500 issues, 8 runs each, 150 candidates, 7 samples | 25,300 (156 MB) | 200 of 200 | 3,697 ms (2,875–4,032) | 7.6 ms (6.6–9.6) | 490x | | Large: 3,000 issues, 10 runs each, 400 candidates, 5 samples | 62,600 (382 MB) | 538 of 538 | 57,055 ms (48,343–66,079) | 31 ms (16–79) | 1,800x | Sizes are set with `BENCH_ISSUES`, `BENCH_RUNS_PER_ISSUE`, `BENCH_CANDIDATES` and `BENCH_ITERATIONS`. The timed query is the real first page (limit 100). Plan (default size): the baseline runs `SubPlan` 270 times. Each run is a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that removes about 12,000 rows by filter, for about 8.2 million buffer reads in total. The treatment runs an `Append` of two `Limit` index scans (`heartbeat_runs_company_native_issue_contract_id_uq` and `heartbeat_runs_company_ctx_issue_created_idx`), for about 3,800 buffer reads in total. Tests: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "terminal GitHub"`: 13 passed, including the 2 new tests. - Mutation check: when the legacy branch does not exclude runs with a native binding, the first new test fails. It passes with the fix. - `cd server && npx tsc --noEmit`: passed. ## Risks - Low risk. The change is one SQL subquery. The match is the same set of runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id = <uuid>`, and a null `native_issue_id` makes the `coalesce` use the legacy context. - The candidate set itself is not bounded by this change. Acknowledged responses that never become final stay in the candidate set. Each candidate is now cheap, but the count still grows with that history. - `readGitHubResponseRun` in `response-run-state.ts` uses the same `coalesce` match. The sweep calls it once for each candidate row, and other GitHub paths also call it. Each call is a separate, smaller statement. This pull request does not change it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8148fc0b1e |
fix(release): fetch draft-notes PR summaries in batches (#20602)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release workflow drafts the next stable's release notes when a beta publishes (`draft_stable_notes`) > - The draft nests each referenced PR's summary under its commit subject, and it fetches each summary with its own `gh pr view` call > - A beta range now references more than 1,500 PRs, so these serial calls do not finish inside the job's 10-minute timeout > - This pull request fetches the summaries in batches of 50 per GraphQL query > - The benefit is that the job finishes in under a minute and pushes the notes branch again ## Linked Issues or Issue Description **What happened?** The `draft_stable_notes` job of [run 42763789753](https://github.com/paperclipai/paperclip/actions/runs/42763789753) (beta `2026.1110.0-beta.0`) reached its 10-minute timeout. GitHub cancelled it during "Draft stable notes from the published beta", while a `gh` process was still running. The beta itself published and its smoke tests passed. The `release-notes/v2026.1110.0-beta.0` branch was not pushed. **Expected behavior** The job writes the notes draft and pushes the branch for every beta, whatever the size of the commit range. **Steps to reproduce** 1. Publish a beta whose range from the last stable has many PR-referencing commits. This range has 1,586 commits, and 1,583 different PRs. 2. `draft-stable-notes.sh` calls `gh pr view <n>` once for each commit, one after another. 3. The job reaches `timeout-minutes: 10` before the loop ends. **Paperclip version or commit** `09e24dd12c36251dd12af2d73dbc715ce990f922` (beta `2026.1110.0-beta.0`). This was the first beta to run the job after #12124 added the PR summaries; earlier betas skipped it. ## What Changed - `scripts/draft-stable-notes.sh`: - Collects the unique PR numbers first. Fetches their bodies 50 per `gh api graphql` query (aliased `pullRequest(number:)` fields), into a temporary cache that `enrich_pr` reads. - Parses the raw response with `jq`, not with `gh --jq`. When any number in a batch is not a PR (for example, an issue reference), gh exits non-zero and does not apply its `--jq` filter. The other PRs in the batch still have usable data. - The output format does not change. A missing `gh` or `jq`, a failed batch, a cache directory that `mktemp` cannot create, or `DRAFT_NOTES_SKIP_PR_ENRICHMENT=1` still gives bare subjects. - No code comments are added (per `CONTRIBUTING.md`); this description holds the explanation. - `scripts/draft-stable-notes.test.mjs`: - The enrichment tests now use a fake `gh` that answers GraphQL like GitHub. - New tests: batching (120 PRs make 3 queries, and a repeated PR is fetched once); a batch with a non-PR number; a failed batch; a failed `mktemp`; prose-only and empty bodies; the opt-out. - Range tests now run with enrichment disabled, so an authenticated `gh` on the test machine cannot reach GitHub. - `package.json`: adds the test file to `test:release-registry`. These tests did not run in CI before. ## Verification - `node --test scripts/draft-stable-notes.test.mjs`: 17 pass. They also pass with macOS bash 3.2 first on `PATH`. - `node scripts/ci-profile/node-test.mjs scripts/draft-stable-notes.test.mjs scripts/release-lib.test.mjs`: 31 pass. - Red check: the new tests against the old script fail 5 cases (all batching and enrichment cases). The `mktemp` test fails against an unguarded `mktemp` call. - Live run on the real range: `./scripts/draft-stable-notes.sh 2026.1110.0-beta.0` finished in 47 seconds. It wrote 4,596 excerpt lines for 1,583 PRs. - Parity: for 40 randomly sampled PRs from that range, the new excerpts are byte-identical to the old per-PR `gh pr view` output. - `shellcheck`: no new findings. The two remaining notices were there before this change. ## Risks - Low. The change touches only the post-publish draft job, and every failure still gives the bare-subject draft. - `jq` is now needed for the PR summaries. GitHub's Ubuntu runners include it. Without `jq`, the draft has no summaries, the same as without `gh`. - I did not change `release.yml`. The job already checks out `master`, so after this merges, re-running the failed job of run 42763789753 should push the missing notes branch with the fixed script. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use (shell, file editing, test runs, read-only GitHub API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3bf7af6c77 |
perf(plugins): read the lifecycle inbox from per-resource cursors (#20576)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins get agent, project and agent runtime changes from the `resource_lifecycle_events` journal. Each plugin acknowledges events one resource at a time. > - #20405 made the inbox page read linear in the company journal size. The journal has no retention, so each page still reads every event of the company to find the unacknowledged ones: about 1.7 s per 1M events. > - The delivery worker reads a page again after each acknowledgment. The idle-sleep check also reads the whole journal when all events are acknowledged. `acknowledge` checks every older event of the resource. > - On a hosted instance, the page read ran 181 times in one window at about 1 s each. > - This pull request adds a per-plugin, per-resource cursor. Reads start at the cursor, so they do not read acknowledged history again. > - The benefit is that page, idle-check and acknowledgment cost follow the number of resources and pending events, not the journal size. Delivery order and results stay the same. ## Linked Issues or Issue Description Refs #20405 **What happened?** The plugin lifecycle inbox page query (`select ... from "resource_lifecycle_events" ... DISTINCT ON ...`) ran 181 times at about 1 s each on a hosted database in one window. Each acknowledgment causes another full read of the company journal. The idle-sleep check (`hasPending`) reads the full journal when all events are acknowledged. `acknowledge` reads the full history of the resource. **Expected behavior** A page read, the idle-sleep check and an acknowledgment must not read acknowledged history again. Their cost must not grow with the journal size. **Steps to reproduce** 1. Give a company 400 agents with 2,500 `agent_runtime` events each (1M events). 2. Acknowledge the first 2,000 events of each agent for one plugin. 3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100, undefined, ["agent", "project", "agent_runtime"])`. 4. The read takes about 1.7 s. `hasPending` takes about 240 ms. **Paperclip version or commit** `8d2bf78323` (master). **Deployment mode** Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18. ## What Changed - `packages/db/src/schema/plugin_lifecycle_resource_cursors.ts`: new table `plugin_lifecycle_resource_cursors`. It has one row per plugin, company, resource type and resource id. `acknowledged_through_id` means: every event of that resource at or below this id is acknowledged by the plugin. Plugin and company deletion cascade. - Migration `0462_plugin_lifecycle_resource_cursors`: creates the table and carries existing acknowledgments forward (see Risks for the previous-release story). - `server/src/services/plugin-lifecycle-inbox.ts`: - `list` and `hasPending` use a backward loose index scan on `resource_lifecycle_events_resource_order_idx`. It visits each resource stream once and gets its latest event id. A stream whose latest event is at or below its cursor is settled and is not read again. For other streams, the read gets one head: the unacknowledged creation, else the first unacknowledged transition above the cursor. Then it applies `afterId`, id order and the limit, as before. - `acknowledge` reads only the events between the cursor and the acknowledged event. It still writes the per-event `plugin_lifecycle_acknowledgments` row. It moves the cursor only over a contiguous acknowledged prefix. A creation recorded after a pending transition does not move the cursor. - An event counts as acknowledged when it is at or below the cursor, or when it has an acknowledgment row. Thus rows written by an earlier release, and gaps in old history, keep their current behavior. - `server/src/__tests__/resource-lifecycle-events.test.ts`: new database tests for the migration carry-forward, gaps, late creations, cursor reads, the rollback rows and the plugin-delete cascade. All tests from #20405 still pass without change. - `scripts/benchmark-plugin-lifecycle-inbox.ts`: runs the migration backfill on the fixture and times it. Adds an acknowledgment case. Runs `VACUUM ANALYZE`, because hosted tables are autovacuumed. - `doc/DATABASE.md`: describes the cursor, its read cost and the migration. - The delivery worker (`plugin-lifecycle-delivery.ts`) is not changed. ## Verification Thesis: the cost is a read of the whole company journal (page, idle check) or the whole resource history (acknowledgment). If a per-resource cursor marks the acknowledged prefix, each read can skip that prefix. Then the cost follows the number of resource streams and pending events. The rows must stay equal. Local SQL timings: embedded PostgreSQL 18.1 on Apple Silicon, synthetic data, warm cache, one warmup, 7 samples, baseline and treatment order alternate per sample. The machine was shared with other jobs (load average about 35), so spread is wide. Values are median (min to max) in ms. Rows are equal (`assert.deepEqual`) for every list and probe case. Fixture A: 400 agents x 2,500 runtime events, 2,000 acknowledged per agent. 1,020,440 events, 800,420 acknowledgments. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,700 (1,633-2,438) | 11.5 (9.5-12.2) | 148x | | page of 100, agent and project | 3.3 (2.5-5.4) | 4.2 (3.6-6.7) | 0.8x (slower) | | idle-sleep check | 238 (174-367) | 2.7 (2.1-3.4) | 88x | | acknowledge one head | 15.3 (9.4-30.1) | 20.3 (12.9-29.9) | neutral (noise) | | migration backfill | - | 9.3 s | - | Fixture B: 40 agents x 25,000 runtime events, 24,000 acknowledged per agent (long history). 1,020,080 events. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,943 (1,642-4,979) | 4.4 (2.9-5.6) | 444x | | page of 100, agent and project | 3.5 (2.1-5.7) | 4.6 (2.8-12.0) | 0.8x (slower) | | idle-sleep check | 234 (220-243) | 2.8 (1.9-6.5) | 85x | | acknowledge one head | 64.6 (48.5-71.6) | 5.0 (2.6-7.1) | 13x | | migration backfill | - | 4.6 s | - | Fixture C: 5,000 agents x 200 runtime events, 190 acknowledged per agent (many resources). 1,025,040 events. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,867 (1,649-1,975) | 76.9 (66.2-497) | 24x | | page of 100, agent and project | 10.3 (8.7-12.4) | 21.6 (20.2-23.2) | 0.5x (slower) | | idle-sleep check | 220 (201-298) | 2.8 (2.0-5.3) | 78x | | acknowledge one head | 4.9 (3.4-8.6) | 6.0 (5.2-7.6) | neutral | | migration backfill | - | 15.6 s | - | `EXPLAIN (ANALYZE, BUFFERS)` for the all-types page on fixture A: before 2,488 ms, 13,597 shared hits plus temp files; after 7.0 ms, 7,926 shared hits. Commands: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=400 --per-agent=2500 --acked=2000 --samples=7 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=40 --per-agent=25000 --acked=24000 --samples=7 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=5000 --per-agent=200 --acked=190 --samples=7 cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/project-instance-secrets.test.ts src/__tests__/agent-runtime-hooks.test.ts pnpm --filter @paperclipai/db check:migrations ``` - `resource-lifecycle-events.test.ts`, `plugin-lifecycle-delivery.test.ts` and `project-instance-secrets.test.ts`: 86 passed. `agent-runtime-hooks.test.ts`: 22 passed. - `tsc --noEmit -p server`: pass. `check:migrations`: pass. ## Risks - **Schema change.** Migration `0462` adds one table and two foreign keys to `plugins` and `companies`. It adds no index on an existing table. The backfill reads the acknowledgment table once. It took 4.6 s to 15.6 s on 1M-event fixtures with 0.8M to 1M acknowledgments. - **Migration order.** The number `0462` is reserved after open PRs that use `0459` to `0461` (#20571, #20568, #20573). Merge this PR after them. Before merge, rebase onto master and chain the snapshot after the latest migration. - **Previous-release story.** - Upgrade: the backfill stores, per plugin and resource, the end of the contiguous acknowledged prefix. A cursor never passes an unacknowledged event, so no event is lost. Acknowledged events above a gap keep their rows, so no event is delivered again. A test runs the migration statement on rows that only an earlier release wrote and checks that `list` returns the same rows before and after. - Rolling deploy: an earlier release that still runs writes only acknowledgment rows. The new reads honor those rows. - Rollback: the new code still writes every acknowledgment row, so an earlier release sees the same progress. - **Ordering assumption.** The cursor is safe because writers record a resource's events while they hold that resource's row lock (`paperclip_record_agent_runtime` locks the agent row; agent status and project writers document the lock). Thus a later commit for the same resource always has a higher id. A writer that breaks this contract could commit an event below the cursor, and that event would not be delivered. A lower id from a different resource is still delivered (existing test). - **Slower path.** A page that asks only for `agent` and `project` streams is 1 to 11 ms slower. These streams are short, so the old full read was cheap, and the new read visits each stream once. The cost is bounded by the number of resources. - **Many resources.** The page cost grows with the number of resource streams of the requested types. At 5,000 agents a page is about 77 ms, compared with about 1.9 s before. - The hosted planner can choose a different plan. The hosted improvement is not yet measured. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8e4c5e81b9 |
perf(issues): read created-from tasks by index and inbox review waits by page (#20573)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task page shows "Created from this task". The inbox shows blocked-inbox attention for a page of tasks. > - On a hosted database, the created-from list ran about 4,000 times in one afternoon at about 499 ms each. The task page sends it each time a task opens. > - Its filter was an `OR` of three provenance paths over the list's own rows. PostgreSQL could not use an index for it. It read every task of the company and tested each one, with the list's other per-row conditions (such as the chat conversation state subquery). > - The blocked-inbox attention map also read external review waits for every in-review task of the company, not only for the tasks of the page. > - This pull request reads created-from tasks from three index probes, and reads review waits only for the in-review tasks that can change a finding of the page. > - The benefit is less database work on each task open and each inbox page. The responses stay the same. ## Linked Issues or Issue Description Refs #20400 (shares review-wait reads between the attention maps of one request). Refs #20551 (authorizes the review waits of a page with one read; this change narrows the waits that the page asks for, and adds a test that pins the batched read). Refs #13241 (adds the created-from list). **What happened** - `GET /api/companies/:id/issues?createdFromIssueId=…` used `createdFromIssueCondition`. Its filter was `origin_issue_id = source OR (origin_issue_id IS NULL AND origin_run_id IN source runs) OR (… AND EXISTS creation activity of a source run)`. The `OR` stops index use, so the statement read every task of the company. The list's other conditions, which include the chat conversation state subquery, ran for each task before the task was excluded. - `listIssueBlockedInboxAttentionMap` read review waits for every in-review task of the company graph. The caller uses the findings of the page rows only. **Expected behavior** The created-from list reads only the created tasks. The blocked-inbox map reads review waits only for in-review tasks that the liveness classifier can read for a page row. The rows, the order, and the attention values do not change. **Steps to reproduce** 1. Use a company with tens of thousands of tasks, runs, and activity rows. 2. Open a task page. Look at the created-from list statement: it reads the full task set of the company. 3. Open an inbox page with blocked-inbox attention. Look at the review-wait read: it gets the ID of every in-review task of the company. ## What Changed - `server/src/services/issue-creation-origin.ts`: `createdFromIssueCondition` is now `issues.id <> source AND issues.id IN (three branches)`. Each branch starts from an index: - recorded origin task: `issues_company_origin_issue_idx`; - legacy origin run: the source task's runs (`heartbeat_runs_company_origin_issue_idx`), then the new partial index on `issues (company_id, origin_run_id)`; - creation activity: `activity_log_run_id_idx` for the source runs, then the task by primary key. The precedence does not change: a recorded origin task wins, the run path applies only without one, and the activity path applies only without both. The activity path matches only an `entity_id` that is the canonical text of the task ID, as before (it also compares `id::text = entity_id`). - `packages/db`: add `issues_company_legacy_origin_run_idx` on `issues (company_id, origin_run_id) WHERE origin_issue_id IS NULL AND origin_run_id IS NOT NULL`. Migration `0461` creates it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` and is registered with the other concurrent migrations. `doc/DATABASE.md` documents the index and the benchmark command. - `server/src/services/issues.ts`: add `blockedInboxReviewWaitIssueIds`. It returns the in-review tasks in the blocker closure of the page rows. The blocked-inbox map reads review waits only for these tasks. The liveness classifier reads a wait only for a row or for a blocker that it reaches from a row, and the caller reads only the findings of the rows. The classifier does not change. - Tests: - `issue-created-from-routes.test.ts`: new cases for precedence in both directions, a source-run comment, an activity row whose task has another origin run, a non-canonical and a non-UUID activity entity ID, the source's own creation activity, and other-company rows. - `blocked-inbox-review-wait-scope.test.ts`: the narrowed set for chains, cycles, shared blockers and off-page tasks. The rows get the same findings with the narrowed waits as with the waits of every in-review task, and each narrowed wait is needed. - `issue-list-review-wait-authorization.integration.test.ts`: on real PostgreSQL with two authorized waits, `listExternalReviewWaits` and one issue-list request with blocked-inbox attention each send one bound-run message-link statement. With the per-wait authorization from before #20551, both tests fail (2 statements). - `scripts/benchmark-created-from-issues.ts`: local benchmark. It loads the baseline `issues.ts` and `issue-creation-origin.ts` from a pinned commit, asserts identical complete row payloads, and reports alternating timings and `EXPLAIN (ANALYZE, BUFFERS)`. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-created-from-routes.test.ts src/__tests__/blocked-inbox-review-wait-scope.test.ts src/__tests__/issue-list-review-wait-authorization.integration.test.ts src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/issue-blocker-attention.test.ts src/__tests__/issue-list-assignee-filter-routes.test.ts src/__tests__/issues-parent-id-alias.test.ts src/__tests__/batched-chat-authorization.integration.test.ts src/__tests__/issue-liveness.test.ts` passes (126 tests). One timing test in `issue-list-assignee-filter-routes` failed one time under machine load and passed on rerun. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/db check:migrations` pass. - `pnpm verify` asks for broader verification because the change adds a file under `scripts/`. Required CI supplies that coverage. - Benchmark: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-created-from-issues.ts --baseline-ref=bd30ef8510` Fixture (synthetic, local PostgreSQL 18.1): one company with 30,000 tasks (10,628 with a recorded origin task, 10,445 with only a legacy origin run), 60,030 runs with wide context snapshots (145 MB), and 261,079 activity rows; a second company with 10,000 tasks and 20,000 runs. The busy source task has 63 created tasks through all three paths, plus decoys. The quiet source task has 3. The UI's page: `sortField=id`, `asc`, `limit=500`, routine executions included. Both versions return identical complete payloads. 9 alternating pairs after 2 warm-up pairs: | Case | List statement before | List statement after | List call before, median (min to max) | List call after, median (min to max) | | --- | --- | --- | --- | --- | | Busy source (63 tasks) | 234.9 ms, 47,437 buffers | 0.90 ms, 437 buffers | 156.0 ms (114.5 to 480.7) | 31.6 ms (19.9 to 62.7) | | Quiet source (3 tasks) | 90.7 ms, 47,413 buffers | 0.25 ms, 46 buffers | 126.1 ms (90.6 to 300.4) | 10.5 ms (7.2 to 26.1) | The list call time includes the enrichment reads (labels, runs, attention), which do not change. Inbox page (50 open tasks, blocked-inbox attention): the review-wait read goes from 2 statements with 1,215 parameters (every in-review task of the company) to 1 statement with 31 parameters. The page time does not change in this fixture (median 801 ms before, 873 ms after, both with a wide spread), because the fixture has no review waits and the other graph reads of the map are most of the time. With real waits, the server does an inherited GitHub review read for each wait it finds, so the gain grows with the number of off-page waits. Migration timing (local PostgreSQL 18.1): `0461` (measured as the same statement under its earlier number) on 500,000 tasks (846 MB, 20 companies, one third with a legacy origin run) takes 1.6 s. The index is 12 MB. ## Risks - The new index adds a small write cost to tasks with only an origin run. New tasks with a source task record `origin_issue_id`, so the index holds mostly older tasks. The migration does not lock writes. - The created-from rewrite must keep the same rows. The new tests and the benchmark payload check cover precedence, company scope, and ID text forms. Visibility and read filters are the list's other conditions and do not change. - The review-wait narrowing depends on the liveness classifier reading waits only for rows and their blocker chains. A unit test compares the findings of the rows with full and narrowed waits. A future classifier change that reads waits of other tasks must update `blockedInboxReviewWaitIssueIds`. The function comment says this. - The local timings are synthetic. Hosted acceptance needs a fresh query-statistics window after deployment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, local PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1805a1d247 |
perf(workspaces): probe workspace run bindings through indexes (#20568)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep checks each Git workspace for recent activity and for live consumers before it expires or moves the workspace > - These checks find the runs of a workspace with one SQL scope, `expirationScope`. The scope used one `OR` over all run bindings, and one branch decoded the context JSON of each run > - PostgreSQL could not use an index for that `OR`. Each check read every run of the company and its large context from TOAST. On a hosted database, each call took 2 to 5 seconds, about 5,200 s in total in one window > - This pull request splits the scope into one index probe per binding and adds one concurrent expression index for the legacy `executionWorkspaceId` context key > - The benefit is that each check reads a few index entries instead of all company runs. The cleanup decisions do not change ## Linked Issues or Issue Description **What existing behavior does this improve?** The workspace activity and cleanup-safety checks: `gitWorkspaceLastActivityAt`, `isGitWorkspaceExpirationEligible`, `workspaceMoveEligible` and the orphan-folder consumer check. All of them use `expirationScope` in `server/src/services/git-workspace-expiration.ts`. **Subsystem affected** Execution workspace cleanup in the server (`git-workspace-expiration.ts`, `isolated-repository-rehome.ts`, `isolated-task-directory-retention.ts` through the shared scope) and the `heartbeat_runs` schema in `packages/db`. **Current behavior** The `relevant_runs` CTE selects company runs where any of these is true: `issue_id` or `native_issue_id` is in the issue tree or the bound issues, the run is a bound issue's checkout or execution run, or the context JSON has a matching `issueId`, `taskId` or `executionWorkspaceId`. The planner does one sequential scan of the company runs and reads each context from TOAST. The cost grows with the run history of the company, not with the size of the workspace. **Proposed behavior** Each binding is a separate index probe. The `UNION` of run IDs goes to a primary-key probe. The SQL gives the same rows. **Reason and benefit** Local SQL measurements show a 34x to 169x decrease in time. The decrease grows with the run history (see Verification). ## What Changed - `expirationScope` builds `relevant_runs` from a `UNION` of index probes, one for each binding. The key sets go in as arrays (`= ANY(ARRAY(...))`). With `IN (subquery)`, the planner chose a hash join over a full scan, because it does not count the cost of reading TOASTed contexts. - The legacy context keys use `context_snapshot ->> 'key'`. This expression matches the existing indexes `heartbeat_runs_company_ctx_issue_created_idx` and `heartbeat_runs_company_ctx_task_created_idx`. For a non-object context, `->>` gives `NULL`, as the strict JSONPath decode did. Such a run can still match through a native binding. - New partial index `heartbeat_runs_company_ctx_execution_workspace_idx` on `(company_id, context_snapshot ->> 'executionWorkspaceId') WHERE context_snapshot ? 'executionWorkspaceId'`. The query repeats the predicate. For an array context, `?` can be true, but `->>` gives `NULL`, so the result does not change. The index is partial so that it serves only this probe. A full index that starts with `company_id` made the planner choose it for an unrelated company-scoped query. Migration 0460 builds it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration is in the list of migrations that run outside a transaction. - New tests: a legacy `issueId` or `executionWorkspaceId` binding on a run that is bound to a different issue still blocks cleanup. A plan test with `enable_seqscan = off` checks that no binding scans `heartbeat_runs` and that the three context indexes are used. - The benchmark script `scripts/benchmark-workspace-activity.ts` gets a `--runs-per-workspace` option. Unrelated fixture runs now have distinct context IDs, as real history does. - `doc/DATABASE.md` describes the probes and the new index. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/git-workspace-activity.test.ts src/__tests__/native-harness-backup-expiration.test.ts`: 132 of 132 tests pass. The second file checks that the native session index stays the planner's choice for its query. These include the existing route matrix (root, child, bound, native, context issue, context task, context workspace, checkout, execution run), exactly-once counts, malformed and uppercase contexts, company isolation, controller leases and lease vetoes. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/db check:migrations` pass. - Benchmark (local SQL measurement, PostgreSQL 18.1, synthetic fixture, 8 workspaces per sample, before and after alternate in each iteration). The script compares the complete ordered caller results and the complete `relevant_runs` multisets of the baseline and treatment. They are equal in every run. `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-activity.ts --baseline-ref=origin/master --runs-per-workspace=<N>` | Fixture | Check | Before median (min–max) | After median (min–max) | Speedup | | --- | --- | --- | --- | --- | | 4,800 runs, 330 MB | last activity | 523 ms (460–598) | 12 ms (8–15) | 44x | | 4,800 runs, 330 MB | eligibility | 452 ms (432–473) | 13 ms (11–15) | 34x | | 4,800 runs, 330 MB | idle eligibility | 465 ms (442–573) | 12 ms (10–13) | 40x | | 24,000 runs, 1.7 GB | last activity | 2,240 ms (2,214–2,454) | 13 ms (11–14) | 169x | | 24,000 runs, 1.7 GB | eligibility | 2,245 ms (2,097–2,377) | 17 ms (14–19) | 133x | | 24,000 runs, 1.7 GB | idle eligibility | 2,196 ms (2,179–2,381) | 15 ms (13–18) | 143x | Before: one `Seq Scan` on `heartbeat_runs`. After: `heartbeat_runs_company_issue_created_idx`, `heartbeat_runs_company_native_issue_contract_id_uq`, the three context indexes and `heartbeat_runs_pkey`. - Index build time: on a synthetic table with 200,000 runs and 8 KiB contexts (1.8 GB), the full form of this index took 3.7 s to build concurrently and used 15 MB. The partial index reads the same rows and holds only runs that have the key. ## Risks - Cleanup safety: a wrong "allowed" result could delete an active workspace. The scope keeps every binding of the old query. The benchmark checks exact equality of the run sets, and the existing tests check each binding route and each veto. - Migration: the index build reads every run context once. It runs concurrently and does not block writes. Until migration 0460 builds the index, the new query is still correct but the `executionWorkspaceId` probe scans. - Plan choice: the array form makes index probes the cheapest plan for a small tree. A very large issue tree gives a large array, but each element is still one index probe. - The sweep frequency does not change. This pull request reduces only the cost of each check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d5c1f48180 |
perf(native): index the backup lease lookup by scope digest (#20571)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native harness backup expiration removes old session backups. Before it removes a backup, it finds the environment leases that reference the backup root > - This lookup reads leases by run ID, by the `metadata.nativeHarnessBackup.sessionScopeSha256` digest, or by the identity's own lease ID, in one `OR` query > - The metadata arm has no index. So PostgreSQL reads every lease of the company, and the metadata of each lease, for each lookup > - On a hosted test database, this statement ran 21,736 times at a mean of 158 ms in one window. After other fixes, it still ran 2,364 times at a mean of 121 ms > - This pull request adds a small partial expression index on the digest > - The benefit is that each lookup reads a few index entries instead of all company leases. The query text and the result do not change ## Linked Issues or Issue Description **What existing behavior does this improve?** The lease lookup in `backupDependencies` in `server/src/services/native-runtime/native-harness-backup-expiration.ts`. `reconcileBackup` calls it for each backup root when the expiration worker loads or checks a root. **Subsystem affected** Native harness backup expiration and the `environment_leases` table schema. **Current behavior** The query is `company_id = $1 and (heartbeat_run_id in (...) or metadata->'nativeHarnessBackup'->>'sessionScopeSha256' = $n or id = $m)`. The run arm and the ID arm have indexes. The digest arm has no index. A `BitmapOr` plan needs an index for every arm, so the planner reads all company leases and evaluates the JSON expression on each row. Lease metadata holds sandbox details, so these reads are large. **Proposed behavior** The partial index `environment_leases_native_backup_scope_idx` keys leases by the digest. It holds only leases that have a backup stamp. Every arm of the `OR` now has an index, and the planner combines three index probes. The SQL and the returned lease IDs do not change. **Reason and benefit** This statement was in the top ten by total database time on a loaded hosted database. A local reproduction shows a query that is more than 6,000x faster and reads 9 buffers instead of about 436,000. **Breaking changes** None. The API, the lease-matching rules and the results do not change. ## What Changed - Add the partial expression index `environment_leases_native_backup_scope_idx` on `(metadata->'nativeHarnessBackup'->>'sessionScopeSha256')`, where that value is not null. - Add migration `0459`. It uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration runner runs it as a top-level statement, the same as migrations 0454–0458. - Add a regression test. It checks the lease-matching rules: run leases, a lease with only the backup stamp, and the identity's own lease match. Another digest, a non-string stamp, a top-level key with the same name, and the same digest in another company do not match. The test also checks that the custom plan and a forced generic plan of the generated SQL use `BitmapOr` with the new index and no sequential scan. - Add `scripts/benchmark-environment-lease-backup-lookup.ts`. It captures the SQL that the real service generates and measures it with and without the index. - Document the index in `doc/DATABASE.md`. ## Verification Cause and thesis. The cost comes from the one `OR` arm without an index, which forces a scan of all company leases and their metadata. Thesis: an index on the digest lets the planner use a `BitmapOr` of three index probes, so the cost no longer depends on the company's lease count. The results stay identical. The benchmark confirms the thesis. I did not change the SQL. With the index, the existing `OR` query already gets the index plan. A `UNION` rewrite would not help without the index, because the digest arm would still scan the company. Local SQL benchmark: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-environment-lease-backup-lookup.ts ``` Fixture (synthetic): one company with 100,000 leases with 4 KiB of incompressible metadata each. 30% of the leases have a backup stamp for another session. A second company has 20,000 leases. The table is 678 MB. The backup root has 3 run leases, 1 lease with only the stamp, and 1 identity lease. The same digest is also in the other company. Embedded PostgreSQL 18 on an Apple laptop, which other work loaded heavily. The hosted database is Neon with different compute, so absolute times are local SQL measurements only. The script measures in the order: no index, index, no index, index. Each measurement has 2 warmups and 11 samples of `EXPLAIN ANALYZE` execution time and of the full `reconcileBackup` call. | Measurement | Without index | With index | | --- | --- | --- | | Lease query execution, run 1 | median 196 ms (138–562 ms) | median 0.010 ms (0.007–0.017 ms) | | Lease query execution, run 2 | median 185 ms (144–295 ms) | median 0.030 ms (0.020–0.045 ms) | | `reconcileBackup` wall time (includes file reads), runs 1 and 2 | median 188 ms and 216 ms | median 1.4 ms and 4.2 ms | | Plan | `Gather > Seq Scan` | `Bitmap Heap Scan > BitmapOr` of `heartbeat_run_idx`, the new index and `pkey` | | Buffers | 365,880 hit + 70,050 read | 9 hit | These numbers are from head `65bd9d7ac6`. The final head only renumbers the migration to 0459 after rebases; the SQL, the index and the service code are the same. Two earlier runs on the same fixture, SQL and index gave 236–425 ms without the index and 0.023–0.033 ms with the index. The machine had heavy load from other work, so the spread without the index is wide. Output check: in all four measurements, the service returned the same 5 lease IDs, and the complete query rows were equal. Index build (`CREATE INDEX CONCURRENTLY` from the migration file): 0.8–2.4 s for the 678 MB table over all runs. Index size: 3.4 MB with 30% of leases stamped. Tests: - `cd server && npx vitest run src/__tests__/native-harness-backup-expiration.test.ts`: 109 passed. The new test failed when I dropped the index before the plan check (confirmed red), and it passes with the index. - `pnpm --filter @paperclipai/db check:migrations`: passed. - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/server typecheck`: passed. - `pnpm verify`: it asks for broader coverage (exit 2) because of the new script. See the CI results for this head. ## Risks - Migration time. A concurrent index build reads the full `environment_leases` table and does not block writes. Locally, the build took 0.8–2.4 s for 678 MB. The time grows linearly with the table size. I did not time it on a production-size clone. I expect a few seconds to tens of seconds on a large instance, which is in the 1–2 minute budget. This estimate is not proven. - If the build stops, PostgreSQL can leave an invalid index. The migration runner repairs invalid concurrent indexes when it runs again, and `IF NOT EXISTS` makes a rerun safe. - The index expression must stay textually equal to the query expression. If someone changes the JSON path, the planner cannot use the index. The new test catches this for custom and generic plans. - The index is partial, so ANALYZE does not collect statistics for the expression. The planner uses the default equality estimate. On the large fixture and in the small test, the planner still selects the index plan. A company with very few leases can still get a sequential scan, which is cheap. - Each lease write with a backup stamp writes one more small index entry. The write cost is small. - This change does not reduce how often the worker calls the lookup. It makes each call cheap. The other digest lookups in backup retention (`native-harness-backup-retention.ts`) and session retention checks can also use the index. - Previous-release story: no persisted state changes. The index is derived from existing lease metadata. - The hosted improvement is not verified until deployment. After deployment, compare a new query-statistics window for this statement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6ed6b0754b |
perf(recovery): probe recovery source runs and stale-continuation runs by key (#20584)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The workspace recovery worker saves the files of failed runs. After each pass, it reads the time of its next deadline (`nextWorkspaceRecoveryAt`) > - That read finds the source run, its finalizer and its leases with `id::text = evidence->>'runId'`. A text comparison cannot use a key. Each exhausted hold read every run, finalizer and lease of its company > - On a hosted staging database this statement used about 4 s per call (21 calls in the sample window). The cost grows with exhausted holds multiplied by the company's run count > - The ownership reconciler's stale-continuation check (`checkStaleDroppedComments`) has the run lookup shape that #20412 fixed in the lost-wake check: one OR across the legacy context and the native column > - This pull request compares the run id as a canonical uuid, and it gives the stale-continuation check the per-binding run probes from #20412 > - The benefit is the same recovery decisions with index-driven work: 11 s to 93 ms for the deadline read on a representative local fixture ## Linked Issues or Issue Description Refs #20412 **What happened?** The workspace recovery deadline read was one of the most expensive statements that remained on a hosted staging database after the fixes in #20393, #20405, #20386, #20412, #20391 and #20400. Query statistics showed about 4 s per call. The statement text starts with `select case when (issue_recovery_actions.…`. The statement evaluates the retained-source fallback and stop-proof conditions for each recovery action. These conditions read the source run (`heartbeat_runs`), its finalizer (`native_run_finalizations`) and its leases (`environment_leases`) with `x.id::text = evidence->>'runId'`. The cast prevents the use of the primary key. PostgreSQL reads all rows of the company for each exhausted hold. The worker's discovery (`retainedWorkspaceCandidates`) uses the same conditions. The stale-continuation check matched a task's runs with `(context_snapshot->>'issueId' = issue OR native_issue_id = issue)`. With the OR, PostgreSQL can join runs only by company. It also hashed every wake request of the company for the stale-wake test. **Expected behavior** Each condition uses a key lookup. The worker and the reconciler make the same decisions for the same saved state. **Steps to reproduce** 1. Seed a company with a deep run history and a few thousand workspace recovery actions. Let a few hundred be exhausted holds that stay active. 2. Call `nextWorkspaceRecoveryAt`. 3. The call takes seconds, and its time grows with holds multiplied by runs. The benchmark below shows this. **Paperclip version or commit** Baseline: `8d2bf78323`. **Deployment mode** Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1. Staging uses hosted PostgreSQL. ## What Changed - `workspace-recovery-fallback-discovery.ts`: the source run, finalizer and lease probes compare `id = canonicalExecutionUuid(evidence->>'runId')`. The helper returns null unless the text is the canonical lowercase uuid form. Thus the probe matches exactly the rows that `id::text = evidence->>'runId'` matched. The helper already exists in `execution-identity.ts`. - `nextWorkspaceRecoveryAt` and worker discovery use these conditions. Their SQL changes only in these probes. - `checkStaleDroppedComments` reads in two stages. The first stage keeps comments that a stale-continuation cancellation dropped and that no open wake carries. The second stage applies the run probes. The run probes use the `taskRuns` helper from #20412, which is now shared by both checks. The stale-wake lookup ends with `offset 0`, so it stays a per-comment index probe. - The final `order by c.created_at limit N`, the projection and the delivery check after the query do not change. - Add `scripts/benchmark-recovery-status-checks.ts`. It loads the changed files from an explicit baseline commit and from the working tree. Before it records time, it compares every changed condition for every recovery action, the complete service results and the positional SQL rows. - Add tests: each run binding in the stale-continuation check (native, legacy, live run on either binding, wrong agent, wrong company, failed run, run outside the comment time, closed by a person, open wake). Also a test that a hold with a non-canonical saved run id is not discovered, as before. - Document both checks and the benchmark in `doc/DATABASE.md`. - No schema change, index, migration, or new startup work. ## Verification - `pnpm exec vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/workspace-recovery-fallback.test.ts src/services/workspace-recovery-stop-proof.test.ts src/__tests__/heartbeat-held-comment-wake.test.ts --maxWorkers=1` (from `server/`). - `pnpm --filter @paperclipai/server typecheck`. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-recovery-status-checks.ts --baseline-ref=8d2bf78323 --samples=7` Benchmark setup: PostgreSQL 18.1 (embedded, local), `shared_buffers` 128 MB, `work_mem` 4 MB. All data is synthetic, with md5-hex filler. Each fixture has two companies (full size and a quarter). Two warmup rounds, then 7 timed samples for each arm, in alternating order. Each sample runs the real service function and asserts the same result. These are local SQL measurements on a shared machine, so the spread is wide. Workspace recovery deadline (`nextWorkspaceRecoveryAt`). About 8% of actions are exhausted holds that stay active. Some actions save a non-canonical (uppercase) run id. Before timing, the benchmark compares six conditions (fallback pending, due time, due; re-arm pending, due time, due) for every action of the fixture. | Fixture | Runs / actions / pending holds | Baseline median (min–max) | Treatment median (min–max) | Shared buffers | | --- | --- | --- | --- | --- | | small | 6,500 / 255 / 36 | 50.9 ms (47–76) | 6.8 ms (6–8) | 19,133 → 1,075 | | representative | 78,750 / 3,813 / 588 | 11,098 ms (8,277–12,790) | 93 ms (35–145) | 3,493,417 → 24,902 | | deep history | 191,250 / 3,813 / 588 | 40,146 ms (31,190–52,521) | 40 ms (36–230) | 13,491,792 → 26,911 | Stale-continuation check (`checkStaleDroppedComments`). The fixture is pessimistic: more than half of recent comments on done tasks have a stale cancelled wake. The table shows statement time from `EXPLAIN ANALYZE` with default settings, and one statement run with `enable_nestloop = off` (the company-only hash plan). The result is checked as equal in both modes. | Fixture | Runs / recent comments / reported | Statement, baseline → treatment | Hash plan, baseline → treatment | | --- | --- | --- | --- | | small | 3,792 / 76 / 24 | 3.2 → 1.5 ms | 5.5 → 3.8 ms | | representative | 75,406 / 1,125 / 200 (limit) | 34 → 26 ms | 66 → 25 ms | | busy hour | 152,031 / 5,625 / 200 (limit) | 328 → 153 ms | 2,302 → 165 ms | - Neutral path: the complete stale-continuation service call stays about the same (busy hour 750 → 655 ms median). The per-comment delivery check after the query (`deliveredQueuedCommentIds`) is most of that time. This change does not touch it. ## Risks - Low risk. Every predicate keeps the same logic. Only the probe shape and the evaluation order change. The benchmark compares each changed condition for every recovery action, including non-canonical saved run ids, and the complete results of both checks. - Previous-release story: no saved state, schema or identity changes. Holds saved by older releases are read with the same rules, so they carry forward unchanged. - Rows with the same comment `created_at` have no defined order, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
765ccd8691 |
perf(chat): index chat-admission checks by company and payload issue id (#20562)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connections save each external message as a `chat_actions` row
before the server admits it as an agent wakeup.
> - The server checks if an issue holds chat input that it did not
admit. Recovery skips such an issue. Manual task start refuses it.
> - On a hosted instance, the recovery read with this check used the
most total database time: 488 calls at about 1 s each. The manual-start
check ran 155 times at about 0.9 s.
> - The check matches `chat_actions.payload->>'issueId'` against the
issue. No index covers this value, so each call reads the full
`chat_actions` table and its payloads.
> - This pull request adds one concurrent partial index for the check.
The check SQL does not change.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The chat-admission check `unadmittedChatWakeupCondition`. Recovery uses
it in the per-issue candidate read. Manual task start uses it in the
held-input check.
**Subsystem affected**
Server chat admission, issue recovery, manual task start, and the
database schema.
**Current behavior**
The check finds `inbound_wakeup` actions by `company_id` and
`payload->>'issueId'`. Two nested checks find `failed_run_retry` actions
and later `inbound_wakeup` actions on the same company and payload issue
ID. `chat_actions` has no index on `company_id` or on the payload issue
ID. Each call reads the full table and reads each payload. On a hosted
instance, the two statements used about 630 s of database time in one
statistics window.
**Proposed behavior**
The check and its nested checks read a small set of rows from one index.
The admission result does not change.
**Reason and benefit**
The hosted database had many concurrent active queries. Request reads
waited behind background work such as this check.
**Breaking changes**
None. The migration adds an index only.
## What Changed
- Migration `0458_chat_admission_issue_index` builds
`chat_actions_company_issue_kind_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. It runs through the existing top-level statement path,
which repairs an invalid index on retry.
- Keys: `(company_id, payload->>'issueId', kind, status)`.
- Partial: `kind IN ('inbound_wakeup', 'failed_run_retry')`.
- One index serves the outer input lookup, the failed-run retry lookup,
and the later-input lookup. All three match on the same company and
payload issue ID.
- The Drizzle schema declares the index. The snapshot chain moves to
`0458` and keeps five snapshots.
- New regression test
`server/src/__tests__/chat-admission-condition.test.ts` pins the
admission semantics on real PostgreSQL:
- Pending and failed input without a receipt holds the issue. Received,
processed and cancelled input does not. Other action kinds do not.
- A skipped receipt, a receipt for another agent, or a receipt in
another company does not admit input. A failed or cancelled run still
counts as admission.
- The issue ID is compared as company-scoped text. Uppercase, array,
null, other-issue and other-company values do not match.
- Only an admitted retry in the same conversation and issue that lists
the action releases skipped input.
- Only later admitted input in the same conversation, issue and session
generation releases a failed input. Pending input stays held.
- The check reads `chat_actions` only through the new index.
- `scripts/benchmark-chat-admission-index.ts` measures both caller
statements with and without the index and fails if the rows differ.
- `scripts/benchmark-chat-admission-index-migration.ts` times the
migration through the normal migration runner.
- `doc/DATABASE.md` documents the index and the benchmark commands.
## Verification
Local SQL measurements on synthetic data. PostgreSQL 18.1 (embedded),
default settings. The fixture has 100,995 `chat_actions` rows (662 MB):
60,000 GitHub actions with 6 KiB incompressible payloads, 2,000 issues
with 20 admitted inbound messages each (3 KiB payloads), and 1,000
failed-run retries. Each case has 2 warmups and 9 samples of `EXPLAIN
ANALYZE` execution time. Runs alternate: no index, index, no index,
index.
| Caller statement | Case | No index, median (range) | Index, median
(range) |
| --- | --- | --- | --- |
| Manual-start held check | quiet issue | 3,093 ms (2,150–3,519) / 3,586
ms (2,375–11,235) | 0.036 ms (0.034–0.052) / 0.040 ms (0.033–0.047) |
| Manual-start held check | admitted input | 3,555 ms (3,310–8,080) /
5,205 ms (3,353–9,525) | 0.049 ms / 0.052 ms |
| Manual-start held check | unadmitted input | 3,002 ms (2,864–3,221) /
4,179 ms (3,200–6,585) | 0.051 ms / 0.044 ms |
| Manual-start held check | failed, later admitted | 3,586 ms / 3,722 ms
| 0.066 ms / 0.054 ms |
| Manual-start held check | admitted retry | 2,733 ms / 4,953 ms | 0.049
ms / 0.053 ms |
| Recovery candidate read | quiet issue | 43 ms (41–48) / 96 ms (44–219)
| 0.047 ms / 0.055 ms |
| Recovery candidate read | admitted input | 203 ms (48–575) / 174 ms
(44–314) | 0.055 ms / 0.058 ms |
| Recovery candidate read | unadmitted input | 66 ms / 144 ms | 0.053 ms
/ 0.054 ms |
- Buffers per call fall from about 40,000 shared hits to 5–23.
- Locally, the recovery read without the index uses a hashed full scan
of `chat_actions`. The held check uses a full scan per call. On the
hosted instance both took about 1 s per call. The local fixture is
smaller than the hosted table.
- Correctness: complete positional rows match across all four runs for
all six cases (quiet, admitted, unadmitted, failed then admitted,
retried, other-company decoy). The script asserts the expected held
result for each case.
- Migration timing: 600,001 `chat_actions` rows (1,759 MB), 300,000 of
them in the index scope. First run through `applyPendingMigrations`: 4.2
s. Rerun after the history row is removed: 0.5 s. Concurrent build
alone: 3.4 s. Index size: 30 MB.
Commands:
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index.ts
OTHER_PAYLOAD_KIB=2 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index-migration.ts
cd server && npx vitest run src/__tests__/chat-admission-condition.test.ts # 6 passed
pnpm --filter @paperclipai/db check:migrations # passed
pnpm --filter @paperclipai/server typecheck # passed
```
## Risks
- The migration builds one index concurrently. It does not block writes.
On the 1.76 GB local fixture it took about 4 s. A larger hosted table
takes longer, but the build does not lock out reads or writes.
- Writes to `inbound_wakeup` and `failed_run_retry` actions maintain one
more index. Status updates on these rows were already not HOT updates,
because other partial indexes use `status`.
- The planner has no expression statistics for a partial index. The new
test pins that the check uses the index on an analyzed table with
unrelated rows.
- The check SQL does not change, so admission behavior does not change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
99edcd28a7 |
perf(attention): read each retention candidate kind in its own bounded branch (#20566)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The attention retention worker archives old "Needs you" items, such as open decisions, pending questions, failed runs and items in review > - To find its work, the worker reads a page of retention candidates. It reads one page per changed scope, and one single-source read for each check of a due item > - On a hosted stack this candidate read ran 2,835 times at about 1 s per call, 4,201 s of database time in one afternoon window > - The read built every candidate of every kind before it applied the keyset limit. Its company and source filters used the `param is null or column = param` form. Under a generic prepared plan these filters cannot use an index. The failed-run branch also matched retry receipts with a bound LIKE parameter, so it could not use the partial index `heartbeat_run_events_retry_exhausted_idx` > - This pull request reads each kind in its own branch, puts every filter into that branch, and limits each branch to one page > - The benefit is a bounded read for each page, and a primary-key read for each single-source check, with the same rows, order and cursors ## Linked Issues or Issue Description **What existing behavior does this improve?** The attention retention candidate scan in `server/src/services/attention-retention.ts`. **Subsystem affected** Server: attention retention background work (database reads only). **Current behavior** Each page reads all candidates of all selected kinds, then sorts them and keeps 100. Optional filters use `($n is null or column = $n)`. A generic prepared plan cannot use an index for these filters. The single-source check compares `id::text`, so it cannot use the primary key. The retry receipt LIKE pattern is a parameter, so the plan cannot use the partial receipt index. **Proposed behavior** Each kind is one branch with its company, source, issue and keyset filters inside it and `limit 100`. Kinds before the cursor are not read. A source id filter uses the uuid primary key. The receipt pattern is a SQL literal that matches the partial index predicate. The page result does not change. Refs #20386 (adds the partial receipt index that this branch now uses). ## What Changed - `server/src/services/attention-retention.ts`: replace the single union scan with `candidateQuery`. Each source kind is a branch that has its own filters, retention join and one-page limit. The outer query still orders by `(source_kind, source_id)` and keeps 100 rows. - Only add a filter when the scope sets it. There is no `param is null or ...` form now. - A single-source read compares `id = $uuid`. It returns no rows for text that is not the canonical lowercase uuid form. The former `id::text = $text` comparison also never matched such text. - Skip the kinds that come before the keyset cursor kind. `ATTENTION_RETENTION_KIND_ORDER` holds the kind order, and a test checks it against the database collation. - The failed-run branch matches `'Bounded retry exhausted%'` as a literal, so it reads `heartbeat_run_events_retry_exhausted_idx`. - Export `listAttentionRetentionCandidates` for the test and the benchmark. The service uses it without a change in behavior. - New test `server/src/__tests__/attention-retention-candidates.test.ts`. It runs on real PostgreSQL and compares every page, row and cursor with the former scan (kept in the test as an oracle). It covers: one company, all companies, one kind, the blocker kind (which ignores retention), an unknown kind, issue scopes (including uppercase and malformed ids), exact source ids (uppercase, braces, no dashes, empty and malformed text return nothing), a `null` source id from a decoded scope, other-company isolation, inactive companies, kept and archived retention, and failed runs without a lifecycle receipt. - New benchmark `scripts/benchmark-attention-retention-candidates.ts`. It loads the baseline service from git, captures the SQL that each version sends for the same scopes, and times both under custom and generic plans. Each round alternates the order. It fails if the rows differ. - No migration. The existing indexes cover each branch: company and status indexes on each source table, the issues company/status index, and the partial receipt index from #20386. ## Verification - `cd server && npx vitest run src/__tests__/attention-retention-candidates.test.ts`: 10 passed. - `cd server && npx vitest run src/__tests__/attention-retention-work.test.ts`: see the note under Risks. - `cd server && npx tsc --noEmit`: pass. - Benchmark (local SQL measurement only): `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-attention-retention-candidates.ts --baseline-ref=8d2bf78323` Embedded PostgreSQL 18.1. Fixture: 12 companies (1 large and 11 at 1/5 size), 38,400 issues, 64,000 runs, 1.29 M run events, 6,400 decisions, 9,600 interactions, 4,800 recovery actions, 7,534 retention rows. Seven samples for each scope, after two warmups, with `plan_cache_mode` forced to custom and then to generic. Rows were equal in every sample. | Plan | Scope | Rows | Before median (min–max) | After median (min–max) | Speedup | EXPLAIN shared blocks before → after | | --- | --- | --- | --- | --- | --- | --- | | generic | single source (check path) | 1 | 4.82 ms (3.81–5.88) | 0.19 ms (0.12–0.66) | 25× | 3,491 → 10 | | generic | company failed runs | 81 | 45.0 ms (40.5–51.7) | 2.9 ms (2.3–4.6) | 15× | 3,752 → 3,752 | | generic | company page 1 | 100 | 39.4 ms (30.1–63.7) | 7.6 ms (5.8–14.3) | 5.2× | 4,396 → 4,719 | | generic | company page 2 | 100 | 33.3 ms (31.2–38.1) | 6.9 ms (5.2–10.3) | 4.8× | 4,237 → 4,712 | | generic | company issue scope | 4 | 11.8 ms (9.8–24.0) | 4.1 ms (3.4–6.4) | 2.9× | 4,243 → 4,225 | | generic | all companies page 1 | 100 | 107.2 ms (100.2–116.0) | 29.3 ms (18.8–64.4) | 3.7× | 13,657 → 14,672 | | custom | single source (check path) | 1 | 2.68 ms (2.01–5.33) | 0.81 ms (0.65–1.50) | 3.3× | 3,491 → 10 | | custom | company failed runs | 81 | 5.7 ms (3.9–15.8) | 4.6 ms (3.2–7.1) | 1.2× | 3,752 → 3,752 | | custom | company page 1 | 100 | 11.7 ms (9.2–13.3) | 10.7 ms (9.4–19.5) | 1.1× (neutral) | 4,396 → 4,719 | | custom | company page 2 | 100 | 11.2 ms (7.6–14.2) | 10.1 ms (8.5–12.0) | 1.1× (neutral) | 4,237 → 4,712 | | custom | company issue scope | 4 | 10.9 ms (9.1–13.3) | 8.4 ms (6.4–10.4) | 1.3× | 4,243 → 4,225 | | custom | all companies page 1 | 100 | 21.3 ms (17.4–25.9) | 24.4 ms (20.3–72.0) | 0.9× (neutral) | 13,688 → 14,672 | - The timing connection uses named prepared statements, as the server does by default. The benchmark checks `pg_prepared_statements` before it collects samples, so the samples reuse the parsed statement. - The driver uses named prepared statements by default, so after five runs PostgreSQL can switch a statement to a generic plan. The generic rows are the relevant case for a busy server. Custom plans were already good for most scopes, and the change is neutral there. - Plan after the change (company page, generic): the failed-run branch reads the partial receipt index (13 blocks), then probes `heartbeat_runs` by primary key. The single-source check reads each primary key and the retention unique index (10 blocks in all). ## Risks - Low behavior risk. The test compares the new pages with the former query on the same data, including cursors in the middle of a kind and at kind boundaries. - Kind skipping depends on the order of kind names. Every kind name is different from the others before its first underscore, so C and linguistic collations give the same order. A test checks the order against the database. - The benchmark fixture is smaller than the hosted database. The hosted cost (about 1 s per call) is larger than the local baseline. I did not measure the hosted improvement. It needs a fresh query-statistics window after deploy. - On this loaded local machine, one full run of `attention-retention-work.test.ts` had 3 tests time out at 30 s (no-run issue finalization, no-run workspace finalization, and interaction create/withdraw). The same 3 tests pass when run alone. CI is the authority for this suite. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
faaae003a9 |
perf(chat): find the conversation head from open delivery work (#20564)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - External chat messages go into a durable delivery queue. Each
conversation drains in provider order.
> - Each drain step reads the head of one conversation. The head query
filters one endpoint's deliveries by state and thread.
> - On a hosted instance, this query ran about 10,000 times at about 300
ms each. The database was busy, and request reads waited behind
background work.
> - PostgreSQL read every processed delivery of the endpoint and its
`normalized_event` value to find a few open rows.
> - This pull request finds the endpoint's open delivery IDs first and
fetches only those rows.
> - The benefit is less background database load. The selected head and
the delivery order do not change.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The inbound conversation head query
(`earliestOpenConversationDelivery`).
**Subsystem affected**
Server chat inbound delivery queue.
**Current behavior**
The query has `endpoint_id = $1 and (state in ('received', 'retry',
'processing') or (state = 'processed' and <pending inbound wake>))`, a
thread filter on `normalized_event`, and a sort on `normalized_event`
keys. No index matches the `or`. PostgreSQL scans all deliveries of the
endpoint. For each row it reads `normalized_event` to test the thread.
It also reads all of `chat_actions` to build the pending-wake set. A
busy endpoint keeps all of its processed history, so the cost grows with
that history.
**Proposed behavior**
The query selects `id in (<open delivery IDs of the endpoint>)`. That
set is a `union all` of two small reads:
- received, retry and processing deliveries of the endpoint, through the
existing `chat_deliveries_work_idx`;
- processed deliveries that an issued or processing `inbound_wakeup`
action of the same endpoint (and company) still owns.
PostgreSQL fetches only those rows by primary key, then applies the same
reaction, thread and order rules.
**Reason and benefit**
The cost no longer grows with processed history. On a local fixture with
60,000 processed deliveries, the head query reads 1,309 shared buffers
instead of 262,630.
**Breaking changes**
None. There is no schema change, migration, API change, timer or poll
change.
Related work: #17971 changed the pending-wake predicate of the same
queue. #20400 added the index for the other chat delivery statement in
the same report (`select state ...
normalized_event->'githubAutomatic'->'context'->>'deliveryId'`). That
statement already uses the #20400 index on master, so this pull request
does not change it.
## What Changed
- `delivery-ledger.ts`: add `openConversationDeliveryIds(endpointId)`.
It returns the endpoint's open delivery IDs. It selects the same set as
the old state predicate for that endpoint.
- `conversation-queue.ts`: the conversation head query uses `id in
(openConversationDeliveryIds(endpointId))` in place of the state `or`.
The thread filter, reaction exclusion, order, limit and lifecycle
dependency read do not change.
- `inbound-queue-discovery.test.ts`: add a real PostgreSQL test. It
compares the new set with the old predicate on three endpoints. It
covers each state, issued and processing wakes, duplicate wakes, wakes
of non-processed deliveries, processed, other-kind and null-delivery
actions, an action on a different endpoint of the same company, and
reactions. It also checks that the real captured head query returns the
oracle's first row.
- `benchmark-inbound-queue-discovery.ts`: add
`--endpoint-history=<rows>` (a busy endpoint with 4 KiB payloads) and
`--only=<fixture>`. Add a work budget: the treatment head query must
visit at most 100 delivery rows on the history fixtures.
- `doc/DATABASE.md`: describe the head query and the benchmark command.
## Verification
Local SQL measurements. PostgreSQL 17.10 (Postgres.app, macOS arm64),
disposable database, synthetic data, default settings with 256 MB shared
buffers. The host was under heavy load from other work, so timings are
noisy. Shared buffers and visited rows are stable. Each arm has two
warmups and five alternating samples (median [min..max]). Baseline:
`origin/master` at `8d2bf78323`.
| Fixture | Query | Baseline | Treatment | Shared buffers | Delivery
rows visited | Output |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| 60,000 processed, 4 KiB | head | 290.09 ms [220.21..554.19] | 7.34 ms
[5.85..7.68] | 262,630 → 1,309 | 60,002 → 4 | Equal |
| 20,000 processed, 4 KiB | head | 276.02 ms [211.77..322.12] | 34.61 ms
[4.45..194.15] | 67,618 → 486 | 20,002 → 4 | Equal |
| 1,500 processed, 32 KiB | head | 15.88 ms [13.66..21.79] | 2.29 ms
[1.64..7.75] | 10,602 → 80 | 1,502 → 4 | Equal |
| 20 processed | head | 1.49 ms [0.92..3.66] | 1.85 ms [1.01..1.97] | 2
→ 45 | 22 → 968 | Equal |
- The deadline and ready selectors do not change. Their buffers are
identical in both arms; their timings move only with host noise.
- The small control is neutral. On a near-empty table, PostgreSQL picks
a sequential scan for the open-state branch. This costs about 45
buffers.
- The 60,000-row run passes `--require-work-budget`. The baseline
exceeds the budget (`baselineExceedsWorkBudget: true`). This is the
red/green proof. There is no timing threshold in CI.
- On the treatment, the remaining head cost is the scan of
`chat_actions` for pending wakes of the endpoint. The baseline also
reads all of `chat_actions` for its hashed subplan.
- `pnpm exec vitest run src/__tests__/inbound-queue-discovery.test.ts`
(server): 6 tests pass, including the new equivalence test.
- `chat-channels.integration.test.ts` filtered with `-t
"reorder|orders|FIFO|conversation drain|lease renewal|receipt
order|edit|lifecycle"`: 148 assembled chat tests pass. They cover Slack,
Teams, Telegram, Discord and GitHub provider-order drains, head-of-line
barriers, lease loss, lifecycle edits and wake retries.
- `pnpm exec tsc --noEmit -p tsconfig.json` (server) passes.
Reproduce:
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=8d2bf78323 --endpoint-history=60000 --only=endpoint-history --require-work-budget
```
Set `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to use an existing disposable
PostgreSQL server. The benchmark uses synthetic data only.
## Risks
- Query plans depend on data and settings. These local numbers do not
prove hosted request latency. A follow-up statistics window after deploy
must confirm the gain.
- The pending-wake branch still scans the endpoint's `chat_actions`. A
partial index on pending `inbound_wakeup` actions would remove that
scan. I did not add it, to avoid a migration.
- Low behavior risk. The new set equals the old predicate for each
endpoint, and the order expression is unchanged. Tests compare both on
the same rows.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code. Extended reasoning was used.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
56aa160a4e |
perf(native): index heartbeat runs by company and native session (#20560)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs share a native session. Retention, recovery, resume and archive code reads all runs of one session from `heartbeat_runs`. > - On a hosted instance, the native retention lookup ran about 67,000 times in one day at about 22 ms, and about 10,000 times at about 11 ms in a later window. > - No index has `native_session_id`. Each read walks the agent's or the company's run history and filters the rows. > - This pull request adds one concurrent partial index on `(company_id, native_session_id)`. The SQL does not change. > - The benefit is less background database load. Request reads wait less, and the results stay the same. ## Linked Issues or Issue Description **What existing behavior does this improve?** The native session-run lookups. The largest caller is `sessionRetentionBlocker` in native harness backup and checkpoint retention. It selects `id, status, finished_at`, an incomplete-recovery flag and the issue ID for one company, agent and native session. **Subsystem affected** Server native runtime (retention, recovery, resume and archive reads) and the database schema. **Current behavior** `heartbeat_runs` has no index on `native_session_id`. PostgreSQL reads the whole run history of the company (or of the agent through `heartbeat_runs_company_agent_started_idx`) and filters by session. The cost grows with the run history, not with the size of the session. **Proposed behavior** The index finds the few runs of one session directly. Every caller already filters by `company_id` and `native_session_id`, so one index serves the agent-scoped and the issue-scoped reads. **Reason and benefit** This statement family used about 1,500 s of database time in one day on a busy hosted database. The local proof below shows 12.3 ms to 0.04 ms per call. ## What Changed - `packages/db`: add `heartbeat_runs_company_native_session_idx` on `(company_id, native_session_id) WHERE native_session_id IS NOT NULL`. Migration `0457_native_session_run_index` builds it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration is registered as a top-level-statement migration, the same way as 0454–0456. - Add a regression test in `native-harness-backup-expiration.test.ts`. It captures the real retention lookup and proves that its plan can use the index. It also pins the company and session scope: a live run of another company with the same session ID does not protect a checkpoint. A live run of the same agent in another session does not protect it. A live run of the same session does protect it. - Add `scripts/benchmark-native-session-run-index.ts`. It runs the real checkpoint retention pass (dry run), captures the exact SQL, and replays it with and without the index. It also times the migration through the normal migration runner. - `doc/DATABASE.md`: describe the index and the benchmark command. ## Verification Local SQL measurements, synthetic data. PostgreSQL 18.1 (embedded test server). Fixture: 100,000 runs (50,000 of the measured agent, 50,000 of a second agent), 4 runs per session, 4 KiB incompressible `runner_profile_json` per run, 638 MB table. Order: no index, index, no index, index. 15 `EXPLAIN (ANALYZE, BUFFERS)` samples after 3 warmups. | Pass | Plan | Median execution | Min / max | Shared buffers | | --- | --- | --- | --- | --- | | No index (1) | Seq Scan | 12.30 ms | 10.33 / 35.27 ms | 3,607 | | Index (2) | Index Scan using `heartbeat_runs_company_native_session_idx` | 0.036 ms | 0.021 / 0.115 ms | 38 | | No index (3) | Seq Scan | 12.38 ms | 10.24 / 26.14 ms | 3,607 | | Index (4) | Index Scan using `heartbeat_runs_company_native_session_idx` | 0.043 ms | 0.022 / 0.098 ms | 38 | - Correctness: the service SQL is the same in every pass. The returned rows (all columns) and the retention result are equal in every pass. The script fails on any difference. - Migration: the first run through the migration runner took 525 ms on the 638 MB fixture. The rerun took 273 ms and skipped the existing index. A direct concurrent build took 145–221 ms. Index size: 1.9 MB. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-native-session-run-index.ts` - `npx vitest run src/__tests__/native-harness-backup-expiration.test.ts` (server): 108 passed. - `pnpm --filter @paperclipai/db check:migrations`: passed. Five snapshots remain. - `pnpm verify` exits 2 (a new script under `scripts/` is a global CI input). Required CI supplies the broad coverage. ## Risks - Low risk. The migration adds one index concurrently and does not lock writes. The SQL of every caller is unchanged. - The index adds a small write cost to `heartbeat_runs` inserts and to updates of `native_session_id`. Rows with a null session are not indexed. - Persisted run state: no saved run, session or checkpoint shape changes. State saved by the previous release carries forward unchanged. - Hosted latency is not measured yet. The numbers above are local SQL measurements. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bd30ef8510 |
perf(chat): authorize the waits of an issue page with one read (#20551)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can work on tasks that start from an external chat, such as Slack. When the agent waits for the next chat message, the task shows "in review" with an external response wait. > - Issue lists and inbox attention show these waits. To show a wait, the server checks that the chat conversation is still authorized: the messages, the destination, the endpoint, and each person who wrote. > - The server did this check one time for each waiting task, and it started all the checks at the same time. A page with 19 waits sent 19 copies of the message-link statement at the same moment, each on its own pool connection, and then 19 copies of each policy read. > - On a hosted database, 19 identical copies of this statement were active at the same moment, about 670 ms each. They were 19 of the 20 active queries. > - This pull request reads the messages of all waits of a page with one statement and shares the policy reads between the waits. Each wait gets the same authorization result as before. > - The benefit is fewer statements and fewer connections for each issue page, so one page cannot fill the pool with copies of one read. ## Linked Issues or Issue Description Refs #20400 (shares wait reads between the callers of one request; this change removes the per-wait fan-out inside that shared read). Refs #18966 (reuses the prepared message-link statement; this change keeps that statement for single runs). **What happened** `listExternalReviewWaits` ran `authorizeChatConversationForBoundRun` one time for each waiting task, inside `Promise.all`. Each call sends the message-link statement, then a resource read, and three reads for each person. A page with N waits sent N copies of each statement at the same time, on up to N pool connections. **Expected behavior** One issue page sends one message-link statement for all of its waits. A resource or a person that is common to many waits is read one time. The authorization result of each wait does not change. **Steps to reproduce** 1. Put many tasks of one company in review with an external chat response wait. 2. Open the issue list or the inbox. 3. Look at the active statements of the database. One copy of the message-link statement runs for each wait. ## What Changed - `server/src/services/native-runtime/chat-attachment-reuse.ts`: - Split `authorizeChatConversationForBoundRun` into steps: read the bound provider and wake comments, read the message rows, and apply the policy to the rows (`authorizeBoundRunLinks`). The single-run path uses the same prepared statement, the same lock modes and the same policy code as before. - Move the message-link join and its policy filters to one query builder, `boundRunLinksQuery`. The single-run statement and the batch statement both use it. - Add `authorizeChatConversationsForBoundRuns(db, companyId, runs)`. It is read-only and takes no locks. It reads the messages of all runs with one statement, then gives each run only the rows that the single-run statement returns for that run (same issue, agent, provider and comments). The runs share each distinct resource and principal read. Each run gets its conversation, or `null` for a policy denial. Other errors are thrown, as before. - Add `isBoundChatAuthorizationDenial` for the three policy denial errors. - `server/src/services/native-runtime/external-review-wait.ts`: `listExternalReviewWaits` uses the batch authorization. The inherited GitHub review-wait reads do not change. - `server/src/__tests__/batched-chat-authorization.integration.test.ts`: new tests on real PostgreSQL. For each case, the batch result must equal the result of separate single-run reads. The cases include each denial: a missing or foreign comment, no comments, no execution marker, a wrong provider, agent or issue, a person without a link or without a membership, a viewer member, a revoked link, a disabled or unavailable destination, a paused endpoint, a closed conversation, an unprocessed delivery, a delivery without a person, and a verifying endpoint. A test also makes sure that one person's decision never applies to another person in the same batch, and that a batch sends one message-link statement. - `scripts/benchmark-chat-bound-authorization.ts`: a local benchmark that loads the pre-fix modules from a baseline commit, checks that both versions return the same waits, and measures them. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/batched-chat-authorization.integration.test.ts src/__tests__/prepared-chat-authorization.integration.test.ts` passes (10 tests). - Mutation check: when I removed the issue filter from the per-run row selection, or shared one principal decision for all people of an endpoint, the new tests failed (1 and 3 failures). - Related suites pass: `heartbeat-reviewed-chat-binding`, `external-chat-wait`, `preserved-native-response-wait`, `chat-attachment-reuse` and `external-review-wait` (392 tests). - `pnpm verify` asks for broader verification because the change adds a file under `scripts/`. Required CI supplies that coverage. - Benchmark: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-bound-authorization.ts --baseline-ref=f4337bb5fb` (the measured pre-fix commit) Benchmark fixture (synthetic, local PostgreSQL 18.1, pool of 10): one company, one Slack endpoint, 20 tasks waiting for a chat response from 5 people, 4 of those waits denied (the person is a suspended member), and 2,000 earlier conversations with 50,000 processed deliveries (6 KiB events, 419 MB). Both versions return the same 16 waits. 15 alternating pairs after warm-up: | One issue page (20 waits) | Before | After | | --- | --- | --- | | Statements sent | 101 | 18 | | Message-link statements | 20 | 1 | | Server time of the message-link statements | 0.50 ms (sum of 20) | 0.13 ms | | Page time, median (min to max) | 8.86 ms (6.94 to 12.05) | 4.05 ms (3.62 to 5.54) | | 8 pages at the same time, median (min to max) | 54.18 ms (48.36 to 89.24) | 14.99 ms (12.60 to 19.59) | ## Risks - The batch must give each wait the same result as a separate read. The batch statement uses the same join and policy filters. It then selects each run's rows with the equality filters of the single-run statement. The policy code is the same function for both paths. The tests compare both paths for each denial case. - Shared reads: the batch reads each resource and each person one time. All reads are plain reads with no locks, as before. A policy change that occurs during one page read can now apply to all waits of the page at the same time, instead of to some of them. - An invalid wake comment id in one wait makes the whole page read fail. This was already true: the failure of one wait rejected the whole `Promise.all`. - The single-run path (attachment reuse, status decisions, run preparation and others) keeps its prepared statement and lock modes. The order of the conditions in its WHERE clause changed; the conditions are the same. - Not proven: why each copy took about 670 ms on the hosted database. Locally one copy takes about 0.03 ms with the expected index plan. The likely causes are the burst of copies on a small compute and cold pages. Verify after deployment that the statement runs at most one time for each issue page request. - No migration and no schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
94cfb84150 |
fix(scripts): build the plugin SDK in the agent worktree preflight (#20473)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors and agents prepare each new git worktree with `node
scripts/agent-preflight.mjs`
> - The server imports `@paperclipai/plugin-sdk` through its built
`dist`. The preflight did not build that `dist`.
> - On a fresh worktree, the server and the restart-chaos harness stop
at startup with `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`
> - The SDK exports `dist` on purpose: plugin workers load it under
plain Node. CI, `paperclipai run` and the typecheck scripts already
build it with `scripts/ensure-plugin-build-deps.mjs`.
> - This pull request makes the preflight run the same builder as a step
> - The benefit is that one preflight command gives a worktree that can
start the server and the harness
## Linked Issues or Issue Description
**What happened?**
After `node scripts/agent-preflight.mjs` on a fresh worktree, `node
cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts ...` stops
at startup. The error is `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported
from `server/src/services/reusable-sandbox-resume-error.ts`. The
workaround is `pnpm --filter @paperclipai/plugin-sdk build`.
**Expected behavior**
The preflight prepares everything that the server needs to start. The
server and the harness import their modules without a manual build.
**Steps to reproduce**
1. Make a new worktree from `origin/master`.
2. Run `node scripts/agent-preflight.mjs`.
3. See that `packages/plugins/sdk/dist` does not exist.
4. From `server/`, import
`./src/services/reusable-sandbox-resume-error.ts` with tsx. The import
fails with `ERR_MODULE_NOT_FOUND`.
**Paperclip version or commit**
`origin/master` at
|
||
|
|
bf45700ca3 |
fix(native): make Retry work after a turn start cut off once Codex recorded the turn (#20516)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run sends `turn/start` through its runner. A control-plane restart can kill the runner and Codex while that request is in flight > - #20507 lets the runner resume by itself when Codex's thread history proves the turn was never created. When Codex did record the turn, the turn may have done work, so the runner still fails closed and nothing replays it > - That branch must still end in a Retry that works. The next run on the same conversation is refused at the runner-state fence before it asks for a runner. #20507 let Retry continue after that run by matching the refusal's error text > - Driving that branch end to end with the real runner exposed a second dead end. When the owner's runner dies, its provider-session close cannot prove suspension, so this server process can keep an operator-required cleanup quarantine for the task. Retry was then refused outright ("Send a new message") > - This pull request replaces the text match with a server-authored run-log event, lets Retry take the same verified-stop path as a new message, and adds a real-runner test of the whole branch: the cut-off owner fails closed, the follow-up is refused with structured proof, Retry is admitted, and the fresh Retry run completes > - The benefit is that this restart window cannot strand a task, and the Retry gate no longer depends on error wording ## Linked Issues or Issue Description **What happened?** Follow-up to #20507 (restart-chaos wave `w-20261110T033426-s108`, t03). #20507 left one item open: prove that a person's Retry produces a working run when the restart cut off `turn.start` after Codex recorded the turn. It also matched the fence refusal by its error text. **Expected behavior** After that cut-off, the owner stays failed closed and the turn is never started again by itself. A person's Retry starts a new run that completes. It never fails with `quarantine_owner_unverified` and never returns 202 with no run. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-cut-turn-start-retry.integration.test.ts`. 2. Without the new event, the test fails: the refused follow-up has no proof that it never launched, so Retry after it would wait on `process_identity_missing`. ## What Changed - `native-session-executor.ts`: when the runner-state fence refuses a run before launch (`prior_owner_*` or `legacy_prior_owner_*`), the executor records the server-authored event `native.prior_owner_fence_refused` with the closed refusal `reason`, then fails the run as before. A failed write leaves the run held (the safe side). - `native-local-process-stop.ts`: `nativeRunNeverLaunched` accepts a failed run that has this event (and still no launch, identity, `adapter.invoke` or runner-sourced event). It no longer reads the error text. - `explicit-native-continuation.ts`: when the task's in-process cleanup quarantine is operator-required, a Retry no longer returns `cleanup_quarantined` at once. Like a new message, it must first pass `verifyStoppedNativeSessionForContinuation`, which releases the quarantine and records `native.stopped_conversation_verified`. A native Retry always starts a fresh conversation (`forceFreshSession`), so it never resumes the interrupted turn. If the proof fails, Retry still returns `cleanup_quarantined`. - `verifyStoppedNativeSessionForContinuation` (`native-session-executor.ts`), three narrow additions: - a run with `native.prior_owner_fence_refused` may stand in for the saved state's owner, like a run refused by the cleanup quarantine (both were refused before launch); - when the session root is gone, it reads the fence's single `identity_indeterminate` quarantined copy of the owner's state (several copies stay ambiguous); - a provider startup attempt in phase `initialization_failed` whose direct child exited and whose process and process group are gone no longer blocks the proof. A same-run restore that fails closed leaves exactly this record. Those PIDs are also re-checked before the quarantine is released. - New real-runnerd integration test `native-cut-turn-start-retry.integration.test.ts` (real runnerd and fake Codex with `--hang-turn-start after`): 1. The owner run's `turn/start` hangs after Codex recorded the turn; the runner and Codex are killed. The owner's provider state keeps the ambiguous start, and the thread history keeps the turn. 2. The same-run resume restores the runner and fails closed with "cannot safely recover an ambiguous Codex turn start". Codex saw exactly one `turn/start`. 3. A follow-up on the same conversation is refused with `prior_owner_*`, never `quarantine_owner_unverified`. It records `native.prior_owner_fence_refused`, launches nothing, and `nativeRunNeverLaunched` is true for it. 4. The owner's operator-required in-process cleanup quarantine is pinned (whether the real close lands there is timing-dependent; about half of the local runs did). 5. Retry of the refused follow-up passes `admitExplicitNativeContinuation` (not a dry run) with no block reason, releases the quarantine and records `native.stopped_conversation_verified`. 6. The Retry run, prepared with a fresh conversation as task Retry does (`forceFreshSession`), uses a different runner-state root and completes with exit code 0. Codex saw one new `thread/start` and one new `turn/start`; the cut-off turn was never started again. - `explicit-native-continuation.test.ts`: the fence rows now insert the event. A new row shows that the refusal text alone keeps the run held. The "worker loss (retry)" case now expects a verified Retry to be admitted like a new message. - `native-recovered-owner-followup.test.ts`: a fixture event moves to a free sequence number, because the refused follow-up now writes its own event. - Docs: `doc/run-log-events.md` (new event) and `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification - Red/green: - Without the event write, the integration test fails (no fence event, so the follow-up is not proven unlaunched). - Without the gate and proof changes, it fails with `expected [ 'cleanup_quarantined' ] to deeply equal []`: Retry is refused while the owner's quarantine is held. - With all changes, it passes. - `vitest run` of `native-recovered-owner-followup`, `explicit-native-continuation` (427 tests), `quarantined-task-retry`, `issue-confirm-run-stopped-routes`, `native-finalization-recovery`, `task-interruption-query`, `execution-projection`, `chat-run-publications`, `closed-native-receipt-accounting`, `agent-live-run-routes` and the new integration test pass. - Server `tsc --noEmit` passes. - Runner code and the pack `contractDigest` do not change. ## Risks - Low-medium risk. The new event is written only on the fence refusal path, which already fails the run. It is local run-log data (no Telemetry or OpenTelemetry). - Retry after an operator-required cleanup quarantine is a behaviour change: it was always refused, and now it is admitted only after the same exact stop proof a new message needs. That proof checks the owner's runner stop record, every recorded provider and startup PID and process group, and the unchanged state fingerprint before it releases the quarantine. - A run refused by a build between #20507 and this change has only the error text. It stays held on `process_identity_missing` until a person sends a new message or the board confirms the stop. #20507 has not shipped in a release yet. - The test models the restart in-process: it kills the runner and Codex and calls the executor again. It does not drive the HTTP Retry route; the route's own behaviour (fresh conversation, admission gate) is covered by existing tests and by `nativeRunNeverLaunched` here. - Open decision, unchanged: a turn Codex recorded before the restart is never replayed or adopted. Continuing is a person's decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20507 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |