mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
89de6cbe3e4fbda30ead2ff76eec40aac41fc65f
8075
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
89de6cbe3e |
perf(server): list skills without a folder transaction when bundled folders exist (#21715)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI polls `GET /api/companies/:companyId/skills`, and the server runs all requests on one Node main thread > - The cost on that thread is per database statement, so a polled read must stay small > - Each skills list read reconciles the bundled skill folders, and that opened two transactions with a company advisory lock, even when all folders already existed > - A write-path guard must not run a lock and a transaction on every poll > - This pull request checks the current folders with a plain read first and takes the lock only when a folder must change > - The benefit is a skills list read with no transaction and no advisory lock, and 13 fewer statements (58 to 45) ## Linked Issues or Issue Description **What existing behavior does this improve?** `folderService.ensureBundledCategory` and `pruneEmptyBundledCategories` always open a transaction and take `pg_advisory_xact_lock` for the company folders. **Subsystem affected** Server: skill folders service, used by the company skills list. **Current behavior** Every `GET /api/companies/:companyId/skills` sends 58 statements, including 2 transactions (BEGIN, lock, reads, COMMIT), even when nothing changes. **Proposed behavior** `ensureBundledCategory` reads the company skill folders once. When the bundled root exists and the category folder exists with the expected name, it returns that folder. `pruneEmptyBundledCategories` computes the empty categories with plain reads and returns when there is nothing to delete. In all other cases, both functions take the lock and run the old code, which reads the folders again under the lock. The same request sends 45 statements with no transaction. **Reason and benefit** The polled read stops taking a company lock, and it sends fewer statements. **Breaking changes** None. ## What Changed - `ensureBundledCategory`: a plain read fast path when the category folder and the bundled root exist and the name matches. Create and rename still run under the company folder lock. - `pruneEmptyBundledCategories`: a plain read pre-check. The delete still runs under the lock, and it computes the empty set again under the lock. - New test `company-skills-list-statements.test.ts`: a warm skills list read has no transaction and no advisory lock and at most 45 statements; two concurrent first ensures create exactly one folder; two concurrent first list requests create no duplicate folder; a renamed category is repaired and an empty category is pruned through the locked path. ## Verification - Red: before the change, the warm read had 2 transactions (`expected 2 to be +0`) and 58 statements. - Green: after the change, the warm read has 0 transactions, no `pg_advisory` statement, and 45 statements. The response body is equal to the first read. - Concurrency: two `ensureBundledCategory` calls at once return the same folder, and there is exactly one row for the system key. - `pnpm exec vitest run` on `company-skills-list-statements`, `company-skills-service`, `company-skills-routes`, `company-skills-runtime-concurrency`, `company-skills-catalog-service`, `company-skills` and the folder tests: 215 tests pass. Two `company-skills.test.ts` tests in "project workspace skill discovery" fail on my machine with and without this change. They test file system discovery and do not use the folders service. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The fast path returns only when the locked path would return the same folder without a write. Every write still happens under the lock, after a fresh read. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.4 |
||
|
|
30c4d2f4a1 |
perf(server): read the issue detail with fewer database statements (#21713)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI loads the task page through `GET /api/issues/:id`, and
the server runs all requests on one Node main thread
> - A CPU profile of the server shows that the cost is per database
statement: SQL building, parameter encoding, result parsing and one
socket write each
> - The issue detail read sent 73 statements for a task with two
ancestors, one blocker and a project mention
> - Several of those statements read the issue row again, only to get
values that the route already loaded
> - This pull request passes the loaded issue to those reads, walks the
ancestor chain in one query, and skips reads whose result the response
does not use
> - The benefit is 9 fewer statements on each task page load (73 to 64),
plus one fewer statement for each extra ancestor level, with a
byte-identical response
## Linked Issues or Issue Description
**What existing behavior does this improve?**
`GET /api/issues/:id` reads the same issue row several times in one
request, and walks the ancestor chain with one statement for each level.
**Subsystem affected**
Server: issue routes and the issue, issue reference and project
services.
**Current behavior**
The route loads the issue, then `getRelationSummaries`,
`getCurrentScheduledRetry`, `findMentionedProjectIds`,
`listIssueReferenceSummary` and `getAncestors` each read the issue row
again. `getAncestors` sends one statement for each ancestor. The inbox
archive fields read two activity aggregates even when the issue has no
archive row. The project embed reads the workspace provider catalog, but
the compact embed does not return it. The test fixture request sends 73
statements.
**Proposed behavior**
The route passes the values it already has. The ancestor walk is one
recursive query. The activity aggregates run only when an archive row
exists. The project embed skips the provider catalog. The same fixture
request sends 64 statements, and the JSON response is the same.
**Reason and benefit**
Fewer statements per request means less CPU on the main thread for every
open task page.
**Breaking changes**
None. Other callers of the changed service functions keep the old
behavior, because the new arguments are optional.
## What Changed
- `issueService.getRelationSummaries` and `getCurrentScheduledRetry`
accept an optional known `companyId` and then skip the issue read.
- `issueService.findMentionedProjectIds` accepts the loaded issue
(`companyId`, `title`, `description`).
- `issueService.getAncestors` accepts the known `parentId` and walks the
chain in one recursive query. It keeps the order (nearest first), the
limit of 50, and the stop at a cycle or at a task that is its own
parent.
- `issueService.getActiveInboxArchiveFields` reads the archive row first
and reads the activity aggregates only when an archive row exists.
- `issueReferenceService.listIssueReferenceSummary` accepts an optional
known `companyId`.
- `issueThreadInteractionService.listForIssue` accepts the loaded issue
(`status`, `companyId`, `executionWorkspaceId`). `GET
/api/issues/:id/interactions`, which the board polls, passes it (11 to
10 statements).
- `projectService.getById` accepts `{ workspaceProvider: false }`. The
issue detail route uses it, because `compactIssueProject` never returns
`workspaceProvider`.
- Test helpers: `createStatementCensus` (statements per request, grouped
by statement shape with the server callers), and a board route fixture
with the real session actor middleware. `meterDatabaseStatements` can
record a call stack for each statement.
- New tests: `issue-detail-statements.test.ts` (statement budget,
byte-identical normalized JSON against a fixture file, access rules,
membership removal on the next request, ancestor walk with a cycle) plus
an interactions poll test, and `board-route-statement-census.test.ts`
(census of the polled and page-load board routes).
## Verification
- Red: before the change, the fixture request sent 73 statements, and
the budget test failed (`expected 73 to be less than or equal to 64`).
- Green: after the change, the request sends 64 statements. The
normalized JSON is byte-identical to the fixture file that I wrote
before the change.
- Access: another company's user gets 404, a missing issue gets 404, an
anonymous request gets 404, a hidden issue stays readable for a member,
and a removed membership gets 404 on the very next request.
- `pnpm exec vitest run` on `issue-detail-statements`,
`board-route-statement-census`, `issues-goal-context-routes`,
`issues-service`, `issue-references-service`, `inbox-archive-routes`,
`heartbeat-task-context`, `project-workspace-provider.integration`,
`workspace-provider-failure`, `issue-telemetry-routes`,
`issue-thread-interaction-routes`, `issue-document-restore-routes`,
`heartbeat-referenced-projects`, `project-environments`: 14 files, 414
tests pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Interactions: the polled route sends 10 statements instead of 11, with
a response equal to the service result without the known issue.
- Census: `PAPERCLIP_STATEMENT_CENSUS=/tmp/census.txt pnpm exec vitest
run src/__tests__/board-route-statement-census.test.ts` writes
statements per request and the top statement shapes for each board
route.
## Risks
- Low risk. The new arguments are optional, so other callers keep the
old reads.
- When the route passes a known `companyId`, a concurrent delete of the
issue no longer gives a 404 from these sub-reads. The route already
loaded and authorized the issue in the same request, so the response is
the same in practice.
- The JSON fixture file fails when someone changes the issue detail
response. Regenerate it with `PAPERCLIP_WRITE_ISSUE_DETAIL_GOLDEN=1` and
review the diff.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
f4f4902707 |
perf(chat): read the run state of a GitHub response sweep page in one batch (#21711)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two sweeps for request response
comments: the acknowledgement sweep (`acknowledgement-work.ts`) and the
terminal-response sweep (`terminal-response-work.ts`). Each reads pages
of up to 100 requests.
> - For each loaded request, both sweeps call `readGitHubResponseRun`
one row at a time. That is one latest-run statement, plus the run
identity context read and the message link read of
`chatRunSourceMessageId`. A page of 100 requests costs about 300
statements.
> - On paperclip.staging, those two provenance reads ran at about 13/s
each. #21705 stops the idle backlog from being reloaded. This PR makes
the reads that remain cost a constant number of statements per page.
> - The per-row read also had a failure mode. A 403 from one run's
provenance check aborted the acknowledgement sweep's whole page, so no
request of that company was acknowledged until the 403 went away.
> - This pull request adds a batched `readGitHubResponseRuns` and uses
it once per page in both sweeps.
> - The benefit is a constant number of statements per page, and a
provenance failure that affects only its own request.
## Linked Issues or Issue Description
Refs #21705 (the same sweeps; that PR stops reloading requests that
cannot progress).
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On paperclip.staging (2026-11-15, about 01:45–02:25Z),
`provenanceReads.context` (`select … from run_identity_contexts where id
= $1 and run_id = $2 and company_id = $3 and status = $4 limit $5`) ran
at 13.37/s, and `provenanceReads.link` (`select
chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join … where chat_message_links.company_id = $3 and
comment_id = $4 …`) ran at 13.20/s. Both GitHub response sweeps reach
them through `readGitHubResponseRun` → `chatRunSourceMessageId(db, run,
{ provider: "github" })`, once for each loaded row. `load()` costs one
page read plus three statements per row. `check()` read the run state
twice for each due request (before and after the acknowledgement).
In the acknowledgement sweep, a 403 from `chatRunSourceMessageId` (for
example, a run whose identity context is no longer accepted) threw out
of `entry()` and aborted the whole page. The terminal sweep already kept
such a row due, so the acknowledgement owner could revoke it.
**Expected behavior**
A page of requests costs a constant number of statements. A failure for
one run affects only the requests of that run.
**Steps to reproduce**
1. Create 120 GitHub response requests on 30 tasks. Give each task an
ended run with an accepted identity context.
2. Register the terminal-response sweep. On master, the first pass costs
one page read plus three statements per row.
3. Mark the requests `received`, revoke one run's identity context, and
register the acknowledgement sweep. On master, the load fails with a
403, and none of the 150 requests is acknowledged.
**Paperclip version or commit**
master at
|
||
|
|
8cc26221af |
perf(native): accept a plugin's in-preparation project-auth check on warm attach (#21710)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On a warm turn Core attaches the kept Runner to the new run: the plugin prepares the new run, then Core checks managed project authentication > - A warm attach has no submit step, so Core sends a separate `check_project_auth` operation after `prepare` > - That separate operation is one more plugin dispatch and Launcher round trip on every warm turn, about 0.12–0.18 s, and it is serial > - This pull request lets the plugin do the check inside `prepare` and say so, so Core does not send the second operation > - The benefit is a faster warm turn with the same proof and the same failure behavior ## Linked Issues or Issue Description **What existing behavior does this improve?** The warm attach of a plugin Runner task (`plugin-runner-task.ts` `prepareLaunch`). When `launched` is true, Core always calls `checkProjectAuth` after `prepare`. **Subsystem affected** Native runtime, plugin Runner tasks, and the plugin SDK environment task contract. **Current behavior** Cold starts fold the check into submit (`projectAuthCheckAtSubmit`). Warm attaches send `prepare`, then a separate `check_project_auth`. Measured on a plugin sandbox, the separate operation costs 0.12–0.18 s on each warm turn. **Proposed behavior** The `prepared` result has a new optional flag, `projectAuthChecked`. A plugin sets it only when it ran the managed project-authentication check inside this preparation, for this launch's configuration, provider and projects, and the check allowed them. Core then records the same lease proof that an allowed `check_project_auth` records, and skips the separate check. **Reason and benefit** One fewer serial round trip on every warm turn. The plugin already runs the check for this task; this removes a dispatch, not a check. **Breaking changes** None. The flag is optional. A plugin that does not set it, or that saw a refusal, gets the current behavior. Plugins detect the flag with the SDK schema before they send it. ## What Changed - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `projectAuthChecked: true` to the `prepared` result, with a doc comment that defines it. - `server/src/services/environment-task-runtime.ts`: when a `prepared` result has `projectAuthChecked`, record `runnerTaskProjectAuthValidation` for the launch's configuration, provider and projects, without `atSubmit`. This is the proof that `submit` and `read_file` require. - `server/src/services/native-runtime/plugin-runner-task.ts`: do not send `check_project_auth` when the prepared result has `projectAuthChecked`. - Tests: a warm attach with the flag makes only `prepare`; without it, `prepare` then `check_project_auth`. The runtime test covers all four combinations of `projectAuthCheckAtSubmit` and `projectAuthChecked`. ## Verification - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server/`: 157 passed. - Not run locally: full typecheck and full suite. CI covers them. ## Risks Low risk. Core trusts the flag the same way it trusts a plugin's `check_project_auth` answer and `projectAuthCheckAtSubmit` today. A refusal still goes through Core's own check, so the conflict error is unchanged. The plugin side ships separately and sends the flag only when its SDK accepts it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, tracing queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b5ba548112 |
feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line that a real-agent ramp harness checks (#21695) > - Four finish-line criteria need numbers from inside each server process: main-thread CPU, run event lateness, agent API time before the handler, and missed lease renewals > - The server had no way to give them, so every harness run was at best "incomplete" > - This pull request adds one instance-admin endpoint that returns those numbers per process over a short sliding window, off by default, and connects the harness to it > - The benefit is that a ramp run can measure every server criterion it can see, with near-zero cost when on and no cost when off ## Linked Issues or Issue Description Refs #21695 **Subsystem affected** Server (instance routes, run event append, native lease renewal), database client (statement counter), and the scale-ramp harness in `scripts/scale-ramp/`. **Problem or motivation** The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the server has no endpoint for it. The criteria `main_thread_cpu`, `run_event_lateness_p99`, `agent_api_pre_handler_p99` and `lease_renewals_missed_zero` are therefore always "not measured", and no run can pass. **Proposed solution** Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of one server process over the last 50 to 60 s. Only an instance admin can read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start, default off) turns on collection and the route. Off, the path answers 404 and no collector starts. **Alternatives considered** - The postgres.js `debug` option for the statement counter: rejected, because it captures a stack (`new Error()`) for every query. - Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected, because it includes GC and libuv threads. It is returned as `processCpu` instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux only, null elsewhere). - Default on: the measured cost is negligible (numbers below), but the event-loop monitor wakes the process every 20 ms when idle, and this is a new instance-wide surface. It stays off; the runbook says how to turn it on for a ramp run. ## What Changed - `server/src/services/server-metrics.ts`: a per-process collector. It uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20 ms), `process.cpuUsage()`, the main thread's `/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native `perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing on the hot path awaits, allocates per call or does I/O. - `server/src/routes/instance-server-metrics.ts`: `GET /api/instance/server-metrics`, guarded by the existing `assertInstanceAdmin` (the check that `/instance/database-backups` and other instance routes use). It returns numbers, a random per-process `processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no SQL, names, IDs or hostnames. - `server/src/app.ts`: when the flag is on, two middlewares time agent API requests from arrival to the API router, the route is mounted, and shutdown stops the collector. When the flag is off, nothing is mounted. - The snapshot also carries `dedicatedConnections` from `dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit` since start. The harness ignores it today; it shows dedicated-connection queueing in a restart burst. - `packages/db/src/statement-count.ts` and `lazy-transaction-queries.ts`: a Drizzle logger that increments one counter per statement replaces Drizzle's no-op logger. No edit to `client.ts`. - `paperclip-control-plane-port.ts`: one call records runner `emittedAt` to committed row for each committed (not duplicate) runner event. - `native-execution-lease.ts`: one call counts a lease whose renewal deadline passed. - Harness: missed lease renewals are summed per `processId` (several processes, or a restart, no longer cancel out). A metrics URL on the target origin gets the board token in the `Authorization` header; `PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin gets no token. - Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the serializer's own output. A server test asserts the serializer produces exactly this file, and a harness test parses it and checks every field. - `rules/company-scope-exceptions.mdc`: adds the snapshot as an instance-wide exception with its reason. The runbook in the plan doc documents the path, the flag, the credential and each field. | Field | Provided | Source | | --- | --- | --- | | `mainThreadCpu` | yes (Linux; null elsewhere) | `/proc/self/task/<pid>/stat` utime+stime | | `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` | | `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s p99 in the window | | `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT or raw driver calls) | | `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue time | | `runEventLatenessP99Ms` | yes (null with no events) | runner event append | | `agentApiPreHandlerP99Ms` | yes (null with no agent requests) | arrival and router-entry middlewares | | `leaseRenewalsMissed` | yes, counter since process start | native lease deadline timer | ## Verification - `cd server && npx vitest run src/__tests__/instance-server-metrics-routes.test.ts src/__tests__/server-metrics.test.ts`: through `createApp` on the embedded test database. 200 with the exact field set for an instance admin; 403 for a company owner who is not an instance admin, for an agent key, and unauthenticated; 404 and no collector when the flag is off; values move after agent requests, statements, a late run event and a missed renewal; the body has no IDs, keys or SQL. - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks (red/green checked for the lease hook by removing the call). - `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts src/client-options.test.ts`: the counter counts statements inside transactions and savepoints. - `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture contract and the board-token header. - `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in `packages/db`: pass. Overhead, measured on a shared, loaded laptop (load average 8 to 20), median with min and max: | Measure | Off | On | | --- | --- | --- | | Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) | 3.36 ns (3.32 to 6.16) | | CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us (20.3 to 42.4) | 32.3 us (20.7 to 45.5) | | CPU per HTTP request, client and server in one process (2,000 x 6) | 67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) | | Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08% (0.05 to 0.09) | | Record one run event (Date.parse + histogram) | - | 148 ns (135 to 188) | | One snapshot | - | 55 us | The statement and request differences are inside the run-to-run spread. The direct cost is about 3 ns per statement, against about 30 us of CPU per statement. ## Risks - Low risk. Off by default; off, only the statement counter runs, and it costs one increment more than Drizzle's no-op logger. - `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other platforms. The harness then uses `eventLoopUtilization`. - Run event lateness compares the runner's clock with the server's clock, so clock skew adds to it. Negative values count as zero. - With several processes, each request reaches one process. The harness keeps the highest gauge and sums counter growth per process, but a process it never samples is not seen. - The statement counter does not count BEGIN/COMMIT or raw postgres.js calls. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
554e6d415c |
fix(restart-chaos): credit a follow-up only to a run that started a provider turn, and post a missed follow-up after the run ends (#21692)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs staging-mix waves across server swaps and grades each task for duplicate side effects, extra turns and follow-up delivery > - In one wave, the grader reported two false failures > - Task t005: the grader credited a follow-up to a run that failed before its provider turn. It then counted the run that really served the follow-up as a duplicate completion and an extra turn > - Task t009: the agent finished before the follow-up window opened. The harness never posted the follow-up, but the grader still required the post > - False failures hide real regressions, so the grader must match what the product did > - This pull request credits a follow-up only to a run that started a provider turn, and posts a missed follow-up after the run ends > - The benefit is that both false failures go away, and the t009 case still tests the product: the product must answer the late follow-up ## Linked Issues or Issue Description Refs #21598 **What happened?** In wave `w-20261114T231451-s7110`, two tasks failed on harness defects, not product defects. - t005 (`T:completion_once`, a duplicate side effect and an extra turn): run 76f30f70 was woken for follow-up comment 3f6a9eb9. It failed before any provider turn, so the agent never received the comment. Recovery run fdf1f793 served the comment. It added the FOLLOWUP line and did not post DONE again (DONE comments = 1). The grader credited 76f30f70 with the answer and flagged fdf1f793 as a second completion and an extra turn. - t009 (`T:follow_up_posted`): the follow-up was due at 23:25:19. The harness posts only during a swap overlap (opened 23:28:02.7) or at the 85% point (about 23:29:15). The agent finished at 23:27:56.0, so the harness never posted. The grader still required the post. **Expected behavior** - t005 grades one completer (60bdd4dc) and no extra turn. fdf1f793 is the run that answered the follow-up. - t009 gets its follow-up after the run ends. The grade then requires the product to answer it. **Steps to reproduce** 1. Build wake evidence with three runs: a done run, a run woken by a follow-up comment that has no provider turn, and a recovery run that re-completes. 2. Grade with `gradeStagingMix`, `completedAtAnchor` and `runsAfterDone`. 3. Drive `stagingMixStep` with the t009 timings: the run ends before the overlap and before the 85% point. ## What Changed - `tests/restart-chaos/wake-evidence.ts`: record `providerTurnStarted` for each run. It is true when the run has a `turn.submitted`, `turn.accepted` or `turn.started` run event. - `tests/restart-chaos/follow-up-delivery.ts`: a run with `providerTurnStarted: false` does not receive its comments. The comments carry to the next run that starts a provider turn. That run is the answering run. Add `undeliveredFollowUpRuns`. Evidence without the field keeps the old behavior. - `tests/restart-chaos/staging-mix-retry.ts`: `runsAfterDone` does not list a run that failed before it could deliver a follow-up as an extra turn. - `tests/restart-chaos/staging-mix.ts`: when no run is active after the hold or the 85% point, `stagingMixStep` posts the follow-up after the run ended. It records the post as an after-end follow-up, so `after_end_follow_ups_served` requires an agent reply and the ledger line. The `follow_up_posted` detail says "after the run ended". - `tests/restart-chaos/observer.ts`: a staging-mix task with an unposted follow-up does not finish yet. - Tests and `tests/restart-chaos/README.md`. ## Verification - Red on master, green with the fix: - `a follow-up whose run failed before any provider turn (w-20261114T231451-s7110 t005) > credits the follow-up to the run that served it, with one completer and no extra turn` - `... > lets only the serving run re-complete, and still fails a later recovery run that answers nothing` - `staging-mix board step > posts a follow-up the run finished before as an after-end follow-up the product must answer (w-20261114T231451-s7110 t009)` - `staging-mix board step > does not post an after-end follow-up between runs before the during-run window closes, or while a run is queued` - `TaskObserver > does not finish a staging-mix task whose run ended before its planned follow-up was posted` - Guards that still grade as failures on master and with the fix: - `... t005 > still fails a recovery re-completion when the run that received the follow-up started its provider turn` - `a done the server rejected as incomplete > keeps a run woken by recovery after a real completion as an extra turn and a duplicate completion (G5c t004)` (#21598) - `completion_once truth table: a later completion that answers nothing` - `wake evidence against real PostgreSQL` now checks `providerTurnStarted` false, then true after a `turn.started` event. - `npx vitest run --config tests/restart-chaos/vitest.config.ts --maxWorkers=3`: 71 files, 823 tests pass. - `tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - Not run: a new wave. The wave's database is gone, so the real t005 evidence cannot get `providerTurnStarted` from the new query. The tests use the real t005 and t009 values. ## Risks - Low risk. The change affects only the restart-chaos harness. - A run that failed after `turn.submitted` but before the provider read the input counts as a delivery. This keeps the grader strict: it can report a false duplicate, but it cannot hide a real one. - A staging-mix task with a planned follow-up that cannot be posted now waits until its deadline instead of resting. The grade still reports the post errors. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45125f7748 |
test(server): stop the accounting identity child from timing out on loaded runners (#21712)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server CI runs the general test suite in 12 shards on shared runners > - One accounting test fails at random on unrelated pull requests and passes on re-run > - Each failure costs a full re-run of the shard and the `verify` job, and it teaches people to re-run instead of read > - The test spawns two helper processes that load about 1,400 TypeScript modules under a 5 second kill timer > - This pull request makes the helper processes plain `node` processes, so they do not load the server module graph > - The benefit is that the test no longer depends on runner load, and it still uses real exited processes ## Linked Issues or Issue Description **What happened?** `server/src/services/run-cost-accounting.test.ts > run cost accounting work > normal accounting Retry reconstructs an uncommitted legacy receipt after its real controller and child exited` failed in "General tests (server (5/12))" three times on 2026-11-15 (UTC), each with the retry also failing: - PR #21687, head `2172a5b26a`, job 132643921517 - PR #21687, head `6a1d2fb0ef`, job 132650244103 - PR #21678, head `fc0e96f23c`, job 132663296537 Every failure has the same assertion: `AssertionError: {"exitCode":null,"signal":"SIGTERM","timedOut":true,...,"stderrTail":"accounting-identity:program-entered\naccounting-identity:module-imported\n"}: expected null to be +0` In five of the six failed attempts, the child stopped after `module-imported` and before `birth-read`. In one attempt, the child printed its result but the 5 second timer fired before it exited. **Expected behavior** The test passes on a loaded runner. The helper processes are not under test. Only the accounting Retry decision is under test. **Steps to reproduce** 1. Slow each module load in the child by 8 ms with an `--import` hook in `NODE_OPTIONS`. The test passes `NODE_OPTIONS` to the child. This simulates a loaded runner. 2. Run `npx vitest run src/services/run-cost-accounting.test.ts -t "normal accounting Retry"` in `server/`. 3. On `master` the test fails with the same `timedOut: true` assertion as CI. **Root cause** The child ran `node --import tsx/esm -e "await import('controller-process-identity.ts'); await currentProcessStartedAtMs(); ..."`. That import pulls in `@paperclipai/shared` and `@paperclipai/db` source through tsx, which is 737 modules. On Linux, `readProcessStartedAt` also dynamically imports the full runner package shim (`server/src/vendor/paperclip-runner/index.ts`) to call the 30-line `/proc` reader. That adds 652 more modules. tsx transpiles all of them in a cold process under a 5 second `timeoutSec`. On a shared CI runner this crosses 5 seconds. **Production impact** None. In the built server, the runner package is already loaded at boot, and `currentProcessStartedAtMs()` on Linux reads `/proc` once per process and caches the result. The cost exists only in a fresh tsx process that imports source. ## What Changed - The accounting identity helper child is now `node -e` with no tsx loader and no module imports. It installs a `SIGUSR2` handler, prints `ready`, and waits. - When the test sees `ready`, it reads the child start time with the product function `observedProcessStartedAt(pid)` and then sends `SIGUSR2`. The child exits with code 0. - The test builds the recorded `ControllerProcess` (random boot id, child pid, `currentProcessSpaceHash()`, observed start time). These are the same values the child computed for itself before. The child process is still real, has its own process group, and has exited before the Retry checks run. - No retries, skips or timeout changes. The 5 second guard now covers a plain `node` start, which takes about 40 ms. ## Verification - Red on `master` with the slow-load hook: `Tests 1 failed`, `timedOut: true`, the same assertion as CI. - Green with this change and the same hook: `Tests 1 passed`. - Full file without the hook: `npx vitest run src/services/run-cost-accounting.test.ts` passes 34/34. - Helper loop, 30 runs per variant, macOS, 18 cores with 36 busy-loop processes (load average about 56): - old child (macOS path): 29/30 exited in 5 s, p50 2,020 ms, p90 2,234 ms, max 5,172 ms - old child plus the runner import (as on Linux CI): 29/30, p50 3,663 ms, p90 4,411 ms, max 5,023 ms - new child: 30/30, p50 38 ms, p90 43 ms, max 46 ms - The full test under the same stress: 6/6 before and 6/6 after on macOS. macOS uses the `ps` path and does not load the runner package, so this loop does not show the Linux failure. The helper loop and the slow-load repro do show it. - `npx tsc --noEmit -p server/tsconfig.json` passes. - Other tests with the same pattern: `native-controller-source-digest.test.ts` also spawns a tsx child, but with a 60 second timeout. It is not the same cause, so this pull request does not change it. ## Risks - Low risk. Only one test changes. No product code changes. - The test no longer checks that a separate process computes its own identity stamp. That was not the purpose of this test. Other tests cover `currentControllerProcessStamp()`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
92308f839d |
test(e2e): use the Coordinator task panel, inbox and plan controls in QA journeys (#21701)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The QA Cloud product journeys check real user paths on a hosted test stack > - #19360 (2026-11-05) made the Coordinator workspace standard; its task side panel starts with no tabs, and the header control opens a tab chooser > - The journeys were written for the classic panel, which opened straight on Properties, and for older inbox and plan controls > - On QA the smoke suite passed 8 of 24 journeys; 13 failures trace to these out-of-date journey steps > - This pull request updates the journeys to use the shipped Coordinator controls the way a person does > - The benefit is that the QA journeys test the product again instead of failing on removed controls ## Linked Issues or Issue Description **What happened?** The QA staging smoke suite passed 8 of 24 journeys. 13 failures come from journey steps that target the classic UI: - Task side panel: the journeys click "Show properties" and expect property rows. In Coordinator the click opens a tab chooser ("Open a side panel tab"), and the user picks Properties or Artifacts. The product's own e2e tests (`tests/e2e/artifact-tab-arrival.spec.ts`, `artifact-uploads.spec.ts`, `disconnect-resilience.spec.ts`) and the chat-catalog runtime cases already use the chooser. - Inbox: desktop Coordinator inbox rows show unread state as a bold title and have no per-row "Mark as read" button (`doc/plans/2026-11-06-chat-contacts-pins.md`). The `r` shortcut marks the hovered row read on the Mine view. - Plan: the Coordinator Plan chip is named "Open Plan revision N in task sidebar", not "Open Plan revision N". **Steps to reproduce** 1. Run `pnpm test:e2e:runner -- --target-config <managed local config>` with task-properties, task-edit-lifecycle, task-document-persistence and project-task-context. 2. See 0/4 pass on master; each stops at the tab chooser. **Expected behavior** The journeys reach the same saved-state checks as before through the shipped controls. ## What Changed - New `product-journeys/task-panel.ts` helper: opens the side panel and selects a tab (Properties or Artifacts) from the chooser, an open tab, or the "Open a new tab" button. It waits for the page instead of a single non-waiting visibility check, and does not close an already open panel. - task-properties, task-edit-lifecycle, task-dependency-resume, project-task-context and task-document-persistence (and routine/agent journeys that reuse `openDocumentAuthoring`) use the helper. task-properties reopens the panel after reload before it checks "You". - inbox-triage: when the row has no visible "Mark as read" button, open the Mine view, blur search, hover the row and press `r` (as `tests/e2e/coordinator-inbox-views.spec.ts` does). - Plan journey: `planPreviewLabel()` accepts the optional " in task sidebar" suffix and still requires the exact revision. - No product code changes. An earlier revision of this PR changed "Show properties" to select Properties; it is reverted because the product e2e suite asserts the chooser is the intended design. All saved-state assertions are unchanged: assignment, labels, status, documents, read state and plan revision are still read back through the API, so a broken feature still fails. ## Verification - Real browser journeys on a managed local instance against unmodified master UI: 4/4 pass (task-edit-lifecycle, task-properties, task-document-persistence, project-task-context); 0/4 before. - Inbox fallback checked by hand on a local instance: an unread task on the Mine view became `isUnreadForMe=false` after blur + hover + `r`. - `tests/runner-e2e` unit tests for product-journeys and plan-revision: 47 files, 1240 tests pass. The task-properties contract test now models the chooser and asserts the opener and the Properties option are used before assignment and after reload. New `planPreviewLabel` test covers revision 12 vs 1 and a missing revision. - QA Cloud rerun follows once the QA harness pin moves to a master commit that contains this change. ## Risks - Low risk: test code only. If the chooser changes again, the helper fails visibly at the panel step. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8adf0a463a |
perf(db): build the agent request authentication statements once per client (#21707)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - One Node process controls every run of a workspace. At 1,000
concurrent agents, main-thread CPU for each database statement is the
scarce resource.
> - A CPU profile of the hosted server shows the cost is per statement:
building SQL text in Drizzle costs about as much as parsing the result
in postgres.js.
> - Every authenticated agent API request (about 0.3 per agent per
second, plus tool calls) runs three fixed statements in
`actorMiddleware`: the agent row, the run authentication projection, and
the responsible user's company memberships. The server built the SQL
text of all three again for every request.
> - A local micro-benchmark shows that building the text is 15 to 40 µs
of CPU per statement for these shapes, and that a reused prepared query
costs the same as a raw driver call. Drizzle's `tracing.ts` wrapper is a
no-op (0.1 µs).
> - This pull request prepares the three statements once per database
client with placeholders, and executes them with the request's values.
The driver receives the same SQL text and parameters as before.
> - The benefit is about 40% less main-thread CPU for each authenticated
agent request in a local benchmark.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The CPU cost of agent JWT and agent key authentication on every agent
API request.
**Subsystem affected**
Server request authentication (`server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts`).
**Current behavior**
For every agent request, `actorMiddleware` builds three Drizzle queries
from scratch: `agents` by id, the `heartbeat_runs` authentication
projection (with `jsonb_build_object` and two `run_identity_contexts`
subqueries), and the `company_memberships` read with an `exists` on
`auth_users`. Building and preparing them costs 14, 38 and 40 µs of CPU
(median of 7 passes of 5,000 builds).
**Proposed behavior**
Build each statement one time for each database client, with
`sql.placeholder` values. Execute the prepared query with the request's
values. The SQL text, the parameters, and the returned rows stay the
same.
**Reason and benefit**
At 1,000 agents, agent API traffic is several hundred authenticated
requests per second on one main thread. Measured CPU per authenticated
request (local, embedded PostgreSQL, `process.cpuUsage()`, 2,000
requests per pass):
| Run (interleaved) | Before: median (min–max) µs | After: median
(min–max) µs | Load average |
| --- | --- | --- | --- |
| 1 | 436 (363–599) | 200 (183–224) | 10.8 / 10.1 |
| 2 | 489 (430–535) | 312 (212–392) | 11.6 / 13.4 |
| 3 | 461 (442–562) | 281 (247–409) | 13.1 / 12.9 |
| separate | 383 (322–810), 301 (268–476) | 169 (156–439), 193 (174–215)
| 9.5 / 9.0 |
The machine is shared (18 cores), so the spread is wide. Every pair
shows the after run lower. The drop is larger than the build cost alone,
because the request also allocates less (fewer builder objects to
collect).
**Breaking changes**
None. The statements, their parameters and the result shapes do not
change.
**Additional context**
Per-statement cost attribution from the new
`packages/db/scripts/benchmark-statement-cpu.ts` (CPU µs per statement,
median of 5 passes of 2,000, two runs, concurrency 1, load average
20–26):
| Statement | Drizzle builder | Build only (`toSQL`) | Drizzle
`.prepare()` reused | `unsafe(text, params)` | `unsafe(..., {prepare:
true})` |
| --- | --- | --- | --- | --- | --- |
| `issues` parent by id | 105 / 122 | 17 / 18 | 59 / 75 | 57 / 63 | 30 /
33 |
| `issues` full row by id | 277 / 326 | 136 / 153 | 96 / 81 | 97 / 77 |
48 / 33 |
| `company_memberships` by principal | 128 / 103 | 35 / 33 | 71 / 66 |
80 / 63 | 37 / 32 |
| join with an in-list of 10 | 169 / 173 | 57 / 57 | 83 / 88 | 109 / 80
| 42 / 40 |
| update by id | 110 / 106 | 25 / 26 | 75 / 80 | 76 / 73 | 39 / 40 |
| insert `activity_log` | 116 / 125 | 34 / 35 | 65 / 57 | 64 / 55 | 41 /
30 |
Drizzle's no-op tracer (`startActiveSpan`) measured 0.1 µs per call.
## What Changed
- `server/src/services/prepared-statements.ts`: add
`preparedStatement(prepare)`, the single-statement form of the existing
`preparedStatements` helper (one prepared query for each `Db`, held in a
`WeakMap`).
- `server/src/middleware/agent-run-authentication.ts`: split the query
into `agentRunAuthenticationQuery(db, binding)`;
`loadAgentRunAuthentication` executes the query prepared once with
placeholders.
- `server/src/middleware/auth.ts`: add `agentAuthenticationQuery` and
`responsibleUserMembershipsQuery`; the JWT path, the agent key path and
`loadResponsibleUserMemberships` execute their prepared forms.
- `server/src/__tests__/agent-auth-prepared-statements.test.ts` (new):
proves that each prepared statement sends the same SQL text and
parameters and returns strictly equal rows, and that repeated agent
requests do not build the statements again.
- `server/src/__tests__/agent-auth-middleware.test.ts`: the test
double's select chain now supports `.prepare().execute()`.
- `server/src/__tests__/prepared-statements.test.ts`: covers
`preparedStatement`.
- `packages/db/scripts/benchmark-statement-cpu.ts` (new): CPU per
statement for the builder, build only, a reused prepared query, `unsafe`
with and without `prepare: true`, and the no-op tracer. Run with `cd
packages/db && ITERATIONS=2000 PASSES=5 node --import tsx
scripts/benchmark-statement-cpu.ts` (`CONCURRENCY`, `ONLY`, `VARIANTS`
are optional).
- `scripts/benchmark-agent-request-auth-cpu.ts` (new): CPU per
authenticated agent JWT request through `actorMiddleware`. Run with
`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-request-auth-cpu.ts`.
Both benchmarks start their own embedded PostgreSQL in a temporary
directory.
## Verification
- Red/green: the "builds the request authentication statements once per
database client" test failed on `origin/master` (`select` called 12
times in 3 requests) and passes with this change.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-auth-prepared-statements.test.ts
src/__tests__/prepared-statements.test.ts
src/__tests__/agent-auth-middleware.test.ts
src/__tests__/agent-run-authentication.test.ts
src/__tests__/agent-jwt-request-authentication.test.ts
src/__tests__/auth-session-route.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/local-agent-process-boundary.test.ts
src/__tests__/liveness.test.ts src/__tests__/inbox-mine-routes.test.ts
src/__tests__/board-bootstrap-routes.test.ts
src/__tests__/agent-auth-identity-transfer.test.ts
src/__tests__/prepared-statements-driver.test.ts
src/services/fresh-continuation-fallback.test.ts`: all pass.
- CPU before and after: the tables above.
## Risks
- Low risk. The prepared query uses Drizzle's own `PgPreparedQuery`, the
same execute path that the builder uses. The test pins the SQL text, the
parameters and the rows against the builder.
- Drizzle's `.prepare(name)` on postgres.js does not send a statement
name to the database (see #21201). With `DATABASE_PREPARED_STATEMENTS`
unset, the driver sends the same unnamed statement as before, so
connection poolers see no change.
- A test double for `Db` must now support `.prepare().execute()` on
these selects. One test double changed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.3
|
||
|
|
ea8c734cb9 |
fix(chat): park GitHub terminal responses that wait for a newer run or task change (#21705)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel posts one App-owned response comment per
accepted request. The terminal-response sweep
(`terminal-response-work.ts`) finds processed, non-final responses whose
task's latest run has ended, and asks the acknowledgement owner to post
the final text.
> - On paperclip.staging, 1,557 processed `github_response_comment` rows
are not final. None changed in the last 5 minutes, and the oldest dates
from 2026-10-28. The sweep reads all of them again on each company-wide
rebuild (every 30 s per company), with one page read plus three
statements per row. Two of those statements run at about 13/s each in
`pg_stat_statements`.
> - The sweep cannot make progress on these rows. Their task's latest
run ended without owning the request (a later request started it), or
the pause is already shown. Nothing that the decision reads changes
until a new run starts, the task status changes, or the row changes.
> - This pull request records that wait on the row and skips it on broad
rebuilds until one of those inputs changes.
> - The benefit is that an idle backlog costs one page read per rebuild
instead of about three statements per row, and no request loses a path
to settle.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On paperclip.staging (2026-11-15, about 01:45–02:25Z), two single-row
statements ran at nearly the same rate:
- `select id, run_id, company_id, status, responsible_user_id,
message_id, parent_context_id from run_identity_contexts where id = $1
and run_id = $2 and company_id = $3 and status = $4 limit $5` at
13.37/s.
- `select chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join chat_conversations … join chat_endpoints … where
chat_message_links.company_id = $3 and comment_id = $4 …` at 13.20/s.
They are `provenanceReads.context` and `provenanceReads.link` in
`run-origin.ts`. The terminal-response sweep reaches them through
`readGitHubResponseRun` for each row it loads.
The backlog of `github_response_comment` actions on staging:
- `processed`, `delivery_id` set, `result.final` not `true`: 1,557 rows
(the terminal sweep's set). None updated in the last 5 minutes. The
oldest `updated_at` is 2026-10-28.
- `result.code` of those rows: null 1,387, `acknowledgement_unconfirmed`
150, `acknowledgement_superseded` 20.
- The acknowledgement set (`received`, or `failed` and retryable) is
empty.
- Other rows: processed and final 6,645, failed and not retryable 265,
cancelled 116.
**Root cause**
`entry()` returns `null` ("nothing to post now") when
`githubResponseExecution` has no body. That happens when the task's
latest run has ended but does not own the request and the task is not
blocked. It also happens when the task is blocked and the row already
shows this pause (`pausedVersion` matches). The page query still selects
these rows, because the latest run has ended (or the task is blocked)
and the row is not final. Nothing records the decision, so each
company-wide rebuild reads every row again: `readGitHubResponseRun` (one
statement) plus `chatRunSourceMessageId` (the context and link reads)
for each row. The codes do not matter. `acknowledgement_unconfirmed` is
a retry code that a later successful acknowledgement keeps, and
`acknowledgement_superseded` marks a request whose shared PR comment has
a newer version. All three groups wait for the same thing: a newer run
that owns the request, or a task status change.
**Expected behavior**
A response that cannot progress costs nothing until the run, the task or
the response changes. A response that can progress settles as before.
**Steps to reproduce**
1. Create a GitHub endpoint and a task with several processed, non-final
response comments (codes null, `acknowledgement_unconfirmed`,
`acknowledgement_superseded`).
2. Give the task a newer ended run whose identity context names a later
request, so the run does not own the older requests.
3. Register the terminal-response sweep and send a company-wide hint
every 30 s.
4. On master, each rebuild reads every row again with about three
statements per row and writes nothing.
**Paperclip version or commit**
master at
|
||
|
|
82b243ab07 |
perf(server): bound dedicated database connections with a fair queue (#21693)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server holds long advisory-lock transactions on dedicated
database connections (`withDedicatedDbConnection`): workspace source and
lifecycle locks, the skill cache lock, agent-directory operations,
status cards and others
> - `withDedicatedDbConnection` kept at most 4 idle connections, but it
had no limit on connections in use. A run start can hold two at once,
because the source lock nests the lifecycle lock
> - After a restart or deploy, a burst of run starts can open hundreds
of connections. That can exceed the database's `max_connections` at the
moment recovery needs the database most. The documented per-process
total (`DATABASE_POOL_MAX` + 2) was also wrong
> - This pull request bounds the dedicated connections in use with a
fair queue, keeps nested and in-transaction callers deadlock-free, and
states the real connection budget
> - The benefit is that a restart burst of 200 or 1,000 run starts
completes within the limit (at most 48 in use with the default limit of
24), where before most of the starts failed with "too many clients"
## Linked Issues or Issue Description
**What existing behavior does this improve?**
`withDedicatedDbConnection`, which gives long advisory-lock transactions
a connection outside the main pool.
**Subsystem affected**
Database client (`packages/db/src/client.ts`) and the connection-sizing
documentation.
**Current behavior**
Nothing bounds the dedicated connections in use; only 4 idle ones are
kept. A run start can hold two at once (the source lock nests the
lifecycle lock). A burst of run starts after a restart can open hundreds
of connections and exceed `max_connections`. The documented per-process
total (`DATABASE_POOL_MAX` + 2) is wrong.
**Proposed behavior**
A per-process limit on dedicated connections in use
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24) with a first-in,
first-out queue. A caller that already holds a dedicated connection, or
runs inside an open transaction, never waits, because its wait could
deadlock with the holder that blocks it. After
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (default 30 s), a queued caller
gets a connection over the limit, so a lock cycle that PostgreSQL cannot
see ends in a delay, not a hang. The documentation states the real
per-process budget.
**Reason and benefit**
At 50 agents, staging showed about 9 new connections per second from
this function. At 1,000 agents, a restart burst can take the database
down when recovery needs it most. With the default limit of 24, a burst
of 200 or 1,000 starts completes with at most 48 dedicated connections
in use.
**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). Reusing the
outer session for a nested lock was considered. The locks are
transaction-level advisory locks (`pg_advisory_xact_lock`), so a nested
lock in the outer transaction would be held until the outer transaction
ends, which changes lock lifetimes and order.
## What Changed
- `packages/db/src/client.ts`: `withDedicatedDbConnection` admits
callers against a per-owner limit (`DATABASE_DEDICATED_CONNECTION_MAX`,
option `dedicatedConnectionMax`, default 24) with a first-in, first-out
queue. The slot is released after the connection is returned or closed.
- Nested callers (an active dedicated hold of the same owner in the
async context) and callers inside an open transaction of a client from
this process are admitted without a wait. The transaction marker is an
`AsyncLocalStorage` scope that ends when the transaction callback ends.
- A queued caller is admitted over the limit after
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (option
`dedicatedConnectionWaitMs`, default 30000).
- `dedicatedDbConnectionStats(db)` reports the limit, in-use, peak,
waiting, queued, over-limit and nested-over-limit counts.
- `PRIVATE_POOL_MAX_CONNECTIONS` names the private pool sizes that were
literals, and `databaseConnectionBudget()` returns the per-process worst
case: main pool + 9 private + 2 listeners + 2 x limit + 4 idle dedicated
= 73 with the defaults.
- `server/src/services/skill-package-store.ts`: an isolated package read
takes its package I/O permit before its dedicated connection, and takes
a second dedicated connection only for the cache trim after a publish.
Before, each of up to 32 queued readers held a dedicated connection
while it waited for one of 2 permits. `skillPackageStorageIoDepth()`
reports the queue for the store's test.
- `withDedicatedDbConnection(db, action, { admitWithoutWaiting: true })`
admits a caller at once. The skill package store passes it for the
dedicated connections it takes while it holds a package permit, so a
permit holder never waits for a slot that a cache-lock owner holds while
that owner waits for the permit. The overshoot is bounded by the permits
(2 storage, 2 response).
- `doc/DATABASE.md` and `docs/deploy/database.md`: document the
admission rules, the two new variables and the real connection budget,
which replaces "`DATABASE_POOL_MAX` + 2".
## Verification
- New `packages/db/src/dedicated-connection-admission.test.ts` (embedded
PostgreSQL, `max_connections` 100). A restart burst with the default
limit: each start holds a source advisory lock on a dedicated connection
and nests a lifecycle lock on a second one, 40 ms in all.
- Before (same burst, scratch test on master): 200 starts gave 151
failures ("sorry, too many clients already") and 99 dedicated sessions
open. 1,000 starts gave 823 failures and 99 sessions.
- After: 200 starts gave 0 failures, 45 sessions, peak in use 48, queue
wait p50 541 ms / p99 815 ms. 1,000 starts gave 0 failures, 40 sessions,
peak in use 48, wait p50 1,767 ms / p99 3,422 ms. The wait is the queue
of this synthetic burst of 40 ms holds. Before, the same callers waited
about as long for connection setup, and most then failed.
- Steady state: 20 concurrent holders under the limit are never queued
(`queued: 0`).
- Nested acquisition: with a limit of 2, two holders each take a nested
connection while a third caller is queued. All finish (no deadlock); the
queued caller runs after them.
- In-transaction caller: with a limit of 1, a pooled transaction holds a
row lock and then asks for a dedicated connection, while the one
dedicated holder waits for that row lock. The caller is admitted at once
and both finish. A hard limit would deadlock here.
- Resource holder: with a limit of 1, the slot holder waits for a
resource that a second caller holds. The second caller, with
`admitWithoutWaiting`, is admitted at once and both finish.
- Wait bound: with a limit of 1 and a 100 ms bound, a queued caller is
admitted over the limit after the bound.
- Budget: the sum of the actual `max` of every private pool equals
`databaseConnectionBudget().privatePools`, and the default total is 73.
- `skill-package-store.test.ts` ("does not load queued legacy bodies for
files reads"): 32 readers queue for package I/O while only the 2 active
readers hold dedicated connections, and the 33rd reader still gets
"busy" at once. With the old order and a limit of 24, CI showed only 22
readers reaching that queue. `skill-package-store`,
`skill-version-cache`, `company-skills-empty-runtime`,
`runtime-skill-preservation`, `company-skills-runtime-concurrency` and
`company-skills-service` pass locally. Two `company-skills.test.ts`
workspace-discovery tests fail locally on master too (same result with
this change), so they are not caused by it.
- `packages/db`: client, lazy-transaction, prepared-query,
response-capture, work-signal and dedicated-pool suites, 134 passed.
- `server`: skill-version-cache, status-card-evaluation-work,
git-workspace-expiration, execution-workspace-runtime-control-conflict,
agent-instruction-working-copies, company-skills-empty-runtime and
decisions-service suites: 413 passed. In one combined run under machine
load, git-workspace-expiration failed at file level with all its tests
passing; alone it passed 96 of 96.
- `npx tsc --noEmit` in `packages/db`: passes.
## Risks
- A caller that holds an application-level lock (for example a
filesystem lock) and then waits in the queue can block a dedicated
holder that needs that lock. The 30 s wait bound ends such a cycle with
a delay. The over-limit counter shows it.
- Nested and in-transaction callers do not wait, so the in-use count can
reach twice the limit with one level of nesting. The documented budget
counts this.
- In a large restart burst, run starts now queue for a slot instead of
opening new connections. Steady state (about 10 to 20 dedicated
connections at 1,000 runs) stays under the default limit of 24, so a
run's first turn does not wait there.
- The checkpoint pool (3 connections) can limit throughput at 1,000 runs
on a hosted database. This pull request does not change it; it needs an
owner decision on the hosted connection budget.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9c230ace94 |
fix(heartbeat): settle deferred chat wakes that can never resume instead of retrying every pass (#21678)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat deferred-receipt worker resumes wakes that waited
behind a running task execution
> - A chat message that arrives while its task runs is saved as a
deferred chat wake
> - When the task finishes (done or cancelled) first, the saved wake can
never resume, but the sweep retried it forever and touched `updated_at`
on every pass
> - An idle instance did one write and one failed resume about 4 times a
second, and the log line did not say why
> - This pull request gives each such wake one recorded outcome, and
makes every other failure wait on a stored, spaced, bounded next attempt
> - The benefit is an idle database stays idle, and a person sees the
existing "not started" result instead of silence
## Linked Issues or Issue Description
**What happened?**
The stranded deferred-queue sweep in
`server/src/services/heartbeat/queue.ts` selects
`deferred_issue_execution` chat-inbound wakes. On every pass it runs
`update agent_wakeup_requests set updated_at = now()` for each wake.
Then `resumeDeferredChatWake` fails because the task is done or
cancelled, so the chat authority denies the work start
(`task_assignment`). Nothing records the failure, so the next pass
repeats it.
**Steps to reproduce**
1. Save a chat message on a task while a run is active, so the message
becomes a deferred chat wake.
2. Let the run finish and mark the task done.
3. Leave the instance idle. The log shows "saved chat source could not
resume after recovery" again and again for the same request ids.
**Expected behavior**
A wake that can never resume gets one terminal outcome with a stored
reason. A wake that can resume later waits on a stored time. An idle
instance runs no statements for these wakes.
**Paperclip version or commit**
master at
|
||
|
|
11b09b7234 |
refactor: remove doc/execution-semantics.md and its references (#21699)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents read the docs under `doc/` for background on run execution > - `doc/execution-semantics.md` grew to 6,310 lines of per-PR notes > - It was hard to read and it drifted from the code > - The code and tests are the source of truth for execution behavior > - This pull request removes the file and every reference to it > - The benefit is one less stale document that readers can mistake for a requirement ## Linked Issues or Issue Description **Issue type** Outdated or unclear documentation. **Where is the issue?** `doc/execution-semantics.md`, and the docs, rules, skills, maps, code comments and log text that link to it. **What's wrong?** The file is 6,310 lines of notes added one PR at a time. Much of it repeats or contradicts the code. Readers and agents treat it as a contract, but the code and tests decide the behavior. **Suggested fix** Delete the file. Remove each link to it. Keep the documents that link to it. Point at code where a pointer is still useful. Do not move the content to another doc. Refs #10664 (an open docs PR that edits this file; it becomes obsolete). ## What Changed - Delete `doc/execution-semantics.md`. - Remove links and pointers in `doc/PRODUCT.md`, `doc/LOW-TRUST-PRESETS.md`, `doc/TASK-WATCHDOG.md`, `doc/composer-stop.md`, `doc/native-workspace-finalization-recovery.md`, `doc/run-log-events.md`, `doc/runner-internals.md` and `doc/workspace-repositories.md`. - `doc/TASK-WATCHDOG.md` now points at `server/src/modules/active-run-watchdog/` for the silent active-run watchdog. - Remove the link in `rules/upgrade-safe-run-state.mdc` and `maps/feature-map/recovery.md`. - Update the skills `diagnose-why-work-stopped`, `fix-failed-paperclip-run` and `terminal-bench-loop`. The `fix-failed-paperclip-run` skill now points at `server/src/services/heartbeat/` and `server/src/services/recovery/`. - Remove the file from the source anchors in `.devin/wiki.json`. - Remove the doc name from four code comments in `server/` and `ui/`. - The failed-attachment log message no longer ends with a link to the removed doc. Two tests now check the message text instead of the link. - Leave two dated plan files under `doc/plans/` unchanged. They are history. ## Verification - `git grep -i execution-semantics` finds no hits outside `doc/plans/`. - Every relative Markdown link in the changed files resolves. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs` passes (808 tests). - `pnpm verify` was not run locally. The worktree has no `node_modules`. CI runs the changed Vitest files. ## Risks - Low risk. Most changes are docs and comments. - One runtime log line is shorter. It no longer has a link. No logic changes. - Open PRs that edit the deleted file, such as #10664, will conflict. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and shell access. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
adc3356ad3 |
fix(native): retry a coordinator claim that loses a deadlock instead of failing the run (#21703)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native (Paperclip Runner) run claims its `native_run_finalizations` coordinator before it starts any provider work. The claim is one transaction. It locks the coordinator row, then the run row. > - A writer that holds the run row and then waits for the coordinator row closes a lock cycle. PostgreSQL then aborts the claim with a deadlock (`40P01`), because the claim waited first. > - The aborted claim rolls back whole and starts no provider work. But the raw database error failed the run before its first turn. > - The failure path then preserved the prepared workspace. That stamped the unclaimed coordinator `native_continuation_requires_reconciliation`, and `failRun` used it as the run's error code. The task waited for reconciliation. > - A transient database fault must never end a run like this. The target rate of runs that need Retry is 0%. > - This pull request retries the claim transaction on transient lock contention with the shared bounded helper. > - The benefit is that a fresh run that loses a lock race at its claim continues to its provider turn by itself. ## Linked Issues or Issue Description **What happened?** On a hosted QA stack, a fresh native assignment run (not a retry) failed after about 50 s. The run status was `failed`. The error code was `native_continuation_requires_reconciliation`. The error text was `Database query failed (40P01)`. The stderr was empty. The task needed reconciliation. **Steps to reproduce** 1. Prepare a native run with an unclaimed coordinator (`observed`, attempt 0), as runtime selection leaves it. 2. In another transaction, lock the run row. 3. Start the native session. Its claim locks the coordinator and waits for the run row. 4. In the other transaction, lock the coordinator row. 5. PostgreSQL aborts the claim with `40P01`, and `executePaperclipNativeSession` rejects with that error. **Expected behavior** The claim retries after the competing transaction commits. The run reaches its provider turn. **Paperclip version or commit** `master` at |
||
|
|
c38a471042 |
perf(server): download stale agent files before the agent row lock (#21694)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Durable agent files live in object storage. Each server keeps a local copy (the projection), marked with the head generation it matches > - When the projection is stale, the next locked read or write restores it: it downloads every file from object storage. This happens for the first run of each agent on a new server (every deploy on hosted ephemeral servers) and after another server wrote > - The download ran inside the transaction that holds the `agents` row lock (`FOR UPDATE`), on the main pool for the run-start read (`readCommittedForRuntime`) and on the 2-connection file-storage pool for the store. A download can take up to 30 s > - After a deploy at 1,000 agents, this is one long row lock and one pinned connection per agent, all at once, and every writer of that agent waits behind it > - This pull request downloads the files outside the transaction and takes the lock again only to swap them in > - The benefit is that the row lock is held for milliseconds instead of the download time (0 ms observed instead of 321 ms with a 300 ms download in the test), and writers do not wait for a download ## Linked Issues or Issue Description **What existing behavior does this improve?** The restore of a stale local copy of an agent's durable files (the first run of each agent on a new server, or after another server wrote). **Subsystem affected** Server: durable agent files (`agent-file-durability.ts`, `agent-file-store.ts`, `agent-instruction-revisions.ts`). **Current behavior** The restore downloads every file from object storage inside the transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled connection. A download can take up to 30 s, and every writer of that agent waits behind it. **Proposed behavior** Under the lock, detect that a download is needed and roll back. Download the head's files, verified by digest, into a private staging directory with no lock. Run the transaction again, and swap the staged files in only when the head generation is unchanged. After two changed heads, restore under the lock as before. **Reason and benefit** After a deploy at 1,000 agents, every agent's first run restores at once. The row lock and connection are now held for milliseconds instead of the download time, and writers do not wait for a download. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path had the same problem for uploads and was fixed earlier with uploads before the lock (`uploadAgentFilesBeforeCommit`); this is the restore side. Reading the head before every transaction was considered; it adds a query to every call, while the deferral costs nothing when the projection is current. ## What Changed - `agent-file-durability.ts`: the download loop moves into `downloadAgentFiles`, unchanged. `withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside the scope, the first restore of an attempt that needs a download throws a private deferral error, which rolls back the transaction. The wrapper stages the download in `.agent-files-<uuid>` next to the root and runs `work` again. `restoreDurableAgentFiles` uses a staged directory only when its generation equals the head it reads under the lock. Staged directories are removed when the scope ends. - Only the first restore of each attempt can defer, so nested restores of other agents keep their old behavior. Outside the scope, the behavior is unchanged. - `agent-file-store.ts` (`locked`, so every store read, write, apply and working-copy preparation) and `agent-instruction-revisions.ts` (`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run their transactions in the scope. ## Verification - New test `server/src/__tests__/agent-file-restore-locking.test.ts` on embedded PostgreSQL, with an object-storage fixture: - Hold time, with a 300 ms download, through `readCommittedForRuntime` (the run-start read). Before: the download ran with the agent row locked, and the row was locked 321 ms of 322 ms. After: the download ran with the row unlocked, and a sampler saw the row locked for 0 ms. - Concurrency: a board write of the same file runs while the reader's download is blocked. Before: the writer waited for the reader's download (5 s timeout). After: the writer commits a new head during the download. The reader then discards its stale staged copy and returns the writer's content. The projection is marked at the writer's generation, and no staging directory is left. - Failure: when object storage fails, the read fails once with the storage error, after one attempt made without the row lock, and leaves no staging directory. - Progress: when the head generation changes during each of two downloads, the third attempt restores under the lock as before (3 downloads), and the projection matches the final head. - `npx vitest run` for agent-file-checkpoint-locking, agent-file-checkpoints, agent-file-durability, agent-file-restore-locking, agent-instruction-revisions, agent-instruction-working-copies, agent-instructions-service, agent-directory-working-copies and agent-instructions-routes: 287 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A deferred attempt runs the transaction body again. The deferral happens at the first restore, before the body writes files, and the transaction rollback undoes database work, so the second attempt starts from the same state. - A staged download uses disk next to the agent root until the scope ends, the same as the temporary directory of the old download. - If the head changes during every download, the third attempt downloads under the lock as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cb9fe3f72d |
perf(server): capture a remote agent-file checkpoint before the directory lock (#21690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote run saves its agent files at the end of every turn (`checkpointWarm`) > - On hosted (ephemeral) servers, this save holds the cross-controller `agent-directory` advisory lock in an open transaction on a dedicated database connection > - The lock was held across the full remote capture: an asset upload, the checkpoint script, the download and the scratch cleanup in the sandbox. Each step is a sandbox round trip of hundreds of milliseconds or more > - At 1,000 concurrent agents, connection-seconds are the scarce resource, and this hold occurs on every turn of every warm remote run. The lock is per agent, so a second run of the same agent got "Agent directory operation is already active" for the whole capture > - This pull request captures the remote files before the lock, and takes the lock only to commit > - The benefit is that the lock and its connection are held for the commit only (85 ms instead of 2.5 s in the test with a 200 ms sandbox), and no sandbox call runs under the lock ## Linked Issues or Issue Description **What existing behavior does this improve?** The per-turn save of agent files for a warm remote run (`checkpointWarm`) on hosted servers. **Subsystem affected** Server: agent directory working copies (`server/src/services/agent-directory-working-copies.ts`). **Current behavior** The save holds the per-agent `agent-directory` advisory lock, an open transaction and a dedicated database connection across every sandbox round trip of the remote capture. A second run of the same agent gets "Agent directory operation is already active" for that whole time. **Proposed behavior** Capture the remote files with no lock. Under the lock, use that capture only when the working-copy row is unchanged. Otherwise discard it and capture again under the lock, as before. **Reason and benefit** At 1,000 concurrent agents, connection-seconds are the scarce resource, and this hold occurs on every turn of every warm remote run. The lock is now held for the commit only. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). A lease row in place of the advisory lock was considered; it is a larger change, and the optimistic capture removes the slow part without a new protocol. ## What Changed - `agent-directory-working-copies.ts`: `checkpointWarm` captures the remote checkpoint before `serial()` for a disposable remote copy (hosted servers only), with the same inputs that the locked capture uses. - Under the lock, `checkpoint()` reuses that capture on its first attempt only when the current row equals the row the capture used (`adoptionAuthority`, which excludes only `updatedAt` and `nextAttemptAt`) and the target is the same. If the row changed, it captures again under the lock, as before. - If the early capture fails, the code logs a warning and captures under the lock, as before. The early capture's scratch is removed after the lock is released. - Local copies and non-hosted servers are unchanged. ## Verification - New test `server/src/__tests__/agent-directory-remote-checkpoint-lock.test.ts` on embedded PostgreSQL, with the real checkpoint transport and a fixture sandbox runner that takes 200 ms per call: - Before: 11 of 11 sandbox calls ran while the `agent-directory` advisory lock was held. The lock was held 2,482 ms of a 2,515 ms checkpoint. - After: 0 of 11 sandbox calls ran under the lock. The lock was held 85 ms of 2,863 ms. - Concurrency: a competing collector (`collectStopped` from a second service instance) runs while the early capture is blocked in the sandbox. Before: the collector failed with "Agent directory operation is already active". After: the collector saves and releases the copy, and the checkpoint discards its early capture and returns the collector's row unchanged. The canonical file has the final turn's content once, and no scratch directory is left. - `npx vitest run src/__tests__/agent-directory-working-copies.test.ts src/__tests__/agent-file-checkpoint-locking.test.ts src/__tests__/heartbeat-agent-directory-busy.test.ts src/__tests__/agent-directory-lock-crash-point.test.ts src/__tests__/agent-instruction-working-copies.test.ts`: 189 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - The capture now runs while another controller can operate on the same copy. The capture only reads the remote directory into private scratch. The commit uses it only when the row is unchanged, so the result is the same as one of the orders the lock allowed before: the checkpoint first, or the other operation first and then the old locked checkpoint. - The agent can change remote files between the capture and the commit. That was also true before, because the agent is not frozen during a checkpoint. - Changed bytes are still uploaded to object storage inside the lock (before the agent row lock). Moving that upload out is a possible follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
328a954cc1 |
test(scaling): add a real-agent ramp harness for hosted instances (#21695)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d6acbf50e6 |
refactor: cut DATABASE.md to setup and schema-change rules (#21700)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` tells every contributor and agent to read `doc/DATABASE.md` before a schema or migration change > - That file grew to 2,611 lines. About 1,950 of them were notes on the tables of one feature, added one pull request at a time > - The setup steps and the migration rules started at line 792, below those notes > - This pull request cuts `doc/DATABASE.md` to 377 lines: setup, and the rules that apply to every schema change > - The benefit is a guide that a person or an agent can read in full before a schema change ## Linked Issues or Issue Description Refs #21670 (the same cleanup for `doc/DEVELOPING.md`). **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/DATABASE.md` **What's wrong?** The file was 2,611 lines. It opened with a Better Auth upgrade note and 492 lines on task query performance. The schema change workflow started at line 1367. **Suggested fix** Keep database setup and the general rules in `doc/DATABASE.md`. Remove the notes on individual tables. The schema and its tests describe those. ## What Changed - `doc/DATABASE.md` is now 377 lines. It keeps the three database modes, client tuning, the schema change workflow, migration compatibility, the migration authoring checklist, migration snapshots, the rule for retries after a lost connection, transaction-aware work signals, plugin namespaces, and backups. The wording of the kept text is unchanged. - New `doc/database-client-internals.md` holds the response-size diagnostic, the notes on the postgres.js patch, and the transaction memory note. The text is moved without wording changes. - "Experimental remote workspace snapshots" moved to `doc/workspace-storage.md`, which links to it. - "Execution admission accounting" moved to `server/src/modules/README.md`, which links to it. - Deleted about 1,950 lines of notes on one feature each. No other file linked to them. The deleted sections are: Better Auth 1.7.16 upgrade, Durable native provider backups, Workspace cohort admission proofs, Original local Git sources for archived workspaces, Proactive agent chat messages, Persistent case handoffs, Task query performance, Tool profile reads, Execution identity row locks, Remote workspace row locks, Managed adapter settings, Cloud configuration migration receipt, Cloud runtime identity singleton, Unclaimed Cloud warm standby, Native finalization history and idle sleep, Feedback history and idle sleep, Resource membership tables, Decision training snapshot retention, Decision queues and triage provenance, Native runner persistence, Chat communication snapshots, Telegram private draft identities, Attachment upload provenance, Question-response delivery receipts, Secret storage, Persistent agent conversations, Resource lifecycle events, Legacy controller ownership, Agent file persistence and legacy revisions, Large API response snapshots, Runner API response reservations, Internal agent commentary, Agent identity keys and backups, Slack app registration, Slack manager workspace grants, Installed skill packages, Project repository skill synchronization, Task workspace records, Execution recovery query performance, Agent runtime synchronization journal, Retired Pipelines storage, Prompt injection classification, Deferred conversation requests, GitHub inherited review-wait reads, Sandbox creation ownership, Object-backed run evidence, Cloud runtime configuration singleton, Independent accounting and budget evaluation, Standard workspace and Classic opt-in, Project environment rolling compatibility. - Also deleted from kept sections: the subscription-table note and the issue-reference backfill note in the embedded mode section, the HTTP access log note and the per-endpoint retry notes in the connection loss section, and the notes on migrations `0355`, `0313`, and `0314` in the authoring checklist. - `AGENTS.md` now states what `doc/DATABASE.md` is for. ## Verification - A script built the new files by line range from the old file. It asserts the old line count and the first line of each range. - A link check on every changed Markdown file, and on the four files that link to a `DATABASE.md` anchor, found 0 missing files and 0 missing anchors. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `pnpm verify` was not run. This worktree has no installed dependencies, and the change has no runtime code. ## Risks - The deleted notes were not all read in full before deletion. The decision was by section: a section about one feature's tables, with no inbound link, was deleted. The text stays in git history. - Two deleted sections describe procedures for an operator: the duplicate-account check before the Better Auth 1.7.16 migration, and the secret storage tables. Restore them into a focused doc if operators still need them. - No runtime behavior changes. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>canary/v2026.1115.0-canary.2 |
||
|
|
958eda6f16 |
fix(native): wait for a live outgoing server before resuming its warm session in a deploy overlap (#21697)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. The outgoing server still serves while the incoming server starts new runs. > - A warm agent session stays with the server that ran its last turn. That server closes the session about 1 s after it leaves new run starts to the replacement (#21627). > - A follow-up that started within about 1 s of its predecessor's finish read the session as suspended, while the outgoing server was still closing it. The follow-up did not wait. It started a second Runner with the wrong OpenCode launch files, and the Runner refused the launch. > - The same-run retry then went to the outgoing server, which had already left run starts. The run failed, and a blocked parent showed a Retry during its scheduled resume. > - This pull request makes the follow-up wait for the live outgoing server, rebuilds the admitted launch files on the new Runner, keeps same-run retries on the replacement, and shows the scheduled resume for a blocked parent. > - The benefit is that a follow-up during a deploy overlap finishes without a failed run or a Retry banner. ## Linked Issues or Issue Description Refs #21627 **What happened?** A zero-downtime release-swap test (Gate B, wave B1 on master |
||
|
|
54cde3353b |
perf(recovery): back off assigned-work re-checks that find the same refused state (#21691)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work sweep checks every open task assigned to an agent
(`todo`, `in_progress`, `blocked`). When a task has nothing running,
nothing pending and no one else's move, it claims one automatic
continuation and asks the wake queue to start it.
> - When the wake queue refuses that continuation, the claim is recorded
as "not started" and the task is checked again 90 seconds later. The
per-task spacing (2, 4, 8, then 15 minutes) lets it claim again. The
refusal is not counted toward the hourly cap, so this never ends.
> - Each cycle reads all of the task's facts, writes an activity entry
and saves a skipped wake row. For 10 such tasks this is about 5,000
statements and 50 claims per hour, forever, on an idle server.
> - This pull request stores a fingerprint of the task's facts with each
refused continuation. A later check that finds the same facts waits,
with reads only, on a backoff that doubles to a 6-hour ceiling.
> - The benefit is that a task that cannot progress costs almost nothing
while idle, and any real change on the task is still picked up within
seconds.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
`act` in `server/src/services/recovery/assigned-work-sweep.ts` repairs a
`no_pending_path` or `retry_offered` task with `deps.enqueueWakeup`.
When admission refuses (for example the assignee has `wakeOnDemand:
false`), `act` returns `enqueue_refused` with `retryAt` 90 seconds
later. The next check claims again as soon as `continuationAllowance`
allows: 2, 4, 8, then 15 minutes. Refused claims do not count toward
`CONTINUATION_CAP`, so the task is re-claimed forever. Each claim writes
an `issue.assigned_work_continued` activity entry ("could not start yet.
It will try again shortly."), and each wake attempt saves a skipped wake
row.
Census on a staging-shaped fixture (real PostgreSQL, real report): 60
tasks in review, 10 running, 5 assigned to a paused agent and 10 whose
continuation is refused. Only the 10 refused tasks are violations
(`retry_offered`). Tasks in review are not in the sweep's scope at all,
and a paused assignee is an "agent paused" wait. So on a hosted instance
where most open tasks are in review, only the tasks whose continuation
is refused pay this cost. One claim check costs 102 statements. A
backed-off re-check costs 34 statements and writes nothing.
**Expected behavior**
A re-check that finds the same state as the last refused continuation
stores nothing and backs off. A real change on the task (a run finishes,
a status change, an assignment, a budget or pause change, a new wake)
re-checks it promptly.
**Steps to reproduce**
1. Create 10 tasks in `in_progress`, assigned to an agent with
`runtimeConfig.heartbeat = { enabled: false, wakeOnDemand: false }`,
each with a failed run.
2. Start the issue-state recovery work (it registers the assigned-work
sweep).
3. Leave the instance idle. The sweep claims and is refused about every
2 to 15 minutes per task, forever.
**Paperclip version or commit**
master at
|
||
|
|
f4e025a1cf |
perf(recovery): wait for an assignee that cannot run instead of retrying its continuation every pass (#21687)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops and recovery decides to continue the task in a
fresh run, it records a pending continuation. The execution-continuation
worker delivers it with `heartbeat.wakeup`.
> - When the task's assignee cannot take work (paused, terminated,
waiting for hire approval, or an invalid reporting chain), the wake
refuses. The worker recorded only an "attempted at" time and tried again
on the next 15-second pass, forever.
> - Each refused wake also saved a new skipped wake row. An idle server
with one such task did about 13,000 statements and 390 wake attempts
every 2 hours.
> - This pull request makes delivery check the assignee with the same
rule the wake uses. When the assignee cannot run, the continuation waits
on a stored retry time and is re-admitted at once when the assignee can
run again.
> - The benefit is that an idle server does almost no work for these
tasks, and a resumed agent still gets its continuation within seconds.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
`deliverReconciledExecutions`
(`server/src/services/execution-recovery-resolution.ts`) offers each
pending continuation (`continuationDelivery: "pending"`) to
`heartbeat.wakeup`. When the assignee is not invokable, the wake throws
"Agent is not invokable in its current state" and saves a skipped
`agent_wakeup_requests` row. The worker catches the refusal, records
`continuationDeliveryAttemptedAt`, and the continuation schedule
(`executionControlSchedule.continuation`) asks for another pass in 15
seconds because a pending row exists. Nothing ever changes, so the loop
never ends.
Measured on real PostgreSQL with the real coordinator and the real wake,
on a fake clock, for one pending continuation over 2 idle hours:
| Assignee status | Wake attempts (master) | Statements (master) |
| --- | --- | --- |
| paused | 380 | about 12,980 |
| terminated | 376 | about 12,820 |
| pending_approval | 395 | about 13,000 |
An agent in `error` is not affected: it is invokable, and the wake
delivers the continuation on the first pass. Only the statuses and
conditions that make an agent not invokable cause the loop.
**Expected behavior**
A continuation that cannot be delivered because its assignee cannot run
does not try again every pass. It waits for a real event (the agent can
run again) or a spaced backstop time.
**Steps to reproduce**
1. Create a task in `in_progress` assigned to an agent, with a failed
run.
2. Record a pending continuation for that run, as the fresh-continuation
fallback does.
3. Pause, terminate or un-approve the agent.
4. Run the execution-continuation worker. It calls the wake about every
18 seconds and saves a skipped wake row each time.
**Paperclip version or commit**
master at
|
||
|
|
c66530da63 |
fix(server): release settled native run process identity at graceful shutdown (#21689)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server process records the local process of each native run in
`heartbeat_runs` (`process_pid`, `process_group_id`,
`process_started_at`, `process_host_hash`)
> - Only a server in the same PID space can prove that such a process
stopped. `proveLocalOwnerGone` does not accept a recorded process group
from another container
> - Each deploy replaces the server process. Settled native runs that
the old server ran keep their process fields, and no later server can
release them. Recovery sweeps had to learn to skip these rows one by one
> - The stopping server is the one process that can check its own PID
space. After its graceful drain, it can see that a settled run's process
exited
> - This pull request adds a graceful-shutdown step that releases the
process identity of those runs, with the same rules that the running
server already uses, in one candidate read and one update
> - The benefit is that a deploy no longer adds stranded process records
for the runs that the stopping server could prove stopped, and a
successor on the same host has nothing to release at startup
## Linked Issues or Issue Description
Refs #21590
**What happened?**
After a graceful shutdown, settled native runs (`failed`, `cancelled`,
`timed_out`, `interrupted`) that the stopping server ran keep
`process_pid` and `process_group_id`. The successor runs in a new
container with a different `process_host_hash`. It cannot prove that the
recorded process stopped, so the rows stay forever. A successor on the
same host releases them at startup with about 10 statements for each
run.
**Expected behavior**
The server that ran the process knows that the process exited. A
graceful shutdown should release these records before the server stops.
An abrupt stop should keep today's behavior.
**Steps to reproduce**
1. Run native work so that some runs settle while the server runs, or
settle during the shutdown drain.
2. Send SIGTERM to the server.
3. Count settled native runs with `process_pid` or `process_group_id`
set: the count does not go down after the deploy.
**Paperclip version or commit**
master at
|
||
|
|
3eabdd6d10 |
perf(server): size run-log uploads and object journal IO from open run logs (#21685)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every run of a workspace, and the target
is 1,000 concurrent agents on one server
> - Each native run writes its run log to object storage. The server
batches a run's log lines and uploads one batch per run each second
> - The server allowed only 4 log uploads and 8 object storage
operations at a time, for the whole process. These numbers were set when
50 agents was a lot
> - At 200 runs, the run logs fell about 8 seconds behind. At 1,000
runs, pending log bytes grow toward the 64 MiB process cap. Past that
cap, run events wait and log appends can fail
> - This pull request sizes the log upload slots and the object storage
operation slots from the number of open run logs, between the old floor
and a ceiling that comes from the storage client's socket budget
> - The benefit is that run logs stay about one flush interval behind at
200 runs, and fall 7.7 seconds behind instead of 44 seconds at 1,000
runs (p50), without more sockets than the storage client allows
## Linked Issues or Issue Description
Refs plan item 1g in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` ("Size the shared
limiters ... from the active run count").
**What existing behavior does this improve?**
The shared run-log upload limiter and the shared object journal
operation limiter.
**Subsystem affected**
Server: object storage run logs (`object-run-log-store.ts`,
`object-byte-journal.ts`) and the S3 storage client.
**Current behavior**
The process allows 4 run-log uploads and 8 object journal operations at
a time, for any number of runs. Each upload is 3 to 4 sequential object
writes. At 200 runs and 50 ms per write, the run logs commit with a p50
delay of 8.0 s, and only 55% of the log lines offered in a 20 s window
commit inside it.
**Proposed behavior**
The upload slots are one per 10 open run logs, at least 4 and at most
21. The operation slots are the upload slots plus 4 for reads, at most
25. That is half of the 50 sockets that the S3 client may open, so
downloads and other storage calls keep the other half. The S3 client now
sets its 50-socket limit explicitly, from the same constant.
**Reason and benefit**
With the change, 200 runs commit their logs at the flush interval (p50
1.1 s), and 1,000 runs commit with a p50 delay of 7.7 s instead of 43.8
s. Fewer pending bytes stay in memory, so the 64 MiB pending cap and its
admission failure are farther away.
**Breaking changes**
None. With 40 or fewer open run logs the limits are the same as before
(4 uploads, 8 operations).
## What Changed
- `server/src/lib/active-run-slots.ts`: a small helper that gives a slot
count from an active run count, clamped between a floor and a ceiling.
- `server/src/storage/s3-client-limits.ts` and `s3-provider.ts`: the S3
client sets `maxSockets` to 50 (the SDK default) from a named constant,
so the budget is explicit.
- `object-byte-journal.ts`: the object operation limit is the upload
slots plus a read reserve of 4, from the number of open run-log writers.
Open writers are counted with `trackObjectJournalWriter()`. When the
limit grows, waiting operations start at once. When it shrinks, running
operations finish and no new ones start until the count is below the
limit.
- `object-byte-journal.ts` also owns the upload limiter
(`acquireObjectJournalUpload`), process-wide, next to the writer count.
When a run log opens, waiting uploads start at once. When run logs
close, running uploads finish and no new upload starts until the running
count is below the new limit.
- `object-run-log-store.ts`: uses the shared upload limiter. A run log
registers when it begins and releases when it is finalized. The waiting
queue stays first in, first out, and each run has at most one upload in
flight, so one run cannot starve another.
- Review fix: the first revision kept the upload limiter per store, so a
run log opened by another store did not wake uploads waiting in this one
(a lost wakeup on growth). The shared limiter fixes it; a new test
covers growth and shrinkage with queued work.
- Tests: limit growth with uploads waiting and shrinkage with uploads
running; helper bounds; the storage socket budget; 200 open runs upload
at least 20 logs at once (4 before); 1,000 open runs stay at the ceiling
and every upload finishes.
## Verification
- `cd server && npx vitest run src/lib/active-run-slots.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts`: all pass.
- Red/green: with the slot count forced back to the fixed floor of 4,
the two new scale tests fail (4 uploads in flight); with the change they
pass.
- Simulation (in-memory object store with 50 ms per write, each run
appends 1.5 KB at 0.7 records/s, the recorded native event rate, 20 s
window, flush every 1 s). Counts are stable on a loaded machine because
the latency is simulated:
| Runs | Build | Peak concurrent writes | Writes/s | Log lines committed
in window / offered | Commit delay p50 / p99 |
| --- | --- | --- | --- | --- | --- |
| 200 | before | 4 | 74 | 1,473 / 2,669 | 8.0 s / 13.5 s |
| 200 | after | 20 | 211 | 2,646 / 2,790 | 1.1 s / 1.3 s |
| 1,000 | before | 4 | 75 | 1,229 / 13,499 | 43.8 s / 62.3 s |
| 1,000 | after | 21 | 392 | 7,789 / 13,667 | 7.7 s / 12.6 s |
## Risks
- More object storage writes per second at high run counts (at 1,000
runs and 50 ms per write, about 390/s instead of about 75/s). This
raises object storage request cost in proportion. The ceiling bounds it.
- The S3 client now states its socket limit. The value is the SDK
default, so behavior does not change.
- The open-writer count is process-wide. A run log that is never
finalized keeps its count, as it already keeps its writer entry. The
ceiling bounds the effect.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
fad8f051ca |
fix(ui): keep a saved task when its plan, run or work-product read answers 404 (#21684)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21671 saves the first chat frame of recently opened tasks in the browser's read cache, so a hard load shows the chat before any response. > - It also removes a task's saved data at once when a read of the task or its chat is refused (401, 403 or 404). > - A task without a plan answers 404 on `GET /issues/:id/documents/plan`. That is a normal answer, but a reader that does not catch it records a query error. > - The cache treated that 404 as "the task is gone" and removed the task and its saved chat. On staging, a task without a plan was therefore never saved, so it never opened from the saved chat. > - This pull request revokes on a 404 only for the task record and its comments. 401 and 403 still revoke on every chat read. > - The benefit is that tasks without a plan (most tasks) open from the saved chat as intended. ## Linked Issues or Issue Description **Describe the bug** After #21671, a task without a plan is not kept in the saved read cache. The first chat frame is not shown from the saved cache for that task. **Steps to reproduce** 1. Open a task that has no plan document (for example PAP-4463 on staging). 2. Look at the saved read cache (`paperclip:query-cache:v1:*` in local storage) after the page settles. 3. The task's record and chat are missing. The page shows `GET /api/issues/:id/documents/plan` answering 404. **Expected behavior** A 404 for a missing plan, active run or work products keeps the task. Only a refusal of the task itself (its record or comments, or any 401/403) removes it. **Actual behavior** Any 404 on a saved chat read removes the task and its saved chat from memory and storage. Refs #21671. ## What Changed - `ui/src/lib/persisted-query-cache.ts`: a chat read revokes the task on 401 or 403, or on 404 only for the recent comments. The task record keeps revoking on 401, 403 and 404. - `ui/src/lib/saved-task-chat.test.ts`: a 404 for a missing plan, active run or work products keeps the task and its saved chat. - `doc/ui-internals.md`: the read-cache rules state which 404s revoke. ## Verification - `cd ui && npx vitest run src/lib/saved-task-chat.test.ts`: 24 passed. With the change in `persisted-query-cache.ts` reverted, the 3 new cases fail. The existing 401/403/404 revocation cases on recent comments still pass. - `src/lib/persisted-query-cache.test.ts` and `src/lib/restored-task-chat-bootstrap.test.ts` pass (90 tests across the three files). `pnpm --filter @paperclipai/ui typecheck` passes. - Staging evidence (build `fc9e4cbbb`, PAP-4463): the only 4xx during the load was `GET /api/issues/:id/documents/plan` 404, and the saved record did not hold the task. ## Risks - Low risk. A 404 on a plan, run or work-product read no longer revokes the task. A task that was deleted or made unreadable still answers 404 or 403 on its record and comments, and that removes it. Any 401 or 403 still removes it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser automation). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.1 |
||
|
|
28761f9c25 |
perf(workspaces): read a downloaded snapshot archive once, in the encrypt pass (#21683)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote agents work in sandboxes, and the server saves each sandbox workspace as an encrypted snapshot in object storage > - One Node main thread on the server controls every run, so per-agent byte work on that thread limits how many agents one server can hold > - The save path read every downloaded archive twice: a main-thread SHA-256 re-hash after the download, then a full read in the off-thread encrypt pass that already computes the same digest > - This pull request removes the main-thread re-hash and checks the encrypt pass's digest and size against the sandbox receipt instead > - The benefit is one full archive read per save instead of two, and no archive hashing on the main thread, with the same integrity guarantee ## Linked Issues or Issue Description Refs the 1,000-concurrent-agents plan in `doc/plans/2026-11-13-thousand-concurrent-agents.md` (Phase 1, "Workspace snapshot" row). **What existing behavior does this improve?** Saving a remote workspace snapshot after a run. **Subsystem affected** Server: remote workspace snapshot save (`remote-workspace-transport.ts`, `remote-workspace-snapshot-storage.ts`, `remote-workspaces.ts`). **Current behavior** `captureSandboxWorkspaceArchive` streams the whole downloaded archive through `createHash` on the main thread to compare it with the sandbox receipt. Then `uploadWorkspaceSnapshot` reads the whole archive again on a worker thread to encrypt it, and computes the same plaintext digest. **Proposed behavior** After the download, the server checks only the file size against the receipt. The encrypt pass compares its plaintext digest and size with the receipt. A mismatch fails with the same `remote_workspace_capture_transfer_mismatch` error before the upload attempt is recorded and before any object is stored. **Reason and benefit** One full read and one SHA-256 of each archive leave the server's main thread (about 0.36 ms of main-thread time per MB on a loaded dev machine; 37 ms for a 100 MB archive, at event-loop utilization 1.0 for that time). ## What Changed - `captureSandboxWorkspaceArchive` no longer re-hashes the downloaded archive. It checks with `lstat` that the download is a file with the receipt's byte size, and fails with `remote_workspace_capture_transfer_mismatch` when it is not. - `uploadWorkspaceSnapshot` accepts an optional `expectedArchive` (`sha256`, `byteSize`). After the encrypt pass, it compares the sealed plaintext digest and size with it. A mismatch throws `remote_workspace_capture_transfer_mismatch` before the attempt enters `metadata.uploads` and before `putFile`. The scratch ciphertext is removed as before. - `saveRemoteWorkspace` passes the sandbox receipt as `expectedArchive` when it captured an archive. - The save-level `verifyMs` timing field is removed, because that step no longer exists. The storage-level `verifyMs` stays. - `doc/run-log-events.md` says that the encrypt step is the only full read and where a mismatch fails. Callers that relied on the early hash keep their behavior: - Unchanged-snapshot reuse (`findUnchangedWorkspaceSnapshot`) compares the sandbox receipt, not the local hash. When it reuses the parent, the local copy is used only to list directories, and `readRemoteWorkspaceArchiveDirectories` verifies the manifest digest. - Reuse of an earlier upload attempt or an already uploaded snapshot does not read the archive. Those objects were verified when they were stored. - The read-only disk-full fallback builds its archive on the server, so its receipt always matches. Its behavior does not change. - The error message and the failure path do not change, so the save failure code, retry and fallback stay the same. ## Verification - Red/green: `remote-workspace-transport.test.ts` "leaves the downloaded archive's single full read to the encrypting upload" counts main-thread read streams of the downloaded archive during capture. On master it is 1 (fails). With this change it is 0 (passes). The encrypt pass is the one remaining read. - New: digest mismatch and size mismatch in `uploadWorkspaceSnapshot` reject with the transfer-mismatch error, do not call `putFile`, and leave the row `pending` with no recorded attempt. A retry with the right receipt saves. - New: an end-to-end `saveRemoteWorkspace` with a transfer that flips one byte (same size) fails with the transfer-mismatch error, stores nothing, keeps the writer and the previous snapshot, and a retry saves and restores the right content. - New: a download whose size differs from the receipt fails in capture. - `cd server && npx vitest run src/services/remote-workspace-snapshot-storage.test.ts src/services/remote-workspace-transport.test.ts src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-transfer-timeout.test.ts src/services/remote-workspace-readonly-capture-budget.test.ts src/services/sandbox-disk-full-fault.test.ts` passes: 6 files, 360 tests. `src/services/remote-workspace-snapshot-reuse.test.ts` passes: 32 tests. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. A same-size corrupt download is now detected in the encrypt pass instead of right after the download. It is still detected before any attempt is recorded or any object is stored. - When the sandbox reports an unchanged workspace, a corrupt local copy no longer fails the save. The save reuses the verified parent snapshot, which matches the sandbox's content. The local copy is used only for its manifest, and the manifest digest is verified. - The save-level `saveTiming.verifyMs` field is gone. No code reads it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cf2a4221e3 |
perf(server): skip the per-request stale instance-admin purge when no row exists (#21680)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud-hosted instances authenticate each request through trusted Cloud headers in the actor middleware > - That path deletes stale `instance_admin` rows for the user on every request, static files and live-events upgrades included > - On an idle hosted instance this delete ran about 1.2 times a second, which is a write on an instance-wide table for each request > - Almost always there is no row to delete, so the statement is wasted work and a database round trip > - This pull request finds out if a stale row exists inside the membership read that the request already does, and deletes only when a row exists > - The benefit is one read and no write for each unchanged authenticated request, with the same purge on the very next request when a stale row appears ## Linked Issues or Issue Description **What existing behavior does this improve?** Cloud trusted-header authentication in the actor middleware. **Subsystem affected** Server authentication (`server/src/middleware/auth.ts`). **Current behavior** `resolveCloudTenantActor` in `server/src/middleware/auth.ts` sends `delete from instance_user_roles where user_id = $1 and role = 'instance_admin'` for every trusted-header request. The actor middleware runs before the static UI, so asset requests run it too. Each live-events WebSocket upgrade runs it again through `resolveCloudActor`. The retry wrapper runs it again after a transient connection error. **Proposed behavior** An unchanged request sends no write. A stale `instance_admin` row is still removed on the next request after it appears. **Reason and benefit** Idle hosted instances send about one delete for each request. This costs a round trip and a write statement on the main database for no change. **Breaking changes** None. The resolved actor and the authorization decisions do not change. ## What Changed - The Cloud trusted-header path reads the user's active memberships together with an `exists` check for an `instance_admin` row, in one statement. - The purge runs only when that check finds a row, when the user has no active membership row yet (first request), or when the read fails. - The session path and the Cloud path share one membership filter function, so both still resolve the same access set. - New test `server/src/__tests__/cloud-tenant-actor-statements.test.ts` meters the statements of one authenticated request on the embedded test database. ## Verification - `cd server && npx vitest run src/__tests__/cloud-tenant-actor-statements.test.ts src/middleware/cloud-tenant-actor.test.ts src/__tests__/cloud-tenant-actor-round-trips.test.ts`: 41 passed. - Red/green: with the `origin/master` version of `auth.ts`, the new test fails. An unchanged member request sends 2 statements, and one of them is the delete. With this change it sends 1 read and 0 writes. - The test inserts a stale `instance_admin` row between two requests. The next request deletes it, and the request after that sends no write again. - The test changes the stack role from member to admin and back. The membership role and the experimental-settings access change on the very next request each time. ## Risks - Low risk. The actor that the middleware returns is the same. Authorization decisions do not change. - The purge now runs after the membership read, not before the tenant sync. The sync does not touch `instance_user_roles`, so the order does not matter. The request still waits for the purge before it continues. - A race between the read and a new row insert existed before too (delete, then insert). The next request removes such a row. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dcba9628c7 |
perf(recovery): keep a finished task's settled attention settled and halve graph hint fan-out (#21676)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Issue recovery records "recovery actions" on tasks. A sweep resolves
the active action of a task that is already done or cancelled.
> - On an idle hosted instance, a debug-log run (from #21640) named a
ping-pong. A native worker re-surfaces a board attention for 13 finished
tasks. Recovery resolves it as `source_terminal`. The unique "one active
action" index then lets the worker insert it again, about every 26
seconds per task. That built 45,008 rows since 2026-10-21. Each insert
and each resolve also signals run dispatch and the task's recovery
graph, which wakes both sides again.
> - Separately, every task-graph recovery hint is published twice per
task queue: once as the graph hint and once as a `{companyId, backstop:
true}` twin. That is half of all idle signals, for a backstop that is
time-gated anyway.
> - This pull request breaks the ping-pong from the recovery side and
removes the backstop twin.
> - The benefit is that a finished task's settled attention stays
settled and wakes nothing. Each graph hint costs half the signals.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
With zero runs active, recovery and native workers kept re-creating and
re-resolving the same board attention on 13 done or cancelled tasks. The
rows are `active_run_watchdog`, owner `board`, cause
`native_restart_recovery_blocked`. The loop added about one row per task
every ~26 s and about 2.6 work-signal commits per second.
**Expected behavior**
Once recovery settles a finished task's attention, the same attention
identity is not surfaced again, and settling it wakes nothing that a
finished task cannot use.
**Steps to reproduce**
1. Create a cancelled task.
2. Surface the native "controller process space unverified" attention
for it (same cause and fingerprint each time), then run
`reconcileActiveRecoveryActions`.
3. Repeat step 2. On master each cycle inserts a new active row and
resolves it, and each resolve signals the task queues and run dispatch.
**Paperclip version or commit**
master at
|
||
|
|
97a167093d |
fix(queue): send a message into a starting run without a cancel or recovery errors (#21661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can queue a message while an agent works, then send it into the running turn > - A run that is still starting takes queued messages into its first turn, and Interrupt hands off to them > - Both rules only accepted a queue that a user started > - A user's message can merge into an earlier automatic wake that waits behind the task, and that wake keeps its system requester > - Then the starting run did not take the message, Interrupt cancelled the boot, and the task showed "Couldn't finish this attempt" with Retry and "Waiting for execution recovery" under the composer until the cancelled boot settled > - This pull request makes both rules judge the messages and the recorded Interrupt intent, not the wake's requester, and shows an Interrupt hand-off as sent > - The benefit is that sending a queued message feels immediate and shows no error ## Linked Issues or Issue Description **What happened?** A run started from a resolved blocker. The user queued a message 20 seconds later, while the runner was still starting. The message merged into an older automatic deferred wake. The queue offered Interrupt instead of sending it. Interrupt cancelled the starting run. For about 7 seconds the task showed "Couldn't finish this attempt. Your message is saved." with Retry, and the composer showed "Waiting for execution recovery. Your message is saved." Then the message was delivered. **Expected behavior** The starting run takes the message into its first turn. When an Interrupt does stop a run, the message shows as sent, with no error banner, Retry, or recovery text. **Steps to reproduce** 1. Have an automatic deferred wake for a task (for example from an earlier blocked execution). 2. Start a run for the task with a Paperclip runner agent. 3. While the runner is still starting, queue a message and click the send control. ## What Changed - `deferredWakeJoinsFirstTurn`: a system or unattributed wake qualifies. Each message must still be the responsible user's own (`firstTurnEligibleComment`). A wake from another user or an agent keeps its own delivery. - Task interruption presentation: find the Interrupt queue by its recorded `queuedCommentInterrupt` intent, not by `requestedByActorType = user`. The hand-off now shows as working for a merged queue too. - Queued-comment read: while a deferred queue carries an Interrupt intent, report an `interrupt_handoff` wait instead of the recovery message. - UI (`TaskChatQueuedMessages`, `IssueDetail`): an `interrupt_handoff` response counts as sent. The messages stay out of the queue and no status or error line shows. ## Verification Red/green tests. Each failed before the fix and passes after it: - `server/src/__tests__/issue-queued-comments-routes.test.ts`: "gives a booting run's first turn the user's message merged into an earlier automatic wake"; the Interrupt fallback case now expects `interrupt_handoff` on the response and on a poll. 144/144 pass. - `server/src/__tests__/task-interruption-query.test.ts`: the hand-off case runs for a user-started and a system-started queue. 32/32 pass. - `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx`: "shows an Interrupt as sent while the stopped turn hands off". 43/43 pass. - `first-turn-queued-input.test.ts` 8/8, `issue-queued-comment-queue` 7/7. Server and UI typecheck pass. Root cause from the staging wake diagnostics: the run started at 20:26:54, the message was coalesced at 20:27:17 into a deferred `issue_execution_deferred` automation wake from the previous day, the Interrupt at 20:27:20 recorded `execution_recovery`, and the false-positive hold resolved at 20:27:26. Not run: local runner E2E steering cells. The OpenCode chat-features cells require the trusted CI host. The Codex and Claude `revise-while-running` chat cells failed before steering on a fixture workspace path (`workspaces/` vs `isolated-workspaces/`), which this change does not touch. Staging verification follows the deploy. ## Risks - Low. A wider set of deferred wakes can join a starting run's first turn. Message eligibility, the claim receipt, and exactly-once delivery are unchanged. - An Interrupt intent that never delivers shows "sent" in the composer. The task notice still shows a failure with Retry after its existing grace period. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ac114d9479 |
refactor: cut DEVELOPING.md to a setup and workflow guide (#21670)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` tells every contributor and agent to read `doc/DEVELOPING.md` for setup, commands, testing, and worktrees > - That file grew to 5,495 lines. Most of it was notes on how one subsystem behaves, added one pull request at a time > - Readers could not find the commands. Agents also read behavior notes as rules > - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and workflow guide. It moves the subsystem notes to focused docs > - The benefit is a guide that a person or an agent can read in full, and a clear place for each kind of note ## Linked Issues or Issue Description **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/DEVELOPING.md` **What's wrong?** The file was 5,495 lines. It opened with 112 lines about case handoffs before it reached Prerequisites. Unrelated sections sat below the "OpenClaw Docker UI One-Command Script" heading. Statements about current behavior read like policy. **Suggested fix** Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each subsystem note to a doc for that subsystem. Add an index. ## What Changed - `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health checks, local run, test drive, Docker, instance layout, database, storage, secrets, worktrees, test commands, Storybook, CLI, and an index of the other docs. - 15 new docs hold the moved text without wording changes: `ci-workflows`, `test-infrastructure`, `worktrees`, `hot-restart-deploys`, `heartbeat-internals`, `runner-internals`, `workspace-repositories`, `workspace-storage`, `skills-and-teams-catalog`, `ui-internals`, `case-handoffs`, `chat-connector-development`, `instance-configuration`, `integration-api-notes`, `database-backups`. - "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`. The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`. - Deleted about 280 lines: the story-by-story Storybook notes, "Agent Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only sections, "Workspace feature checks (2026-10-09)", and "Chat and task navigation caching". The test-drive build-lock paragraph is now four lines. - Repointed every inbound link and every script or workflow comment that named a moved section. - `AGENTS.md` now says where a behavior note goes, so the guide does not grow again. ## Verification - A script split the file by line range. It asserts that each of the 5,495 source lines is kept, moved, or dropped exactly one time. - A link check on every changed Markdown file found 0 missing files and 0 missing anchors. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `node --check` passes on the three scripts with changed comments. - `pnpm verify` was not run. This worktree has no installed dependencies, and the change has no runtime code. ## Risks - Low risk. The change is documentation and comments only. - A link to a moved section from outside this repository will land on the top of `doc/DEVELOPING.md`. - The new docs are not in the audited prose list in `.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An edit to `doc/DEVELOPING.md` does not. This PR does not change that list. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
648b238be1 |
perf(native-runtime): stop re-finalizing native results whose finalization error is recorded (#21674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime releases owners of failed runs. When the owner's controller is in a process space that Paperclip cannot verify, it shows controller namespace attention on the task instead. > - On a hosted instance with zero active runs, this writer made about 21% of all work signals. It inserted a new active attention row for the same task about every 26 seconds. > - The tasks were already done or cancelled. Recovery resolved each row as `source_terminal`. The active slot was then free, so the next owner-release pass inserted a new row and published a graph recovery signal again. The loop did not stop. > - A live task had the same defect: after the fresh-continuation fallback resolved its attention, the next pass made the same incident again. > - This pull request stops the writer from surfacing attention for finished tasks. It also stops the writer from making an incident again when that run authority was resolved after its lease expired. Those owners are no longer release candidates and do not set the release deadline. > - The benefit is that an idle instance does not write or publish anything for these runs. Attention still shows once for a live task. ## Linked Issues or Issue Description **What happened?** The failed-run owner release (`settleFailedRunOwner`, expired-owner path) called `surfaceNativeControllerNamespaceAttention` for every expired foreign owner that had no active attention. It did this even when the source task was `done` or `cancelled`. Recovery resolved that attention as `source_terminal`. The next pass found no active attention and inserted a new board row. Each insert published `notifyIssueGraphRecovery` and the other execution-control signals. The stranded-task check then ran again for the task graph. A hosted instance collected about 45,000 rows on 13 finished tasks in this way. **Expected behavior** A pass that changes nothing writes nothing and publishes nothing. A finished task never gets controller namespace attention. A live task gets one incident for each exact run authority. A resolved incident stays resolved until the authority changes. **Steps to reproduce** 1. Make a native run that failed, with an `observed` coordinator whose foreign lease has expired and whose controller host is not recorded. 2. Mark the task done or cancelled, or let the fresh-continuation fallback resolve the attention of a live task. 3. Run the execution-finalization worker with the issue recovery workers for 30 minutes. 4. On master, each task gets about 40 to 60 new attention rows. The writer publishes thousands of signals. ## What Changed - `surfaceNativeControllerNamespaceAttention` (shared helper): before it creates attention, it checks for a resolved incident for this run that was resolved at or after the owner's current lease expiry. If one exists, it writes nothing. - New `settledNativeControllerNamespaceAttention` SQL predicate. It is shared by the helper, the release candidate page, and the release deadline query, so all three use the same rule. - `settleFailedRunOwner` (expired-owner path) does not surface attention when the task is `done` or `cancelled`. The restart-recovery claim path has the same guard. - `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt` skip foreign owners whose task is finished or whose incident is settled. A pass that has nothing to do selects nothing, and no immediate deadline is set again. Local owners keep the normal proof-based release. - The owner release proofs did not change. A foreign owner is still never released without proof. - Tests: `server/src/__tests__/native-failed-run-owner-release-idle.test.ts` (real PostgreSQL, real coordinator and workers, fake clock, statement and signal counters) and three new cases in `native-failed-run-owner-release.test.ts`. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/native-failed-run-owner-release-idle.test.ts src/services/native-runtime/native-failed-run-owner-release.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/services/native-runtime/native-restart-recovery-isolation.test.ts`: 80 tests pass. - Red on master, green with this change. The test seeds one done task and one cancelled task (each with an earlier resolved `source_terminal` incident) and one live task, then runs 30 idle minutes: - master: 126 attention inserts, 249 attention writes, 7,024 signals, 8,025 statements. - this branch: 1 attention insert (the live task, surfaced once), 2 attention writes (that insert and the fallback's resolution), 36 signals, 350 statements. All of this is startup work. Nothing repeats. - After the fresh continuation settles a live task's attention, six unrelated finalization wakes over 30 minutes: - master: 3 attention inserts, 90 signals, 551 statements, and the release deadline stays due. - this branch: 0 attention writes, 0 run writes, 6 signals (the test's own wakes), 153 statements, and the release deadline is null. - New unit cases: a done or cancelled task never surfaces, and surfaces once when it is reopened. A settled incident stays settled over repeated passes. A renewed authority (a new lease expiry) surfaces one new incident. - The three new unit cases fail on master and pass on this branch. - `tsc --noEmit` for `server` passes. ## Risks - Low risk. The change only removes writes. It does not weaken a release proof, an authorization, a budget, a pause, or an approval check. - An incident that a person resolved or dismissed after the lease expired is not made again for the same authority. A renewed authority (a later lease expiry) still makes a new incident. - A superseded incident (status `cancelled`) still shows again when the superseding action clears. This keeps the current behavior. - The historical rows that the loop made are not removed. This change only stops new rows. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21621, #21616, #21617 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.0 |
||
|
|
64c7122dae |
perf(native-runtime): stop re-surfacing settled controller namespace attention (#21675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime releases owners of failed runs. When the owner's controller is in a process space that Paperclip cannot verify, it shows controller namespace attention on the task instead. > - On a hosted instance with zero active runs, this writer made about 21% of all work signals. It inserted a new active attention row for the same task about every 26 seconds. > - The tasks were already done or cancelled. Recovery resolved each row as `source_terminal`. The active slot was then free, so the next owner-release pass inserted a new row and published a graph recovery signal again. The loop did not stop. > - A live task had the same defect: after the fresh-continuation fallback resolved its attention, the next pass made the same incident again. > - This pull request stops the writer from surfacing attention for finished tasks. It also stops the writer from making an incident again when that run authority was resolved after its lease expired. Those owners are no longer release candidates and do not set the release deadline. > - The benefit is that an idle instance does not write or publish anything for these runs. Attention still shows once for a live task. ## Linked Issues or Issue Description **What happened?** The failed-run owner release (`settleFailedRunOwner`, expired-owner path) called `surfaceNativeControllerNamespaceAttention` for every expired foreign owner that had no active attention. It did this even when the source task was `done` or `cancelled`. Recovery resolved that attention as `source_terminal`. The next pass found no active attention and inserted a new board row. Each insert published `notifyIssueGraphRecovery` and the other execution-control signals. The stranded-task check then ran again for the task graph. A hosted instance collected about 45,000 rows on 13 finished tasks in this way. **Expected behavior** A pass that changes nothing writes nothing and publishes nothing. A finished task never gets controller namespace attention. A live task gets one incident for each exact run authority. A resolved incident stays resolved until the authority changes. **Steps to reproduce** 1. Make a native run that failed, with an `observed` coordinator whose foreign lease has expired and whose controller host is not recorded. 2. Mark the task done or cancelled, or let the fresh-continuation fallback resolve the attention of a live task. 3. Run the execution-finalization worker with the issue recovery workers for 30 minutes. 4. On master, each task gets about 40 to 60 new attention rows. The writer publishes thousands of signals. ## What Changed - `surfaceNativeControllerNamespaceAttention` (shared helper): before it creates attention, it checks for a resolved incident for this run that was resolved at or after the owner's current lease expiry. If one exists, it writes nothing. - New `settledNativeControllerNamespaceAttention` SQL predicate. It is shared by the helper, the release candidate page, and the release deadline query, so all three use the same rule. - `settleFailedRunOwner` (expired-owner path) does not surface attention when the task is `done` or `cancelled`. The restart-recovery claim path has the same guard. - `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt` skip foreign owners whose task is finished or whose incident is settled. A pass that has nothing to do selects nothing, and no immediate deadline is set again. Local owners keep the normal proof-based release. - The owner release proofs did not change. A foreign owner is still never released without proof. - Tests: `server/src/__tests__/native-failed-run-owner-release-idle.test.ts` (real PostgreSQL, real coordinator and workers, fake clock, statement and signal counters) and three new cases in `native-failed-run-owner-release.test.ts`. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/native-failed-run-owner-release-idle.test.ts src/services/native-runtime/native-failed-run-owner-release.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/services/native-runtime/native-restart-recovery-isolation.test.ts`: 80 tests pass. - Red on master, green with this change. The test seeds one done task and one cancelled task (each with an earlier resolved `source_terminal` incident) and one live task, then runs 30 idle minutes: - master: 126 attention inserts, 249 attention writes, 7,024 signals, 8,025 statements. - this branch: 1 attention insert (the live task, surfaced once), 2 attention writes (that insert and the fallback's resolution), 36 signals, 350 statements. All of this is startup work. Nothing repeats. - After the fresh continuation settles a live task's attention, six unrelated finalization wakes over 30 minutes: - master: 3 attention inserts, 90 signals, 551 statements, and the release deadline stays due. - this branch: 0 attention writes, 0 run writes, 6 signals (the test's own wakes), 153 statements, and the release deadline is null. - New unit cases: a done or cancelled task never surfaces, and surfaces once when it is reopened. A settled incident stays settled over repeated passes. A renewed authority (a new lease expiry) surfaces one new incident. - The three new unit cases fail on master and pass on this branch. - `tsc --noEmit` for `server` passes. ## Risks - Low risk. The change only removes writes. It does not weaken a release proof, an authorization, a budget, a pause, or an approval check. - An incident that a person resolved or dismissed after the lease expired is not made again for the same authority. A renewed authority (a later lease expiry) still makes a new incident. - A superseded incident (status `cancelled`) still shows again when the superseding action clears. This keeps the current behavior. - The historical rows that the loop made are not removed. This change only stops new rows. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21621, #21616, #21617 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fc9e4cbbb3 |
perf(ui): show a returning task's saved chat before any server response (#21671)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI opens a task as a chat. A returning browser restores a saved read cache from local storage before the first render. > - #21650 made a returning hard load share the one chat bootstrap. But the saved cache held no chat, so the first chat frame still waited for that response. > - The owner approved saving recent comments in the browser on 2026-11-14, so that cached tasks show their chat at once. > - This pull request saves the first chat frame of the 20 most recently opened tasks, with size, age, revocation and isolation guardrails. The restored chat shows at once and revalidates through the bootstrap. > - It also adopts the hard-load bootstrap at app start and sends the other live-run readers through it, so readers that start before the task page mounts do not send their own requests. > - The benefit is that a hard load of a task the user opened before shows its conversation before any server response. ## Linked Issues or Issue Description **Describe the enhancement** A hard load of a task that the user opened before still waits for one server round trip before its chat appears. The saved read cache did not include comments. **Current behavior** With every API response delayed by 800 ms, a returning load shows the chat at 892 ms (median). The frame waits for the chat bootstrap response. **Proposed behavior** The saved read cache also keeps the first chat frame of recently opened tasks. A returning load shows that frame before any response, then revalidates through the bootstrap. **Why** The owner approved saving recent comments in the browser for this on 2026-11-14. Cold and cached loads must both be as fast as possible. Related: #21650 (returning load shares the chat bootstrap), #20594 and #20591 (returning inbox from the saved cache), #20592 and #21557 (cold task load from one bootstrap). ## What Changed - `ui/src/lib/persisted-query-cache.ts`: saves the reads the first chat frame waits for (`taskChatFirstFrameRef`): recent comments, their input facts, recent activity, the recent run window, interactions, attachment metadata, live and active runs, work products and the plan document. - Only for the 20 saved recent tasks, in the record of the task's own user and company. - At most 50 comments per task (the newest; `hasMore` is set), 192 KiB per task, 1 MiB in total, newest tasks first. - If storage refuses a record, the record is saved again without chat, so chat never displaces the rest of the cache. - All other field exclusions (tokens, credentials, environment values, embedded documents) still apply. Only `comments` is allowed, and only for these keys. - Expiry is the same as every other entry: 24 hours after the read. - A 401, 403 or 404 on the task or on any of its chat reads removes the task and its saved chat from memory and storage at once. - `ui/src/lib/issue-chat-bootstrap.ts` and `ui/src/main.tsx`: `adoptHardLoadChatBootstrap` adopts the bootstrap that the HTML started, at app start, under the task's restored id and identifier. Readers that start before the task page mounts share it. - `IssueDetail.tsx`, `LiveRunWidget.tsx`, `IssueRunLedger.tsx`, `useTaskPluginPanels.ts`: their `live-runs` reads share the loading bootstrap. Before, whichever observer mounted first could send its own request. - `doc/DEVELOPING.md`: the read-cache rules now state what is saved, the bounds, why, and the owner's approval on 2026-11-14. - Tests: `ui/src/lib/saved-task-chat.test.ts` (guardrails), a new case in `restored-task-chat-bootstrap.test.ts`, and an always-on perf test: "a returning task load shows its saved chat before any API response, then revalidates". ## Verification Local production build, local_trusted instance, CDP delay on every API request, fresh browser context per run, JavaScript in the HTTP cache. Time to the revealed history with the latest message and an editable composer. Saved read cache present against cleared (other storage kept), 5 alternating pairs each. | API delay | Build | Cleared | Present | Requests before frame (present) | | ---: | --- | ---: | ---: | ---: | | 800 ms | master `36101ebcb9` | 1,911 ms (1,895–1,920) | 892 ms (887–898), released by the bootstrap | 33 | | 800 ms | this PR | 1,922 ms (1,912–1,936) | 571 ms (556–588), before any response | 36 | | 300 ms | master | 916 ms (907–920) | 570 ms (564–576) | 37 | | 300 ms | this PR | 928 ms (910–998) | 586 ms (568–622) | 37 | - At 300 ms the bootstrap answers before the page mounts on both builds, so they are the same. At 800 ms the saved chat removes the wait for the bootstrap: −321 ms against master. - The new perf test holds every response for 2.5 s. It fails on master (frame at 2,595 ms, first response at 2,524 ms) and passes with this PR. - Guardrail tests, red/green: I disabled each guardrail in turn in `persisted-query-cache.ts` and ran `saved-task-chat.test.ts`. Each mutant made its test fail, and all 19 tests pass with the code. Mutants: no chat saved, no 50-comment cap, no 20-task cap, no per-task budget, no total budget, no retry without chat when storage refuses, no 24-hour expiry, no revocation on a chat read, chat stored in the wrong company record, and no field exclusions. - Sign-out and other-user isolation use the existing per-user records and their clearing (no new code). The tests cover them for saved chat. - "Never overwrites newer data" is the query cache's hydrate rule; the test covers it. "The bootstrap replaces saved data" uses the restored-entry rule from #21650; the test covers it. - App-start adoption test: it fails when the adoption call is removed, and passes with it. - Focused Vitest: 25 files that use the changed modules, including `IssueDetail.test.tsx` and `TaskChatThread.test.tsx`: 1,156 passed. `pnpm --filter @paperclipai/ui typecheck` passes. - Browser journeys against this build: `layout-stability.spec.ts`, `plan-startup.spec.ts`, `cold-bootstrap.perf.spec.ts`: 20 passed (2 opt-in benchmarks skipped). Leftover early reads on staging (one run after #21650 started `issues/:ref?view=core` and `issues/:ref/live-runs` before the frame): I could not reproduce them locally. That included sidebar Recents and a bootstrap delayed to 1.2 s with CDP initiator stacks. Both reads would go to the server only if no bootstrap was being tracked for the task, or if an observer used a direct read. This PR removes both openings: the bootstrap is adopted at app start, and every `live-runs` observer shares it. I will check the request timeline on staging after deploy. ## Risks - Privacy: recent comments of up to 20 recent tasks now stay in local storage for up to 24 hours, or until sign-out. The owner approved this on 2026-11-14. The bounds, the revocation and the field exclusions are tested. Saved data is per user and company, and sign-out clears it in every tab. - Storage: saved chat is limited to 1 MiB in total. A record that storage refuses is saved again without chat. - A saved chat can be up to 24 hours old when it shows. It is marked for revalidation, and the bootstrap answer replaces it within one round trip. A refused revalidation removes it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser automation through Playwright and Chrome). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
36101ebcb9 |
fix(evals): grade five cohort-14 chat cases by saved state and clicked links (#21667)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 14 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`) failed five cases where the agent behaviour was correct or acceptable. > - Four failures came from graders that read free-form reply text or required a setup label to be absent. One failure came from a harness race with a client-side route rewrite. > - This pull request fixes those graders and that harness step. It does not change the product. > - The benefit is that these cases measure the feature again: the retained correct replies pass, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened?** In chat-eval cohort 14 (run 43637708886), five cases failed although the agent behaved correctly: - `projects.share-project`: a regex over the free-form reply rejected a correct board-only handoff that linked the project access settings. - `teams.install-team`: the grader required the literal skill slug, name or key. The reply named the blocked skill "document-maintenance"; the slug is `doc-maintenance`. - `steering.discard-queued-message`: the final plan removed the contact section but cited the setup label in a "Latest direction" note. The grader required the label to be absent everywhere. - `workspaces.inspect-git`: the grader required the reply to name `existing-notes.txt`. The person asked about "website files", and the agent correctly left the unrelated file out. - `runs-adapters.reset-session`: the harness opened the agent actions menu on `/agents/<id>/runtime`. The app then rewrote the route to `/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions" click timed out. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not reply wording. Broken behaviour (an agent-made grant, a kept contact section, a changed unrelated file, a missing first reply) still fails. **Steps to reproduce** 1. Use the retained cohort-14 evidence for the five cases. 2. Run the old graders against the retained replies and revisions. Each case fails. 3. For reset-session, open an agent profile by id while the app rewrites the route after a delay, then open the actions menu before the rewrite completes. **Paperclip version or commit** master `d4498ae7ac` (cohort 14); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`): removed the reply regex. The grade now requires (a) a rendered link to the project's configuration page (where Manage access lives) that the harness clicks and that opens the Manage access control, and (b) saved state with no agent-made access grant: Riley is absent from the membership before the owner acts, and no agent `project.access_member_added` or visibility `project.updated` activity exists. The owner step still completes when the link is missing, so later phases keep their evidence. - `teams.install-team` (`external-team-conditional.ts`): removed `namesTeamSkill`. The case contract is "reports conflicts or partial failures truthfully"; truthfulness is meaning, so a person reviews it. The partial outcome is already graded from saved state: the catalog skill is still missing beside the local skill that holds its slug, and the installer recorded a warning. A skill link was not chosen because the blocked catalog skill has no installed page to link, and the contract only asks for links to the installed agents and project. The check is now "a first reply was delivered". - `steering.discard-queued-message` (`runtime-discard.ts`): the later contact request now names a contact person and an attempt-specific phone (a setup change in the harness act step, not the YAML). The grade collects the contact section's distinctive whole lines from the revision that saved it: lines in the section whose heading carries the label, contact name or phone, plus lines that carry one of those markers, excluding lines that also appear elsewhere in that revision. None of those whole lines may remain in the final revision, and the contact name and phone may not appear anywhere in it (substring match on those distinctive literals). A note that quotes the label inside a longer line, and shared lines such as `---`, do not count. - `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring `existing-notes.txt` in the reply. The question asks about website files, and the unrelated file's bytes are still checked unchanged. No YAML change, so no TestOps manifest change. - `runs-adapters.reset-session` (`runtime-support.ts`): `openRuntimeAgentProfile` now waits until the URL settles on the agent's canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app uses) before callers use profile controls. This also covers the other callers of this helper. ## Verification - Red/green tests: - `internal-privacy-share.test.ts`: the retained cohort-14 reply and a reworded reply pass. An agent grant, an agent visibility change, a pre-owner membership, a root-only link and an unopened link fail. The old regex returns false on the retained reply. - `external-team-conditional.test.ts`: the retained cohort-14 reply passes. The same reply fails when saved state shows no blocked skill or no installer warning. A missing first reply fails. - `runtime-discard.test.ts`: the retained cohort-14 plan passes with the label still cited in a note. Shared lines remaining passes. Removing only the contact name and phone fails. Keeping the heading line, a contact detail line, or reformatted details with the contact name fails. I also ran the new check on the full retained revisions: retained passes, kept section fails. - `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed unrelated file still fails. - `runtime-support-profile-route.test.ts` (real Chromium): the profile rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset sessions" is clicked on the settled page. Without the fix, the click times out (the cohort-14 failure). - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: 291 files passed. - `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout under load passed on rerun). - `pnpm test:e2e:runner:typecheck`: passed. - Narrow paid run of the five full case ids on this branch: run 43643474282 (results are added in a comment). ## Risks - Low risk. Only eval graders, fixtures and one harness wait change. - `teams.install-team` no longer checks how the reply names the blocked skill. A person reviews that meaning; saved state still proves the partial outcome. - `projects.share-project` now needs a link to the project configuration page. A reply that links only the project overview fails `owner-access-page-linked`. - `steering.discard-queued-message` does not detect a contact section that the final revision rewrites line by line with new wording. A person reviews that. The later request text changed (it now lists a contact name and phone), so earlier retained runs used a different request. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.18 |
||
|
|
172c267338 |
fix(runner): count returned validation rejections and give a copyable call in repeated-failure corrections (#21666)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents change Paperclip records through the `call_api` tool, and the native runtime runs each tool call through a repeated-identical-failure guard (#21360, #21398) > - Some model hosts drop arguments. A model then sends the same failing call again and again, and each attempt costs a model turn > - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input hash > - The guard counted these refusals and stopped running the call after the second failure. But its correction showed only a `"body": {...}` placeholder, so the model had no concrete call to copy. Also, the guard did not count a route's own 400/422 when it came back as an `ok: false` result > - This pull request puts a copyable call in each correction and counts returned 400/422 rejections. It never counts transient failures > - The benefit is that a model gets the exact call to send on the second failure, and the third identical validation failure does not run, whether the rejection is thrown or returned ## Linked Issues or Issue Description Refs #21360, #21398 **What happened?** In chat-eval cohort #14, `questions-and-approvals.review-plan` (run 470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH /api/issues/{id}` with no `body` 7 and 10 times, all with the same input hash. The pre-dispatch check (`assertBodyPresent`) refused each call and returned HTTP 400. The guard corrected the second failure and refused the rest. But the correction said only `Resend with a top-level body argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The model kept sending the identical call, and each refusal cost another model turn: 2 min 18 s in review-plan and 24 s in private-task. Also, when the route itself returns a 400/422 validation response, `call_api` gives back `{ ok: false, status: 422, ... }` as a result and does not throw. The guard did not count that, so an identical invalid request reached the route every time. **Expected behavior** On the second identical deterministic failure, the agent gets a correction with a call it can copy that passes validation. On the third, Paperclip refuses to run the call and tells the agent to change the arguments or to stop and report. Transient failures (5xx, timeouts, rate limits, conflicts) always run again. **Steps to reproduce** 1. Run a native-runtime agent on a host that drops the `body` argument. 2. Have it update a task with `call_api PATCH /api/issues/{id}`. 3. Look at the tool results. They contain only a `{...}` placeholder. A route's 422 returned as `ok: false` is never counted. **Paperclip version or commit** mastercanary/v2026.1114.0-canary.17 |
||
|
|
c603f591dc |
fix(runner): bound ACPX instructions only where the sandbox delivers them (#21646)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner prepares a private sandbox for each ACPX agent before it
opens the provider session
> - `validateProviderPolicy` in that sandbox stage refuses system
instructions over 32 KiB for every ACPX agent
> - Only Pi gets the text from the sandbox stage. Claude, Codex, Grok
and Antigravity do not use it there. The host accepts 256 KiB
> - An agent with a large instruction bundle fails every run at
`session.open`. The error says only `classification=admission_sandbox`,
so the operator cannot see the cause
> - This pull request applies the 32 KiB sandbox bound only to Pi, and
gives every instructions rejection its own closed admission code
> - The benefit is that Claude and Codex agents with large instructions
can start, and an over-bound rejection names its real cause
## Linked Issues or Issue Description
Fixes: #20814
Refs #14430 (this pull request added the 32 KiB check, with no stated
reason for the other providers)
Related: #21644 (also fixes #20814, but only for Claude, and does not
change the classification)
**What happened?**
Every run of a Claude ACPX agent failed with
`provider_session_open_rejected`. The message was `ACPX sidecar command
session.open was rejected (retryable=false,
classification=admission_sandbox)`. A reproduction in a sandbox from the
same image showed that all the filesystem steps of
`prepareAcpxRuntimeSandbox` succeed. The real cause: the agent's
instructions entry file was 54,734 bytes. That is more than the 32 KiB
check in `validateProviderPolicy`. The host bound (`boundedInstructions`
in `runtime-host.ts`) is 256 KiB. Agents with smaller instructions (12.7
KB) work.
**Expected behavior**
The sandbox stage must refuse only the text that it delivers. #14430
added the check when only Pi consumed `systemInstructions` in the
sandbox (as the `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var). The check
runs before any per-provider branch, so it also refuses Claude, Codex,
Grok and Antigravity. Those providers get the text in the session
request. When a bound refuses the text, the error must name the
instructions, not the sandbox.
**Steps to reproduce**
1. Create a Claude ACPX agent whose composed instructions are more than
32 KiB (for example, an `AGENTS.md` of 54 KB).
2. Start a run of the agent on the Paperclip Runner.
3. See that `session.open` fails with
`classification=admission_sandbox`.
**Paperclip version/commit**
`master` at
|
||
|
|
ecdeed5e7e |
perf(ui): open a task restored from the saved read cache through the chat bootstrap (#21650)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI opens a task page as a chat. A hard load of a task URL starts one chat bootstrap request before the app's JavaScript runs. > - A returning browser also restores a saved read cache from local storage. This cache holds the core fields of recently opened tasks, but no chat history. > - The task page started the bootstrap only when the core task was not cached. A restored core task skipped it. The early bootstrap request ran, but nothing used its answer. > - So every first-frame read went to its own endpoint, and the comment input facts waited for the recent comments. A returning load was slower than a cold load: on staging, 3.6–3.8 s against 1.9–2.4 s. > - This pull request treats a task that only the saved cache holds as cold for the chat. The page adopts the bootstrap, the restored task's own revalidation shares it, and the answer replaces the restored data. > - The benefit is that a returning load is now faster than a cold load, not slower. ## Linked Issues or Issue Description **Describe the bug** A hard load of a task page is slower when the browser has a saved read cache than when it has none. Measured on staging on 2026-11-14 (hard load, JavaScript in the HTTP cache, 3 alternating pairs on one task): saved cache cleared 1.9–2.4 s, saved cache present 3.6–3.8 s, until the chat composer exists. **Steps to reproduce** 1. Open a task page so that the saved read cache (`paperclip:query-cache:v1:*`) holds it. 2. Close the tab and open the same task URL again. 3. Compare the time to the first chat frame with a load where the saved read cache is cleared. **Expected behavior** A load with a saved read cache is at least as fast as a load without one. The chat's first-frame reads share the one bootstrap request. **Actual behavior** With a saved read cache, the bootstrap request runs but its answer is not used. About 16 more per-endpoint reads run before the frame, and the frame waits for a second, serial wave of reads. Related: #20355 (chat bootstrap endpoints), #20592 (cold task open from one bootstrap), #20591 and #20594 (returning Mine inbox from the saved read cache), #21557 (cold-load perf spec). ## What Changed - `ui/src/lib/persisted-query-cache.ts`: records which queries a page restored from the saved read cache. `isRestoredSavedRead(client, key)` is true until a read or a local write in this page replaces that data. - `ui/src/pages/IssueDetail.tsx`: starts (or adopts) the chat bootstrap when the core task is missing or only restored. It passes the restored task's id and identifier as aliases, so reads keyed by the id share the request while it loads. - `ui/src/lib/issue-chat-bootstrap.ts`: `readFromIssueChatBootstrap` lets a key that holds only restored data share the bootstrap, like an empty key. `startIssueChatBootstrap` accepts aliases. - `ui/src/lib/issueChatBootstrapCache.ts` and `ui/src/lib/issueDetailCache.ts`: the bootstrap answer replaces restored data (core task under every ref). The restored core task's revalidation shares the bootstrap. A core task that was read in this page keeps its own small read. - `ui/src/hooks/useIssuePlanDocument.ts`: the first plan read shares the bootstrap's plan projection instead of reading `documents/plan` before the frame. - `tests/perf/issue-detail/cold-bootstrap.perf.spec.ts`: an always-on request-shape test for a returning load with a saved read cache, and an opt-in benchmark of alternating pairs (saved cache present and cleared). The harness now clears storage after the warm-up page closes, because closing the app saves its read cache again. - `ui/src/lib/restored-task-chat-bootstrap.test.ts`: unit tests for the restore-then-bootstrap path. - `tests/perf/issue-detail/README.md`: documents the returning-load test and benchmark. ## Verification Local production build (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`), local_trusted instance, CDP delay of 300 ms on every API request, fresh browser context per run, JavaScript in the HTTP cache. Time from navigation start to the revealed history with the latest message and an editable composer. | Build | Saved read cache | Runs | Median | Spread | API requests before frame | Frame released by | | --- | --- | ---: | ---: | ---: | ---: | --- | | master `95a6ab31e6` | cleared | 5 | 913 ms | 908–957 | 33 | company list (shell stage 2) | | master `95a6ab31e6` | present | 5 | 1,204 ms | 1,178–1,215 | 49 | `activity?view=comment-inputs` (serial second wave) | | this PR | cleared | 7 | 942 ms | 925–997 | 35 | company list (shell stage 2) | | this PR | present | 7 | 573 ms | 566–658 | 37 | app boot and route mount (all task data is the bootstrap) | - Paired delta with this PR (present − cleared): median −376 ms over 7 pairs, all negative (−267 to −400). On master it was +292 ms over 5 pairs, all positive. - With an 800 ms API delay (this PR, 5 pairs): present 890 ms, released by the bootstrap response; cleared 1,910 ms. - Cold load with no app storage (existing benchmark, 7 runs): master 910 ms, this PR 939 ms. The host load average was about 20 during the PR run (other work on the machine). Requests before the frame are 35 in both. I make no claim of a cold change. - Request-shape tests (`cold-bootstrap.perf.spec.ts`): the new returning-load test fails on master (the timeline shows `?view=core`, `recent-comments`, `activity`, `runs`, `interactions`, `attachments`, `live-runs` before the frame) and passes with this PR. The existing cold test passes. - Unit tests: `restored-task-chat-bootstrap.test.ts` has 10 cases. With the behavior changes reverted (the restore marker kept so the file imports), 3 fail: the bootstrap refreshes the restored task and fills the chat, id-keyed reads share the loading bootstrap, and a partial answer. All 10 pass with this PR. The other 7 cases are guards that hold on both: a stale refresh reads the server, a failed bootstrap falls back, a 403 removes the restored task from memory and storage, another company's answer is not seeded, an answer is not used after another user signs in, and another user's saved task is not restored. - Focused Vitest: 27 files that use the changed modules, including `IssueDetail.test.tsx` and `TaskChatThread.test.tsx`: 1,208 passed. - Browser layout journeys against the PR build: `layout-stability.spec.ts` and `plan-startup.spec.ts`, 17 of 17 passed (reveal order, reading anchor, older history, Back, comment links, drafts during background refresh, failed and stalled reads with Retry, plan projection with stalled plan reads). - `pnpm verify` exit 2 (the README is a global CI input), so required CI supplies the broad coverage. Commands: ```sh cd ui && npx vitest run src/lib/restored-task-chat-bootstrap.test.ts PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts cold-bootstrap PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:<port> PAPERCLIP_ISSUE_PERF_COLD_RUNS=7 pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts cold-bootstrap -g returning ``` ## Risks - Low risk. The change only affects data that the saved read cache restored and that no read in this page has confirmed. Data read in this page keeps the old rule: a refresh asks the server. - The saved read cache content is unchanged. It still excludes comments and bootstrap payloads, so the first chat frame still needs one response (the bootstrap). It cannot appear before any task response without saving chat history, which the documented allowlist excludes. - If the bootstrap fails, stalls or omits the task, each read falls back to its own request, as on a cold load. A 403 or 404 on the restored task's fallback read removes it from memory and storage, as before. - Account changes: a bootstrap answer is bound to the session's user, as before. The saved cache restores data only for the same user. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser automation through Playwright). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0ab73d4e30 |
feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner can keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started. Core can wait for that turn. The transport can ask for it (#21607), and a controller restart during it is safe (#21612). > - No run does this yet, because Core does not ask the Runner for it. > - This pull request makes the native runtime of Core ask, for ACPX Claude. It is the only part that changes what a production run does. > - The benefit is the goal of the whole work: a run with a background command that is still running continues when the command finishes, and does not fail with `native_finalization_missing`. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. #21607 (merged): the transport can ask the Runner for provider-started turns. No caller asks. 2. #21612: the recovery after a controller restart during a wait or during a provider-started turn. 3. **This pull request:** Core sets the option for native ACPX Claude runs. This is part 3. Its base branch is `fix/provider-turn-restart-recovery`. Refs #19912 (design). **What existing behavior does this improve?** What a native ACPX Claude run does after a turn that ends with no result while a background command is still running. **Subsystem affected** Core, the native runtime: `server/src/services/native-runtime/native-session-executor.ts` and `native-provider-turn-wait.ts`. ACPX Claude only. **Current behavior** - The run ends at the end of the turn. The Runner restarts Claude, the background command stops, and the run fails with `native_finalization_missing`. **Proposed behavior** - The native runtime sets `providerStartedTurns: true` in the transport options for ACPX Claude, at the two places that build those options. One function, `nativeProviderStartedTurns`, decides this and also decides if the consumer gets the wait options, so the two cannot be different. - A turn that ends with no result while a background command is still running now waits, for a maximum of 10 minutes, for the turn that Claude starts by itself. That turn is consumed as the next turn of the same run, and its result settles the run. - Other providers and other ACPX agents do the same as today. - Limit of the Runner (see #21421): after a follow-up turn, the Runner waits again only while a background command is still running. A command that finishes while Claude reports on an earlier command gets no follow-up. **Reason and benefit** - The opt-in is in the last part, so each earlier part can merge with no change for a production run. ## What Changed - `server/src/services/native-runtime/native-provider-turn-wait.ts` (5 lines added, 1 removed): `nativeProviderStartedTurns`. - `server/src/services/native-runtime/native-session-executor.ts` (4 added, 2 removed): the option at the two places that build the transport options, and the same function for the wait options. - `server/src/services/native-runtime/native-session-executor.test.ts` (4 added, 2 removed): the two native paths set the option for ACPX Claude and not for other providers. ## State saved by the previous release - No persisted state changes in this part. - **A Runner that does not know the field** ignores it, and the run does the same as today. - **A deploy during a run:** a turn that started before the deploy has no opt-in and ends as today. The next turn has the opt-in. - **Rollback of Core:** the previous Core does not set the option, so the Runner does not wait. ## Verification The items below were run on the tree that was verified before the split into three parts (commit `1b15ded1c4` has that tree, byte for byte). The head of this branch (`23e2c13ad3`) is that commit on the new head of #21612; its three server files are the same. - `pnpm typecheck` and `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run build:typescript` in `packages/paperclip-runner`: all passed. - `server/src/services/native-runtime/native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed, 21 skipped. - The end-to-end behavior is tested in parts 1 and 2 with the real `paperclip-runnerd`, where the test sets the option in the way that this part does. - Live run with a real Claude account, not in CI, at the adapter level of the Runner (see #21316): the cycle of Claude was given as a turn with its own usage, and each Paperclip tool call was admitted. Not run: a live run through Core, `paperclip-runnerd` and the real sidecar with a real Claude account (that path needs an API key, not a local sign-in); a remote sandbox; the full suite. CI supplies the broad coverage. ## Risks - **This part turns the behavior on for each native ACPX Claude run.** A run with no background command is not changed: the Runner waits only when a shell is still running at the end of a turn that has no result. - **This part must not merge before #21612.** Without #21612, a controller restart during a wait or during a provider-started turn fails the run. - **A run can now stay open up to 10 minutes longer** when the background command does not finish or Claude does not start a turn. The agent computer stays active for that time. - **A waiting run can show as "suspicious" after 5 minutes of silence** in the run watchdog. The watchdog only labels a native run; it does not stop it. - **Follow-ups that the Runner cannot wait for are lost,** as the limit above says. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
6ba46fd080 |
fix(github): read only the webhook secrets each delivery step uses (#21651)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents receive GitHub work through a GitHub App chat channel. GitHub
sends each event as a signed webhook delivery.
> - Each delivery passes three steps: durable staging, the durable
replay worker, and the replayed request's preflight under the credential
lease.
> - Each step resolved every stored GitHub App secret. A GitHub App
connection stores seven secrets, so each delivery decrypted and audited
about 21 secrets. The steps use only the signing secret, the bound
installation id and the App id.
> - On our staging instance, these reads are now the largest source of
secret reads and `secret_access_events` rows.
> - This pull request makes each webhook step read only its own keys,
and only when it needs them.
> - The benefit is fewer vault reads and audit inserts per delivery, and
the App private key and OAuth client secret stay sealed on the webhook
path.
## Linked Issues or Issue Description
Refs #21641 (fixed the other source: review-check polls now reuse cached
installation tokens).
**What happened?**
On our staging instance (serving
|
||
|
|
ddfbbfff20 |
fix(runner): recover provider-started turns after a controller restart (#21612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - The Runner can keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started. Core can wait for that turn. #21607 lets a caller ask the transport for it. > - A controller can restart at any time, for example at a deploy. A wait can last 10 minutes and a provider-started turn can last longer, so a restart during one of them is a normal event. > - Tests with the real `paperclip-runnerd` showed that such a restart failed the session. The checkpoint that Core had was the checkpoint of the turn before, and the recovery of the driver refused or mislabelled the turns that it did not know. > - This pull request makes that restart safe. It changes no production run, because no caller asks for provider-started turns before part 3. > - The benefit is that part 3 can turn the behavior on without a failed run at each deploy. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. #21607 (merged): the transport can ask the Runner for provider-started turns. No caller asks. 2. **This pull request:** the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. This is part 2. Its base branch is `master`. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It makes a controller restart safe for a run that asked for provider-started turns. **Subsystem affected** Paperclip Runner package: the consumer (`src/native-session-runtime.ts`), the session backend (`src/backends/harness-driver-backend.ts`), the driver (`src/drivers/codex/`) and the transport (`src/live/`). Only for a run that asked for provider-started turns. **Current behavior** - The consumer saves no checkpoint when a wait starts or when a provider-started turn starts. During a wait and during a provider-started turn, the checkpoint that Core has is the checkpoint from the start of the turn before. It names that turn as active. - A restart from that checkpoint fails: the driver has the old turn as active, gets the events of a provider-started turn, and stops the session with `turn_binding_mismatch`. - The recovery of the driver adopts one turn that it does not know as a disposition recovery turn, and refuses two or more. With two or more, it closes the transport, and that stops the Runner and Claude. - The Runner does not send again an event that the controller acknowledged. If the start of a provider-started turn was acknowledged before the restart, the new driver gets the other events of that turn with no start. - A turn that ends waiting for a provider-started turn sets the terminal of the session, as a turn that ends the run does. After a restart, if the provider-started turn already has its result when recovery completes, the consumer finds a result and a terminal, settles the run, and does not read the events of that turn. The start, the items, the usage event and the terminal of that turn do not reach Core. - The recovered driver reads notifications from its construction, and `reconcile()` maps the terminal of the saved turn only after its reads. If the saved checkpoint is still the checkpoint of the controller turn and the start of the provider-started turn is already queued, the driver judges that start too early and stops the session with `turn_binding_mismatch`. - A checkpoint that waits can also have a consumed disposition marker (a recovery turn after an earlier restart ended waiting). The driver restores that session as closed and reads no notifications, and the consumer settles the run on the earlier terminal with no result. **Proposed behavior** - **Checkpoints.** The consumer saves a checkpoint when a wait starts and when a provider-started turn starts, with the function that it already uses for a governed wait. A checkpoint that fails stops the turn, as for a governed wait. - **A checkpoint as of the event that Core stored.** The driver gives each event its sequence when it reads it from the provider, before the consumer stores it. A snapshot that is taken later can be ahead of Core. For these two checkpoints, the driver captures its snapshot when it has handled the notification that made the event (the `turn.completed` that waits, or the `turn.started` of the provider), with the sequence of that event. The consumer asks for the snapshot as of the event that it has just appended. The saved state and cursor are then those of an event that Core has. - **Session terminal.** A turn that ends waiting for a provider-started turn does not set the terminal of the session (`HarnessNativeSession`). The terminal of the session is the end of the run, and that turn does not end the run. A checkpoint that is saved during a wait or during a provider-started turn has no terminal, so a recovered session settles only from the events that it reads, in order, to the terminal of the turn that has the result. - **Driver, a checkpoint that waits.** Recovery keeps the wait. It does not adopt and does not refuse the turns after the last terminal that it knows. The Runner sends again, in order, each event that was not acknowledged, and the driver admits each provider-started turn from that stream, as it does with no restart. - **Driver, a checkpoint with an active turn.** `reconcile()` accepts one turn in progress that comes after the saved active turn, when the saved turn ended with the wait marker. It refuses each other unexpected active turn as before, and it refuses this one too when the reconciled turn settled a result. - **Driver, a start that is queued before recovery reconciled the saved turn.** A `turn/started` of the provider that arrives while the session does not wait for one first waits until `reconcile()` has settled, and is then judged as before. `reconcile()` judges the live state after its reads; if the stream already settled the saved turn into a wait, it accepts exactly one turn in progress after the last terminal turn and does not adopt it. The driver admits that turn at its start. - **A wait with a consumed disposition marker.** The driver keeps reading notifications for a restored session that waits, as it does for a wait with no restart. It still refuses a second controller-started turn. The consumer does not take the consumed-marker shortcut for a recovered snapshot that waits, so the wait is resumed. - **Transport, the start comes first.** For each driver, the transport gives the `turn/started` of a provider-started turn before each other notification and before a tool call of that turn. If it did not give that start to this driver, it builds it from the `turn.started` event in its stored events. It does not do this for the turn that the recovered driver already has as active (`resumeActiveTurnId`). It binds the events and the tool call of a provider-started turn to that turn, not to the turn that the Runner reports as active at the time of the recovery read. - **Transport, which start owns an event.** The stored events can have gaps: the window removes old events but keeps the latest of each anchor type, and the controller checkpoint drops streamed deltas. The owner of an event is the last stored `turn.started` before it, if it is a provider start, and if the first turn terminal that is stored after that start is not before the event and is the terminal of the same provider turn (`providerTurnId`). If the first stored terminal after the start is the terminal of a different turn, the end of the provider turn is not known, and the transport builds no start. **Reason and benefit** - The checkpoint is the cause. With a true checkpoint, the two states of a restart are states that recovery can handle: a wait, or an active turn. - The Runner already sends again each event that was not acknowledged. Recovery uses that stream and does not build the turns again from a read. - The window removes a terminal only while a later terminal of the same type is stored. Thus the first stored terminal after a start shows if a terminal of that turn was removed. A rule that only ignores the gaps is not safe: the Runner can make a `turn.failed` for a turn that did not emit its start (`provider_turn_lost_on_restore`). ## What Changed Production code: 113 lines added, 19 removed. - `src/native-session-runtime.ts` (15 added, 4 removed): the two checkpoints in `consumeTurn`, with the sequence of the appended event, and the resumed wait with a consumed disposition marker. - `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed), `codex-session-terminal.ts` (2 added), `codex-session-notifications.ts` (2 added), `codex-harness-session.ts` (24 added, 7 removed), `codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at the event, and the recovery of the driver. - `src/contracts/harness-driver.ts` and `src/contracts/native-session-backend.ts` (1 added each): an optional `boundarySnapshot` on a driver session, and an optional sequence in the snapshot options. Other drivers do not implement it and are not changed. - `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the session terminal, and the snapshot as of a sequence. - `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and `src/live/provider-turn-tool-order.ts` (18 added): the start-first rule and the owner of an event. - Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5 removed), `src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348 added), `src/live/runnerd-codex-transport.test.ts` (137 added), `src/control-plane/durable-prp-control-plane.test.ts` (84 added), `src/live/provider-turn-tool-order.test.ts` (new, 61), `src/native-session-runtime.provider-turn.test.ts` (92 added, 6 removed), `src/backends/harness-driver-backend.test.ts` (56 added). - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only, 123 added, 14 removed): a delay before a provider-started turn, a goal state at its terminal, a list of provider-started turns for one session, a gap between them, a turn that stays open until the new controller reads the snapshot, and a turn that starts after that read. The modes that existed are not changed, and the fake reads its input on one thread as before. ## State saved by the previous release - No new persisted field. The driver session already has `awaitingProviderTurn` (#21453). - **More checkpoints:** one when a wait starts and one when a provider-started turn starts. An ordinary turn saves the same number of checkpoints as before (4 in the unit test, measured before and after). - **No terminal in a checkpoint of a wait.** A checkpoint that is saved during a wait or during a provider-started turn has `terminal: null`. When a wait ends with no provider-started turn (the limit, a comment of a person, or the Runner ends the run), the last checkpoint of the run also has `terminal: null`; before, it had the terminal of the turn that waited. One reader uses that field with no result: the text of a cleanup failure (`nativeCheckpointRecordsCompletedTurn`), which then does not say that the turn completed. In the tests with the real daemon (the limit, the provider process lost, the sidecar exits), the run still ends on the path of today (`native_finalization_missing`). - **A checkpoint of the previous release** has no waiting state. Recovery takes the path of today. - **Rollback of the Runner package during a wait:** the previous code reads the checkpoint, ignores `awaitingProviderTurn`, and takes its recovery path of today. ## Verification With nothing from part 3. The Rust checks and the build checks in the first item were run before the last seven commits; these commits change no Rust. The other items were run at the head of this branch (`36d5e5e587`), or at an earlier commit of this branch for a suite whose files the later commits do not change. - Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm run build:typescript` and `node --test test/acpx-sidecar-contract.test.mjs` (18 passed) in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt --check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the `acpx` unit tests (102 passed): all passed. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in `server`: all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the production code of this head, 26 of 26 passed in each. The Core of these tests follows the production append (`appendNativeSourceEvent`): it stores an event that is out of order, keeps its contiguous cursor below a gap, and reports a duplicate or a conflict for an event that it has. It does not reject a gap in the sequence; production does not. The restart cases, all with the real `paperclip-runnerd`: - **From the checkpoint that the real consumer saved.** The first `executeNativeSession` is fenced and detached as the server does for a restart, and a second one recovers from the last checkpoint that the first one saved. (a) Restart during the wait; the provider-started turn starts after the new controller is attached. (b) Restart while a provider-started turn is in progress. In each: the test asserts the shape of that checkpoint, the run settles with the result of the provider-started turn, each event is appended one time and in order, the turn has one `turn.started` with `origin: "provider"`, and the session closes with no error. - **Without the two checkpoints, each of these cases fails:** the saved checkpoint is the checkpoint of the turn before, and the second controller fails with `turn_binding_mismatch` or with a replay conflict. - **A slow `appendEvent`, with events that arrive behind the stored event** (3 cases, each run 3 times with the fix and 3 times without it, with the same result in each run). R7: the append of the `turn.completed` that waits is slow, and the provider-started turn starts behind it. With the snapshot at the event, the checkpoint is the waiting state at the sequence that Core has, and Core gets the `turn.started` of the turn one time. Without it, the checkpoint names the provider-started turn as active, the transport builds no start, and Core never gets that `turn.started`. R8: the first controller answers the `paperclip_finish` call while the append of the provider `turn.started` is slow. With the snapshot at the event, the checkpoint has no result and the run ends with `native_finalization_missing` (see Risks). Without it, the checkpoint has a result that Core does not have, and recovery stops with "session cannot start another turn". R6: the append of the provider `turn.started` is slow and a delta is queued behind it. With the snapshot at the event, the new controller continues the sequence that Core has. Without it, the only difference is one sequence number that Core never gets. - **The start was acknowledged before the restart** (3 cases: one turn open across the restart; two turns; a turn that waits for the answer to its `paperclip_finish` call). The new driver gets the start of each turn one time before its other events, the finish call is answered exactly one time, and the session closes with no error. - **The turn ran with no controller attached** (2 cases, with and with no result) and **two turns in one outage** (2 cases): the turns arrive in order after the recovery. - **A provider-started turn that is active in the checkpoint** gets no second start, and the next provider-started turn arrives after it. - **The turn already has its result when recovery completes** (3 cases: the turn made its `paperclip_finish` call with no controller attached; the turn waited for the answer to that call at the restart; two turns across the restart). The test asserts the exact list of events that Core gets: each provider-started turn has one start, its usage item, its message item and its terminal, and then `run.result.accepted` and `run.terminal`. `completeRun` is called one time with the turn that has the result. Before the change to the session terminal, the second controller appended only `run.result.accepted` and `run.terminal` in each of these cases. - **The wait checkpoint is not saved, then a restart** (R10). The last checkpoint is the checkpoint of the controller turn, Core has the terminal of that turn, and the start of the provider-started turn is queued. The run settles with the result of the provider-started turn, Core has one `turn.started` for it, and the finish call is answered one time. Without the wait in the driver, recovery stops with `turn_binding_mismatch`. - **An ordinary turn**, with no option. One that waits for the answer to its finish call at a restart: Core gets its result, its terminal and the run terminal one time. One whose checkpoint after `startTurn` is ahead of Core at a restart (R9): the run settles, and Core does not have only the one event that the first controller did not store. These cases pass with and without this pull request. - **No checkpoint with a terminal and no result** is saved by the two controllers in the cases above. - `src/live/provider-turn-tool-order.test.ts` and `src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests cover a stored start with removed events after it, a dropped delta, the terminal of the turn before and after the event, a terminal of a different turn after a removed terminal, and a controller start. With no option, with a different agent or provider, and for a turn that the controller started, the notifications of the transport are the same as before this pull request. - `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed. Three new tests use the real commit path: a full window of 4,096 events removes the events after a provider start and the start is found; the window removes the `turn.failed` of a provider turn after a later `turn.failed` with no start, and no start is found; a restored checkpoint that dropped the deltas of the turn, and the start is found. - `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`: 491 passed. The driver tests cover a provider start that is queued before `recoverSession()` is called, the wait state with one turn in progress at `reconcile()` (and the two cases that it must still refuse), a restored wait with a consumed disposition marker (the provider turn is admitted and `startTurn` is refused), the snapshot at the two events (also after the driver moved on, and for a restart continuation that ends waiting), one later turn in progress, one completed and a second in progress, two completed with a result, and the cases that `reconcile()` must still refuse. The backend tests cover the session terminal and the snapshot as of a sequence. - `src/native-session-runtime.provider-turn.test.ts` and `src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a restored wait with a consumed disposition marker: the provider-started turn arrives and settles the run, or the wait ends at its limit and the run takes the path of today with no new turn. - `server`: `native-session-executor.test.ts`, `native-provider-turn-wait.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21 skipped. - Mutation checks, each made named tests fail: no checkpoint at the start of a wait; no checkpoint at the start of a provider-started turn; the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in the sequence, no `turn.started` in Core, a different failure); the backend ignores the sequence; the driver has no snapshot at the event; no start-first rule; no exclusion of the active turn of the recovered driver; the turn binding only at the first delivery; the contiguous sequence required again; no check of the first terminal after the start; the adoption of later turns put back; no exception in `reconcile()`; no check after the settlement in `reconcile()`; the terminal of the session set again by a turn that ends waiting; no wait of the provider start for `reconcile()`; no exception in `reconcile()` for a session that waits; the pump deferred for a restored wait; the consumed-marker shortcut taken for a snapshot that waits. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** Each change is behind the waiting state, a provider `turn.started`, or the option of the transport. No caller sets the option before part 3. - **An event that the first controller received but Core did not store is not sent again after a restart.** This is the same on `master` and is not changed here. Only the terminal of a turn is built again. If the lost event is the result of a provider-started turn (the first controller answered the `paperclip_finish` call and stopped before Core stored the proposal), the run ends on the path of today for a turn with no result. R8 shows this. R9 shows the same loss for one event of an ordinary turn. - **A second `turn.started` for one provider-started turn can reach Core.** This occurs if the restart comes after Core stored the start and before the checkpoint of that start is saved (the time of one checkpoint write). The last checkpoint then waits, and the transport builds the start again for the new driver. Seen with the real daemon when the checkpoint write is held; not in a committed test. The usage accounting of the server uses the turn id as its key, so there is no second count, but the second start resets the "restored" mark of that turn. The transcript does not read `turn.started`. - **The older checkpoints can also save a cursor that is ahead of Core** (measured for the checkpoint after `startTurn` of an ordinary turn). Production accepts the gap in the sequence that follows, so this is not a failure there. It is on `master` and is not changed here. - **When the terminal of a provider-started turn was removed from the stored events,** the transport builds no start for an event of that turn, and the session stops on that event, as before. This needs a later terminal of the same type with no start of its own (a turn that was lost while it started) and a full window. - **The last checkpoint of a wait that ends with no provider-started turn has no terminal,** as the section above says. The change is only in a run that waits, and no run waits before part 3. - **A wait with a consumed disposition marker has no test with the real daemon.** The state needs two restarts in one run, and the test helper does one. Unit tests of the driver and of the consumer cover it. - **A provider start that waits for `reconcile()`** waits as long as the reads of `reconcile()` take. Each read ends at the command deadline of the Runner (30 seconds by default) or when the transport closes. - **`item.delta` events across a restart are not asserted.** A streamed delta that the first controller received is not sent again. The completed items are asserted. - **The driver keeps a maximum of 8 snapshots at events** and drops each one when the consumer asks for it or for a later one. If the consumer is more than 8 such events behind, the checkpoint uses the current snapshot, as before this change. - **A second provider-started turn that also ends during the same outage** is not tested with the real daemon: the fake gives each terminal the same goal revision, and the Runner then holds the second terminal until a controller is attached. The end of the reconnect time of the Runner is not tested here. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, decided the designs, reviewed the diffs, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with tool use (shell, file edits, local test runs), wrote the change and the tests, reproduced the review findings with the real daemon, and verified this slice alone. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>canary/v2026.1114.0-canary.16 |
||
|
|
d4498ae7ac |
fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.
Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.
**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.
**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.
**Paperclip version or commit**
`master` at `df1b12ad8e`.
## What Changed
- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.
## Verification
- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.
## Risks
- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9b581d689b |
fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In agent chat, a finished delegated task or a declined tool action wakes the chat agent so it tells the person > - #21401 stopped a second completion report when an earlier reply already gave it > - But the suppressed completion still created a zero-second heartbeat run, and declined tool actions still got a second report > - Run accounting must count only real turns, and the person must get each outcome once > - This pull request settles an already-reported outcome before it becomes a run, and applies the same rule to declined tool actions > - The benefit is one report per outcome, no phantom run, and no extra model call ## Linked Issues or Issue Description Refs #21401 **What happened?** Chat eval cohort 13 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`) showed three problems after #21401. 1. `questions-and-approvals.duplicate-answer`: the completion wake was suppressed, but it still created a 0-second run (`runtimeMode: legacy`, no usage). The bounded-native-runs check rejected that run. 2. `skills.studio-test`: the CEO's single reply reported the Studio result after it read the test task done (0 passed, 1 failed, links COM-2). The completion was then suppressed, but the eval waited for a separate completion run and timed out after 240 s. 3. `questions-and-approvals.decline-tool`: the person declined the tool review one second after the turn for their message started. That turn read the declined action and said so. The decline continuation, which waited behind that turn, then posted the same declined outcome again. **Expected behavior** Each finished task or declined tool action is reported once in the conversation. An outcome that an earlier reply already reported creates no heartbeat run. **Steps to reproduce** 1. In chat, delegate a short task, or start a governed tool read that needs approval. 2. While it waits, send a message that wakes the chat agent. 3. Let the task finish, or decline the tool review, while that turn runs. The turn reads the outcome and reports it. 4. See the completion wake become a 0-second run, or see a second decline report from the decline continuation. **Paperclip version or commit** master `df1b12ad8e` ## What Changed - `server/src/services/chat-completion-delivery.ts`: `settleReportedChatCompletions` marks each undispatched completion that an earlier reply reported as delivered, with that reply as its response. It returns true only when nothing else is owed. The outbox `deliver` also checks this before it requests a wake, so no wake and no run exist for an already-reported completion. - `server/src/modules/wake-queue`: a new transaction port, `skipAlreadyReportedChatWake`, runs in the release drain before a deferred wake is promoted. When the earlier reply already reported every outcome of the wake, the wake ends `skipped` ("Already reported by an earlier reply in this conversation") and no run is created. Wakes that carry queued comments or a failed-chat retry are never skipped. - `server/src/services/chat-interaction-outcome-report.ts` (new): the same structural rule for declined tool actions. A conversation agent run that lists its interactions records `chat_interaction.declined_read` for each declined tool action. The decline continuation is settled when the same agent replied in the conversation after the decline, from a run with that receipt, posted after the read. The rule covers only declined tool-action confirmations. Answers, plan reviews and accepted actions keep their own turn. - `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls the recorder for agent runs. - The run-start check from #21401 stays as a fallback for a completion turn that was admitted before the earlier reply landed. - `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`: the Studio oracle (v5) accepts either the completion turn's reply or an earlier CEO reply as the delivery. The earlier reply counts only from saved state: no completion run exists, the replying run has a `chat_handoff.completed_read` receipt for the test task, the reply was posted after that read, it is the only such reply, and it links the saved result. No wording check is added. - `doc/run-log-events.md`: documents both read receipts and the skipped wake outcome. ## Verification - `cd server && npx vitest run src/__tests__/chat-completion-delivery.test.ts src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts src/modules/wake-queue/application/use-cases.test.ts`: all pass. - Red: with the drain check and the outbox check disabled, 5 tests fail: "starts no completion wake or run ...", "settles a deferred completion wake without a run ... (true)", "settles the decline continuation with no run ...", and both "settles an already-reported chat wake without a run" cases. - Negative controls stay green: a linking reply without a done read, a decline the earlier turn never read, a non-tool-action card, and an accepted card all still run their continuation. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`: 34 pass. Red on the old oracle: "accepts the earlier reply the server took as the completion report" fails. New negative cases reject no done read, a read of another task, a reply before the read, a reply from another run, two replies, an extra completion run, and a missing task link. - A narrow paid eval run of `duplicate-answer`, `studio-test` and `decline-tool` on this branch follows in a comment. ## Risks - A completion turn admitted before the earlier reply landed (a narrow race) still ends as a short succeeded run with no model call, as in #21401. - If an earlier reply read the outcome but did not mention it, the person does not get a second message. The read receipt, the reply-after-read order and, for completions, the task link make this unlikely. - The decline rule covers only declined tool-action confirmations. Other interaction outcomes are unchanged. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
64f3abbdef |
test(evals): write the hard-stop setup note in plain words (#21487)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog checks that the chat agent explains a budget hard stop to a person > - The `budgets-costs.explain-hard-stop` setup puts a note on the conversation. The note said the receipt was "an imported synthetic accounting fixture, not live model-provider billing" > - In chat-eval cohort 13, the agent repeated that eval jargon to the person > - This pull request rewrites the setup note in plain words, the same way #21408 and #21429 did for spend-breakdown > - The benefit is that the eval tests the explanation a real person would get, without setup jargon to copy ## Linked Issues or Issue Description Refs #21408 Refs #21429 **What happened?** Chat-eval cohort 13 ran OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`. The `budgets-costs.explain-hard-stop` reply gave a correct diagnosis, but it said "The charge is an imported accounting fixture, not live model-provider billing". That wording came from the eval's own setup note. **Expected behavior** The setup note uses plain words. It keeps the references the scenario needs, and it does not give the agent test jargon to repeat. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.budgets-costs.explain-hard-stop`. 2. Read the conversation note and the reply. ## What Changed - `budgetIncidentSourceDescription` in `tests/runner-e2e/chat-catalog-domains/identity-budget-incidents.ts` builds the note. The note names the task, the recorded hard-stop incident and the separately paused agent. It no longer says that the receipt is an imported synthetic fixture. - The grader is unchanged. It already proves the receipt's origin from saved state (`receiptOrigin`), not from the reply. The note's task identifier, incident reference and paused-agent name are unchanged. The grader matches no literal from the note. - TestOps manifest: no change is necessary. The case's oracle version, source file list, query, context and expected outcome are unchanged. ## Verification - `npx vitest run -c tests/runner-e2e/vitest.config.ts chat-catalog-domains/identity-budget-incidents.test.ts` (16 tests pass). - Red/green: the new test checks that the note keeps the task, incident and paused-agent references and contains none of the setup jargon. It fails on master (no plain-words note) and passes with this change. - A narrow paid run of `budgets-costs.explain-hard-stop` (with the two sibling cohort-13 PRs) ran on a combined branch. The results are in a PR comment. ## Risks - Low risk. Only eval setup text changes. The scenario difficulty is the same: the note states the same facts about the stop and the manual pause, without the test-origin remark. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8a97037bd2 |
fix(runs): say who started a retry run in the issue run read (#21486)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat agent answers questions about failed runs by reading the
task's run list (`GET /api/issues/{id}/runs`)
> - For a retry run, that read returned only raw invocation fields
(`on_demand`, `manual`) and an owner id. It did not say who started the
retry
> - In chat-eval cohort 13, the agent told the person "Paperclip retried
automatically" about a run that the board started with Retry
> - This pull request adds a plain-words `retriedBy` field to each retry
run, and tells the chat agent to use it
> - The benefit is that people get a true account of who retried their
work, with a name and no raw ids
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
In `runs-adapters.inspect-failed-run`, the reply said "Paperclip retried
automatically on a supported model". The successful run was a person's
manual Retry: `invocationSource: on_demand`, `triggerDetail: manual`,
wake reason `retry_failed_run`, and the responsible user was the board.
The agent's own reasoning hedged ("probably automatic"), because the run
read did not say who started the retry.
**Expected behavior**
The run reads that the agent uses say who started each retry: a person,
by display name, or Paperclip's automatic recovery, in plain words and
with no raw ids.
**Steps to reproduce**
1. Let a run fail on a task.
2. Press Retry on the failed run as a board user.
3. Read `GET /api/issues/{id}/runs`. Before this change, nothing in the
retry row says that a person pressed Retry.
## What Changed
- `runsForIssue` (`server/src/services/activity.ts`) reads the wake
request of each retry run. It adds `retriedBy` to each row:
- `{ kind: "person", personName, summary }` when a user wake with reason
`retry_failed_run` or `task_retry` started the run. The name comes from
the user record, or "Board" for the local board, with the same fallback
as activity actor names.
- `{ kind: "automatic", summary }` for an `automation` retry run.
- `null` for a run that is not a retry, or a retry whose starter is not
recorded. The read does not guess. It does not use `responsibleUserId`,
because automatic retries keep the original responsible user.
- The OpenAPI description of `GET /api/issues/{id}/runs` and the chat
agent directive tell the agent to use `retriedBy` and not to call a
retry automatic unless `retriedBy` says so.
- The UI `RunForIssue` type has the new optional field.
## Verification
- `cd server && npx vitest run src/__tests__/activity-service.test.ts
src/__tests__/agent-conversations.test.ts
src/__tests__/activity-routes.test.ts
src/__tests__/openapi-routes.test.ts` passes.
- Red/green: the new test "says who started each retry run" fails
without the service change (`expected undefined to be null`) and passes
with it. It covers a board Retry, a named member's Retry, an automatic
retry that keeps the board as responsible user, a retry with no recorded
wake, and no raw ids in the output.
- `npx tsc --noEmit -p server` passes.
- A narrow paid run of `runs-adapters.inspect-failed-run` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.
## Risks
- Low risk. The change adds one read-only field and one bounded query by
primary key on the wake requests of retry runs in the page.
- No persisted state changes.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
95a6ab31e6 |
feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.
## Linked Issues or Issue Description
No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).
**Current behavior**
A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.
**Proposed behavior**
An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.
**Reason and benefit**
Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.
## What Changed
- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
- `signal` in `signalDatabaseWork`;
- `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
- `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
- each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
- `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
- the process instance is disabled without the env;
- allowlist filtering, and spec parsing including colon topics;
- per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
- real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
- suppressed records never capture a stack;
- buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
- no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.
## Verification
- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:
```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```
## Risks
- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #21621.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
163d9f323c |
perf(github): reuse repository installation tokens until shortly before expiry (#21641)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel lets an agent review pull requests and answer comments through a GitHub App > - Each GitHub call gets a repository-scoped installation token from `githubBotRepositoryToken` > - That function minted a new token on every call. To mint, it resolved every App secret through the secret service, and each resolution wrote one audit row > - The review-check head poll calls it about every 5 minutes for each waiting check. On staging this made the secret audit insert one of the busiest statements in the database > - This pull request keeps a minted token in memory until shortly before it expires, and checks the connection's current state on every call > - The benefit is far fewer database writes and reads, fewer GitHub token mints, and an audit log that records real secret use ## Linked Issues or Issue Description **What happened?** On paperclip.staging (commitcanary/v2026.1114.0-canary.15 |
||
|
|
a18479bbba |
perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The terminal workspace sweep walks every candidate workspace in a full pass. On staging that is about 4,000 finished remote workspaces, and each one costs a persistence probe and a work product read. > - #21617 limited full passes to startup and an hourly backstop that only activity arms. But three other paths in the worker still started a full pass at once: > - a wake whose hints do not parse to workspace or task targets; > - more than 50 pending targets; > - any run that finds no targets. > - Across one staging deploy the probe count rose by about 2.8 full passes in 26 minutes. At 1,000 concurrent agents, more than 50 run-finish hints between passes would be normal, so the overflow path alone would bring back passes proportional to history. > - This pull request works through targets in batches of 50 and turns hintless wakes into the backstop. It gives the one caller that needs a broad look, a workspace provider plugin becoming ready, its own hint that inspects only plugin workspaces. > - The benefit is that a full pass happens only at startup and at most once per backstop hour, whatever the activity. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On staging, the hourly steady state holds (one full pass per hour, no probes between). But across a deploy the probe count rose by 11,351 in under 26 minutes, about 2.8 full passes over the roughly 4,000 finished remote workspaces. **Steps to reproduce** The new test file `server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs the real coordinator and worker against real PostgreSQL on a fake clock, with finished remote workspaces that cannot be archived (the staging state). These are the counts on master. The numbers are workspaces checked by full passes; a pass over N workspaces checks N: | Scenario | Full-pass checks on master | Expected | | --- | --- | --- | | A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2 extra passes) | 0 | | One hintless wake and one unparseable hint, 120 workspaces, 30 minutes | 1,490 (about 12 passes) | 0, then 1 at the backstop | | A workspace provider plugin becomes ready (hintless notify from `plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin workspaces | | Restart: old instance drains while hints arrive, new instance starts and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30 minutes | 2,424 (12 passes) | 202 (the startup pass) | **Cause of the passes across a deploy:** - The new instance runs its startup pass. - Each workspace provider plugin that turns `ready` sends a hintless notification (`plugin-registry.ts`), which started a full pass. - Run recovery at startup finishes many runs, and their `issue:` hints overflow 50, which started another full pass. - Any later wake without targets also started a full pass, including the coordinator's 5-minute retry timer. In the fixture this produced two back-to-back passes every 5 minutes. - The draining old instance starts nothing, because the worker's `canRun` is false during drain. The test confirms this. **Expected behavior** A full pass runs only at startup and when the activity-armed backstop is due, at most once per hour. Targets are worked through in targeted batches. ## What Changed - `server/src/services/terminal-workspace-work.ts`: - **Pending targets.** Workspace and task targets go into pending sets with no cap. Each run inspects at most 50 of them, and the rest stay queued for the next run, which is scheduled at once. A failed targeted run puts its targets back. - **Hintless or unparseable wakes** only arm the hourly backstop. - **No targets, nothing due:** the run does nothing. - **Due deadlines:** the due workspaces are queued as targets, including isolated re-home and nested clone deadlines keyed by a workspace. A due non-workspace deadline (the isolated source orphan sweep, task directory work) runs a targeted sweep for its prelude and arms the backstop. - **A failed plugin workspace listing** restores the plugin refresh flag, so the coordinator's retry lists again. - **Full passes** happen only at startup, when the backstop is due, and on continuation pages of a pass already under way. - **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`): it queues the ids of plugin-provider candidate workspaces from `listPluginTerminalWorkspaceIds`. - The worker now always tells the service `targeted: true`, so a hint during a pass no longer requests a rescan lap. - `server/src/services/execution-workspaces.ts`: - New `listPluginTerminalWorkspaceIds` reads the plugin-provider candidates. - A targeted sweep reads all matches of its at most 50 targets, with no page limit and no rescan request. - A failed targeted sweep no longer requests a rescan lap, because the worker re-queues its targets. - `server/src/services/plugin-registry.ts`: the two workspace-provider notifications send `plugin-workspaces` instead of no hint. - `server/src/__tests__/terminal-workspace-work.test.ts`: a new test, `retries the plugin workspace listing after it fails instead of waiting for the backstop`, fails without the retry fix. Three existing tests now describe the new behaviour: - unscoped commits go to the backstop; - more than 50 hinted workspaces are inspected in batches of 50 and 1; - a due deadline on a service without targeting still runs a full pass. - `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`: four new tests, the scenarios above. The startup pass stays. The 1,000-workspace test in `terminal-workspace-deadlines.test.ts` shows that durable deadlines keep an idle instance quiet after startup. But only the startup pass reconciles workspaces whose hints were lost while no instance was running, so removing it would need a stored "last full pass" marker first. ## Verification - The four new tests fail on master with the counts in the table and pass after the change: - 0 full-pass checks for the burst, worked as targeted batches of at most 50 (120 checks in total); - 0 full-pass checks for 30 minutes after hintless and unparseable wakes, then 1 pass at the backstop; - only the 3 plugin workspaces are inspected after the plugin hint; - exactly 1 full pass across the restart: none on the draining instance, only the startup pass on the new one. - Prompt archiving and the missed-hint backstop keep their existing tests in `terminal-workspace-deadlines.test.ts`: - the release of a holding run inspects its workspace at once; - a run made terminal outside `writeRunStatus` is archived within 1 hour 1 minute; - 1,000 finished remote workspaces stay unread at idle. - `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`, `terminal-workspace-work`, `terminal-workspace-deadlines`, `terminal-workspace-idle`, `terminal-workspace-page-batch`, `execution-workspaces-service`, `plugin-workspaces.integration`, `workspace-operations-reconciliation`, `execution-workspace-reopen`, `isolated-task-directory-retention`, `git-workspace-expiration` and `workspace-delivery-cleanup.integration`. - `pnpm check:module-boundaries` passes. ## Risks - **A notification that relied on an immediate broad pass now waits up to an hour.** The only caller I found that sent such a notification on purpose is the workspace provider readiness in `plugin-registry.ts`, and it now has its own targeted hint. Other hintless notifications arm the backstop. - **The isolated source orphan sweep** runs only at the start of a full pass. Its due deadline now arms the backstop instead of starting a pass at once. This only matters when the isolated source re-home feature is enabled. - **A targeted sweep has no page limit.** Its at most 50 targets bound the matched workspaces to those tasks, their ancestors and their execution workspaces. - **A burst of N hints costs N/50 targeted runs, back to back.** The cost scales with activity, not with history. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.14 |
||
|
|
890dcbdd66 |
fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap: the outgoing server still serves while the incoming server starts new runs > - A warm agent session stays open in the server process that ran the last turn, after that run finished > - The incoming server took over a warm session while the outgoing server was alive and still held it, because the finished run's finalization lease ends at commit > - The takeover then failed, the task went blocked with a Retry, and in one case a retry loop ran 27 times on the same sandbox > - This pull request makes the incoming server wait while the server that holds the warm session is alive, makes the outgoing server close idle warm sessions as soon as it leaves new runs to the replacement, and stops the retry loop when the promised fresh sandbox does not come > - The benefit is that a follow-up during a deploy finishes without a person, and no Runner ever gets two controllers ## Linked Issues or Issue Description **What happened?** A zero-downtime release-swap test (Gate B, wave B1, mastercanary/v2026.1114.0-canary.13 |
||
|
|
45531b719e |
perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Stranded-work recovery checks tasks that are assigned to an agent
but have no live run. It runs as a scoped-deadline queue that a change
notification or a stored deadline wakes.
> - For a failed legacy run, the check calls
`terminalizeLegacyExecution` with the run's current status. When the
run's recovery decision is already recorded (for example "continue in a
fresh run from saved work"), that call changes nothing. But it still
updated the run row and notified about 18 delivery queues, including the
run's own recovery graph.
> - The graph notification re-checked the same task, which re-asserted
the same run again. The loop ran about once a second per waiting task,
forever, with zero runs active. On a hosted instance this shows up as
constant idle transactions.
> - This pull request makes a re-assertion that changes nothing return
before it writes or notifies.
> - The benefit is that an idle instance does no stranded-recovery work
for tasks that wait on a recorded decision. A real change (another run
fails, a comment, a status change) still notifies and recovers at once.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
With zero runs active, the server runs many transactions per second. A
CPU profile groups most samples under `reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`). A local fixture on real
PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent,
whose latest legacy run failed and whose recovery decision is recorded.
Each check reaches the branch that terminalizes a failed legacy run that
needs reconciliation (`service.ts`, the
`legacyExecutionNeedsReconciliationWithEvidence` branch, and the same
call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution`
then:
- updates the run with the same status (only `updated_at` changes);
- calls `notifyExecutionRunWork`, `notifyInstructionCollection` and
`notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the
task and about 18 queues;
- finds the recorded decision and returns.
The graph hint re-checks the same task after the queue's 1-second pass
interval, so the loop never ends.
**Expected behavior**
A check that finds nothing to change stores nothing and wakes nothing.
Only a real change re-checks the task.
**Steps to reproduce**
1. Create a task in `in_progress` assigned to an agent, with a failed
legacy run (`adapter_failed`).
2. Start the issue-state recovery work. The boot pass records a
fresh-continuation decision for the run.
3. Leave the instance idle. On master: 591,045 statements in 30 minutes
for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update
"heartbeat_runs"` per task per second.
**Paperclip version or commit**
master at
|
||
|
|
68d5c9c120 |
feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner can already keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started (#21423, #21316). Core can already wait for that turn (#21453, #21454, #21455). No run does this yet, because no controller asks the Runner for it. > - The request goes through the transport of the Runner package. This pull request gives the transport an option for it and the behavior that the option needs. > - No caller sets the option in this pull request. It changes no production run. > - The benefit is a small base for the two pull requests after it: the recovery after a controller restart, and the switch that turns the behavior on. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. **This pull request:** the transport can ask the Runner for provider-started turns. No caller asks. 2. #21612: the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It adds the support that parts 2 and 3 use. **Subsystem affected** Paperclip Runner package, the controller side: `src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a caller that sets the new option. **Current behavior** - The transport has no way to ask the Runner for provider-started turns. **Proposed behavior** - The transport options have a new field, `providerStartedTurns`. The transport uses it only for ACPX with the agent `claude`. In each other case the field has no effect. - With the option, the transport sends `providerStartedTurns: true` in `turn.start`. It does not send it for a compaction turn. - With the option, a tool call in a provider-started turn waits until the transport gave each earlier event to the driver. Without this, a `paperclip_finish` of that turn could reach the driver before the `turn.started` of that turn, and the driver would refuse it. - With the option, the transport gives a `run.terminal` of the Runner to the driver as a marked internal notification (`paperclip/runTerminal`). The driver uses it only during a wait, so a wait ends at once when the Runner says that the run is over. - With no option, the transport does the same as before this pull request. Live sessions, evals, the CLI runner and the probes do not set the option. **Reason and benefit** - The opt-in is a choice of the caller, not a property of the agent. Only a consumer that can wait for a provider-started turn must ask for one. `CapabilityLiveSession` completes a turn at the first `turn/completed`, so it must not get the behavior because its agent is Claude. ## What Changed - `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the option, the field in `turn.start`, the order of the tool call, and the `paperclip/runTerminal` notification. - `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the source sequence of a tool call that belongs to a provider-started turn. - `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7 cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the real consumer of Core. - `src/live/live-session.test.ts` (12 lines added): a Claude live session gives the transport no such option. - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only): the mode `provider-result-resolved`. In this mode the provider-started turn calls `paperclip_finish` and waits for the answer of Core through the Runner (`tool.resolve`), as the real sidecar does. The mode `provider-result` is not changed. ## State saved by the previous release - No persisted state changes in this part. - **A Runner that does not know the field** ignores it. - **Rollback of the Runner package:** the previous transport does not send the field, so the Runner does not wait. ## Verification At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and 3: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node --test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed in each. The cases: - background work, then the turn of Claude calls `paperclip_finish`; Core answers; the Runner gives the answer to the sidecar with exactly one `tool.resolve`; the turn completes; the run finalizes with one `run.result.accepted`; - a Claude run with no option from the caller takes the path of today: no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`, no wait, and no `session.provider_turn` to the sidecar; - the limit ends the wait, the run takes the path of today, and the provider processes had exited before the cleanup of the test; - the provider process is lost during the wait, and the run settles in seconds; - the sidecar exits during the wait, and the run settles in seconds; - a turn with no background work does not wait; - an ACPX agent that is not Claude is never opted in, also when the caller sets the option. - Each case asserts the result of the close of the session: clean in 6 cases, and a named failure in the case where the sidecar is killed. Each clean case also asserts that there is no `native_session_settlement_incomplete` diagnostic and no "conflicting duplicate tool result" in the durable state. - `src/live/live-session.test.ts`, `src/live/runnerd-codex-transport.test.ts`, `src/live/runnerd-acpx-provider-turn-wait.test.ts`, `src/native-session-runtime.provider-turn.test.ts`: 434 passed. - Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102 passed). - Mutation checks, run on the full change before the split: with the option ignored, 6 of the 7 assertions of the case with no option fail; with the old mode of the fake, the case with the result fails at the close of the session. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No caller sets the option. Core sets it in part 3. - **The transport is shared by each provider.** The three new behaviors are behind one condition: the option, ACPX, and the agent `claude`. With the condition false, the code path is the path of today. - **The tool-call ordering uses a 10 ms poll** while earlier events are not delivered, only in a provider-started turn. - **A controller restart during a wait or during a provider-started turn is not handled in this part.** Part 2 handles it. Part 3 must not merge before part 2. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1114.0-canary.12 |
||
|
|
7db92ea319 |
perf(workspaces): read a sweep page's task trees and active runs in two statements (#21618)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep inspects candidate workspaces 50 per
page. For each workspace it read the task tree with its own recursive
query. For many of them it also ran one or two queries to check for an
active run.
> - The cost of a page therefore grew with the page: about 50 to 110
statements. On the hosted instance, building the SQL text of these
statements costs about as much main-thread time as parsing their
results.
> - The archive statement re-checks the terminal tree and the absence of
a live run in the same statement, and the remote and plugin cleanups
re-check active runs under the lifecycle lock. So these early reads are
only pre-filters.
> - This pull request reads the task trees and the active-run workspaces
of a whole page in two fixed-text statements.
> - The benefit is that a page costs a constant number of these reads,
whatever its size.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The per-page reads of the terminal workspace sweep
(`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).
**Current behavior**
Each candidate calls `listWorkspaceIssueTree`, one recursive query. A
candidate that reaches the active-run checks calls
`workspaceHasActiveRun`: one query for the linked tasks' run ids, then
one for live runs.
**Proposed behavior**
- `readPageIssueTrees` reads the tree rows of every root on the page in
one statement. Its SQL text is fixed: the roots are one `jsonb`
parameter. The trees are then built in memory and keep the same row
shape.
- `readPageActiveRunWorkspaces` returns, in one fixed-text statement,
the page workspaces that have a `queued` or `running` run on a linked
task.
- The two pre-filter checks in the loop use these results. A workspace
whose source task changed earlier in the same pass (rehome) still reads
its own tree.
- The authoritative checks are unchanged: the archive `UPDATE` with its
terminal-tree and live-run predicates, the active-run check inside the
remote and plugin cleanup under the lifecycle lock, and the
per-workspace check before a stale reopen flag is cleared.
**Reason and benefit**
One rotation pass should issue a bounded number of statements per page.
For a page of 50 workspaces (40 with open trees, 10 held by a run), the
tree reads fall from 50 to 1 and the active-run reads from 20 to 1.
## What Changed
- `server/src/services/execution-workspaces.ts`: add
`readPageIssueTrees` and `readPageActiveRunWorkspaces`, and use them for
the per-page pre-filter.
- `server/src/__tests__/terminal-workspace-page-batch.test.ts`: two new
tests on real PostgreSQL.
## Verification
- `reads the task trees and active runs of a page in a fixed number of
statements`: 2 companies, 50 workspaces, each with a child task, 10 held
by a `running` run.
- Before the change it fails: 50 tree reads.
- After the change: one tree read, one active-run read, fewer than 10
statements for the page. The results are the same (`checked: 50`,
`skippedNonTerminalTree: 40`, `skippedActiveRun: 10`).
- `does not archive a workspace whose run starts after the page was
read`:
- The page is read first. Then, during that workspace's close-readiness
check, a `running` run is inserted and set as its task's checkout run.
- The sweep counts the workspace `eligible: 1`, `archived: 0`,
`skippedRace: 1`, and it stays `active`.
- After the run succeeds, the next sweep archives it (`archived: 1`).
- `npx vitest run src/__tests__/terminal-workspace-page-batch.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/plugin-workspaces.integration.test.ts` (in `server/`).
- `pnpm check:module-boundaries` passes.
## Risks
- **The pre-filter reads happen once, at the start of the page.** A page
can take long, because cleanups do remote and Git I/O. A run that starts
mid-page on a workspace the pre-filter saw as idle is still caught by a
check made at the destructive step itself, in the same statement or
under a lock. Line numbers are on this head:
- **Archive** of an active workspace: the archive `UPDATE` in
`sweepTerminalWorkspaces` carries `NOT EXISTS (… live_run.status IN
('queued', 'running'))`
(`server/src/services/execution-workspaces.ts:4291`). It changes 0 rows
and the workspace counts as `skippedRace` (`:4340`).
- **Remote cleanup** (destroying the sandbox):
`runTerminalWorkspaceCleanup` re-runs `workspaceHasActiveRun` under the
lifecycle lock before `cleanupRemoteWorkspaceAllocation` (`:2702`).
- **Plugin workspace removal**: the same check under the lifecycle lock
before `removePluginWorkspace` (`:2706`).
- **Git cache expiration** and **isolated task directory expiry**:
`isGitWorkspaceExpirationEligible` re-reads the tree and refuses any run
that is not finished, inside the locked transaction
(`server/src/services/git-workspace-expiration.ts:311` and `:416`,
`server/src/services/isolated-task-directory-retention.ts:244` and
`:290`).
- A run that ends mid-page leaves its workspace for the run-release hint
or the next inspection.
- **Tree rows are now deduplicated** (`UNION` instead of `UNION ALL`).
Parent cycles therefore cannot repeat rows.
- Low risk otherwise. No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.11
|