Commit Graph
200 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 b1e70ab6dd fix(db): recover prepared statements that a pooler or schema change invalidates (#21718)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs every database statement from one Node main thread.
With `prepare: true`, postgres.js sends a statement in one round trip
instead of two, and the CPU cost of each statement is about half
(benchmark from #21707).
> - `DATABASE_PREPARED_STATEMENTS=true` turns on preparation for ORM
queries (#17673). Hosted instances connect through a transaction-mode
pooler (PgBouncer 1.22 or later, Neon `-pooler` hosts), which supports
protocol-level prepared statements.
> - I tested the flag locally through PgBouncer 1.26.1 in transaction
mode. Evictions, server connection churn, `RECONNECT`, transactions,
pipelining, `SET LOCAL`, advisory transaction locks and savepoints all
work. One failure is not acceptable: after a migration changes a column
type, every prepared statement that reads that column fails, for every
client and for every new process.
> - PgBouncer shares one server-side prepared statement by SQL text.
PostgreSQL never repairs a named statement whose result type changed, so
it fails with `cached plan must not change result type` until PgBouncer
drops it. The postgres.js retry prepares the same text again, and
PgBouncer gives it the same broken statement. postgres.js also caches
the failed statement without columns, so the connection throws
`TypeError: Cannot read properties of undefined (reading 'length')` for
that text even after the schema is correct again.
> - This pull request makes the client recover. When a prepared
statement fails with a prepared-statement error, the client stops
preparing that text, retries once without preparation where that is
safe, and counts it. It also caps the number of prepared texts.
> - The benefit is that the flag can be turned on behind a pooler
without a deploy migration that breaks reads. Behavior with the flag
unset or `false` does not change.

## Linked Issues or Issue Description

Refs #17673 (the opt-in), #10795 (the flag), #21707 (the CPU benchmark).

**What happened?**

With `DATABASE_PREPARED_STATEMENTS=true` behind PgBouncer 1.26.1 in
transaction mode (`max_prepared_statements=5`, local PostgreSQL 17), a
change of a column type made every later execution of an affected
statement fail. A new client process failed in the same way. Measured
with the local probe, 20 executions after the change and 10 from a new
process:

| Change | Before: failures | After: failures |
| --- | --- | --- |
| `int` to `bigint`, ORM read outside a transaction | 20/20, new process
10/10 (0A000) | 0/20, 0/10 |
| `varchar` length change, ORM read | 20/20, 10/10 | 0/20, 0/10 |
| Add column, `select *` | 20/20, 10/10 | 0/20, 0/10 |
| `int` to `bigint`, first statement of a transaction | 20/20, 10/10 |
0/20, 0/10 |
| `int` to `bigint`, second statement of a transaction | 20/20, 10/10 |
1/20, 1/10 |
| Table recreated with its original shape, same client | 10/10
(TypeError in `DataRow`) | 0/10 |
| Add column, explicit-column ORM read | 0/20 | 0/20 |

**Expected behavior**

A statement whose pooled prepared statement is stale or lost runs
anyway, and the failure does not reach the caller.

**Steps to reproduce**

1. Run PostgreSQL behind PgBouncer 1.22 or later with `pool_mode =
transaction` and `max_prepared_statements` above 0.
2. Start Paperclip with `DATABASE_PREPARED_STATEMENTS=true` and run a
read of a table a few times.
3. Change the type of a column that the read returns, for example `alter
table ... alter column n type bigint`.
4. Run the read again from the same process or from a new process: it
fails with `cached plan must not change result type`.

## What Changed

- `packages/db/src/prepared-queries.ts`:
- The prepare decision for each `unsafe` query is made when the driver
builds the query, not when the caller creates it, so a query that waits
on a busy connection sees a text that was demoted after it was queued.
- A failure with 26000 (`prepared statement does not exist`), 42P05
(duplicate statement name), or 0A000 `cached plan must not change result
type` demotes the text: the client does not prepare it again in that
process.
- A pooled statement that fails in that way is retried once with
`prepare: false`. Statements in a reserved connection, cursors,
`forEach`, `describe` and queries with `onexecute` are not retried.
- When the failed statement is the first statement that the transaction
ran (only `SET TRANSACTION` before it), the client restarts the
transaction on the same connection: `ROLLBACK`, `BEGIN` with the same
options, the same `SET TRANSACTION` statements, then the statement
unprepared. The callback runs once and receives the result. Tagged
statements and statements in savepoints count toward the execution
order. The client does not restart when another statement of the
transaction is queued, and it never restarts inside a savepoint.
- At most 1,000 distinct texts are prepared for each client and its
private pools. Texts longer than 8 KB are not prepared. Explicit
per-query `prepare` options, simple queries and the flag-off paths are
unchanged.
- `packages/db/src/client.ts`, `packages/db/src/index.ts`: a client and
its dedicated, liveness, checkpoint, file-storage and listener pools
share one state. `preparedStatementStats(db)` returns the prepared,
unprepared, demoted and retried counts.
- `packages/db/src/prepared-queries-postgres.test.ts`: nine tests
against embedded PostgreSQL: restart after a stale plan and after a lost
statement (the callback runs once and its later work commits),
`repeatable read read only` kept after a restart for driver and ORM
transactions, no restart after a later statement, after a tagged first
statement, or for an out-of-order await, a savepoint whose first
statement fails while the outer transaction continues once, the pooled
retry with `.values()`, and the cap.
- `doc/DATABASE.md`, `docs/deploy/database.md`: the cap and the
recovery.

## Verification

- Red/green, `pnpm --filter @paperclipai/db exec vitest run
src/prepared-queries-postgres.test.ts`. With the old
`prepared-queries.ts` (stub stats exports only), the five new tests
fail: two transactions fail with the stale-plan and the
missing-statement errors, the ORM read fails with 42P05, the invalidated
text is prepared again, and 5 texts are prepared where the cap is 2.
With this change, 23/23 pass.
- `pnpm --filter @paperclipai/db exec vitest run` on
`prepared-queries-postgres`, `client-options`, `client`,
`dedicated-connection-pool`, `dedicated-connection-admission`,
`response-capture-postgres`, `lazy-transaction-queries`,
`client-disconnect-replay`: 91/91 pass on the final revision. `tsc
--noEmit -p packages/db` passes.
- 31 server test files with `DATABASE_PREPARED_STATEMENTS=true`
(services, routes, inbox, recovery, heartbeat events, agent auth,
prepared-statement helpers): 829/830 pass. The one failure
(`execution-workspaces-service`, a git-scan timing test) passes alone
with the flag on and off. The machine load average was 9 to 20.
- Local PgBouncer probe (not in CI, because CI has no PgBouncer):
PgBouncer 1.26.1, PostgreSQL 17, postgres.js 3.4.9 (patched), Node 26.4.
The results in the table above, plus 0 failures for 4,800 statements
over 60 distinct texts with `max_prepared_statements=5`, 0 failures with
`pg_terminate_backend` and `RECONNECT` every 25 statements, and 0
failures for transactions with `SET LOCAL`, pipelines of 20 statements,
`pg_advisory_xact_lock` and savepoint rollback.

- Through PgBouncer 1.26.1 (transaction mode,
`max_prepared_statements=5`, 4 server connections) and PostgreSQL 17, at
head 84732c8f67, compared with the master `prepared-queries.ts`. Each
row ran 10 times on its own client, warmed before the migration. "Type"
is `int` to `bigint` under an ORM read; "star" is an added column under
`select *`. Failures before → after:

| Row | Type change | Added column, `select *` |
| --- | --- | --- |
| (a) outside a transaction, same process | 10 → 0 | 10 → 0 |
| (b) first statement of an ORM transaction (callback runs once) | 10 →
0 | 10 → 0 |
| (b) first statement, `begin isolation level repeatable read read only`
| 0 → 0 (the statement was evicted and prepared again) | 10 → 0 |
| (b) first statement, ORM `repeatable read`, `read only` (`SET
TRANSACTION` repeated) | 10 → 0 | 10 → 0 |
| (c) later statement of a transaction, same process | 10 → 1 | 10 → 1 |
| (d) new process after the migration: (a) / (b) / (b) driver / (b) ORM
/ (c) | 10/10/0/0/0 → 0/0/0/0/1 | 10/10/10/10/10 → 0/0/0/0/1 |
| (e) same client after the table is restored: (a) / (b) / (b) driver /
(b) ORM / (c) | 10/10/0/10/10 → 0 | 10 TypeError each → 0 |

Every restarted transaction checked `transaction_isolation = repeatable
read` and `transaction_read_only = on` after the restart. The PostgreSQL
log shows each restart as `ROLLBACK`, `BEGIN` with the same options,
then the statement as an unnamed statement. With 12 clients restarting
at once over 2 server connections and 6 clients of background load,
12/12 transactions succeeded. In the 15 restarts I paired by
`application_name`, the second `BEGIN` ran on the same server connection
as the `ROLLBACK`, because both are sent together. A different server
connection is also safe: only `SET TRANSACTION` ran before the restart
point, and the client repeats it after the new `BEGIN`.
- Silent rollback check through PgBouncer with
`max_prepared_statements=2`: 3 clients ran 600 transactions, each with
an insert and 6 statements over 40 distinct texts, and more distinct
statements between transactions. PgBouncer parsed 6,409 statements on
the server for 1,089 client parses, so it evicted and prepared again the
shared `commit` (`PGBOUNCER_15`) many times. All 600 `COMMIT`s
succeeded, 600/600 rows were present, and no 26000 error was logged. The
silent rollback needs a `DEALLOCATE` on the session, which PgBouncer
does not allow (it closes the client) and Paperclip does not send.

## Risks

- A transaction restart happens below the callback. Code in the callback
before its first statement runs once. The restarted transaction has a
new transaction id.
- A postgres.js behavior that this PR does not change: with driver-level
preparation (the flag `true`, or unset for tagged SQL), the tagged
`commit` of `sql.begin` is a prepared statement. If something
deallocates it on the session, the driver retries `COMMIT` in the
aborted transaction, which PostgreSQL turns into a rollback without an
error. Through PgBouncer this did not happen in 600 transactions with
the `commit` statement evicted repeatedly (see Verification). It needs a
manual `DEALLOCATE` on a direct session.
- A later statement in a transaction still fails once for each affected
text in each process after a migration that changes a column type. The
text then runs unprepared.
- A demoted text stays unprepared until the process restarts. The cap
admits texts in first-use order.
- A client that sends `DEALLOCATE ALL` through PgBouncer is disconnected
by PgBouncer. Paperclip does not send it.
- With the flag unset or `false`, the code path is the same as before.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, local PostgreSQL and PgBouncer in a throwaway
container, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:57:04 -06:00
DottaandClaude Opus 5.5 82b243ab07 perf(server): bound dedicated database connections with a fair queue (#21693)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server holds long advisory-lock transactions on dedicated
database connections (`withDedicatedDbConnection`): workspace source and
lifecycle locks, the skill cache lock, agent-directory operations,
status cards and others
> - `withDedicatedDbConnection` kept at most 4 idle connections, but it
had no limit on connections in use. A run start can hold two at once,
because the source lock nests the lifecycle lock
> - After a restart or deploy, a burst of run starts can open hundreds
of connections. That can exceed the database's `max_connections` at the
moment recovery needs the database most. The documented per-process
total (`DATABASE_POOL_MAX` + 2) was also wrong
> - This pull request bounds the dedicated connections in use with a
fair queue, keeps nested and in-transaction callers deadlock-free, and
states the real connection budget
> - The benefit is that a restart burst of 200 or 1,000 run starts
completes within the limit (at most 48 in use with the default limit of
24), where before most of the starts failed with "too many clients"

## Linked Issues or Issue Description

**What existing behavior does this improve?**
`withDedicatedDbConnection`, which gives long advisory-lock transactions
a connection outside the main pool.

**Subsystem affected**
Database client (`packages/db/src/client.ts`) and the connection-sizing
documentation.

**Current behavior**
Nothing bounds the dedicated connections in use; only 4 idle ones are
kept. A run start can hold two at once (the source lock nests the
lifecycle lock). A burst of run starts after a restart can open hundreds
of connections and exceed `max_connections`. The documented per-process
total (`DATABASE_POOL_MAX` + 2) is wrong.

**Proposed behavior**
A per-process limit on dedicated connections in use
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24) with a first-in,
first-out queue. A caller that already holds a dedicated connection, or
runs inside an open transaction, never waits, because its wait could
deadlock with the holder that blocks it. After
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (default 30 s), a queued caller
gets a connection over the limit, so a lock cycle that PostgreSQL cannot
see ends in a delay, not a hang. The documentation states the real
per-process budget.

**Reason and benefit**
At 50 agents, staging showed about 9 new connections per second from
this function. At 1,000 agents, a restart burst can take the database
down when recovery needs it most. With the default limit of 24, a burst
of 200 or 1,000 starts completes with at most 48 dedicated connections
in use.

**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). Reusing the
outer session for a nested lock was considered. The locks are
transaction-level advisory locks (`pg_advisory_xact_lock`), so a nested
lock in the outer transaction would be held until the outer transaction
ends, which changes lock lifetimes and order.

## What Changed

- `packages/db/src/client.ts`: `withDedicatedDbConnection` admits
callers against a per-owner limit (`DATABASE_DEDICATED_CONNECTION_MAX`,
option `dedicatedConnectionMax`, default 24) with a first-in, first-out
queue. The slot is released after the connection is returned or closed.
- Nested callers (an active dedicated hold of the same owner in the
async context) and callers inside an open transaction of a client from
this process are admitted without a wait. The transaction marker is an
`AsyncLocalStorage` scope that ends when the transaction callback ends.
- A queued caller is admitted over the limit after
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (option
`dedicatedConnectionWaitMs`, default 30000).
- `dedicatedDbConnectionStats(db)` reports the limit, in-use, peak,
waiting, queued, over-limit and nested-over-limit counts.
- `PRIVATE_POOL_MAX_CONNECTIONS` names the private pool sizes that were
literals, and `databaseConnectionBudget()` returns the per-process worst
case: main pool + 9 private + 2 listeners + 2 x limit + 4 idle dedicated
= 73 with the defaults.
- `server/src/services/skill-package-store.ts`: an isolated package read
takes its package I/O permit before its dedicated connection, and takes
a second dedicated connection only for the cache trim after a publish.
Before, each of up to 32 queued readers held a dedicated connection
while it waited for one of 2 permits. `skillPackageStorageIoDepth()`
reports the queue for the store's test.
- `withDedicatedDbConnection(db, action, { admitWithoutWaiting: true })`
admits a caller at once. The skill package store passes it for the
dedicated connections it takes while it holds a package permit, so a
permit holder never waits for a slot that a cache-lock owner holds while
that owner waits for the permit. The overshoot is bounded by the permits
(2 storage, 2 response).
- `doc/DATABASE.md` and `docs/deploy/database.md`: document the
admission rules, the two new variables and the real connection budget,
which replaces "`DATABASE_POOL_MAX` + 2".

## Verification

- New `packages/db/src/dedicated-connection-admission.test.ts` (embedded
PostgreSQL, `max_connections` 100). A restart burst with the default
limit: each start holds a source advisory lock on a dedicated connection
and nests a lifecycle lock on a second one, 40 ms in all.
- Before (same burst, scratch test on master): 200 starts gave 151
failures ("sorry, too many clients already") and 99 dedicated sessions
open. 1,000 starts gave 823 failures and 99 sessions.
- After: 200 starts gave 0 failures, 45 sessions, peak in use 48, queue
wait p50 541 ms / p99 815 ms. 1,000 starts gave 0 failures, 40 sessions,
peak in use 48, wait p50 1,767 ms / p99 3,422 ms. The wait is the queue
of this synthetic burst of 40 ms holds. Before, the same callers waited
about as long for connection setup, and most then failed.
- Steady state: 20 concurrent holders under the limit are never queued
(`queued: 0`).
- Nested acquisition: with a limit of 2, two holders each take a nested
connection while a third caller is queued. All finish (no deadlock); the
queued caller runs after them.
- In-transaction caller: with a limit of 1, a pooled transaction holds a
row lock and then asks for a dedicated connection, while the one
dedicated holder waits for that row lock. The caller is admitted at once
and both finish. A hard limit would deadlock here.
- Resource holder: with a limit of 1, the slot holder waits for a
resource that a second caller holds. The second caller, with
`admitWithoutWaiting`, is admitted at once and both finish.
- Wait bound: with a limit of 1 and a 100 ms bound, a queued caller is
admitted over the limit after the bound.
- Budget: the sum of the actual `max` of every private pool equals
`databaseConnectionBudget().privatePools`, and the default total is 73.
- `skill-package-store.test.ts` ("does not load queued legacy bodies for
files reads"): 32 readers queue for package I/O while only the 2 active
readers hold dedicated connections, and the 33rd reader still gets
"busy" at once. With the old order and a limit of 24, CI showed only 22
readers reaching that queue. `skill-package-store`,
`skill-version-cache`, `company-skills-empty-runtime`,
`runtime-skill-preservation`, `company-skills-runtime-concurrency` and
`company-skills-service` pass locally. Two `company-skills.test.ts`
workspace-discovery tests fail locally on master too (same result with
this change), so they are not caused by it.
- `packages/db`: client, lazy-transaction, prepared-query,
response-capture, work-signal and dedicated-pool suites, 134 passed.
- `server`: skill-version-cache, status-card-evaluation-work,
git-workspace-expiration, execution-workspace-runtime-control-conflict,
agent-instruction-working-copies, company-skills-empty-runtime and
decisions-service suites: 413 passed. In one combined run under machine
load, git-workspace-expiration failed at file level with all its tests
passing; alone it passed 96 of 96.
- `npx tsc --noEmit` in `packages/db`: passes.

## Risks

- A caller that holds an application-level lock (for example a
filesystem lock) and then waits in the queue can block a dedicated
holder that needs that lock. The 30 s wait bound ends such a cycle with
a delay. The over-limit counter shows it.
- Nested and in-transaction callers do not wait, so the in-use count can
reach twice the limit with one level of nesting. The documented budget
counts this.
- In a large restart burst, run starts now queue for a slot instead of
opening new connections. Steady state (about 10 to 20 dedicated
connections at 1,000 runs) stays under the default limit of 24, so a
run's first turn does not wait there.
- The checkpoint pool (3 connections) can limit throughput at 1,000 runs
on a hosted database. This pull request does not change it; it needs an
owner decision on the hosted connection budget.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:33:14 -06:00
DottaandClaude Fable 5.2 d6acbf50e6 refactor: cut DATABASE.md to setup and schema-change rules (#21700)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DATABASE.md` before a schema or migration change
> - That file grew to 2,611 lines. About 1,950 of them were notes on the
tables of one feature, added one pull request at a time
> - The setup steps and the migration rules started at line 792, below
those notes
> - This pull request cuts `doc/DATABASE.md` to 377 lines: setup, and
the rules that apply to every schema change
> - The benefit is a guide that a person or an agent can read in full
before a schema change

## Linked Issues or Issue Description

Refs #21670 (the same cleanup for `doc/DEVELOPING.md`).

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DATABASE.md`

**What's wrong?**

The file was 2,611 lines. It opened with a Better Auth upgrade note and
492 lines on task query performance. The schema change workflow started
at line 1367.

**Suggested fix**

Keep database setup and the general rules in `doc/DATABASE.md`. Remove
the notes on individual tables. The schema and its tests describe those.

## What Changed

- `doc/DATABASE.md` is now 377 lines. It keeps the three database modes,
client tuning, the schema change workflow, migration compatibility, the
migration authoring checklist, migration snapshots, the rule for retries
after a lost connection, transaction-aware work signals, plugin
namespaces, and backups. The wording of the kept text is unchanged.
- New `doc/database-client-internals.md` holds the response-size
diagnostic, the notes on the postgres.js patch, and the transaction
memory note. The text is moved without wording changes.
- "Experimental remote workspace snapshots" moved to
`doc/workspace-storage.md`, which links to it.
- "Execution admission accounting" moved to
`server/src/modules/README.md`, which links to it.
- Deleted about 1,950 lines of notes on one feature each. No other file
linked to them. The deleted sections are: Better Auth 1.7.16 upgrade,
Durable native provider backups, Workspace cohort admission proofs,
Original local Git sources for archived workspaces, Proactive agent chat
messages, Persistent case handoffs, Task query performance, Tool profile
reads, Execution identity row locks, Remote workspace row locks, Managed
adapter settings, Cloud configuration migration receipt, Cloud runtime
identity singleton, Unclaimed Cloud warm standby, Native finalization
history and idle sleep, Feedback history and idle sleep, Resource
membership tables, Decision training snapshot retention, Decision queues
and triage provenance, Native runner persistence, Chat communication
snapshots, Telegram private draft identities, Attachment upload
provenance, Question-response delivery receipts, Secret storage,
Persistent agent conversations, Resource lifecycle events, Legacy
controller ownership, Agent file persistence and legacy revisions, Large
API response snapshots, Runner API response reservations, Internal agent
commentary, Agent identity keys and backups, Slack app registration,
Slack manager workspace grants, Installed skill packages, Project
repository skill synchronization, Task workspace records, Execution
recovery query performance, Agent runtime synchronization journal,
Retired Pipelines storage, Prompt injection classification, Deferred
conversation requests, GitHub inherited review-wait reads, Sandbox
creation ownership, Object-backed run evidence, Cloud runtime
configuration singleton, Independent accounting and budget evaluation,
Standard workspace and Classic opt-in, Project environment rolling
compatibility.
- Also deleted from kept sections: the subscription-table note and the
issue-reference backfill note in the embedded mode section, the HTTP
access log note and the per-endpoint retry notes in the connection loss
section, and the notes on migrations `0355`, `0313`, and `0314` in the
authoring checklist.
- `AGENTS.md` now states what `doc/DATABASE.md` is for.

## Verification

- A script built the new files by line range from the old file. It
asserts the old line count and the first line of each range.
- A link check on every changed Markdown file, and on the four files
that link to a `DATABASE.md` anchor, found 0 missing files and 0 missing
anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- The deleted notes were not all read in full before deletion. The
decision was by section: a section about one feature's tables, with no
inbound link, was deleted. The text stays in git history.
- Two deleted sections describe procedures for an operator: the
duplicate-account check before the Better Auth 1.7.16 migration, and the
secret storage tables. Restore them into a focused doc if operators
still need them.
- No runtime behavior changes.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 20:01:28 -06:00
DottaandClaude Fable 5.2 ac114d9479 refactor: cut DEVELOPING.md to a setup and workflow guide (#21670)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DEVELOPING.md` for setup, commands, testing, and worktrees
> - That file grew to 5,495 lines. Most of it was notes on how one
subsystem behaves, added one pull request at a time
> - Readers could not find the commands. Agents also read behavior notes
as rules
> - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and
workflow guide. It moves the subsystem notes to focused docs
> - The benefit is a guide that a person or an agent can read in full,
and a clear place for each kind of note

## Linked Issues or Issue Description

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DEVELOPING.md`

**What's wrong?**

The file was 5,495 lines. It opened with 112 lines about case handoffs
before it reached Prerequisites. Unrelated sections sat below the
"OpenClaw Docker UI One-Command Script" heading. Statements about
current behavior read like policy.

**Suggested fix**

Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each
subsystem note to a doc for that subsystem. Add an index.

## What Changed

- `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health
checks, local run, test drive, Docker, instance layout, database,
storage, secrets, worktrees, test commands, Storybook, CLI, and an index
of the other docs.
- 15 new docs hold the moved text without wording changes:
`ci-workflows`, `test-infrastructure`, `worktrees`,
`hot-restart-deploys`, `heartbeat-internals`, `runner-internals`,
`workspace-repositories`, `workspace-storage`,
`skills-and-teams-catalog`, `ui-internals`, `case-handoffs`,
`chat-connector-development`, `instance-configuration`,
`integration-api-notes`, `database-backups`.
- "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`.
The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`.
- Deleted about 280 lines: the story-by-story Storybook notes, "Agent
Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only
sections, "Workspace feature checks (2026-10-09)", and "Chat and task
navigation caching". The test-drive build-lock paragraph is now four
lines.
- Repointed every inbound link and every script or workflow comment that
named a moved section.
- `AGENTS.md` now says where a behavior note goes, so the guide does not
grow again.

## Verification

- A script split the file by line range. It asserts that each of the
5,495 source lines is kept, moved, or dropped exactly one time.
- A link check on every changed Markdown file found 0 missing files and
0 missing anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `node --check` passes on the three scripts with changed comments.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- Low risk. The change is documentation and comments only.
- A link to a moved section from outside this repository will land on
the top of `doc/DEVELOPING.md`.
- The new docs are not in the audited prose list in
`.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An
edit to `doc/DEVELOPING.md` does not. This PR does not change that list.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 18:02:27 -06:00
DottaandClaude Opus 5.5 d25667f370 perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.

The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.

**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.

**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.

## What Changed

- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:

  ```diff
       remove: (x) => {
  -      const index = xs.indexOf(x)
  -      return index === -1
  +      const at = xs.indexOf(x, index)
  +      return at === -1
           ? null
  -        : (xs.splice(index, 1), x)
  +        : (xs.splice(at, 1), x)
       },
  ...
         if (index === xs.length) {
           index = 0
           xs = []
  +      } else if (index > 1024 && index * 2 > xs.length) {
  +        xs = xs.slice(index)
  +        index = 0
         } else {
           xs[index - 1] = undefined
         }
  ```

The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:

  | Measurement | Before | After |
  | --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |

## Risks

- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:46:41 -06:00
DottaandClaude Opus 5.5 f452b449fd fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:29:23 -06:00
DottaandClaude Opus 5.5 3c3e6c2d75 fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").

**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.

**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.

**Paperclip version or commit**
`master` at 6dd0349d52

**Additional context**
The local reproduction showed the same four statements at equal call
counts, as on the hosted instance (see Verification).

## What Changed

- `pinned()` returns the first reference that holds the version (current
pointer, importing parent, operation, unfinished child, run, or recovery
action) and its ID, not a boolean.
- The sweep stores that reference in `metadata.retentionPin` when it
pins a version. It writes again only when the reference changes or a
deadline must be cleared.
- The candidate query leaves out a version with no deadline while its
recorded reference still holds. Each pin condition is one SQL builder.
`pinned()` and the candidate query use the same builder, so the
exclusion can never hold a version that `pinned()` would release.
- When the recorded reference releases, the version comes back, and
`pinned()` checks every reference again. It records the next reference
that holds, or schedules or retires the version.
- The record is removed when the sweep schedules or claims a version.
- The worker coalesces wakes for up to one second
(`REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS`). Retention deadlines are days,
so this delay has no effect on retirement.
- `doc/DATABASE.md` describes the record.
- No schema migration. The record is in the existing `metadata` JSONB
column.

## Verification

Red/green regression tests in
`server/src/services/remote-workspace-snapshot-retention.test.ts`:
- `leaves %s-pinned history out of later passes, across restarts, until
that pin is released`. This runs once for each of the six pin kinds. It
pins the version and checks that a new service instance (a restart)
examines zero rows. Then it releases the pin and checks that the version
is deleted.
- `re-examines a pinned snapshot when its recorded pin is released but
another still holds it`.
- `revisits a failed save's recorded pin once its abandoned upload is
settled`.
- `does not re-examine unchanged pinned history on execution
notifications`. This uses the coordinator and `notifyExecutionRunWork`.

On the parent commit, the first eight new tests fail (6 pin kinds, the
second-pin case, and the notification case). With the fix, all 31 tests
pass.

Related suites pass locally (212 tests):
`remote-workspaces.integration.test.ts`,
`remote-workspace-snapshot-reuse.test.ts`,
`remote-workspace-cohort-migration.test.ts`,
`snapshot-attempt-upload-abort.test.ts`, and
`remote-workspace-snapshot-retention.test.ts`. `tsc --noEmit -p
server/tsconfig.json` is clean.

Load measurement. Local PostgreSQL 18 with `pg_stat_statements`. 300
pinned rows. About 13 run-work notifications each second for 60 seconds,
after the first pass finished:

| | Before | After |
| --- | --- | --- |
| Pinned rows examined again | 30,250 – 39,000 / min | 0 / min |
| Sweep statements | ~151,000 – 196,000 / min | 88 / min |
| All statements, including the test's own notification transactions |
153,327 – 197,205 / min | 1,647 / min |
| Sweep database execution time | ~1,285 ms / min | ~52 ms / min |
| Candidate query | 0.07 ms, ~10 / min | 0.88 ms, 58 / min |

Before: `begin`, the locked row read, the workspace read, the
importing-parent read and the operation read each ran 30,250 times in 60
seconds. This is the same signature as on the hosted instance. After:
the sweep ran 58 candidate queries and 30 deadline reads. With
per-notification wakes and no coalescing, the sweep ran 776 candidate
queries and 776 deadline reads (1,552 / min).

I did not verify the change on a hosted instance.

## Risks

- A recorded pin is checked against only its own reference. The check
uses the same SQL condition as `pinned()`, so a version cannot stay
excluded after that reference releases. The parametrized test covers
each pin kind.
- A version can be pinned by more than one reference. When the recorded
reference releases, the sweep checks all of them again and records the
next one.
- A producer that rewrites the row's metadata and drops the record
causes one extra examination. A producer that sets a deadline makes the
record inactive, because the exclusion applies only when `delete_after`
is null.
- Versions that have no record yet, which includes all existing pinned
rows, are examined once after deploy. The examination writes the record.
- The candidate query does one primary-key lookup for each pinned row.
It takes about 0.9 ms with 300 pinned rows, and the time grows linearly
with the number of pinned rows.
- Pinned history from failed migration cohorts stays pinned. This change
does not alter that hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:05:32 -06:00
DottaandClaude Opus 5.5 3ae7aba643 perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server uses `withDedicatedDbConnection` (packages/db) when an
action must hold one database session outside the main pool: advisory
transaction locks, lock keepalive, and `pg_settings` reads for workspace
lifecycle, finalization, source locks, skill caches and status cards
> - Each call built a new postgres.js client, opened a new physical
connection (TCP, TLS and startup through the pooler), built a new
drizzle client with the full schema, and then ended all of it
> - On staging the dedicated application name shows about 21 connection
setups per second, and the main thread is pinned (event-loop utilization
0.965)
> - This pull request keeps a small set of idle single-connection
clients per owner and reuses them
> - The benefit is that a dedicated action costs about 0.2 ms of
main-thread time instead of 4 to 8 ms, with the same exclusive-session
guarantee

## Linked Issues or Issue Description

**What existing behavior does this improve?**

`withDedicatedDbConnection` opens and closes one new database client and
one new physical connection for every call.

**Subsystem affected**

Database client (`packages/db`) and the server services that take
dedicated advisory-lock sessions.

**Current behavior**

Each call builds a new postgres.js client, opens a new connection (TCP,
TLS and startup through the pooler), and builds a new drizzle client
with the full schema. On staging (Neon statement statistics, 326 s
window) `SET application_name='paperclip-workspace-finalization-lock'`
ran 21.4 times per second. Only this factory uses that application name.
In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2%
inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that
is this per-call setup.

**Proposed behavior**

A caller still gets exclusive use of one session for the whole action.
The session and its drizzle client are reused by later actions, so the
per-call cost is the action's own queries, not a full connection setup.

**Reason and benefit**

The server main thread is saturated on staging (event-loop utilization
0.965). Reuse removes about 5 ms of main-thread work per call.

**Breaking changes**

None. The exported function keeps its signature.
`closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are
new exports.

## What Changed

- `packages/db/src/client.ts`: each `createDb` owner now has one set of
dedicated single-connection clients, shared with every private client
derived from that owner (so nested dedicated actions use the same set).
- An action takes an idle client (last in, first out) or opens a new
one. It never waits. Callers hold advisory locks and nest dedicated
actions, so a waiting queue could deadlock.
- On success the client returns to the idle set if fewer than
`DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed.
- When the action throws, the client is closed and never reused. This
covers a failed or cancelled transaction, a broken socket, and an
interrupted lock wait.
- When an action ends the client itself (the source-lock cancel path
calls `$client.end`), the client is never reused.
- Idle clients keep the owner's idle timeout and max lifetime, so
postgres.js still closes idle physical connections after
`DATABASE_IDLE_TIMEOUT_SECONDS`.
- New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle
set and stops retention. Ending the owner `$client` does the same. A
later action still works with a connection that closes on release.
- `server/src/index.ts`: shutdown closes the dedicated set next to the
other private pools.
- `doc/DATABASE.md`: documents the idle cap, the "not an admission
limit" rule, and the transaction-scoped session state contract.
- Session state audit: every current caller uses `pg_advisory_xact_lock`
/ `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller
uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or
temporary tables. A reset round trip per call is therefore not added.

## Verification

- New `packages/db/src/dedicated-connection-pool.test.ts` (embedded
PostgreSQL, 10 tests).
- Red on `origin/master`: 200 sequential calls used 200 distinct
sessions (`expected 200 to be 1`). All 10 tests failed.
- Green with this change, 3 runs in a row: 200 sequential calls use 1
session; 7 concurrent callers each hold a different session for the
whole transaction, and afterwards at most 4 stay open and later calls
reuse them; a failed transaction's connection is closed; a terminated
socket during the action is not reused; an idle connection terminated by
the server reconnects; a client the action ended is not reused;
transaction-local settings do not leak; shutdown and owner `$client.end`
close every retained connection; nested actions do not block.
- Focused suites pass: packages/db `work-signals`,
`prepared-queries-postgres`, `response-capture-postgres`,
`checkpoint-pool`, `client`, `client-options`,
`client-teardown-registry`, `postgres-connection-recovery`,
`client-disconnect-replay` (132 tests). Server suites that use dedicated
connections: `skill-package-store`, `status-card-evaluation-work`
(updated: it now proves advisory locks are released, since sessions are
reused), `agent-instruction-working-copies`,
`workspace-recovery-retry-copyback`, `native-execution-lease-batch`,
`company-skills-empty-runtime`, `decisions-service`,
`legacy-controller-lease`, `server-startup-feedback-export` (467 tests).
- `pnpm verify --explain` selects 2779 files for a
`packages/db/src/client.ts` change. Per AGENTS.md I ran the focused
checks above and let CI give the broad coverage.
- Benchmark: 200 calls of `withDedicatedDbConnection(db, d =>
d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))`
against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating
before/after process pairs per mode. Main-thread time is
`eventLoopUtilization().active`.

| Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call
(median, range) |
| --- | --- | --- | --- |
| TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) |
| TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) |
| Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) |
| Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) |

At the staging rate of about 21 calls per second, this removes roughly
100 ms of main-thread work per second (TLS figures; staging CPUs may be
slower).

## Risks

- A retained connection that the server or network drops while idle:
postgres.js reconnects on the next query when it has seen the close
(tested). If the drop and the reuse race, the first query of that action
fails like a main-pool query would. Before this change, a dedicated
action always started on a fresh connection.
- Through a transaction-mode pooler, `SET application_name=...` is a
server-side parameter switch, so its count may not drop as much as the
client-side connection count. Measure the deployed effect with pooler
client connections and main-thread CPU, not only that statement.
- Concurrency above 4 opens extra connections exactly as before; they
close on release. Total retained connections per owner grow by at most 4
idle clients, which still close after the idle timeout.
- An action that sets session-level state would now leak it to the next
action. The contract is documented on `withDedicatedDbConnection` and in
`doc/DATABASE.md`; no current caller does this.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:37:04 -06:00
DottaandPaperclip 2b1d0fc825 fix(connectors): keep result classification alive during run progress (#20763)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - App connectors screen returned content before they give it to an
agent.
> - The model screen checks run identity and reserves its budget in one
transaction.
> - That transaction took a repeatable-read snapshot before it locked
the run.
> - A run event could change the row first and cause all four attempts
to fail.
> - This pull request uses READ COMMITTED for this admission under the
existing locks.
> - The benefit is that connector results can pass screening while run
progress continues.

## Linked Issues or Issue Description

Refs #20385 and #20484. These changes fixed other run accounting paths.
They did not change classifier admission.

**What happened?**

A connector read can return the accounting contention error before the
model screen starts. A real gateway test reproduces SQLSTATE `40001` at
`prompt-injection-prevention.ts` when it reads `heartbeat_runs FOR
SHARE`. Each attempt commits a run event after the transaction starts
its snapshot.

**Expected behavior**

Return screened connector content and save its classifier charge while
run events continue. Refuse paid classification if the run stops, its
identity changes, or access or budget checks fail.

**Steps to reproduce**

Run this test with the production source from `9574065a28`:

`cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts -t 'returns a screened
connector result'`

The test fails after four `40001` errors. It passes with this change.

## What Changed

- Use READ COMMITTED only for prompt-injection admission. Keep company,
project, configuration, agent, run and secret share locks.
- Keep classifier settlement at repeatable read. Keep provider calls
outside transaction retries.
- Test screened results, stored charges and settled holds under run
progress. Test four concurrent reads and refusal after a run stops or
changes identity.
- Document the transaction boundary.

### Hypothesis and thesis for a fix

**Hypothesis:** the classifier admission snapshot becomes stale before
its run share lock. The gateway regression confirms this cause in
current source. The historical connector error alone does not identify
its SQL statement.

**Thesis for a fix:** read current run state under the existing locks at
READ COMMITTED. This removes stale snapshots caused by run events. The
change belongs to classifier admission because every screened connector
result uses this service.

Investigation source: [Braintrust
pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/d5369c32-4c0f-43a1-af94-a01fef0a697b).
Access is required to read the traces.

**Proof:** baseline `9574065a28` failed with `40001` after four
attempts. The same single-read regression passed after the change. The
full focused suites then passed, including four concurrent reads, stored
charges and refusal controls.

## Verification

- Real PostgreSQL gateway regression: baseline red in 28.07 seconds;
candidate green in 23.16 seconds.
- `cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts
src/__tests__/prompt-injection-prevention.test.ts`: 54 tests passed.
After sync with master `1ac8b6a711`, all 54 passed again in 34.46
seconds.
- The tests use the real gateway, content guards, classifier service and
database. The MCP server and paid model are simulated. This is not a
live deployment test.
- Existing controls cover company boundaries, budgets, credentials,
configuration changes, interruption and receipt recovery.
- Staging health reported `16024ffa2`. Its classifier and accounting
source matches the reproduced baseline and includes #20385 and #20484.
The revision at the earlier failure times remains unknown.
- `pnpm verify` selected 2,512 of 3,303 test files because its
dependency policy retained 2,510 files conservatively. The broad local
run was stopped after the selection became clear. It is not a pass. The
focused 54-test result above is the local proof; required CI supplies
broad coverage. An earlier selection was also invalidated by a
concurrent commit.
- Full preflight could not finish because this host has no Cargo. Pinned
dependencies and shared/plugin SDK builds completed. The focused tests
passed without Rust.
- [Greptile reviewed this head at
5/5](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790238737).
No inline thread is open. [Paperclip completed its assessment at 5/5
with no
findings](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790312753).
[Required CI passed for head
`643bbe51faa70dcb17f372b9e85e3cbbcb724b9d`](https://github.com/paperclipai/paperclip/actions/runs/42981473514),
including typecheck, build, 12 server test shards, nine serialized
suites, eight E2E shards and Runner checks. Snyk and Socket passed. The
full inventory contains 58 completed check runs; the neutral Paperclip
check is a completed 5/5 assessment, not an incomplete review.

## Risks

- READ COMMITTED uses a new snapshot for each statement. Existing
admission locks still protect the run status, identity, configuration
and secret checks. Budget checks and reservation writes still commit
together.
- Concurrent independent admissions retain existing budget behavior.
This change adds no retry of paid provider work.
- No schema, API or saved receipt format changes.
- Historical failing SQL is unavailable. The source regression proves
this failure path; it does not prove that every historical example has
the same cause.
- The change is unmerged and not deployed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment model ID and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:37 -06:00
DottaandClaude Opus 5.5 2de4787f41 fix(db): release a postgres.js connection's receive buffer once it is fully read (#21432)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - A hosted server keeps about 0.5 GB of `external` memory (Buffers)
for its whole life, with 340 open database connections
> - postgres.js keeps a zero-length view of the buffer of each
connection's last read. The view keeps the whole buffer alive until that
connection reads again
> - An idle connection therefore pins the socket chunk or the whole
buffer of its largest recent message, which can be one MB or more
> - This pull request replaces a fully read buffer with an empty one in
the driver patch
> - The benefit is that idle connections hold no receive buffers. At 340
connections that each read a 1 MB row, external memory drops from about
320 MB to 0

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server holds 455 to 520 MB of `external` memory for
its whole life. It has about 340 open database connections. Each idle
postgres.js connection keeps the buffer of its last read.

**Steps to reproduce**
1. Open 80 or 340 pool connections.
2. Read a 1 MB row on each connection.
3. Drop the results, let the connections go idle, and force garbage
collection.
4. Measure `process.memoryUsage().external`. Each idle connection still
holds about 0.95 MB.

**Expected behavior**
An idle connection holds no buffer from a result that it has completely
read.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18.

## What Changed

- `patches/postgres@3.4.9.patch`: in `data()` of `src/connection.js` and
`cjs/src/connection.js`, the driver now replaces the receive buffer with
an empty buffer when it has handled every message in it. A partly read
buffer is not changed. The other hunks of the patch are unchanged. I
regenerated the patch with `pnpm patch postgres@3.4.9`, edited the two
files, and ran `pnpm patch-commit`. I did not commit the lockfile change
from that command.
- `packages/db/src/postgres-receive-buffer.test.ts` and
`packages/db/src/__fixtures__/postgres-receive-buffer.mjs`: a regression
test for ESM and CommonJS on real PostgreSQL.
- It checks that results are reassembled correctly for a 3 MB row, for
161 row sizes around one 64 KB socket chunk (to land message boundaries
at the end of a chunk), and for 20,000 small rows that arrive many to a
chunk.
- It records a weak reference to each large receive buffer and checks
that none of them stays alive after the connection becomes idle.
- `packages/db/scripts/benchmark-postgres-receive-buffer.ts`: one
command that reports heap, external and rss per connection, before and
after large rows.
- `doc/DATABASE.md`: documents the new patch behavior, the test and the
benchmark.

## Verification

- Red/green: with the new line removed from the installed driver, the
fixture fails for ESM and CommonJS ("an idle connection must not keep
the receive buffer of its last result"). With the line in place it
passes, repeated twice for each format.
- `pnpm --filter @paperclipai/db exec vitest run
src/postgres-receive-buffer.test.ts
src/postgres-connection-recovery.test.ts`: 20 passed.
- `pnpm --filter @paperclipai/db exec vitest run src/backup-lib.test.ts
src/backup-lib-binary.test.ts src/client-disconnect-replay.test.ts
src/client-teardown-registry.test.ts src/work-signals.test.ts`: 36
passed. In 1 of 4 runs, the JavaScript backup test (COPY and cursor
reads) timed out on a machine with load average 20 to 34. It took 39 s
and 25 s with the change and 21 s without it in passing runs. Read
throughput is the same with and without the change: a 200,000-row select
plus a COPY TO STDOUT has a median of 319 and 348 ms with it, and 329 ms
without it.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/prepared-statements-driver.test.ts
src/__tests__/prepared-statements.test.ts`: 5 passed.
- `pnpm --filter @paperclipai/db typecheck`: passed.
- Benchmark: `node --expose-gc --import tsx
packages/db/scripts/benchmark-postgres-receive-buffer.ts` with a 1 MB
row on each connection, numbers after garbage collection:

| Connections | External per idle connection, before | After | External
held in total, before | After |
| --- | --- | --- | --- | --- |
| 80 | 917 KB | 1 KB | 72 MB | 0 MB |
| 340 | 960 KB | 0 KB | 319 MB | 0 MB |

The rss per connection also went down (80 connections: 1,555 to 1,375
KB; 340 connections: 1,308 to 714 KB). The allocator keeps some freed
memory, so rss changes less than external memory.

## Risks

- Low risk. The change runs only when the buffer has no unread bytes. A
message split across chunks keeps the same reassembly path. The test
covers split messages, exact chunk boundaries and many messages in one
chunk.
- This saves external memory (Buffers outside the JavaScript heap). It
does not reduce `heapUsed`. It does not fix the out-of-memory crashes of
the hosted server by itself.
- The patch must be removed or ported when Paperclip upgrades
postgres.js. The test fails if a driver version keeps the buffer again.

## Model Used

- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), used in
Claude Code with tool use (shell, file editing, local PostgreSQL), heap
snapshots and allocation measurement.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:38:32 -06:00
DottaandClaude Opus 5.5 49f0030f06 fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:17:57 -06:00
Nicky LeachandClaude Opus 5.5 717aae060a fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute on environment leases, and `createOwnedSandbox`
records each sandbox creation so that a lost reply never creates a
second sandbox
> - The record key is company, environment, run and request key. The
plugin request key is the environment fingerprint, so it is the same for
every attempt of one run
> - A same-run recovery can need a new environment after the first lease
is closed with successful cleanup and a termination receipt
> - The old record is still `attached` to the closed lease, so the
recovery fails with "Sandbox creation has already transferred ownership"
before the provider is called
> - This pull request lets the creation fence accept the same stop proof
that recovery already requires. A creation record whose sandbox is
proven stopped becomes resolved, and a resolved record no longer holds
its key
> - The benefit is that same-run recovery gets one replacement sandbox,
while unconfirmed cleanup still blocks a second sandbox

## Linked Issues or Issue Description

Fixes #21214

Refs #20755. That pull request fixes the same failure with a different
rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when
the recorded lease has ended, and it accepts any ended status, including
`failed`, without a termination receipt. A lease whose cleanup is not
confirmed can then produce a second sandbox while the first one can
still run. This pull request keeps the original key and requires the
same stop proof that Core uses elsewhere
(`hasRemoteTerminationReceipt`). It does not add a key generation.

## What Changed

- `server/src/services/sandbox-creation-operations.ts`: when
`createOwnedSandbox` finds the key held by an `attached` operation whose
lease is not active, it checks the same stop proof that same-run
recovery requires before it acquires:
`hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for
the sandbox that this operation created (`creationProviderLeaseId`, or
the lease's `providerLeaseId`).
- With that proof, it runs the same admission transaction one more time.
That transaction first marks the old operation `resolved` (`WHERE id AND
state = 'attached' AND lease_id`), and then inserts the new operation.
  - Without that proof, it throws exactly as before.
- The check covers every source of a stop receipt: `releaseLease`, the
Runner `complete` operation (which writes receipts with direct SQL for
reusable leases), stop-and-retain settlement, and the receipt that
`resolveRemoteTerminationLease` derives for an expired reusable lease.
- A task replacement can leave an earlier operation attached to a lease
that now holds another sandbox. That operation is not proven and keeps
its key.
- Concurrent recoveries serialize on the row update. The second one
updates nothing, finds the first one's unresolved operation through the
partial index, and replays or refuses as before. There is one provider
call.
- Migration `0475_sandbox_creation_owner_unresolved.sql` recreates
`sandbox_creation_operations_owner_idx` under the same name as a partial
unique index (`state <> 'resolved'`). The drop and the create are in one
migration transaction. The Drizzle schema matches.
- The conflict lookup in `createOwnedSandbox` ignores resolved rows. An
active attached lease is still replayed. Pending and
`needs_reconciliation` rows still refuse. `releaseLease` does not
change.
- `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the
operation lifecycle.

## Verification

- `pnpm --filter @paperclipai/db generate`
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`
- `vitest run src/migration-snapshot-drift.test.ts` (db)
- `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48.
With the service change reverted, the 5 replacement tests fail.
- The 9 other server test files that read `sandbox_creation_operations`,
plus `environment-runtime.test.ts`: 564/564
- `pnpm check:migrations` passes

New tests in `sandbox-creation-operations.test.ts`:

- A stop receipt from `releaseLease` (sandbox driver, and the plugin
driver through `acquireRunLease` in the #21214 sequence), from the
`complete` operation's direct write on a retained reusable lease, and
derived only from the destroyed successor of an expired reusable lease
each let the next creation resolve the old operation and create exactly
one replacement. A retry replays that replacement. These tests fail
before the fix.
- Failed cleanup, success without a receipt, a receipt for another lease
or another provider task, a retained lease, and pending cleanup each
still throw "transferred ownership" with no provider call.
- An earlier task's operation that a task replacement left attached
keeps its key.
- Two concurrent recoveries make one provider call. This test fails
before the fix.
- Reapplying migration 0418 keeps the partial index.

I did not run `pnpm verify` or the full suite locally. CI runs them.

## Risks

- Migration: the drop and the rebuild of the index take an exclusive
lock on `sandbox_creation_operations`. The table has one small row for
each sandbox creation, so the lock is short.
- Rolling deploy: the migration is marked `online_compatible`. The
previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also
matches a partial unique index. Its conflict lookup has no state filter,
so it can find a resolved row and throw. That fails closed and never
creates a second sandbox.
- A legacy runner-task operation without `creationProviderLeaseId`
matches any lease it is attached to. Runner-task operations record this
field before they attach, so this is only a theoretical gap.
- An operation stays `attached` until the next creation with the same
key checks it. This is the same as before when no recovery comes.
- Stop-and-retain settlement is not tested directly. The shared receipt
check covers it.
- The fence is unchanged for every case without a stop receipt.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:24:34 -08:00
DottaandClaude Fable 5.2 31c3412bbb perf(db): build tx.query on first read instead of for every transaction (#21148)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every server module reaches PostgreSQL through one client,
`createDb` in `packages/db`. The server has about 1,260 `transaction(`
call sites. Run events, runner checkpoints and the run queue use a
transaction for each write.
> - Drizzle builds one relational query builder for every schema table
when it creates a database object. It creates a database object for each
transaction and for each savepoint.
> - The Paperclip schema has 321 tables. One `BEGIN` therefore allocates
about 100 KB and uses about 80 microseconds of CPU before the first
statement. One `SAVEPOINT` costs the same again. Five places read the
relational API inside a transaction.
> - On a hosted instance the server main thread was near one full core,
and garbage collection used a quarter of it. A fixed 100 KB for each
transaction is garbage that no caller asked for, and it grows with every
new table.
> - This pull request makes `createDb` build `tx.query` the first time a
transaction reads it.
> - The benefit is 82% less allocation and 7 times fewer garbage
collections for an empty transaction, and 45% less allocation for one
native run event. Transaction behavior does not change.

## Linked Issues or Issue Description

Related: #21131 (one transaction of two statements for a native run
event). That pull request reported this cost as a follow-up. No issue
exists for it.

**What existing behavior does this improve?**
The cost of `db.transaction()` and of a nested `tx.transaction()`
(savepoint) on the client that `createDb` returns.

**Subsystem affected**
Database client (`packages/db`).

**Current behavior**
`createDb` calls `drizzle(sql, { schema })`. Drizzle's `PgDatabase`
constructor then runs this loop for the root client, for every
transaction and for every savepoint:

```js
for (const [tableName, columns] of Object.entries(this._.schema)) {
  this.query[tableName] = new RelationalQueryBuilder(...);
}
```

A transaction is a `PgTransaction`, which extends `PgDatabase`
(`drizzle-orm/pg-core/session.js`). `PostgresJsSession.transaction` and
`PostgresJsTransaction.transaction` construct one for each `BEGIN` and
each `SAVEPOINT` (`drizzle-orm/postgres-js/session.js`).

Measured on `master` at 4e932fd178, Node 26, local PostgreSQL, 3,000
iterations, median of 5 passes:

| Scenario | Allocated | CPU | Collections per 1,000 |
| --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 27 µs | 0.3 |
| Empty transaction | 125.8 KB | 131 µs | 5.0 |
| Transaction with one statement | 136.6 KB | 153 µs | 5.3 |
| Transaction with an empty savepoint | 231.6 KB | 235 µs | 9.7 |
| Empty transaction, Drizzle without a schema | 18.5 KB | 53 µs | 0.7 |

The allocation profile of an empty transaction attributes 96.8 KB of
115.6 KB sampled to the `PgDatabase` constructor (`pg-core/db.js:17`),
reached from `PgTransaction` and `postgres-js/session.js:109`. About 50
KB is the builders and the `query` object. About 46 KB is
`Object.entries` of the schema.

The relational API is in use, always as a single-table `findFirst({
where[, columns] })`. It has 22 call sites. Five of them run on a
transaction: `services/announcements.ts:19` and `:25`,
`services/decision-training.ts:331`, and
`services/resource-memberships.ts:361` and `:369` (the service is built
on a transaction in `updateAgent` and in
`services/primary-agent.ts:60`). The other 17 run on the root client.

**Proposed behavior**
A transaction or savepoint that does not read `tx.query` builds no
relational query builder. A transaction that reads `tx.query` gets the
same builders as before, bound to that transaction.

**Reason and benefit**
Less garbage and less CPU on the server main thread for every write
path. Hosted statement statistics showed 13.4 native run events per
second and 5.3 checkpoint inserts per second. Each is one transaction,
so those two paths alone discarded about 2 MB per second. The total
transaction rate of the hosted instance was not measured.

**Breaking changes**
None. No schema change, no API change, and the `Db` type is identical.

## What Changed

- `packages/db/src/lazy-transaction-queries.ts` (new):
`drizzleWithLazyTransactionQueries(client, schema)`.
- It creates the root client with the schema, as before. `db.query` and
`db._` do not change.
- It creates a second Drizzle client on the same postgres.js client,
without the schema, and `db.transaction` opens transactions through it.
`BEGIN`, `SAVEPOINT`, commit, rollback and connection reservation are
still Drizzle's and the driver's own code. No Drizzle internals are
patched.
- Each transaction and savepoint gets a `query` accessor. The first read
builds one `RelationalQueryBuilder` for each table from the root
client's schema description and the transaction's session, then stores
the result as a plain property. `tx._.schema`, `tx._.fullSchema` and
`tx._.tableNamesMap` keep their values.
- `packages/db/src/client.ts`: `createDbClient` calls the new function.
The work-signal and response-capture wrappers install on top of it in
the same order as before.
- `packages/db/scripts/benchmark-transaction-allocation.ts` (new): the
benchmark in this description.
- `doc/DATABASE.md`: new section "Relational queries inside
transactions".

## Verification

Benchmark, same script before and after (`createDb` client):

```sh
cd packages/db
TSX_DISABLE_CACHE=1 node --expose-gc --import tsx scripts/benchmark-transaction-allocation.ts 3000 5
```

| Scenario | Allocated before | Allocated after | CPU before | CPU after
| Collections per 1,000 before | after |
| --- | --- | --- | --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 11.3 KB | 27 µs | 31 µs |
0.3 | 0.3 |
| Empty transaction | 125.8 KB | 22.4 KB | 131 µs | 54 µs | 5.0 | 0.7 |
| Transaction with one statement | 136.6 KB | 35.6 KB | 153 µs | 82 µs |
5.3 | 1.0 |
| Transaction with an empty savepoint | 231.6 KB | 35.6 KB | 235 µs |
111 µs | 9.7 | 1.3 |
| Transaction with a savepoint of one statement | 242.7 KB | 44.8 KB |
331 µs | 217 µs | 10.7 | 2.0 |

After the change, no Drizzle frame is in the allocation profile of an
empty transaction (19.8 KB sampled). The script also runs Drizzle with
the schema and Drizzle without a schema in the same process, so the
comparison can be repeated on any later revision.

The machine was shared and under load (load average 6 to 35 on 18 CPUs).
Allocation and collection counts were stable across four runs. CPU
values moved by a factor of two between runs. The CPU columns come from
the two runs with the lowest load.

One native run event, through the real append path:

```sh
TSX_DISABLE_CACHE=1 EVENTS=1500 PASSES=3 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-run-event-append.ts
```

`native-stream`: 220.4 -> 120.9 KB allocated for each event, 12.0 -> 5.3
collection pauses per 1,000 events. `native-burst`: 221.1 -> 121.0 KB,
10.6 -> 5.3 pauses. Statements for each event do not change (4.6).

Tests, all on real PostgreSQL
(`packages/db/src/lazy-transaction-queries.test.ts`, 15 tests):

- Red and green for the mechanism. On `master` these fail, and with the
change they pass:
- `builds none for a transaction or savepoint that never reads
tx.query`: 321 builders for the transaction and 321 for the savepoint on
`master`, 0 with the change.
- `allocates the same for an empty transaction as a client without the
schema`: 101 KB more than a schema-less client on `master`, below the 16
KB limit with the change. It reads V8's allocation counter, not a timer,
and it skips on a Node.js without that counter.
- Semantics. These pass on `master` and with the change, which shows the
behavior is the same:
  - `commits and returns the callback's value`
- `rolls back and rejects with the thrown error` (thrown error,
`tx.rollback()`, SQL error)
- `keeps the outer transaction usable after a savepoint rolls back`
(three levels)
- `discards a released savepoint when the outer transaction rolls back`
  - `applies the isolation level and access mode`
- `runs concurrent transactions on separate reserved connections and
releases them` (12 transactions on a pool of 3)
- `returns the only connection to the pool after a commit, a rollback
and a failed savepoint` (pool of 1)
  - `behaves the same with prepared statements enabled`
- Instrumentation. These pass on `master` and with the change:
- `registers each transaction and savepoint with the client's work
signals`: the transaction and the savepoint have the client's
work-signal scope, the registration statement runs on the transaction's
own connection, listeners get `intent` at once and `settled` only after
the outer commit, and a rollback reconciles. When transactions bypass
the work-signal wrapper, this test and five tests in
`work-signals.test.ts` fail.
- `response capture counts every statement of a transaction, a savepoint
and tx.query`: six statements, counted for each operation and table,
including two relational reads.
- Every `tx.query` pattern in the codebase:
- `reads tx.query through the transaction, as the announcement and
decision-training services do`
  - `reads tx.query inside a savepoint and follows its rollback`
- `gives tx.query one builder per schema table, the same object on every
read`

Commands run locally on the final revision:

- `pnpm exec vitest run src/lazy-transaction-queries.test.ts
src/work-signals.test.ts src/response-capture-postgres.test.ts` in
`packages/db` on the final revision: 43 passed.
`prepared-queries-postgres.test.ts` passed on the first commit.
- Before the rebase onto current `master`, also
`response-capture.test.ts`, `response-capture-bundle.test.ts`,
`client-options.test.ts`, `client-teardown-registry.test.ts`,
`checkpoint-pool.test.ts`, `client-disconnect-replay.test.ts`: 111
passed in 10 files.
- On the first commit, server callers of `tx.query` and `db.query`, and
hot transaction paths: `announcements-routes`, `decision-training`,
`heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`,
`primary-agent`, `resource-memberships-routes`, `workspace-preferences`,
`sidebar-preferences-routes`, `paperclip-control-plane-port`,
`native-runner-checkpoint-authority.integration`, `dispatch-work`,
`decision-target-work`, `issue-thread-interactions-service`: all passed.
`workspace-preferences` timed out once in its database start hook while
eight suites ran in parallel, and passed alone.
- `pnpm exec tsc --noEmit` in `packages/db`: passed. A type probe showed
that `Db` is the same type as before.
- `pnpm check:module-boundaries`: passed.
- `pnpm verify` exits 2: `packages/db` is a broad input. CI supplies the
full suites. I did not run the full local suite, repo-wide typecheck or
build.

Not verified: the hosted transaction rate, and the effect on a hosted
instance. CI uses Node 24, which has no V8 allocation counter. On CI the
allocation test skips itself, and the builder-count test covers the
mechanism. The benchmark needs Node.js 25 or later and stops with that
message on Node 24. I did not run that error path, because this machine
has Node 26 only.

## Risks

- Every module uses this client. The mitigation is that the transaction
path is still Drizzle's own `transaction()` on the same postgres.js
client. Only the schema argument is absent. The semantics tests pass
unchanged on `master` and on this branch.
- The change relies on Drizzle surface that is not documented API. The
lazy `tx.query` calls the `RelationalQueryBuilder` constructor from
`drizzle-orm/pg-core/query-builders/query`. That subpath is in Drizzle's
export map and the constructor is typed, but `drizzle-orm/pg-core` does
not re-export it. The change also writes the three schema fields of
`tx._`, which the types mark `readonly`. In `drizzle-orm@1.0.0-beta.22`
the same path exports a different class. An upgrade that changes the
constructor fails the typecheck and the `tx.query` tests. An upgrade
that renames `db._.schema` makes the function return the plain eager
client: behavior stays correct, the saving is lost, and the
builder-count test fails. The repository owner decides whether this
dependency on Drizzle is acceptable. A version that uses only documented
API removes the 22 relational reads from the server and creates the
client without the schema.
- `tx.query` is now an accessor until its first read. Code that reads
the property descriptor of `query` sees the difference. No code in the
repository does that.
- The second client and the builders use Drizzle's default dialect
options, as the root client does. If `createDb` ever passes `casing`,
`logger` or `cache` to Drizzle, the same option must go to all three.
The source comment says so.
- A transaction that reads `tx.query` pays the old cost once, on first
read.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use
and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 14:31:21 -06:00
DottaandClaude Fable 5.2 5b698b68cf perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database
connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update,
insert, count/max read, COMMIT). The query builder built five of them
again for every event, and each event created two transaction objects.

A native source event now takes the run lock and then runs one statement
that finds a replayed event or reserves the sequence and inserts, and
reads the retained count and maximum for the acknowledgement. The lock
stays a statement of its own so the append reads a snapshot taken after
the lock was granted. The control-plane port passes its controller check
to the append instead of opening an outer transaction, so there is no
SAVEPOINT. The hot statements are fixed SQL text.

Events that wake a queue still register the wake before the write and
only when the event is new. Order, sequence numbers and durability do
not change: every event still commits before its append returns.

Local benchmark (server/scripts/benchmark-run-event-append.ts), one
native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated,
1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events.
One ordinary event: 105 -> 27 KB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:58:31 -06:00
DottaandClaude Fable 5.2 e8e548e856 feat(db): classify migrations for application under the running release (#20913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time

## Linked Issues or Issue Description

**Subsystem affected**

Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.

**Problem or motivation**

Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.

**Proposed solution**

Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.

**Alternatives considered**

- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.

**Roadmap alignment**

This is release infrastructure. It adds no product feature.

## What Changed

- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".

Published shape:

```json
"migrationCompatibility": {
  "version": 1,
  "migrations": [
    {
      "idx": 468,
      "tag": "0468_busy_wendell_rand",
      "sqlSha256": "56a6e272…",
      "compatibility": "online_compatible",
      "classified": "online_compatible",
      "override": false,
      "reasons": [
        { "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
      ]
    }
  ]
}
```

`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).

### Distribution over the existing migrations

No existing migration is rewritten or annotated.

| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |

Migrations per blocking reason (one migration can have several):

| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.

## Risks

- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.

## Model Used

- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 14:14:40 -06:00
DottaandPaperclip cda23f08ac fix(db): preserve legacy project reads during rolling upgrades (#20860)
## Thinking Path

> - Paperclip controls agent work while web servers change versions.
> - The old and new servers can share a database during that change.
> - Older servers still read and write the project environment column.
> - Migration 0466 removes that column before those servers stop
serving.
> - A later repair alone leaves a committed missing-column window and
loses values.
> - This pull request applies the exact drop and repair in one
transaction.
> - Old reads and saved values survive while current placement behavior
stays unchanged.

## Linked Issues or Issue Description

Refs #20835 and #20852.

**What happened?**

Migration 0466 drops `projects.environment_id`. The previous server
still selects that column and reads it during admission. Applying a
separate repair afterward leaves a window in which those requests fail.
The drop also removes saved values.

**Expected behavior**

A database that has not yet applied 0466 must keep the column and its
values visible to concurrent old servers throughout the upgrade. Both
published migration entries must be recorded truthfully. Current project
responses must keep the new workspace placement contract.

**Steps to reproduce**

1. Use a database before migration 0466 with nonempty project
environment values.
2. Keep an old project writer transaction open and start the pending
migrator.
3. Start an old project read, then commit the writer.
4. Without this repair, the saved values disappear. With this repair,
the reader sees the preserved values and both migrations commit
together.

## What Changed

- Add migration 0467 to restore the legacy nullable UUID column and its
original foreign key. Leave the published 0466 SQL unchanged.
- Apply only the exact SHA-bound 0466/0467 pair in one transaction. Lock
the project table, retain nonempty values in a temporary table, execute
both real SQL files, restore values, and record both real journal
entries before commit.
- Apply the additive 0467 repair normally when 0466 was already
committed.
- Keep a deprecated database-only field and omit it from current project
API responses. Current placement logic stays unchanged.
- Add PostgreSQL regressions for concurrent old reads and writes,
idempotence, rollback, and an already-applied drop. Document the overlap
contract and retain the five newest generated snapshots.

## Verification

- Node 24 cached agent preflight passed.
- `pnpm exec vitest run
packages/db/src/project-environment-overlap-migration.test.ts
server/src/__tests__/project-environments.test.ts -t 'project
environment rolling migration|saves storage, not'`: 4 passed in 32.57
seconds against real disposable PostgreSQL.
- Before the owner fix, the regression had two failures: saved values
disappeared and the restore-failure case committed. The fixed case
proves that both DDL and journal entries roll back together.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/db check:migrations`: passed.
- `git diff --check`: passed. Migration 0466 bytes are unchanged. The
generated snapshot set is 0463 through 0467.
- Review found a raw DELETE response that still exposed the
database-only field. The delete projection now omits it; a real
authorized HTTP DELETE passed with successful cleanup. The concurrency
test now observes the old reader waiting inside PostgreSQL before
releasing the old writer. All three migration cases passed with that
barrier. Earlier HTTP fixture setup/cleanup failures remain recorded as
failed attempts; they were not product-regression passes.
- Candidate `047178538c0df3a9e382259a51635a6e46da5646` has fresh
Greptile 5/5 with no open findings. [Required final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/43062354612)
passed, including build, typecheck, migration checks, all test shards
and the aggregate verification gate. The separate Superagent security
scan ended inconclusive/neutral; it did not produce a passed scan.
Deployed acceptance remains pending. No hosted database or app
deployment was changed by this branch.

## Risks

The pair takes a table lock for its transaction. Operators must retain
their normal lock and statement timeouts. Both SQL files and their exact
hashes must match before the pair runs. A failed restore rolls back the
drop and both journal entries.

If 0466 was already committed, 0467 restores the column and foreign key.
It cannot recover values that were already deleted by the earlier
release. This repair does not claim to recover those values.

## Model Used

OpenAI Codex (GPT-6), with tool-assisted implementation and real
PostgreSQL qualification. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 11:08:30 -06:00
DottaandClaude Fable 5.2 464e1de9b1 perf(issues): decide task reads for a list in one batch (#20858)
The task routes that filter several rows for one actor (summaries, lists, blockers, relation edges and the chat bootstrap) asked for one authorization decision per task, all at the same time. Each decision sent a task-row read and a copy of the task-privacy statement, which is more than 1 KB of SQL, so a request for 50 summaries sent 127 to 279 statements on every pool connection at once.

The routes now ask the authorization service for all decisions in one batched read. It reads the task rows and applies the same privacy predicate to all of them in one statement, and reads membership, agent and run policy once. Within a batch, the authorizing user's membership is read once, and the boundary checks of a low-trust agent, which stay per task, run at most eight at a time.

A request for 50 summaries now sends 22 to 28 statements (65 for a low-trust agent). Tests compare the batch with one check per task for 24 actors.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 10:42:21 -06:00
DottaandClaude Fable 5.2 8ab955776d perf(github): check the requester provenance of an issue list's review waits in one batch (#20851)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can review a GitHub pull request. After the agent publishes
the review, the task waits in review for the next GitHub reply. A later
run of the same task can inherit this wait.
> - Issue lists, issue summaries and inbox attention show these waits.
For each waiting task, the server first checks the requester provenance
of the waiting run (`chatRunSourceMessageId`). This check is an
authorization check: it walks the identity-context lineage of the run
and rejects a changed requester, another task, another company or
another provider.
> - #20673 batched the review reads and the publication reads of one
list request. It kept the provenance check as one call for each waiting
run, inside `Promise.all`.
> - Thus a request with N waits still built and sent at least 2N
statements at the same moment. The database is fast for each statement,
but the server builds each statement on its main thread.
> - On a hosted instance, a CPU profile of the server showed the main
thread at 96% of one core. SQL text building, per-statement timers,
socket writes and garbage collection had the largest shares. Connection
samples showed 235 to 292 pool connections whose last statement was the
source-message link read of this check, and 53 to 112 whose last
statement was its identity-context read.
> - This pull request checks the provenance of all the waiting runs of
one request in one batch. The runs that need the same kind of read share
one statement. Each run gets the same decision as a separate call, from
the same walk and the same reads.
> - The benefit is that a list request with 250 waits sends 17
statements, not 658, and uses about one quarter of the CPU time. The
statement count no longer follows the number of waits.

## Linked Issues or Issue Description

Refs #20673 (batched the review reads and the publication reads of the
same request; this change does the same for the provenance reads that
#20673 kept for each run).
Refs #20551 (authorizes the waits of one issue page with one read).

**What happened**

`readInheritedGitHubReviewWaits` called `chatRunSourceMessageId` one
time for each waiting run, inside `Promise.all`. Each call sent an
identity-context read and a source-message link read. A run that
continues an earlier run also sent a parent-context read and a
source-run read for each continuation in its lineage.
`listExternalReviewWaits` calls this function for the issue list, the
issue summaries and the inbox attention of a company, so each of those
requests repeated the same reads for every waiting task.

Evidence from a hosted instance (sanitized):

- CPU profile of the server process, 30 s, 5 ms samples: main thread at
96% of one core. Building SQL text for select statements: 17.7%
inclusive. The idle-timer restart that the database client does for each
statement: 7.6%. Socket writes: 6.2%. Garbage collection: 13%.
- Three samples of the pool connections (about 494 connections). 235 to
292 connections had the source-message link read of
`chatRunSourceMessageId` as their last statement. 53 to 112 connections
had its identity-context read as their last statement.
- The `DISTINCT ON` review read that #20673 added ran about 29 times
each minute. It runs one time for each list request, so each request
started hundreds of provenance reads.

**Expected behavior**

One list request sends a number of provenance statements that does not
follow the number of waiting tasks. The decision for each run does not
change.

**Steps to reproduce**

1. Put many tasks of one company in review, each with an inherited
GitHub review wait.
2. Open the issue list or the inbox.
3. Count the statements that the server sends. The identity-context read
and the source-message link read run one time for each waiting task.

The benchmark below does these steps on a local PostgreSQL.

## What Changed

- `server/src/services/chat-channels/run-origin.ts`:
- The provenance decision is now one walk (`sourceMessageWalk`). The
walk asks for one read at a time and decides on the returned rows only.
Its checks, their order and its error messages are the checks of the
previous `chatRunSourceMessageId`.
- The five reads of the walk (active identity context, source-message
link, board message, parent context, source run) are defined one time in
`provenanceReads`. Each definition takes its values either as literals
or as SQL columns.
- `chatRunSourceMessageId` (single run) drives the walk with one
statement for each read, as before. The joins, the company scope and the
filters are the same. The identity-context reads now select the seven
columns that the walk uses instead of every column.
- New `chatRunSourceMessageIds(db, runs, options)` (batch). It drives
one walk for each run. In each round, the runs that wait for the same
kind of read share one statement: `unnest(...)` of the request values,
`CROSS JOIN LATERAL` the same single-row read with the request columns
in place of the literals. Thus each run gets the row that its own single
read returns, with `LIMIT 1` and the company scope on each join. The
result for each run is `{ messageId }` or `{ forbidden }` (the 403 that
the single call throws). A database error rejects the whole batch, as
`Promise.all` did.
- `server/src/services/chat-channels/github/review-wait.ts`:
`readInheritedGitHubReviewWaits` checks the provenance of its runs with
the batch. A 403 still gives `revoked`. The single-run
`readInheritedGitHubReviewWait` does not change.
- `server/src/services/chat-channels/run-origin.test.ts`: two new tests
(see Verification).
- `server/src/__tests__/github-review-wait.test.ts`: one new regression
test for the statement count.
- `server/scripts/benchmark-review-wait-provenance.ts`: the reproduction
and the before/after measurement. It loads the pre-fix modules from
`--baseline-ref` and fails unless both versions return the same waits.
- `scripts/benchmark-github-review-wait.ts`: removed an assertion that
`run-origin.ts` is byte-identical to its baseline. Both sides of that
benchmark import the current helper, so the comparison stays fair.
Without this change, the documented command of that benchmark fails.
- `doc/DATABASE.md`: describes the batch provenance check and the new
benchmark command.

Statement count of one batch: the walk of one run does at most 3 + 4L
reads, where L is the number of continuations in its requester lineage.
The batch does the same number of rounds for its longest lineage. Each
round sends at most one statement for each kind of read (5 kinds), for
any number of runs. A request whose waiting runs accepted their GitHub
comment directly sends 2 provenance statements.

## Verification

Red/green regression test
(`server/src/__tests__/github-review-wait.test.ts`):

- `reads the waits of forty runs with the statements of four runs`.
Before the fix: `expected { few: 10, many: 82 } to deeply equal { few:
4, many: 4 }`. After the fix: passes.

Equivalence tests
(`server/src/services/chat-channels/run-origin.test.ts`), on real
PostgreSQL:

- `gives every run of a batch the decision of its own check, in every
provenance state`. 33 runs, each in a different provenance state, are
checked with the single function and with the batch for five option sets
(no options, legacy wake allowed, GitHub, GitHub with legacy wake,
Slack). The test compares the decision of each run (message, `null`, or
the 403 and its message) for: the full batch, the reversed batch, and a
batch of one run. The states include the negative cases: a run of
another company, a message that is linked only in another company, a
message that is linked to another task, a conversation that moved to
another task, a continuation on another task, a changed requester (run,
identity context, continuation, board message), a link of another
provider, a missing link, an outbound link only, a deleted message, an
agent message, a rejected or pending identity context, a missing parent,
another agent and a lineage cycle. The test also states the expected
GitHub decision for each state, so the comparison cannot pass on two
wrong answers.
- `checks the runs of a batch with statements that follow the lineage,
not the number of runs`. 4 runs and 40 runs each send 2 statements. 40
separate checks send 80. One continuation in each lineage gives 4
statements for the batch.
- A mutation check: when the batch gives a row to the wrong request,
both tests fail.

Commands:

- `pnpm vitest run server/src/services/chat-channels/run-origin.test.ts
server/src/__tests__/github-review-wait.test.ts`
- Focused run of 15 test files that import the changed modules, their
callers or the shared seed helper, including
`external-chat-wait.integration.test.ts`, `chat-github-tools.test.ts`,
`slack-authority.test.ts` and
`issue-list-review-wait-authorization.integration.test.ts`: 482 tests
passed. In `idle-completed-history.test.ts`, `afterEach` hooks (a
`TRUNCATE` of the test database) timed out at 30 s for 1 test in this
run and for 3 tests in a second run of the file, while other agents
loaded the machine. No assertion failed. The three tests of that file
that reach the changed code pass when they run alone. CI runs the full
file.
- `server/src/__tests__/chat-channels.integration.test.ts` also uses the
single check. It was not run locally (very large file). CI runs it.
- Typecheck of the changed files and of the callers of
`chatRunSourceMessageId` with a focused `tsc` program: no errors in
these files. The full server typecheck was not run locally (shared
machine). CI supplies it.
- The first CI run of this pull request failed `ci / policy`: the module
boundary check rejects an insert into the agents table outside the agent
lifecycle module, and the benchmark seeded its fixture agent that way.
The second commit seeds that agent with SQL. `pnpm
check:module-boundaries` passes locally, and the benchmark gives the
same statement counts.
- `pnpm verify` returns exit 2 (broader verification required) because
the change adds a file under `server/scripts/`. CI supplies the broad
coverage.

Before/after measurement:

```sh
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-wait-provenance.ts --baseline-ref=e2cf9a670fc5d5f5058fd0972beb71650f77c554
```

One call of `listExternalReviewWaits` (the read of one issue list
request). Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after
seeding. One company, one agent, one GitHub repository. Every wait has a
published review. Every tenth waiting run continues three earlier runs.
3 warm-up calls, then 15 pairs in alternating order. Both versions
return the same waits and the same inherited wait for each run.

| Waits | Version | Statements | SQL text bytes | Wall ms, median
(min–max) | CPU ms, median (min–max) |
| --- | --- | --- | --- | --- | --- |
| 20 | before | 60 | 31,084 | 14.1 (12.8–19.6) | 13.7 (8.7–39.3) |
| 20 | after | 17 | 15,847 | 12.5 (9.4–22.7) | 8.8 (4.1–87.1) |
| 250 | before | 658 | 300,611 | 63.3 (58.1–89.6) | 78.1 (64.8–103.3) |
| 250 | after | 17 | 17,171 | 20.8 (18.9–29.2) | 18.7 (14.1–27.3) |
| 1000 | before | 2,608 | 1,179,710 | 256.0 (221.0–363.2) | 303.3
(262.2–507.8) |
| 1000 | after | 17 | 21,695 | 54.9 (46.7–81.7) | 51.0 (41.6–72.2) |

Of the 658 statements before the fix at 250 waits, 650 are provenance
reads: 325 identity-context reads, 250 source-message link reads and 75
source-run reads. After the fix, 9 statements do the same reads. The
other 8 statements of the request do not change. Sizes are set with
`WAITS=20` and `WAITS=1000`. Other agents used the same machine during
the measurement, so the maximum values have noise.

- The batch statements use index scans only
(`run_identity_contexts_pkey`, `run_identity_contexts_company_run_idx`,
`chat_message_links_company_comment_idx`, `chat_conversations_pkey`,
`chat_endpoints_company_id_uq`, `heartbeat_runs_pkey`). The 9 batch
statements of a request with 250 waits take about 1 ms of server
execution in total.
- Wall time and CPU time are almost equal before the fix. This shows
that the cost is in the server process, not in the database.

Not verified:

- The effect on the hosted instance. This pull request does not deploy.
The share of the main-thread CPU that this path uses there is not known;
the profile gives the share of all statement building.

## Risks

- The provenance check is an authorization check. The risk is a batch
decision that differs from the single decision. Mitigation: both paths
run the same walk and build each read from the same definition. The
batch read is the single read inside `CROSS JOIN LATERAL`, for each
request row. The equivalence test compares both paths in every state.
- A database error (for example an invalid UUID in a legacy task
binding) rejects the whole batch. The previous `Promise.all` also
rejected the whole request. The callers of the batch pass the native
task of each run.
- The statement count of the batch follows the longest lineage of its
runs. A batch with one very long lineage sends as many statements as
that run sent before, in sequence. It does not send more.
- The batch sends its statements in sequence on one connection. Before,
the reads of N runs used up to N pool connections at the same moment.
- The single-run callers (GitHub tools, review publication, response
comments, Slack authority, the finalizer, the status committer) keep one
statement for each read. Their identity-context reads select fewer
columns.
- No migration. No new index.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 10:19:29 -06:00
DottaandPaperclip 81c9143fe4 fix: upgrade Better Auth to 1.7.16 with account migration (#18906)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The board uses Better Auth for credential accounts and sessions.
> - The server pins Better Auth 1.7.2.
> - The November security release fixes 12 issues in Better Auth.
> - The new version stops writing the old required account issuer field.
> - This change updates the packages and the database contract together.
> - New users can sign up, and existing users can still sign in.

## Linked Issues or Issue Description

**What happened?**
The server uses Better Auth 1.7.2. The [security
release](https://better-auth.com/blog/security-release-2026-11-03) fixes
12 issues in 1.7.16. Earlier update PRs #17909 and #17911 target 1.7.7,
which is below this security release.

**Expected behavior**
Use the patched stable release. Keep new and existing credential
accounts usable.

**Steps to reproduce**
Read the server dependency pin. Upgrade it without a schema change.
Better Auth then rejects the required account issuer column. See the
[upgrade guide](https://better-auth.com/docs/guides/1-7-upgrade-guide).

## What Changed

- Pin Better Auth to 1.7.16. The repository refresh bot owns the
committed Core lockfile.
- Make the old issuer column nullable. Keep existing issuer values.
- Replace the issuer index with a unique provider/account index.
- Generate the migration and retain the five newest schema snapshots.
- Test new signup, new-account sign-in, legacy issuer sign-in, duplicate
rejection, and separate provider identities with real PostgreSQL.
- Update the database schema test. Test the historical issuer backfill
separately. Verify upgrades over existing rows and rollback on duplicate
provider/account keys.
- Document the duplicate check and migration order.
- Add the approved two-line dependency resolution step before the backup
job’s frozen install. Leave the frozen check and backup tests intact.

## Verification

- Core head: `af70282c7e24335b8688c8f3cc88fdf65045632c`; parent
`0f1481e4462100357a7caa496b875cf0bc500d95` includes master `84fc67317`.
- Nine changed files. The only workflow edit is the approved two-line
backup dependency resolution step; no managed lockfile edits.
- Migration `0465_better_auth_account_identity.sql` is three SQL
statements; snapshot changes only `public.account`, five newest
snapshots retained.
- DB typecheck and 23 focused tests passed on the parent. Both real
PostgreSQL migration/signup suites were rerun November 11: all 4 tests
passed. Application source and tests are unchanged in this workflow-only
commit.
- The parent’s sandbox-provider lane passed all 205 file-sync tests but
reported an unhandled EPIPE during the symlink rejection case; its
aggregate verify therefore failed. No unrelated source change was made.
The new head reruns the full applicable CI.
- Reproduced the old backup install failure locally. Resolving manifests
followed by a frozen install passed locally; restored the bot-owned
committed lock afterward. Hosted backup qualification remains required.
- Current-head CI and review are running; no final-head green claim or
merge yet.

## Risks

Apply the migration before the new server starts. Check for duplicate
provider/account pairs first. Duplicate rows stop index creation. Keep
separate users separate when resolving duplicates. The migration keeps
issuer data. Do not restore its NOT NULL constraint while new rows
contain null. No deployment is included.

## Model Used

OpenAI GPT-6 Codex, with reasoning and tool use. The runtime did not
expose a more specific model ID or context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 08:26:58 -06:00
DottaandClaude Opus 5.5 3bf7af6c77 perf(plugins): read the lifecycle inbox from per-resource cursors (#20576)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins get agent, project and agent runtime changes from the
`resource_lifecycle_events` journal. Each plugin acknowledges events one
resource at a time.
> - #20405 made the inbox page read linear in the company journal size.
The journal has no retention, so each page still reads every event of
the company to find the unacknowledged ones: about 1.7 s per 1M events.
> - The delivery worker reads a page again after each acknowledgment.
The idle-sleep check also reads the whole journal when all events are
acknowledged. `acknowledge` checks every older event of the resource.
> - On a hosted instance, the page read ran 181 times in one window at
about 1 s each.
> - This pull request adds a per-plugin, per-resource cursor. Reads
start at the cursor, so they do not read acknowledged history again.
> - The benefit is that page, idle-check and acknowledgment cost follow
the number of resources and pending events, not the journal size.
Delivery order and results stay the same.

## Linked Issues or Issue Description

Refs #20405

**What happened?**
The plugin lifecycle inbox page query (`select ... from
"resource_lifecycle_events" ... DISTINCT ON ...`) ran 181 times at about
1 s each on a hosted database in one window. Each acknowledgment causes
another full read of the company journal. The idle-sleep check
(`hasPending`) reads the full journal when all events are acknowledged.
`acknowledge` reads the full history of the resource.

**Expected behavior**
A page read, the idle-sleep check and an acknowledgment must not read
acknowledged history again. Their cost must not grow with the journal
size.

**Steps to reproduce**
1. Give a company 400 agents with 2,500 `agent_runtime` events each (1M
events).
2. Acknowledge the first 2,000 events of each agent for one plugin.
3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100,
undefined, ["agent", "project", "agent_runtime"])`.
4. The read takes about 1.7 s. `hasPending` takes about 240 ms.

**Paperclip version or commit**
`8d2bf78323` (master).

**Deployment mode**
Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18.

## What Changed

- `packages/db/src/schema/plugin_lifecycle_resource_cursors.ts`: new
table `plugin_lifecycle_resource_cursors`. It has one row per plugin,
company, resource type and resource id. `acknowledged_through_id` means:
every event of that resource at or below this id is acknowledged by the
plugin. Plugin and company deletion cascade.
- Migration `0462_plugin_lifecycle_resource_cursors`: creates the table
and carries existing acknowledgments forward (see Risks for the
previous-release story).
- `server/src/services/plugin-lifecycle-inbox.ts`:
- `list` and `hasPending` use a backward loose index scan on
`resource_lifecycle_events_resource_order_idx`. It visits each resource
stream once and gets its latest event id. A stream whose latest event is
at or below its cursor is settled and is not read again. For other
streams, the read gets one head: the unacknowledged creation, else the
first unacknowledged transition above the cursor. Then it applies
`afterId`, id order and the limit, as before.
- `acknowledge` reads only the events between the cursor and the
acknowledged event. It still writes the per-event
`plugin_lifecycle_acknowledgments` row. It moves the cursor only over a
contiguous acknowledged prefix. A creation recorded after a pending
transition does not move the cursor.
- An event counts as acknowledged when it is at or below the cursor, or
when it has an acknowledgment row. Thus rows written by an earlier
release, and gaps in old history, keep their current behavior.
- `server/src/__tests__/resource-lifecycle-events.test.ts`: new database
tests for the migration carry-forward, gaps, late creations, cursor
reads, the rollback rows and the plugin-delete cascade. All tests from
#20405 still pass without change.
- `scripts/benchmark-plugin-lifecycle-inbox.ts`: runs the migration
backfill on the fixture and times it. Adds an acknowledgment case. Runs
`VACUUM ANALYZE`, because hosted tables are autovacuumed.
- `doc/DATABASE.md`: describes the cursor, its read cost and the
migration.
- The delivery worker (`plugin-lifecycle-delivery.ts`) is not changed.

## Verification

Thesis: the cost is a read of the whole company journal (page, idle
check) or the whole resource history (acknowledgment). If a per-resource
cursor marks the acknowledged prefix, each read can skip that prefix.
Then the cost follows the number of resource streams and pending events.
The rows must stay equal.

Local SQL timings: embedded PostgreSQL 18.1 on Apple Silicon, synthetic
data, warm cache, one warmup, 7 samples, baseline and treatment order
alternate per sample. The machine was shared with other jobs (load
average about 35), so spread is wide. Values are median (min to max) in
ms. Rows are equal (`assert.deepEqual`) for every list and probe case.

Fixture A: 400 agents x 2,500 runtime events, 2,000 acknowledged per
agent. 1,020,440 events, 800,420 acknowledgments.

| Case | Before | After | Speedup |
| --- | --- | --- | --- |
| page of 100, all types | 1,700 (1,633-2,438) | 11.5 (9.5-12.2) | 148x
|
| page of 100, agent and project | 3.3 (2.5-5.4) | 4.2 (3.6-6.7) | 0.8x
(slower) |
| idle-sleep check | 238 (174-367) | 2.7 (2.1-3.4) | 88x |
| acknowledge one head | 15.3 (9.4-30.1) | 20.3 (12.9-29.9) | neutral
(noise) |
| migration backfill | - | 9.3 s | - |

Fixture B: 40 agents x 25,000 runtime events, 24,000 acknowledged per
agent (long history). 1,020,080 events.

| Case | Before | After | Speedup |
| --- | --- | --- | --- |
| page of 100, all types | 1,943 (1,642-4,979) | 4.4 (2.9-5.6) | 444x |
| page of 100, agent and project | 3.5 (2.1-5.7) | 4.6 (2.8-12.0) | 0.8x
(slower) |
| idle-sleep check | 234 (220-243) | 2.8 (1.9-6.5) | 85x |
| acknowledge one head | 64.6 (48.5-71.6) | 5.0 (2.6-7.1) | 13x |
| migration backfill | - | 4.6 s | - |

Fixture C: 5,000 agents x 200 runtime events, 190 acknowledged per agent
(many resources). 1,025,040 events.

| Case | Before | After | Speedup |
| --- | --- | --- | --- |
| page of 100, all types | 1,867 (1,649-1,975) | 76.9 (66.2-497) | 24x |
| page of 100, agent and project | 10.3 (8.7-12.4) | 21.6 (20.2-23.2) |
0.5x (slower) |
| idle-sleep check | 220 (201-298) | 2.8 (2.0-5.3) | 78x |
| acknowledge one head | 4.9 (3.4-8.6) | 6.0 (5.2-7.6) | neutral |
| migration backfill | - | 15.6 s | - |

`EXPLAIN (ANALYZE, BUFFERS)` for the all-types page on fixture A: before
2,488 ms, 13,597 shared hits plus temp files; after 7.0 ms, 7,926 shared
hits.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=400 --per-agent=2500 --acked=2000 --samples=7
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=40 --per-agent=25000 --acked=24000 --samples=7
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=5000 --per-agent=200 --acked=190 --samples=7
cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/project-instance-secrets.test.ts src/__tests__/agent-runtime-hooks.test.ts
pnpm --filter @paperclipai/db check:migrations
```

- `resource-lifecycle-events.test.ts`,
`plugin-lifecycle-delivery.test.ts` and
`project-instance-secrets.test.ts`: 86 passed.
`agent-runtime-hooks.test.ts`: 22 passed.
- `tsc --noEmit -p server`: pass. `check:migrations`: pass.

## Risks

- **Schema change.** Migration `0462` adds one table and two foreign
keys to `plugins` and `companies`. It adds no index on an existing
table. The backfill reads the acknowledgment table once. It took 4.6 s
to 15.6 s on 1M-event fixtures with 0.8M to 1M acknowledgments.
- **Migration order.** The number `0462` is reserved after open PRs that
use `0459` to `0461` (#20571, #20568, #20573). Merge this PR after them.
Before merge, rebase onto master and chain the snapshot after the latest
migration.
- **Previous-release story.**
- Upgrade: the backfill stores, per plugin and resource, the end of the
contiguous acknowledged prefix. A cursor never passes an unacknowledged
event, so no event is lost. Acknowledged events above a gap keep their
rows, so no event is delivered again. A test runs the migration
statement on rows that only an earlier release wrote and checks that
`list` returns the same rows before and after.
- Rolling deploy: an earlier release that still runs writes only
acknowledgment rows. The new reads honor those rows.
- Rollback: the new code still writes every acknowledgment row, so an
earlier release sees the same progress.
- **Ordering assumption.** The cursor is safe because writers record a
resource's events while they hold that resource's row lock
(`paperclip_record_agent_runtime` locks the agent row; agent status and
project writers document the lock). Thus a later commit for the same
resource always has a higher id. A writer that breaks this contract
could commit an event below the cursor, and that event would not be
delivered. A lower id from a different resource is still delivered
(existing test).
- **Slower path.** A page that asks only for `agent` and `project`
streams is 1 to 11 ms slower. These streams are short, so the old full
read was cheap, and the new read visits each stream once. The cost is
bounded by the number of resources.
- **Many resources.** The page cost grows with the number of resource
streams of the requested types. At 5,000 agents a page is about 77 ms,
compared with about 1.9 s before.
- The hosted planner can choose a different plan. The hosted improvement
is not yet measured.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:58:32 -06:00
DottaandClaude Opus 5.5 8e4c5e81b9 perf(issues): read created-from tasks by index and inbox review waits by page (#20573)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task page shows "Created from this task". The inbox shows
blocked-inbox attention for a page of tasks.
> - On a hosted database, the created-from list ran about 4,000 times in
one afternoon at about 499 ms each. The task page sends it each time a
task opens.
> - Its filter was an `OR` of three provenance paths over the list's own
rows. PostgreSQL could not use an index for it. It read every task of
the company and tested each one, with the list's other per-row
conditions (such as the chat conversation state subquery).
> - The blocked-inbox attention map also read external review waits for
every in-review task of the company, not only for the tasks of the page.
> - This pull request reads created-from tasks from three index probes,
and reads review waits only for the in-review tasks that can change a
finding of the page.
> - The benefit is less database work on each task open and each inbox
page. The responses stay the same.

## Linked Issues or Issue Description

Refs #20400 (shares review-wait reads between the attention maps of one
request).
Refs #20551 (authorizes the review waits of a page with one read; this
change narrows the waits that the page asks for, and adds a test that
pins the batched read).
Refs #13241 (adds the created-from list).

**What happened**

- `GET /api/companies/:id/issues?createdFromIssueId=…` used
`createdFromIssueCondition`. Its filter was `origin_issue_id = source OR
(origin_issue_id IS NULL AND origin_run_id IN source runs) OR (… AND
EXISTS creation activity of a source run)`. The `OR` stops index use, so
the statement read every task of the company. The list's other
conditions, which include the chat conversation state subquery, ran for
each task before the task was excluded.
- `listIssueBlockedInboxAttentionMap` read review waits for every
in-review task of the company graph. The caller uses the findings of the
page rows only.

**Expected behavior**

The created-from list reads only the created tasks. The blocked-inbox
map reads review waits only for in-review tasks that the liveness
classifier can read for a page row. The rows, the order, and the
attention values do not change.

**Steps to reproduce**

1. Use a company with tens of thousands of tasks, runs, and activity
rows.
2. Open a task page. Look at the created-from list statement: it reads
the full task set of the company.
3. Open an inbox page with blocked-inbox attention. Look at the
review-wait read: it gets the ID of every in-review task of the company.

## What Changed

- `server/src/services/issue-creation-origin.ts`:
`createdFromIssueCondition` is now `issues.id <> source AND issues.id IN
(three branches)`. Each branch starts from an index:
  - recorded origin task: `issues_company_origin_issue_idx`;
- legacy origin run: the source task's runs
(`heartbeat_runs_company_origin_issue_idx`), then the new partial index
on `issues (company_id, origin_run_id)`;
- creation activity: `activity_log_run_id_idx` for the source runs, then
the task by primary key.
The precedence does not change: a recorded origin task wins, the run
path applies only without one, and the activity path applies only
without both. The activity path matches only an `entity_id` that is the
canonical text of the task ID, as before (it also compares `id::text =
entity_id`).
- `packages/db`: add `issues_company_legacy_origin_run_idx` on `issues
(company_id, origin_run_id) WHERE origin_issue_id IS NULL AND
origin_run_id IS NOT NULL`. Migration `0461` creates it with `CREATE
INDEX CONCURRENTLY IF NOT EXISTS` and is registered with the other
concurrent migrations. `doc/DATABASE.md` documents the index and the
benchmark command.
- `server/src/services/issues.ts`: add `blockedInboxReviewWaitIssueIds`.
It returns the in-review tasks in the blocker closure of the page rows.
The blocked-inbox map reads review waits only for these tasks. The
liveness classifier reads a wait only for a row or for a blocker that it
reaches from a row, and the caller reads only the findings of the rows.
The classifier does not change.
- Tests:
- `issue-created-from-routes.test.ts`: new cases for precedence in both
directions, a source-run comment, an activity row whose task has another
origin run, a non-canonical and a non-UUID activity entity ID, the
source's own creation activity, and other-company rows.
- `blocked-inbox-review-wait-scope.test.ts`: the narrowed set for
chains, cycles, shared blockers and off-page tasks. The rows get the
same findings with the narrowed waits as with the waits of every
in-review task, and each narrowed wait is needed.
- `issue-list-review-wait-authorization.integration.test.ts`: on real
PostgreSQL with two authorized waits, `listExternalReviewWaits` and one
issue-list request with blocked-inbox attention each send one bound-run
message-link statement. With the per-wait authorization from before
#20551, both tests fail (2 statements).
- `scripts/benchmark-created-from-issues.ts`: local benchmark. It loads
the baseline `issues.ts` and `issue-creation-origin.ts` from a pinned
commit, asserts identical complete row payloads, and reports alternating
timings and `EXPLAIN (ANALYZE, BUFFERS)`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-created-from-routes.test.ts
src/__tests__/blocked-inbox-review-wait-scope.test.ts
src/__tests__/issue-list-review-wait-authorization.integration.test.ts
src/__tests__/agent-task-hierarchy-depth.test.ts
src/__tests__/issue-blocker-attention.test.ts
src/__tests__/issue-list-assignee-filter-routes.test.ts
src/__tests__/issues-parent-id-alias.test.ts
src/__tests__/batched-chat-authorization.integration.test.ts
src/__tests__/issue-liveness.test.ts` passes (126 tests). One timing
test in `issue-list-assignee-filter-routes` failed one time under
machine load and passed on rerun.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/db check:migrations` pass.
- `pnpm verify` asks for broader verification because the change adds a
file under `scripts/`. Required CI supplies that coverage.
- Benchmark: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-created-from-issues.ts --baseline-ref=bd30ef8510`

Fixture (synthetic, local PostgreSQL 18.1): one company with 30,000
tasks (10,628 with a recorded origin task, 10,445 with only a legacy
origin run), 60,030 runs with wide context snapshots (145 MB), and
261,079 activity rows; a second company with 10,000 tasks and 20,000
runs. The busy source task has 63 created tasks through all three paths,
plus decoys. The quiet source task has 3. The UI's page: `sortField=id`,
`asc`, `limit=500`, routine executions included. Both versions return
identical complete payloads. 9 alternating pairs after 2 warm-up pairs:

| Case | List statement before | List statement after | List call
before, median (min to max) | List call after, median (min to max) |
| --- | --- | --- | --- | --- |
| Busy source (63 tasks) | 234.9 ms, 47,437 buffers | 0.90 ms, 437
buffers | 156.0 ms (114.5 to 480.7) | 31.6 ms (19.9 to 62.7) |
| Quiet source (3 tasks) | 90.7 ms, 47,413 buffers | 0.25 ms, 46 buffers
| 126.1 ms (90.6 to 300.4) | 10.5 ms (7.2 to 26.1) |

The list call time includes the enrichment reads (labels, runs,
attention), which do not change.

Inbox page (50 open tasks, blocked-inbox attention): the review-wait
read goes from 2 statements with 1,215 parameters (every in-review task
of the company) to 1 statement with 31 parameters. The page time does
not change in this fixture (median 801 ms before, 873 ms after, both
with a wide spread), because the fixture has no review waits and the
other graph reads of the map are most of the time. With real waits, the
server does an inherited GitHub review read for each wait it finds, so
the gain grows with the number of off-page waits.

Migration timing (local PostgreSQL 18.1): `0461` (measured as the same
statement under its earlier number) on 500,000 tasks (846 MB, 20
companies, one third with a legacy origin run) takes 1.6 s. The index is
12 MB.

## Risks

- The new index adds a small write cost to tasks with only an origin
run. New tasks with a source task record `origin_issue_id`, so the index
holds mostly older tasks. The migration does not lock writes.
- The created-from rewrite must keep the same rows. The new tests and
the benchmark payload check cover precedence, company scope, and ID text
forms. Visibility and read filters are the list's other conditions and
do not change.
- The review-wait narrowing depends on the liveness classifier reading
waits only for rows and their blocker chains. A unit test compares the
findings of the rows with full and narrowed waits. A future classifier
change that reads waits of other tasks must update
`blockedInboxReviewWaitIssueIds`. The function comment says this.
- The local timings are synthetic. Hosted acceptance needs a fresh
query-statistics window after deployment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, local PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:38:51 -06:00
DottaandClaude Opus 5.5 1805a1d247 perf(workspaces): probe workspace run bindings through indexes (#20568)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep checks each Git workspace for recent
activity and for live consumers before it expires or moves the workspace
> - These checks find the runs of a workspace with one SQL scope,
`expirationScope`. The scope used one `OR` over all run bindings, and
one branch decoded the context JSON of each run
> - PostgreSQL could not use an index for that `OR`. Each check read
every run of the company and its large context from TOAST. On a hosted
database, each call took 2 to 5 seconds, about 5,200 s in total in one
window
> - This pull request splits the scope into one index probe per binding
and adds one concurrent expression index for the legacy
`executionWorkspaceId` context key
> - The benefit is that each check reads a few index entries instead of
all company runs. The cleanup decisions do not change

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The workspace activity and cleanup-safety checks:
`gitWorkspaceLastActivityAt`, `isGitWorkspaceExpirationEligible`,
`workspaceMoveEligible` and the orphan-folder consumer check. All of
them use `expirationScope` in
`server/src/services/git-workspace-expiration.ts`.

**Subsystem affected**
Execution workspace cleanup in the server
(`git-workspace-expiration.ts`, `isolated-repository-rehome.ts`,
`isolated-task-directory-retention.ts` through the shared scope) and the
`heartbeat_runs` schema in `packages/db`.

**Current behavior**
The `relevant_runs` CTE selects company runs where any of these is true:
`issue_id` or `native_issue_id` is in the issue tree or the bound
issues, the run is a bound issue's checkout or execution run, or the
context JSON has a matching `issueId`, `taskId` or
`executionWorkspaceId`. The planner does one sequential scan of the
company runs and reads each context from TOAST. The cost grows with the
run history of the company, not with the size of the workspace.

**Proposed behavior**
Each binding is a separate index probe. The `UNION` of run IDs goes to a
primary-key probe. The SQL gives the same rows.

**Reason and benefit**
Local SQL measurements show a 34x to 169x decrease in time. The decrease
grows with the run history (see Verification).

## What Changed

- `expirationScope` builds `relevant_runs` from a `UNION` of index
probes, one for each binding. The key sets go in as arrays (`=
ANY(ARRAY(...))`). With `IN (subquery)`, the planner chose a hash join
over a full scan, because it does not count the cost of reading TOASTed
contexts.
- The legacy context keys use `context_snapshot ->> 'key'`. This
expression matches the existing indexes
`heartbeat_runs_company_ctx_issue_created_idx` and
`heartbeat_runs_company_ctx_task_created_idx`. For a non-object context,
`->>` gives `NULL`, as the strict JSONPath decode did. Such a run can
still match through a native binding.
- New partial index `heartbeat_runs_company_ctx_execution_workspace_idx`
on `(company_id, context_snapshot ->> 'executionWorkspaceId') WHERE
context_snapshot ? 'executionWorkspaceId'`. The query repeats the
predicate. For an array context, `?` can be true, but `->>` gives
`NULL`, so the result does not change. The index is partial so that it
serves only this probe. A full index that starts with `company_id` made
the planner choose it for an unrelated company-scoped query. Migration
0460 builds it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The
migration is in the list of migrations that run outside a transaction.
- New tests: a legacy `issueId` or `executionWorkspaceId` binding on a
run that is bound to a different issue still blocks cleanup. A plan test
with `enable_seqscan = off` checks that no binding scans
`heartbeat_runs` and that the three context indexes are used.
- The benchmark script `scripts/benchmark-workspace-activity.ts` gets a
`--runs-per-workspace` option. Unrelated fixture runs now have distinct
context IDs, as real history does.
- `doc/DATABASE.md` describes the probes and the new index.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/git-workspace-activity.test.ts
src/__tests__/native-harness-backup-expiration.test.ts`: 132 of 132
tests pass. The second file checks that the native session index stays
the planner's choice for its query. These include the existing route
matrix (root, child, bound, native, context issue, context task, context
workspace, checkout, execution run), exactly-once counts, malformed and
uppercase contexts, company isolation, controller leases and lease
vetoes.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/db check:migrations` pass.
- Benchmark (local SQL measurement, PostgreSQL 18.1, synthetic fixture,
8 workspaces per sample, before and after alternate in each iteration).
The script compares the complete ordered caller results and the complete
`relevant_runs` multisets of the baseline and treatment. They are equal
in every run.

`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-workspace-activity.ts --baseline-ref=origin/master
--runs-per-workspace=<N>`

| Fixture | Check | Before median (min–max) | After median (min–max) |
Speedup |
  | --- | --- | --- | --- | --- |
| 4,800 runs, 330 MB | last activity | 523 ms (460–598) | 12 ms (8–15) |
44x |
| 4,800 runs, 330 MB | eligibility | 452 ms (432–473) | 13 ms (11–15) |
34x |
| 4,800 runs, 330 MB | idle eligibility | 465 ms (442–573) | 12 ms
(10–13) | 40x |
| 24,000 runs, 1.7 GB | last activity | 2,240 ms (2,214–2,454) | 13 ms
(11–14) | 169x |
| 24,000 runs, 1.7 GB | eligibility | 2,245 ms (2,097–2,377) | 17 ms
(14–19) | 133x |
| 24,000 runs, 1.7 GB | idle eligibility | 2,196 ms (2,179–2,381) | 15
ms (13–18) | 143x |

Before: one `Seq Scan` on `heartbeat_runs`. After:
`heartbeat_runs_company_issue_created_idx`,
`heartbeat_runs_company_native_issue_contract_id_uq`, the three context
indexes and `heartbeat_runs_pkey`.
- Index build time: on a synthetic table with 200,000 runs and 8 KiB
contexts (1.8 GB), the full form of this index took 3.7 s to build
concurrently and used 15 MB. The partial index reads the same rows and
holds only runs that have the key.

## Risks

- Cleanup safety: a wrong "allowed" result could delete an active
workspace. The scope keeps every binding of the old query. The benchmark
checks exact equality of the run sets, and the existing tests check each
binding route and each veto.
- Migration: the index build reads every run context once. It runs
concurrently and does not block writes. Until migration 0460 builds the
index, the new query is still correct but the `executionWorkspaceId`
probe scans.
- Plan choice: the array form makes index probes the cheapest plan for a
small tree. A very large issue tree gives a large array, but each
element is still one index probe.
- The sweep frequency does not change. This pull request reduces only
the cost of each check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:17:46 -06:00
DottaandClaude Opus 5.5 d5c1f48180 perf(native): index the backup lease lookup by scope digest (#20571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native harness backup expiration removes old session backups. Before
it removes a backup, it finds the environment leases that reference the
backup root
> - This lookup reads leases by run ID, by the
`metadata.nativeHarnessBackup.sessionScopeSha256` digest, or by the
identity's own lease ID, in one `OR` query
> - The metadata arm has no index. So PostgreSQL reads every lease of
the company, and the metadata of each lease, for each lookup
> - On a hosted test database, this statement ran 21,736 times at a mean
of 158 ms in one window. After other fixes, it still ran 2,364 times at
a mean of 121 ms
> - This pull request adds a small partial expression index on the
digest
> - The benefit is that each lookup reads a few index entries instead of
all company leases. The query text and the result do not change

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The lease lookup in `backupDependencies` in
`server/src/services/native-runtime/native-harness-backup-expiration.ts`.
`reconcileBackup` calls it for each backup root when the expiration
worker loads or checks a root.

**Subsystem affected**
Native harness backup expiration and the `environment_leases` table
schema.

**Current behavior**
The query is `company_id = $1 and (heartbeat_run_id in (...) or
metadata->'nativeHarnessBackup'->>'sessionScopeSha256' = $n or id =
$m)`. The run arm and the ID arm have indexes. The digest arm has no
index. A `BitmapOr` plan needs an index for every arm, so the planner
reads all company leases and evaluates the JSON expression on each row.
Lease metadata holds sandbox details, so these reads are large.

**Proposed behavior**
The partial index `environment_leases_native_backup_scope_idx` keys
leases by the digest. It holds only leases that have a backup stamp.
Every arm of the `OR` now has an index, and the planner combines three
index probes. The SQL and the returned lease IDs do not change.

**Reason and benefit**
This statement was in the top ten by total database time on a loaded
hosted database. A local reproduction shows a query that is more than
6,000x faster and reads 9 buffers instead of about 436,000.

**Breaking changes**
None. The API, the lease-matching rules and the results do not change.

## What Changed

- Add the partial expression index
`environment_leases_native_backup_scope_idx` on
`(metadata->'nativeHarnessBackup'->>'sessionScopeSha256')`, where that
value is not null.
- Add migration `0459`. It uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`. The migration runner runs it as a top-level statement, the same
as migrations 0454–0458.
- Add a regression test. It checks the lease-matching rules: run leases,
a lease with only the backup stamp, and the identity's own lease match.
Another digest, a non-string stamp, a top-level key with the same name,
and the same digest in another company do not match. The test also
checks that the custom plan and a forced generic plan of the generated
SQL use `BitmapOr` with the new index and no sequential scan.
- Add `scripts/benchmark-environment-lease-backup-lookup.ts`. It
captures the SQL that the real service generates and measures it with
and without the index.
- Document the index in `doc/DATABASE.md`.

## Verification

Cause and thesis. The cost comes from the one `OR` arm without an index,
which forces a scan of all company leases and their metadata. Thesis: an
index on the digest lets the planner use a `BitmapOr` of three index
probes, so the cost no longer depends on the company's lease count. The
results stay identical. The benchmark confirms the thesis.

I did not change the SQL. With the index, the existing `OR` query
already gets the index plan. A `UNION` rewrite would not help without
the index, because the digest arm would still scan the company.

Local SQL benchmark:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-environment-lease-backup-lookup.ts
```

Fixture (synthetic): one company with 100,000 leases with 4 KiB of
incompressible metadata each. 30% of the leases have a backup stamp for
another session. A second company has 20,000 leases. The table is 678
MB. The backup root has 3 run leases, 1 lease with only the stamp, and 1
identity lease. The same digest is also in the other company. Embedded
PostgreSQL 18 on an Apple laptop, which other work loaded heavily. The
hosted database is Neon with different compute, so absolute times are
local SQL measurements only.

The script measures in the order: no index, index, no index, index. Each
measurement has 2 warmups and 11 samples of `EXPLAIN ANALYZE` execution
time and of the full `reconcileBackup` call.

| Measurement | Without index | With index |
| --- | --- | --- |
| Lease query execution, run 1 | median 196 ms (138–562 ms) | median
0.010 ms (0.007–0.017 ms) |
| Lease query execution, run 2 | median 185 ms (144–295 ms) | median
0.030 ms (0.020–0.045 ms) |
| `reconcileBackup` wall time (includes file reads), runs 1 and 2 |
median 188 ms and 216 ms | median 1.4 ms and 4.2 ms |
| Plan | `Gather > Seq Scan` | `Bitmap Heap Scan > BitmapOr` of
`heartbeat_run_idx`, the new index and `pkey` |
| Buffers | 365,880 hit + 70,050 read | 9 hit |

These numbers are from head `65bd9d7ac6`. The final head only renumbers
the migration to 0459 after rebases; the SQL, the index and the service
code are the same. Two earlier runs on the same fixture, SQL and index
gave 236–425 ms without the index and 0.023–0.033 ms with the index. The
machine had heavy load from other work, so the spread without the index
is wide. Output check: in all four measurements, the service returned
the same 5 lease IDs, and the complete query rows were equal.

Index build (`CREATE INDEX CONCURRENTLY` from the migration file):
0.8–2.4 s for the 678 MB table over all runs. Index size: 3.4 MB with
30% of leases stamped.

Tests:

- `cd server && npx vitest run
src/__tests__/native-harness-backup-expiration.test.ts`: 109 passed. The
new test failed when I dropped the index before the plan check
(confirmed red), and it passes with the index.
- `pnpm --filter @paperclipai/db check:migrations`: passed.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`: passed.
- `pnpm verify`: it asks for broader coverage (exit 2) because of the
new script. See the CI results for this head.

## Risks

- Migration time. A concurrent index build reads the full
`environment_leases` table and does not block writes. Locally, the build
took 0.8–2.4 s for 678 MB. The time grows linearly with the table size.
I did not time it on a production-size clone. I expect a few seconds to
tens of seconds on a large instance, which is in the 1–2 minute budget.
This estimate is not proven.
- If the build stops, PostgreSQL can leave an invalid index. The
migration runner repairs invalid concurrent indexes when it runs again,
and `IF NOT EXISTS` makes a rerun safe.
- The index expression must stay textually equal to the query
expression. If someone changes the JSON path, the planner cannot use the
index. The new test catches this for custom and generic plans.
- The index is partial, so ANALYZE does not collect statistics for the
expression. The planner uses the default equality estimate. On the large
fixture and in the small test, the planner still selects the index plan.
A company with very few leases can still get a sequential scan, which is
cheap.
- Each lease write with a backup stamp writes one more small index
entry. The write cost is small.
- This change does not reduce how often the worker calls the lookup. It
makes each call cheap. The other digest lookups in backup retention
(`native-harness-backup-retention.ts`) and session retention checks can
also use the index.
- Previous-release story: no persisted state changes. The index is
derived from existing lease metadata.
- The hosted improvement is not verified until deployment. After
deployment, compare a new query-statistics window for this statement.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:30:05 -06:00
DottaandClaude Opus 5.5 6ed6b0754b perf(recovery): probe recovery source runs and stale-continuation runs by key (#20584)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The workspace recovery worker saves the files of failed runs. After
each pass, it reads the time of its next deadline
(`nextWorkspaceRecoveryAt`)
> - That read finds the source run, its finalizer and its leases with
`id::text = evidence->>'runId'`. A text comparison cannot use a key.
Each exhausted hold read every run, finalizer and lease of its company
> - On a hosted staging database this statement used about 4 s per call
(21 calls in the sample window). The cost grows with exhausted holds
multiplied by the company's run count
> - The ownership reconciler's stale-continuation check
(`checkStaleDroppedComments`) has the run lookup shape that #20412 fixed
in the lost-wake check: one OR across the legacy context and the native
column
> - This pull request compares the run id as a canonical uuid, and it
gives the stale-continuation check the per-binding run probes from
#20412
> - The benefit is the same recovery decisions with index-driven work:
11 s to 93 ms for the deadline read on a representative local fixture

## Linked Issues or Issue Description

Refs #20412

**What happened?**

The workspace recovery deadline read was one of the most expensive
statements that remained on a hosted staging database after the fixes in
#20393, #20405, #20386, #20412, #20391 and #20400. Query statistics
showed about 4 s per call. The statement text starts with `select case
when (issue_recovery_actions.…`.

The statement evaluates the retained-source fallback and stop-proof
conditions for each recovery action. These conditions read the source
run (`heartbeat_runs`), its finalizer (`native_run_finalizations`) and
its leases (`environment_leases`) with `x.id::text =
evidence->>'runId'`. The cast prevents the use of the primary key.
PostgreSQL reads all rows of the company for each exhausted hold. The
worker's discovery (`retainedWorkspaceCandidates`) uses the same
conditions.

The stale-continuation check matched a task's runs with
`(context_snapshot->>'issueId' = issue OR native_issue_id = issue)`.
With the OR, PostgreSQL can join runs only by company. It also hashed
every wake request of the company for the stale-wake test.

**Expected behavior**

Each condition uses a key lookup. The worker and the reconciler make the
same decisions for the same saved state.

**Steps to reproduce**

1. Seed a company with a deep run history and a few thousand workspace
recovery actions. Let a few hundred be exhausted holds that stay active.
2. Call `nextWorkspaceRecoveryAt`.
3. The call takes seconds, and its time grows with holds multiplied by
runs. The benchmark below shows this.

**Paperclip version or commit**

Baseline: `8d2bf78323`.

**Deployment mode**

Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1.
Staging uses hosted PostgreSQL.

## What Changed

- `workspace-recovery-fallback-discovery.ts`: the source run, finalizer
and lease probes compare `id =
canonicalExecutionUuid(evidence->>'runId')`. The helper returns null
unless the text is the canonical lowercase uuid form. Thus the probe
matches exactly the rows that `id::text = evidence->>'runId'` matched.
The helper already exists in `execution-identity.ts`.
- `nextWorkspaceRecoveryAt` and worker discovery use these conditions.
Their SQL changes only in these probes.
- `checkStaleDroppedComments` reads in two stages. The first stage keeps
comments that a stale-continuation cancellation dropped and that no open
wake carries. The second stage applies the run probes. The run probes
use the `taskRuns` helper from #20412, which is now shared by both
checks. The stale-wake lookup ends with `offset 0`, so it stays a
per-comment index probe.
- The final `order by c.created_at limit N`, the projection and the
delivery check after the query do not change.
- Add `scripts/benchmark-recovery-status-checks.ts`. It loads the
changed files from an explicit baseline commit and from the working
tree. Before it records time, it compares every changed condition for
every recovery action, the complete service results and the positional
SQL rows.
- Add tests: each run binding in the stale-continuation check (native,
legacy, live run on either binding, wrong agent, wrong company, failed
run, run outside the comment time, closed by a person, open wake). Also
a test that a hold with a non-canonical saved run id is not discovered,
as before.
- Document both checks and the benchmark in `doc/DATABASE.md`.
- No schema change, index, migration, or new startup work.

## Verification

- `pnpm exec vitest run
src/__tests__/ownership-reconciliation-report.test.ts
src/services/workspace-recovery-fallback.test.ts
src/services/workspace-recovery-stop-proof.test.ts
src/__tests__/heartbeat-held-comment-wake.test.ts --maxWorkers=1` (from
`server/`).
- `pnpm --filter @paperclipai/server typecheck`.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-recovery-status-checks.ts --baseline-ref=8d2bf78323
--samples=7`

Benchmark setup: PostgreSQL 18.1 (embedded, local), `shared_buffers` 128
MB, `work_mem` 4 MB. All data is synthetic, with md5-hex filler. Each
fixture has two companies (full size and a quarter). Two warmup rounds,
then 7 timed samples for each arm, in alternating order. Each sample
runs the real service function and asserts the same result. These are
local SQL measurements on a shared machine, so the spread is wide.

Workspace recovery deadline (`nextWorkspaceRecoveryAt`). About 8% of
actions are exhausted holds that stay active. Some actions save a
non-canonical (uppercase) run id. Before timing, the benchmark compares
six conditions (fallback pending, due time, due; re-arm pending, due
time, due) for every action of the fixture.

| Fixture | Runs / actions / pending holds | Baseline median (min–max) |
Treatment median (min–max) | Shared buffers |
| --- | --- | --- | --- | --- |
| small | 6,500 / 255 / 36 | 50.9 ms (47–76) | 6.8 ms (6–8) | 19,133 →
1,075 |
| representative | 78,750 / 3,813 / 588 | 11,098 ms (8,277–12,790) | 93
ms (35–145) | 3,493,417 → 24,902 |
| deep history | 191,250 / 3,813 / 588 | 40,146 ms (31,190–52,521) | 40
ms (36–230) | 13,491,792 → 26,911 |

Stale-continuation check (`checkStaleDroppedComments`). The fixture is
pessimistic: more than half of recent comments on done tasks have a
stale cancelled wake. The table shows statement time from `EXPLAIN
ANALYZE` with default settings, and one statement run with
`enable_nestloop = off` (the company-only hash plan). The result is
checked as equal in both modes.

| Fixture | Runs / recent comments / reported | Statement, baseline →
treatment | Hash plan, baseline → treatment |
| --- | --- | --- | --- |
| small | 3,792 / 76 / 24 | 3.2 → 1.5 ms | 5.5 → 3.8 ms |
| representative | 75,406 / 1,125 / 200 (limit) | 34 → 26 ms | 66 → 25
ms |
| busy hour | 152,031 / 5,625 / 200 (limit) | 328 → 153 ms | 2,302 → 165
ms |

- Neutral path: the complete stale-continuation service call stays about
the same (busy hour 750 → 655 ms median). The per-comment delivery check
after the query (`deliveredQueuedCommentIds`) is most of that time. This
change does not touch it.

## Risks

- Low risk. Every predicate keeps the same logic. Only the probe shape
and the evaluation order change. The benchmark compares each changed
condition for every recovery action, including non-canonical saved run
ids, and the complete results of both checks.
- Previous-release story: no saved state, schema or identity changes.
Holds saved by older releases are read with the same rules, so they
carry forward unchanged.
- Rows with the same comment `created_at` have no defined order, as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:12:41 -06:00
DottaandClaude Opus 5.5 d82e5efeb9 perf(ui): cut repeated reads while a task is open (#20563)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI keeps an open task current with a shared company
websocket and with polling fallbacks
> - In a busy company, one open task sent 35 to 60 API requests in 17
seconds, and each request waits on the database
> - Most of these requests were repeated reads: task summaries 22 times,
the task run list 11 times, task browsers 6 times, and the 60 KB agent
list 6 times
> - The cause is not the open task. Other agents' run and task events
invalidate company-wide lists on every 300 ms batch, and the task view
polls runs and browsers on fixed intervals even while the websocket is
open
> - This pull request spaces company-wide refreshes, and it lets the
websocket replace run and browser polling while it is connected
> - The benefit is about 79% fewer reads from an open task in a busy
company, with the same polling fallback when the connection is down

## Linked Issues or Issue Description

Refs #17608
Refs #9627

**What happened?**

An open task in a busy company sent the same reads again and again. In
one 17-second visit, the UI requested `issues/summaries` 22 times, the
task live runs 11 times, the task browsers 6 times, and the company
agent list 6 times. The company live runs and the task's child lists
(`parentId`, `createdFromIssueId`, `descendantOf`) were requested 2 to 3
times each.

**Expected behavior**

A busy company must not make an open task refetch company-wide lists on
every event batch. While the live connection is open, the UI must use
events, not fixed fast intervals, for task runs and browsers. The UI
must keep its polling fallback when the connection is closed.

**Steps to reproduce**

1. Open a task in a company where other agents run work.
2. Keep the task open for 17 seconds.
3. Count the API requests in the browser network panel.

**Paperclip version or commit**

Based on master at bd30ef8510.

## What Changed

- `ui/src/lib/query-invalidation-batcher.ts`: Add a minimum spacing
option to the invalidation batcher. A quiet lane flushes after the
normal batch interval. A busy lane flushes at most once per spacing
window. Add `createLaneRoutingBatcher`, which sends each invalidation to
one of two batchers by query key.
- `ui/src/context/LiveUpdatesProvider.tsx`: Route company-wide keys to a
5-second ambient lane. These keys are task lists and summaries, the
agent list, heartbeats, company live runs, sidebar badges, the
dashboard, costs and the company activity feed. All other keys, such as
task detail, comments, task runs and browsers, keep the 300 ms lane.
- `ui/src/lib/live-runs-cache.ts`: While the company connection is open,
a successful active run list or active run polls every 15 seconds
instead of every 1 or 3 seconds. Run lifecycle events already refresh
these queries, and progress events patch them.
- `ui/src/hooks/useTaskBrowsers.ts`: While the company connection is
open, stop browser polling when no browser can change state. Keep the
3-second poll for a starting, running, idle or stopping browser, and
when the connection is closed. The two callers pass the task's company.
Socket open and reconnect already reconcile this active query.
- `ui/src/lib/task-browsers-refresh.ts` and `LiveUpdatesProvider.tsx`:
On `browser.*` activity for a task, refresh that task's browser list. If
a read is in flight, one shared follow-up read runs after it settles, so
an older empty result cannot stop polling. Two views of the same task
share that follow-up. A list that no view observes becomes stale, so a
returning view reads it. When the connection opens or resyncs, each
browser list with a read in flight also gets one follow-up read, because
missed activity is not replayed.
- `doc/DATABASE.md`: Document the ambient lane and the new polling
contract.
- Tests: Add a fan-out test that replays a busy-company event stream for
17 seconds against an open task, plus unit tests for spacing, lane
routing, key classification, the run interval and the browser interval.
Update the recovery test for the connected active-run interval.

## Verification

Request count over a fixed 17-second open window
(`ui/src/context/LiveUpdatesProvider.fanout.test.ts`). The test mounts
the reads that an open, running task keeps active. It replays a busy
company: one run status event every second and one task activity event
every 0.7 seconds, none for the open task. Fake timers make each run
deterministic, so the spread is zero. The baseline used the same test
with master's batcher, run interval and browser interval.

| Read | Before | After |
| --- | ---: | ---: |
| Task summaries (sidebar) | 24 | 5 |
| Company agent list | 17 | 5 |
| Company live runs | 17 | 5 |
| Child list: parent | 24 | 5 |
| Child list: created from | 24 | 5 |
| Child list: descendants | 24 | 5 |
| Task live runs | 18 | 2 |
| Task browsers | 6 | 1 |
| **Total** | **154** | **33** |

The baseline numbers match the staging capture well (summaries 22,
browsers 6).

- `pnpm exec vitest run src/context/LiveUpdatesProvider*.test.*
src/lib/live-runs-cache.test.ts
src/lib/query-invalidation-batcher.test.ts
src/hooks/useTaskBrowsers.test.ts src/lib/task-browsers-refresh.test.ts
src/hooks/useTaskPluginPanels.test.tsx
src/components/task-side-panel/TaskSidePanel.test.tsx
src/pages/IssueDetail.test.tsx` (in `ui/`): all pass.
- `pnpm exec tsc --noEmit -p ui`: no errors in `ui/src`.
- `pnpm check:token-gates`: clean.
- Tests also prove these cases:
- With the connection closed, task runs still poll every second and
browsers every 3 seconds.
- With a quiet company, one event refreshes the summaries and child
lists after one normal batch.
- `browser.started` on the open task refreshes its browsers. The same
event on another task does not.
- A browser that starts during a slow empty read appears after that read
settles (`ui/src/lib/task-browsers-refresh.test.ts`). This test fails
when the refresh joins the older read.
- Two views and repeated events during one read cause one follow-up
read. This test fails without the shared queue.
- A list cached while the task was closed is read again when a view
returns.
- A connection that opens during an older empty browser read causes one
more read, so a browser that started while the socket was closed
appears. This test fails without the follow-up.
- Task detail, comments, task runs, browsers, attention and another
company's keys stay on the fast lane.
  - An unrecognized active result keeps the fallback interval.

This is a local request-count proof. It does not include a timed browser
journey on a deployed instance.

## Risks

- Company-wide lists refresh at most once every 5 seconds from live
events, so they can be up to 5 seconds late. After 5 quiet seconds, an
event refreshes them in about 300 ms. Own mutations invalidate their
queries directly, so they do not wait.
- An active task run refreshes from events. Fields without an event,
such as output silence, can be up to 15 seconds late while the
connection is open. Reconnect still reconciles missed events.
- A browser that appears without activity is not shown until the next
activity or reconnect. Every browser lifecycle step logs activity on the
task, so this case is not expected.
- No server, schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:01:53 -06:00
DottaandClaude 84007892e9 refactor: move AGENTS.md section detail to docs and rules, add a bug-fixing section (#20585)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` is the instruction file that each human and AI
contributor reads before a change
> - Pull request #20557 shortened the file, but several sections still
held detail that applies to only some changes
> - The scoped-rules section told agents to read each matching rule in
full, and most changes match five rules
> - The core engineering rules mixed three product invariants with
process conventions
> - This pull request moves that detail to the docs and to scoped rules,
and adds a section on how to fix a bug
> - The benefit is a 211-line file that states what applies to every
change and points to the rest

## Linked Issues or Issue Description

Refs #20557

**Issue type**

Outdated or unclear contributor documentation.

**Where is the issue?**

`AGENTS.md` at the repository root, and `rules/`.

**What's wrong?**

- The scoped-rules section made agents read the full body of each
matching rule before a change. The rule bodies total about 47 KB.
- Sections 4, 6, 8, 9 and 10 repeated detail that belongs in
`doc/DEVELOPING.md`, `doc/DATABASE.md`, a route rule, `DESIGN.md` and
`CONTRIBUTING.md`.
- Section 4 had a database reset command for `data/pglite`. That
directory does not exist.
- Section 5 had seven rules. Four of them were process conventions or
path-specific detail.
- The file did not say how to fix a bug.

**Suggested fix**

Make the rule headers the index. Move section detail to the document or
rule that owns it. Keep only the three invariants in section 5. Add a
bug-fixing section.

## What Changed

- The scoped-rules section now tells agents to list the rule headers
with one `grep` command. Agents read a rule body only for a design
decision in its area, when the description is not sufficient, or when a
reviewer cites it.
- `rules/README.md` says that each `description` must state the required
behavior in one sentence.
- `rules/pull-request-review.mdc` has a description that states its
requirements.
- Section 4 (dev setup) and section 11 (pull requests) now point to
`doc/DEVELOPING.md` and `CONTRIBUTING.md`.
- The schema change workflow moved from section 6 to `doc/DATABASE.md`.
- The API and auth expectations moved from section 8 to the new
`rules/api-route-expectations.mdc`.
- Section 9 links to `DESIGN.md` for design rules. `AGENTS.md` no longer
tells contributors to run `pnpm check:token-gates`.
- Added section 10, "Fixing a Bug": reproduce the error, write a failing
test, write the fix, see the test pass, and verify the fix.
- Section 5 now has three rules: company scope, contract
synchronization, and the control-plane invariants.
- The company-scope exceptions moved to the new
`rules/company-scope-exceptions.mdc`.
- Removed the strategic-docs rule. The plan-doc convention moved to the
repo map. The artifact rule moved to section 7.
- Removed rule 7. Five code comments that cited `AGENTS.md §5.7` now
cite `rules/data-paths.mdc`.
- Regenerated the SDK public source pin, because one of those comments
is in `packages/shared/src/crash-points.ts`.
- Added both new rules to `.greptile/files.json`.
- The API route rule applies only to routes that read or write company
data. Its status code list is a list of common codes.
- The company-scope rule lists the sidebar company order as an
instance-wide exception.
- The `honeycomb-tracing` skill now points to `rules/data-paths.mdc` for
the Telemetry workflow.

## Verification

- Open `AGENTS.md` and confirm that each link resolves. A local script
checked this.
- Run `grep -E '^(description|globs|alwaysApply):' rules/*.mdc` and
confirm that each rule prints a description.
- Run `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD>
--check`. It passes locally.
- `packages/shared/src/crash-points.test.ts` passed locally before the
rebase onto `master`.
- Five source files changed in comments only. No other test suite ran
locally.

## Risks

- An agent that reads only rule descriptions can miss a detail in a rule
body. Reviewers still apply the full rules.
- Pull requests and the definition of done are now sections 11 and 12.
Sections 1 to 9 keep their numbers.
- Dated plan documents under `doc/plans/` still cite `AGENTS.md §5.7`.
They are historical records.
- Pull request #20565 also changes `AGENTS.md`. The second one to merge
needs a conflict fix in section 1 and section 2.

## Model Used

- Claude (Anthropic), in Claude Code with tool use.
- `claude-fable-5-2`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 11:00:31 -06:00
DottaandClaude Opus 5.5 765ccd8691 perf(chat): index chat-admission checks by company and payload issue id (#20562)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connections save each external message as a `chat_actions` row
before the server admits it as an agent wakeup.
> - The server checks if an issue holds chat input that it did not
admit. Recovery skips such an issue. Manual task start refuses it.
> - On a hosted instance, the recovery read with this check used the
most total database time: 488 calls at about 1 s each. The manual-start
check ran 155 times at about 0.9 s.
> - The check matches `chat_actions.payload->>'issueId'` against the
issue. No index covers this value, so each call reads the full
`chat_actions` table and its payloads.
> - This pull request adds one concurrent partial index for the check.
The check SQL does not change.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The chat-admission check `unadmittedChatWakeupCondition`. Recovery uses
it in the per-issue candidate read. Manual task start uses it in the
held-input check.

**Subsystem affected**
Server chat admission, issue recovery, manual task start, and the
database schema.

**Current behavior**
The check finds `inbound_wakeup` actions by `company_id` and
`payload->>'issueId'`. Two nested checks find `failed_run_retry` actions
and later `inbound_wakeup` actions on the same company and payload issue
ID. `chat_actions` has no index on `company_id` or on the payload issue
ID. Each call reads the full table and reads each payload. On a hosted
instance, the two statements used about 630 s of database time in one
statistics window.

**Proposed behavior**
The check and its nested checks read a small set of rows from one index.
The admission result does not change.

**Reason and benefit**
The hosted database had many concurrent active queries. Request reads
waited behind background work such as this check.

**Breaking changes**
None. The migration adds an index only.

## What Changed

- Migration `0458_chat_admission_issue_index` builds
`chat_actions_company_issue_kind_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. It runs through the existing top-level statement path,
which repairs an invalid index on retry.
  - Keys: `(company_id, payload->>'issueId', kind, status)`.
  - Partial: `kind IN ('inbound_wakeup', 'failed_run_retry')`.
- One index serves the outer input lookup, the failed-run retry lookup,
and the later-input lookup. All three match on the same company and
payload issue ID.
- The Drizzle schema declares the index. The snapshot chain moves to
`0458` and keeps five snapshots.
- New regression test
`server/src/__tests__/chat-admission-condition.test.ts` pins the
admission semantics on real PostgreSQL:
- Pending and failed input without a receipt holds the issue. Received,
processed and cancelled input does not. Other action kinds do not.
- A skipped receipt, a receipt for another agent, or a receipt in
another company does not admit input. A failed or cancelled run still
counts as admission.
- The issue ID is compared as company-scoped text. Uppercase, array,
null, other-issue and other-company values do not match.
- Only an admitted retry in the same conversation and issue that lists
the action releases skipped input.
- Only later admitted input in the same conversation, issue and session
generation releases a failed input. Pending input stays held.
  - The check reads `chat_actions` only through the new index.
- `scripts/benchmark-chat-admission-index.ts` measures both caller
statements with and without the index and fails if the rows differ.
- `scripts/benchmark-chat-admission-index-migration.ts` times the
migration through the normal migration runner.
- `doc/DATABASE.md` documents the index and the benchmark commands.

## Verification

Local SQL measurements on synthetic data. PostgreSQL 18.1 (embedded),
default settings. The fixture has 100,995 `chat_actions` rows (662 MB):
60,000 GitHub actions with 6 KiB incompressible payloads, 2,000 issues
with 20 admitted inbound messages each (3 KiB payloads), and 1,000
failed-run retries. Each case has 2 warmups and 9 samples of `EXPLAIN
ANALYZE` execution time. Runs alternate: no index, index, no index,
index.

| Caller statement | Case | No index, median (range) | Index, median
(range) |
| --- | --- | --- | --- |
| Manual-start held check | quiet issue | 3,093 ms (2,150–3,519) / 3,586
ms (2,375–11,235) | 0.036 ms (0.034–0.052) / 0.040 ms (0.033–0.047) |
| Manual-start held check | admitted input | 3,555 ms (3,310–8,080) /
5,205 ms (3,353–9,525) | 0.049 ms / 0.052 ms |
| Manual-start held check | unadmitted input | 3,002 ms (2,864–3,221) /
4,179 ms (3,200–6,585) | 0.051 ms / 0.044 ms |
| Manual-start held check | failed, later admitted | 3,586 ms / 3,722 ms
| 0.066 ms / 0.054 ms |
| Manual-start held check | admitted retry | 2,733 ms / 4,953 ms | 0.049
ms / 0.053 ms |
| Recovery candidate read | quiet issue | 43 ms (41–48) / 96 ms (44–219)
| 0.047 ms / 0.055 ms |
| Recovery candidate read | admitted input | 203 ms (48–575) / 174 ms
(44–314) | 0.055 ms / 0.058 ms |
| Recovery candidate read | unadmitted input | 66 ms / 144 ms | 0.053 ms
/ 0.054 ms |

- Buffers per call fall from about 40,000 shared hits to 5–23.
- Locally, the recovery read without the index uses a hashed full scan
of `chat_actions`. The held check uses a full scan per call. On the
hosted instance both took about 1 s per call. The local fixture is
smaller than the hosted table.
- Correctness: complete positional rows match across all four runs for
all six cases (quiet, admitted, unadmitted, failed then admitted,
retried, other-company decoy). The script asserts the expected held
result for each case.
- Migration timing: 600,001 `chat_actions` rows (1,759 MB), 300,000 of
them in the index scope. First run through `applyPendingMigrations`: 4.2
s. Rerun after the history row is removed: 0.5 s. Concurrent build
alone: 3.4 s. Index size: 30 MB.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index.ts
OTHER_PAYLOAD_KIB=2 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index-migration.ts
cd server && npx vitest run src/__tests__/chat-admission-condition.test.ts   # 6 passed
pnpm --filter @paperclipai/db check:migrations                               # passed
pnpm --filter @paperclipai/server typecheck                                  # passed
```

## Risks

- The migration builds one index concurrently. It does not block writes.
On the 1.76 GB local fixture it took about 4 s. A larger hosted table
takes longer, but the build does not lock out reads or writes.
- Writes to `inbound_wakeup` and `failed_run_retry` actions maintain one
more index. Status updates on these rows were already not HOT updates,
because other partial indexes use `status`.
- The planner has no expression statistics for a partial index. The new
test pins that the check uses the index on an analyzed table with
unrelated rows.
- The check SQL does not change, so admission behavior does not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:44:51 -06:00
DottaandClaude Opus 5.5 faaae003a9 perf(chat): find the conversation head from open delivery work (#20564)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - External chat messages go into a durable delivery queue. Each
conversation drains in provider order.
> - Each drain step reads the head of one conversation. The head query
filters one endpoint's deliveries by state and thread.
> - On a hosted instance, this query ran about 10,000 times at about 300
ms each. The database was busy, and request reads waited behind
background work.
> - PostgreSQL read every processed delivery of the endpoint and its
`normalized_event` value to find a few open rows.
> - This pull request finds the endpoint's open delivery IDs first and
fetches only those rows.
> - The benefit is less background database load. The selected head and
the delivery order do not change.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The inbound conversation head query
(`earliestOpenConversationDelivery`).

**Subsystem affected**
Server chat inbound delivery queue.

**Current behavior**
The query has `endpoint_id = $1 and (state in ('received', 'retry',
'processing') or (state = 'processed' and <pending inbound wake>))`, a
thread filter on `normalized_event`, and a sort on `normalized_event`
keys. No index matches the `or`. PostgreSQL scans all deliveries of the
endpoint. For each row it reads `normalized_event` to test the thread.
It also reads all of `chat_actions` to build the pending-wake set. A
busy endpoint keeps all of its processed history, so the cost grows with
that history.

**Proposed behavior**
The query selects `id in (<open delivery IDs of the endpoint>)`. That
set is a `union all` of two small reads:
- received, retry and processing deliveries of the endpoint, through the
existing `chat_deliveries_work_idx`;
- processed deliveries that an issued or processing `inbound_wakeup`
action of the same endpoint (and company) still owns.

PostgreSQL fetches only those rows by primary key, then applies the same
reaction, thread and order rules.

**Reason and benefit**
The cost no longer grows with processed history. On a local fixture with
60,000 processed deliveries, the head query reads 1,309 shared buffers
instead of 262,630.

**Breaking changes**
None. There is no schema change, migration, API change, timer or poll
change.

Related work: #17971 changed the pending-wake predicate of the same
queue. #20400 added the index for the other chat delivery statement in
the same report (`select state ...
normalized_event->'githubAutomatic'->'context'->>'deliveryId'`). That
statement already uses the #20400 index on master, so this pull request
does not change it.

## What Changed

- `delivery-ledger.ts`: add `openConversationDeliveryIds(endpointId)`.
It returns the endpoint's open delivery IDs. It selects the same set as
the old state predicate for that endpoint.
- `conversation-queue.ts`: the conversation head query uses `id in
(openConversationDeliveryIds(endpointId))` in place of the state `or`.
The thread filter, reaction exclusion, order, limit and lifecycle
dependency read do not change.
- `inbound-queue-discovery.test.ts`: add a real PostgreSQL test. It
compares the new set with the old predicate on three endpoints. It
covers each state, issued and processing wakes, duplicate wakes, wakes
of non-processed deliveries, processed, other-kind and null-delivery
actions, an action on a different endpoint of the same company, and
reactions. It also checks that the real captured head query returns the
oracle's first row.
- `benchmark-inbound-queue-discovery.ts`: add
`--endpoint-history=<rows>` (a busy endpoint with 4 KiB payloads) and
`--only=<fixture>`. Add a work budget: the treatment head query must
visit at most 100 delivery rows on the history fixtures.
- `doc/DATABASE.md`: describe the head query and the benchmark command.

## Verification

Local SQL measurements. PostgreSQL 17.10 (Postgres.app, macOS arm64),
disposable database, synthetic data, default settings with 256 MB shared
buffers. The host was under heavy load from other work, so timings are
noisy. Shared buffers and visited rows are stable. Each arm has two
warmups and five alternating samples (median [min..max]). Baseline:
`origin/master` at `8d2bf78323`.

| Fixture | Query | Baseline | Treatment | Shared buffers | Delivery
rows visited | Output |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| 60,000 processed, 4 KiB | head | 290.09 ms [220.21..554.19] | 7.34 ms
[5.85..7.68] | 262,630 → 1,309 | 60,002 → 4 | Equal |
| 20,000 processed, 4 KiB | head | 276.02 ms [211.77..322.12] | 34.61 ms
[4.45..194.15] | 67,618 → 486 | 20,002 → 4 | Equal |
| 1,500 processed, 32 KiB | head | 15.88 ms [13.66..21.79] | 2.29 ms
[1.64..7.75] | 10,602 → 80 | 1,502 → 4 | Equal |
| 20 processed | head | 1.49 ms [0.92..3.66] | 1.85 ms [1.01..1.97] | 2
→ 45 | 22 → 968 | Equal |

- The deadline and ready selectors do not change. Their buffers are
identical in both arms; their timings move only with host noise.
- The small control is neutral. On a near-empty table, PostgreSQL picks
a sequential scan for the open-state branch. This costs about 45
buffers.
- The 60,000-row run passes `--require-work-budget`. The baseline
exceeds the budget (`baselineExceedsWorkBudget: true`). This is the
red/green proof. There is no timing threshold in CI.
- On the treatment, the remaining head cost is the scan of
`chat_actions` for pending wakes of the endpoint. The baseline also
reads all of `chat_actions` for its hashed subplan.
- `pnpm exec vitest run src/__tests__/inbound-queue-discovery.test.ts`
(server): 6 tests pass, including the new equivalence test.
- `chat-channels.integration.test.ts` filtered with `-t
"reorder|orders|FIFO|conversation drain|lease renewal|receipt
order|edit|lifecycle"`: 148 assembled chat tests pass. They cover Slack,
Teams, Telegram, Discord and GitHub provider-order drains, head-of-line
barriers, lease loss, lifecycle edits and wake retries.
- `pnpm exec tsc --noEmit -p tsconfig.json` (server) passes.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=8d2bf78323 --endpoint-history=60000 --only=endpoint-history --require-work-budget
```

Set `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to use an existing disposable
PostgreSQL server. The benchmark uses synthetic data only.

## Risks

- Query plans depend on data and settings. These local numbers do not
prove hosted request latency. A follow-up statistics window after deploy
must confirm the gain.
- The pending-wake branch still scans the endpoint's `chat_actions`. A
partial index on pending `inbound_wakeup` actions would remove that
scan. I did not add it, to avoid a migration.
- Low behavior risk. The new set equals the old predicate for each
endpoint, and the order expression is unchanged. Tests compare both on
the same rows.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:15:29 -06:00
DottaandClaude Opus 5.5 56aa160a4e perf(native): index heartbeat runs by company and native session (#20560)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs share a native session. Retention, recovery, resume and
archive code reads all runs of one session from `heartbeat_runs`.
> - On a hosted instance, the native retention lookup ran about 67,000
times in one day at about 22 ms, and about 10,000 times at about 11 ms
in a later window.
> - No index has `native_session_id`. Each read walks the agent's or the
company's run history and filters the rows.
> - This pull request adds one concurrent partial index on `(company_id,
native_session_id)`. The SQL does not change.
> - The benefit is less background database load. Request reads wait
less, and the results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The native session-run lookups. The largest caller is
`sessionRetentionBlocker` in native harness backup and checkpoint
retention. It selects `id, status, finished_at`, an incomplete-recovery
flag and the issue ID for one company, agent and native session.

**Subsystem affected**
Server native runtime (retention, recovery, resume and archive reads)
and the database schema.

**Current behavior**
`heartbeat_runs` has no index on `native_session_id`. PostgreSQL reads
the whole run history of the company (or of the agent through
`heartbeat_runs_company_agent_started_idx`) and filters by session. The
cost grows with the run history, not with the size of the session.

**Proposed behavior**
The index finds the few runs of one session directly. Every caller
already filters by `company_id` and `native_session_id`, so one index
serves the agent-scoped and the issue-scoped reads.

**Reason and benefit**
This statement family used about 1,500 s of database time in one day on
a busy hosted database. The local proof below shows 12.3 ms to 0.04 ms
per call.

## What Changed

- `packages/db`: add `heartbeat_runs_company_native_session_idx` on
`(company_id, native_session_id) WHERE native_session_id IS NOT NULL`.
Migration `0457_native_session_run_index` builds it with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. The migration is registered as a
top-level-statement migration, the same way as 0454–0456.
- Add a regression test in `native-harness-backup-expiration.test.ts`.
It captures the real retention lookup and proves that its plan can use
the index. It also pins the company and session scope: a live run of
another company with the same session ID does not protect a checkpoint.
A live run of the same agent in another session does not protect it. A
live run of the same session does protect it.
- Add `scripts/benchmark-native-session-run-index.ts`. It runs the real
checkpoint retention pass (dry run), captures the exact SQL, and replays
it with and without the index. It also times the migration through the
normal migration runner.
- `doc/DATABASE.md`: describe the index and the benchmark command.

## Verification

Local SQL measurements, synthetic data. PostgreSQL 18.1 (embedded test
server). Fixture: 100,000 runs (50,000 of the measured agent, 50,000 of
a second agent), 4 runs per session, 4 KiB incompressible
`runner_profile_json` per run, 638 MB table. Order: no index, index, no
index, index. 15 `EXPLAIN (ANALYZE, BUFFERS)` samples after 3 warmups.

| Pass | Plan | Median execution | Min / max | Shared buffers |
| --- | --- | --- | --- | --- |
| No index (1) | Seq Scan | 12.30 ms | 10.33 / 35.27 ms | 3,607 |
| Index (2) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.036 ms | 0.021 / 0.115
ms | 38 |
| No index (3) | Seq Scan | 12.38 ms | 10.24 / 26.14 ms | 3,607 |
| Index (4) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.043 ms | 0.022 / 0.098
ms | 38 |

- Correctness: the service SQL is the same in every pass. The returned
rows (all columns) and the retention result are equal in every pass. The
script fails on any difference.
- Migration: the first run through the migration runner took 525 ms on
the 638 MB fixture. The rerun took 273 ms and skipped the existing
index. A direct concurrent build took 145–221 ms. Index size: 1.9 MB.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-native-session-run-index.ts`
- `npx vitest run
src/__tests__/native-harness-backup-expiration.test.ts` (server): 108
passed.
- `pnpm --filter @paperclipai/db check:migrations`: passed. Five
snapshots remain.
- `pnpm verify` exits 2 (a new script under `scripts/` is a global CI
input). Required CI supplies the broad coverage.

## Risks

- Low risk. The migration adds one index concurrently and does not lock
writes. The SQL of every caller is unchanged.
- The index adds a small write cost to `heartbeat_runs` inserts and to
updates of `native_session_id`. Rows with a null session are not
indexed.
- Persisted run state: no saved run, session or checkpoint shape
changes. State saved by the previous release carries forward unchanged.
- Hosted latency is not measured yet. The numbers above are local SQL
measurements.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:04:22 -06:00
DottaandClaude Opus 5.5 0b20ea6fe9 fix(accounting): keep budget admission and usage handoff alive under run-row updates (#20484)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Before a run starts provider work, Paperclip reserves its budget and
binds a usage recorder. Both write the run's `heartbeat_runs` row.
> - Run-log appends, output progress and lifecycle events update that
same row while the run prepares and streams.
> - #20385 fixed this race for the standalone usage-receipt write only.
Budget admission and the recorder handoff still take a repeatable-read
snapshot before they lock the run row.
> - Each concurrent row update makes that snapshot stale. Four retries
run out with SQLSTATE `40001`, and the run ends `adapter_failed` with
raw SQL as its error.
> - This pull request applies the #20385 pattern (READ COMMITTED under
the existing locks) to those writers and to the final dispatch gate. It
also gives exhausted retries a bounded cause.
> - The benefit is that busy runs start and account normally, and a
remaining failure names its cause.

## Linked Issues or Issue Description

Refs #20385. Builds on #20483 (bounded failure cause and
`withoutSqlStatementText`).

**What happened?**

The restart-chaos L1 workload with `--db-pressure 4` (harness writers
that update each running run row, one row per statement, like a fast
stream) failed on master. In one wave, 0 of 4 tasks passed. Every first
run failed in `reserveRunBudget` before spawn. In an earlier wave, a run
failed in `createRunUsageRecorder`. The stack in both cases is `could
not serialize access due to concurrent update` (`40001`) on the run-row
`FOR UPDATE`. The board `runError` showed the full `select ... from
heartbeat_runs ... for update` statement. The server log line `heartbeat
execution failed` dropped the database cause.

Realism: the harness writers are synthetic, but the product has the same
writers on the same row. `appendHeartbeatRunEvent` increments
`next_event_seq`, output progress updates `last_output_*`, and lifecycle
events land while a run prepares. On a hosted database the window
between snapshot and row lock is wider than locally, so a busy run
reaches the same race without the harness.

**Expected behavior**

Budget admission and the recorder handoff commit while other writers
update the run row. Budget invariants stay intact: one reservation per
run, a reached hard stop still refuses work, and committed holds count
against later admissions. Usage receipts are journaled once. If retries
still run out, the run shows a bounded cause without SQL, and the server
log records the SQLSTATE.

**Steps to reproduce**

Run `server/src/services/run-accounting-contention.test.ts` against the
parent commit. The deterministic cases commit a real
`appendHeartbeatRunEvent` after the writer's transaction checks the
company and before it locks the run. Six of the eight cases fail with
`40001`.

## What Changed

- `reserveRunBudget` runs at READ COMMITTED. All mutable reads follow
the shared admission fences, the agent `FOR SHARE` lock and the run-row
`FOR UPDATE`. Exposure stays one SQL statement (hold-to-ledger atomic).
The `budget_reservations_run_idx` unique index and the run lock still
prevent a second hold.
- The final dispatch gate's accounting phase
(`dispatchResolvedInteractionIfCurrent` through `withIssueThenRunLocks`
in `run-dispatch/adapters/postgres.ts`) runs at READ COMMITTED. Its
ownership phase already read the same staleness facts at READ COMMITTED
under the same issue and run locks. Retry promotion and stale-run
cancellation keep their repeatable-read snapshot. The first L1 proof
wave found this writer exhausting `40001` once the other three were
fixed.
- `createRunUsageRecorder` runs its receipt-source handoff and its
capture-failure fence at READ COMMITTED. Both lock the run row first,
the same as `persistUsageReceipt` after #20385.
- `withOperationAccountingTransaction` raises
`AccountingContentionError` when retries run out. It has a bounded
message, `code` keeps the SQLSTATE (`40001`/`40P01`), `cause` keeps the
database error, and a warning logs both.
- `failRun` shows that bounded message. The code stays `adapter_failed`,
so chat and recovery retry paths still apply. #20483's `failureCause`
records `40001`. All other SQL text handling comes from #20483
(`withoutSqlStatementText`), unchanged here.
- `doc/DATABASE.md` documents the isolation choice and the failure
contract.

### Inventory of REPEATABLE READ / SERIALIZABLE transactions that write
run-scoped rows

| Writer | Status | Notes |
| --- | --- | --- |
| `persistUsageReceipt` (`usage-receipts.ts`) | Fixed in #20385 | READ
COMMITTED under the run lock. |
| `reserveRunBudget` (`budget-reservations.ts`) | Fixed here | Seen in
waves (`budget-reservations.ts:90`). |
| `createRunUsageRecorder` handoff (`usage-receipts.ts`) | Fixed here |
Seen in waves (`usage-receipts.ts:572`). |
| `createRunUsageRecorder` capture-failure fence (`persistFailure`) |
Fixed here | Same shape: run lock first, runs mid-stream. |
| Native finalizer `claimCoordinator` (`native-run-finalizer.ts`) | Safe
| A default READ COMMITTED transaction with `FOR UPDATE` on the
coordinator row. It cannot raise `40001`. The wave errors there were
`native_finalization_missing`, a symptom of the failed reservation (no
coordinator existed yet). |
| Final dispatch gate, accounting phase
(`dispatchResolvedInteractionIfCurrent`) | Fixed here | Seen in proof
wave `w-20261110T075857-s20386` (runs `140ddb10`, `c028c6c4`,
`fea0ee5a`). Its ownership phase already used READ COMMITTED under the
same locks with the same staleness reads. |
| Retry promotion and stale-run cancellation (`promoteOrCancelDueRetry`,
`cancelStaleQueuedRun`, same lock helper) | Follow-up | Kept at
repeatable read: `loadGateFacts` combines wakes, comments, dependencies,
tree holds and repair state in one decision. They act on queued or
scheduled runs, not streaming ones. Not seen failing in waves. If
retries run out, they now fail with the bounded cause. |
| Run settlement `accountRunCost` (`run-cost-accounting.ts`) | Follow-up
| Seen with `40001` in a wave log. Leaves the run pending;
`reconcileRunCosts` retries it durably. Multi-aggregate (coordinator,
receipts, images, closure). |
| Tool cost events (`paperclip-runner-tool-authority.ts`, `costs.ts`) |
Safe | Insert `cost_events` only; the run row is referenced, not locked
or updated. |
| Budget enforcement and evaluation (`budgets.ts`), billing
reconciliation, accounting-integrity repair | Safe | Lock policy,
invoice or company rows, not a streaming run row. Run reads are plain
selects. |
| Browser use, decisions, prompt-injection, fast responses | Safe |
Write their own invocation/reservation rows, not `heartbeat_runs`. |
| Read-only RR snapshots (`withAccountingReadSnapshot`, idle-sleep, tool
gateway, customer success, `routes/decisions.ts`) | Safe | Read only, or
no run-scoped writes. |

## Verification

- Red on the parent commit, green on this branch:
`server/src/services/run-accounting-contention.test.ts` (real embedded
PostgreSQL).
- `reserveRunBudget > reserves once when run events commit between its
company check and its run lock` (red: `40001`)
- `reserveRunBudget > reserves every run while back-to-back writers
update the run row` (red: `40001`)
- `reserveRunBudget > counts every committed hold, so successive
admissions never exceed the budget` (red)
- `reserveRunBudget > still refuses a run once the hard stop is reached`
(red: `40001` instead of `409`)
- `createRunUsageRecorder > binds its receipt source when run events
commit between its company check and its run lock` (red: `40001`; green
asserts the spooled receipt is journaled exactly once)
- `createRunUsageRecorder > binds while back-to-back writers update the
run row` (red: `40001`)
- `reserveRunBudget > never creates a second reservation for one run
when two dispatches race` (invariant; passes before and after)
- `final dispatch gate > admits and hands off a reserved run when run
events commit between its company check and its run lock` (red:
`AccountingContentionError` after 4 × `40001`)
- `createRunUsageRecorder > persists its capture-failure fence once when
run events commit between its company check and its run lock`
- `final dispatch gate > admits while back-to-back writers update the
run row` (red: `40001`)
- `exhausted retries > fail with a bounded, SQL-free cause that keeps
the SQLSTATE` (new contract)
- `run-completion.test.ts > saves a bounded cause and keeps the
retryable generic code when accounting retries are exhausted` (saved
code, message and `failureCause`, and log fields, through `failRun`)
- Related suites pass: `usage-receipts-read`,
`admission-independent-of-cleanup`, `budget-live-attempts`,
`idle-accounting-work`, `budget-enforcement-scoping`,
`closed-native-receipt-accounting`, `budget-evaluation-work`,
`run-dispatch/adapters/postgres`, `heartbeat/run-completion` (all 126
run-completion cases pass; its `afterAll` embedded-PostgreSQL teardown
exceeded 30 s on this loaded host, the known local flake).
- `pnpm --filter @paperclipai/server typecheck` passes.
- Latency of `reserveRunBudget` on the normal path (no contention,
embedded PostgreSQL, 350 measured reservations per run, 3 runs each):
before p50 14.7 / 15.0 / 18.6 ms, p95 25.6 / 31.4 / 39.4 ms; after p50
14.2 / 14.0 / 16.6 ms, p95 29.7 / 24.0 / 29.3 ms. No added latency: the
statements are the same, only the isolation level changes. Under
contention it removes the retry backoff.
- Live restart-chaos L1 waves (`--seed 20386 --db-pressure 4`, codex and
claude, K1):
- Master before this change: `w-20261110T050818-s20386`, 0/4. Every
first run failed in `reserveRunBudget` with `40001`.
- Head `90530789b0` (first three writers fixed):
`w-20261110T075857-s20386`, 2/4. Both codex tasks passed. Both claude
failures (t02, t03) trace to the dispatch gate, now fixed. Those runs
already showed the bounded cause, with no SQL on the board. The failure
was logged with `sqlState` 40001. No `reserveRunBudget` or recorder
`40001` remained.
- Head `d5ed21926f` (dispatch gate fixed): `w-20261110T090531-s20386`.
All 4 tasks reached done with no retry and R1–R6 passed. There were no
failed runs and zero `40001` in the server log, under 310k pressure
updates. Each task still failed the T-oracle `company_spend_settled`
with `pending_runs=1`. That pending run is t02's continuation run
`601ae80a`, cancelled before provider work as `issue_terminal_status`,
whose accounting debt was not yet settled when graded about 15 s later.
It is not a contention failure. Caveat: I edited the worktree during
this wave (`worktree_changed_mid_wave`), so a clean wave on the final
head follows.
- Final head `9f1c480717`, clean worktree: `w-20261110T094417-s20386`.
All 4 tasks reached done with no retry, and R1–R6 passed for each. There
were no failed runs and zero serialization failures in the server log,
under about 523k pressure updates. Each task still fails only the
T-oracle `company_spend_settled` (`pending_runs=1`). It is the same
cancelled pre-provider continuation run (`issue_terminal_status`), whose
accounting debt was still unsettled at grading. That is a separate
settlement-timing follow-up, not database contention. Master was 0/4 on
the same seed.

## Risks

- READ COMMITTED gives each statement its own snapshot. Admission reads
policy, scope and exposure in separate statements. The shared
company/project fences and the agent share lock exclude concurrent pause
and policy edits, and exposure stays one statement, so each read is at
least as fresh as the old snapshot.
- Concurrent admissions of different runs stay best-effort, as
`doc/SPEC-implementation.md` already states ("concurrent starts can
exceed the same observed balance"). This change does not widen that
window: each statement snapshot is newer than the old transaction
snapshot. It does not add a company mutex.
- A run that fails after exhausted retries now records
`accounting_contention` instead of `adapter_failed`. Retry rules that
key on `adapter_failed` spawn-like messages did not match this failure
before, so no automatic retry changes.
- No schema, migration or persisted-format change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with shell, file-edit and code-execution tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 03:47:48 -06:00
DottaandPaperclip 0991d9bda1 fix(workspaces): protect migration sources and add guarded recovery service (#20443)
## Thinking Path

> - Paperclip manages agents and their task workspaces.
> - Online migration saves files before it publishes remote workspace
authority.
> - A saved migration journal can survive the importing process.
> - Git cache expiration did not count that journal as a consumer.
> - Expiration could remove a source and change its fingerprint while
its import was incomplete.
> - This PR protects unfinished imports and exposes a guarded internal
service transition for an already affected cohort.
> - New imports can preserve the current sources while all old recovery
data stays retained.

## Linked Issues or Issue Description

**What happened?**

Git cache expiration could archive and remove a workspace after an
import process ended but before its saved cohort finished. The saved
source fingerprint then differed from the current archived workspace.
Exact cohort resume correctly refused that changed source. Reopening the
workspace cannot recreate the old metadata or fingerprint.

**Expected behavior**

An unfinished migration remains a durable consumer after process loss.
If an older server already expired some sources, one canonical
transition can fail the exact unpublished cohort without losing its
snapshots, objects, source history, or recovery obligations. New
attempts use the current verified sources.

**Steps to reproduce**

1. Start a cohort import of eight Git worktrees with local changes.
2. Interrupt it after four snapshot uploads.
3. Let an older server expire two idle source worktrees.
4. Resume the saved cohort. It refuses the changed archived sources.

Related: #20429 preserves exact cohort identity when independent
companion imports change only the scope proof. #20424 preserves prepared
recovery state for archived sources. This PR handles completed source
expiration, not a scope-only change. Public searches found related
expiration work and no matching compensation owner.

## What Changed

- Count saved migration markers and unfinished migration journals as Git
expiration consumers. Keep failed migration retention holds as consumers
too.
- Expose `supersedeExpiredRemoteWorkspaceMigrationCohort` as an internal
service API for a privileged operator with a real database handle and
reviewed company/cohort inputs. It is called directly by the
PostgreSQL/Git test and reviewed one-time administrative qualification.
It adds no CLI mode or HTTP route. The existing CLI still refuses a
damaged cohort; this PR does not claim automatic CLI recovery.
- Require an exact unpublished cohort of up to eight members. It
requires completed expiration and an unexpired restore recipe for each
missing member. The other members must retain their saved source
fingerprints.
- Recheck current source and recovery state under source, lifecycle,
finalizer, and row locks. One transaction fails all old journals, writes
audit records, and removes only their exact matched active markers.
- Keep old cohort proofs, operations, snapshots, and object references.
Genuine failure receipts use the existing failed-save retention hold.
This transition does not report migration success.
- Reuse the existing bounded source-lock queue when `sourceLockWaitMs`
is set. Keep fast refusal by default. Cancellation targets only this
waiter, and current checks still run after admission.
- Record predecessor IDs on new attempts and reserve a snapshot
generation above the existing ledger maximum. New success does not
retire the old hold.
- Document the owning transition and retention rules.

## Verification

- The two new journal-consumer cases failed under the old expiration
predicate and passed under the new predicate.
- The full new recovery case passed in 21.12 seconds with real
PostgreSQL locks, Git repositories, encrypted local object storage, and
archive restores. It covers four retained uploads, two real expirations,
unchanged old resume refusal, authority change refusal, transaction
rollback, a held native finalizer, idempotence, eight new imports,
dirty/untracked file restores, and a retention pass 30 days later with
no object deletion.
- Server TypeScript and module-boundary checks passed.
- On `7cec8c915acfad2d08874eaf26abaa852f560219`, the queued-admission
regression failed before forwarding the option (24.07 seconds), then
passed (22.25 seconds). It verifies a real held project lock,
cancellation that preserves the holder and all journals, release into
full compensation, retention, and eight restored fresh imports. Current
server TypeScript and module/diff checks passed. The only production
follow-up changes are the optional queue input and its existing-owner
call.
- The earlier `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c` CI run had one
Runner support failure: the unchanged credential-free Python/Pi
calibration exceeded its outer 25-second subprocess deadline. That job
passed 6,926 other tests and skipped four. The timeout is retained as a
nonpass. The changed logical head requires fresh CI and review; no test
deadline was raised.
- The affected cohort, Git expiration, and eligibility suites passed all
148 cases in 228.41 seconds on
`aa589f3f447e36d0277c32f8b44cbe5e84a673be`. Merged current master
`a9d0946e64`; the only conflict retained both independent added test
blocks. The complete merged cohort suite passed all 30 cases in 70.00
seconds on `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c`, including
upstream larger-cohort controls. Final-head server TypeScript,
incremental preflight, module-boundary, and diff checks passed.
Unchanged expiration/eligibility evidence remains valid. Fresh
`809ab704e619812d8f1bf38cf5de9c65893861cc` CI and review are pending.
- On `809ab704e619812d8f1bf38cf5de9c65893861cc`, the complete recovery
fixture passed in 21.38 seconds after adding one valid archived recipe
with no restore point. It refuses with
`migration_supersession_expiration_unverified` and keeps all workspaces,
journals, and snapshots unchanged. The same case then verifies queue
cancellation/release, full compensation, retained history, fresh
imports, and restored bytes. Server TypeScript passed. This follow-up
changes only the test and service-scope documentation; product bytes
remain identical to qualified
`7cec8c915acfad2d08874eaf26abaa852f560219`.
- Earlier fixture attempts did not pass. They exposed the selection of a
held source and an incorrect expected refusal string. The fixture now
selects eligible sources and asserts the canonical authority refusal.
Production guards were not relaxed.
- Live qualification of the exact owning product bytes completed in
230.419 seconds. All eight old operations became failed with auditable
supersession lineage. All eight old snapshot rows remained byte-for-byte
identical, and their object references remain retained. The existing
source-lock queue waited 1.151 seconds for a normal finalizer handoff.
The three complete source/recovery passes took about 218 seconds
including their transaction, which explains why the earlier 180-second
total budget was insufficient for that measured workload. No source
proof or publication guard was skipped. Compensation made no provider or
object-storage calls.
- Earlier live qualifications remain nonpasses: the 180-second attempt
did not finish; its instrumented successor completed source checks and
refused a busy source lock before mutation. The successful bounded-queue
attempt retained those failure records. Fresh archived/online imports
are a separate subsequent step and are not claimed complete here.
- The local checks do not claim browser acceptance, app deployment,
fleet enrollment, or a completed zero-downtime cutover.

## Risks

- This owner mutates migration bookkeeping. It preserves all-or-nothing
row comparisons and current source/recovery locks. A changed journal,
authority, writer, finalizer, expired recipe, or partial transaction
refuses the transition.
- Historical snapshots and objects remain pinned after new success. This
intentionally retains storage. Release requires a separate reviewed
action.
- An older importer cannot use the successor generation contract. Use
the containing canonical owner for recovery. Normal app delivery must
also carry the expiration guard; a private maintenance graph does not
patch a running server's cleanup worker.
- The migration owner crosses 1,000 lines. The bounded compensation
transition remains beside its private source inspection, fingerprint,
fence, reservation, and publication contracts. Moving it now would
require exporting or duplicating those safety owners. No unrelated
refactor or new framework is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source editing, Git, and
local execution tools. The exact serving model ID and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 21:02:02 -06:00
Dotta f73ff4ace9 fix(native): reprove stale filesystem preservation receipts (#20438)
## Thinking Path

> - Paperclip manages agent work and preserves saved native sessions
across releases.
> - Filesystem independence receipts bind database facts to preserved
controller and skill evidence.
> - A workspace migration can add run metadata without changing those
preserved bytes.
> - That change makes the saved facts digest stale. Full native
preservation then refuses the existing receipt, even after fresh source
checks.
> - WEB preservation already supports a full current proof and guarded
receipt refresh.
> - This pull request applies that pattern to full native receipts,
reuses the authenticated archive, and keeps the existing ownership and
destruction checks.
> - The benefit is a current receipt without a new archive or provider
replay.

## Linked Issues or Issue Description

Refs: #20322.

**What happened?**

Adding the canonical `legacyWorkspaceRecoveryMigration` marker to a
settled native run makes its saved receipt stale. The full native
preservation owner checks current evidence but then refuses the old
facts digest.

**Expected behavior**

The owner can refresh the proof metadata after it proves all current
facts and preserved bindings again. Changed evidence or authority must
leave the prior receipt intact.

**Steps to reproduce**

Use the added PostgreSQL regression. It saves a valid receipt for a
succeeded source with 453 checkpoint parts, adds the producer-shaped
migration marker, and calls preservation again. The unchanged owner
fails with `native_harness_filesystem_independence_unverified`.

## What Changed

- Reuse the prior controller archive through the existing authenticated
readback path.
- Require the old receipt to match every current immutable binding
before a facts refresh. Reconstruct its exact digest from unchanged
bindings and the prior proof fields.
- Recheck current database facts, assets, controller and local files
under the existing finalizer, selection, task and run fences.
- Publish with an exact prior-receipt comparison. Keep a valid
concurrent winner and refuse changed receipts or new owners.
- Audit the prior receipt digest and four proof fields. Add focused
source, integrity, race and DB-only validation tests, and update the
database guide.

## Verification

- Local Node24 canonical preflight and plugin SDK prerequisites passed.
The Runner build came from the content cache. After a normal merge of
fresh master, preflight confirmed unchanged build inputs.
- Genuine RED on `3040448da5`: the saved receipt is valid before the
migration marker and stale afterward. Reproof fails at the old receipt
guard (one failed, 102 intentionally not selected). Both retained RED
attempts reached that same guard.
- GREEN on final local merge `00f4b5d678083106afc88fb2f7e0588961b46d6d`:
19 PostgreSQL controls passed, 91 intentionally not selected (17.75s).
The wrapper completed cleanup.
- The positive preserves the same archive, controller bytes, assets,
local proof and all 453 checkpoint parts. It performs authenticated
object reads and zero archive PUTs. Runs, results, finalizers and leases
remain unchanged except for the receipt. The audit reconstructs the
exact prior receipt.
- DB-only validation passes with the original home, state directory and
volume absent, while storage reads throw.
- Refusal and race controls cover changed asset bindings, raw controller
bytes, missing destruction evidence, unknown receipt fields, a valid
concurrent winner, changed receipts and a late active owner. Existing
WEB refresh, physical finalization-lock and saved-binding controls also
pass.
- Server TypeScript passed, including the adjacent integration test.
Fresh Greptile is 5/5 on exact head
`00f4b5d678083106afc88fb2f7e0588961b46d6d`, with no review threads
([review](https://github.com/paperclipai/paperclip/pull/20438#issuecomment-6756548830)).
Final-head CI is terminal: 56 success, 4 skipped and 1 neutral. The
aggregate `ci / verify` passed
([run](https://github.com/paperclipai/paperclip/actions/runs/42737704110)).
- The first Runner-support job had one unrelated process-inspection
nonpass: `server-stop.test.ts` expected a successful wrapper close, but
its bounded process inspector returned no table. Its cause remains
unknown. The unchanged test, process helpers, workflow and dependency
inputs match the passed master control. One authorized same-head job
retry passed all 328 files / 6,927 tests (4 skipped); the exact
previously failed case passed in 665ms
([retry](https://github.com/paperclipai/paperclip/actions/runs/42737704110/job/129576420893)).
No timeout, assertion or source change was made for the retry.

```sh
node scripts/agent-preflight.mjs --skip-rust-debug
pnpm --filter @paperclipai/plugin-sdk ensure-build-deps
node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts -t "reproves stale native facts|retains stale native proof|guards stale native reproof|invalidates a published receipt after changed|actual physical-finalization lock|refreshes stale WEB facts|refuses a WEB facts refresh|retains or refuses the existing receipt when an earlier capture finishes late"
pnpm --filter @paperclipai/server exec tsc --noEmit
```

## Risks

- Old receipt schemas and immutable bindings remain required. Full fresh
proof carries compatible saved receipts forward; unknown fields or
changed evidence refuse refresh and preserve the old receipt.
- Full native preservation still requires canonical destroyed-provider
receipts. The change retains the existing source capability,
cancellation, finalizer and publication checks.
- Only proof metadata changes. No provider operation, archive
replacement, hosted recovery or deployment was performed for this PR.
Live qualification remains with the operator after delivery.

## Model Used

OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does
not expose a more specific model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 20:05:37 -06:00
DottaandPaperclip 3040448da5 fix(workspaces): refresh saved cohort source proofs on resume (#20429)
## Thinking Path

> - Paperclip preserves agent work when a control-plane server is
replaced.
> - Online workspace imports save exact cohort, operation and snapshot
identities for restart.
> - A cohort proof also includes the complete current companion source
scope.
> - An independent companion import removes that companion from the
local scope.
> - The saved cohort then fails resume even when every saved member is
unchanged.
> - This pull request rechecks the current scope under fresh locks and
refreshes its proof for all members in one transaction.
> - The benefit is restartable online migration while independent work
progresses.

## Linked Issues or Issue Description

Refs: #19873, #20394, #20418. Related PR search found the original
cohort owner, recovery-proof refresh and separate child-verification
reuse. This fix addresses cohort scope freshness.

**What happened?**

An interrupted cohort rejects resume with
`workspace_cohort_membership_changed` after a nonmember companion is
migrated. The saved source proof includes companion workspace and task
IDs. The exact saved members and their fingerprints can remain
unchanged.

**Expected behavior**

The same saved cohort must resume after fresh admission. It must keep
all member, operation and snapshot IDs. Changed members, active owners,
changed journals and lost source locks must still refuse publication.

**Steps to reproduce**

1. Start a two-member cohort and interrupt the second upload after the
first archive is saved.
2. Import an archived nonmember companion through the existing canonical
migration owner.
3. Resume the same two-member cohort. The original implementation
refuses before it acquires the source fence.

**Paperclip version or commit**

Reproduced on master `3a5572c9e5`.

**Deployment mode**

Online administrative workspace migration with PostgreSQL and
authenticated snapshot storage.

## What Changed

- Admit a changed aggregate scope proof only when every saved member has
a running journal and its current fingerprint matches both saved
fingerprints.
- Keep partial reservations on the original strict aggregate-proof path.
Keep exact cohort and operation membership checks.
- Run the existing source, lifecycle, recovery and finalizer admission
before refresh. Lock member rows, inspect the current scope again, and
compare every journal in one transaction.
- Change only `cohortMigration.sourceProof`. Preserve all other
metadata, identities, archive and recovery records. Abort the complete
refresh on cancellation, a changed journal or a lost fence.
- Extend the existing real PostgreSQL fixture with partial upload,
canonical companion migration, refusal controls, transaction rollback,
cold archive restore and replay.
- Document the proof lifetime and resume boundary.

## Verification

- Canonical Node24 worktree preflight passed. Plugin SDK build
prerequisites passed.
- Genuine RED on unchanged product: the real partial-upload and
companion-import sequence fails with
`workspace_cohort_membership_changed` (1 failed; 22 intentionally not
selected; 10.44 seconds).
- The identical positive passes after the fix (1 passed; 22
intentionally not selected; 9.87 seconds).
- The complete focused cohort file passes: 23 tests in 33.33 seconds.
Existing controls include changed member fingerprints, foreign and
completed journals, active operations, peer cancellation, actual
source-fence connection loss and outer-commit failure.
- The new sequence also checks a held source lock, a held finalizer
lock, cancellation before refresh, and an injected second-member update
failure. The failed update rolls back every proof change. Both saved
archives restore tracked and untracked work, and replay uses the same
cohort.
- After one final read-only preflight assertion, that complete new
sequence passes again in 14.52 seconds with 22 existing cases
intentionally not selected. Server TypeScript passes on the final files.
Module boundaries and `git diff --check` pass.
- Commands: `pnpm exec vitest run
server/src/services/remote-workspace-cohort-migration.test.ts`; final
case selection: `-t 'same uploaded cohort'`; `pnpm --filter
@paperclipai/server exec tsc --noEmit`; `node
scripts/check-module-boundaries.mjs`.
- Initial local setup attempts remain nonpasses: sandbox PostgreSQL
could not bind localhost, then the new archived companion fixture used a
noncanonical repository folder. The corrected fixture uses the existing
managed repository path. Neither setup failure is counted as the product
RED.
- Fresh [Greptile
review](https://github.com/paperclipai/paperclip/pull/20429#issuecomment-6755862789)
is 5/5 on exact head `6c640c4667b136e5416c1375f491729cf4bb9b49`, with
zero inline threads. Required final-head CI is running. Hosted
acceptance is not claimed by this PR.

## Risks

- The old aggregate proof contains only a hash. Refresh therefore
requires durable fingerprints for every saved member and repeats all
current admission guards. A partially reserved cohort cannot use this
path.
- A stale proof must never hide a changed member or a new writer.
Current source inspection, P/I/L/F locks, locked member rows, complete
journal compare-and-swap and the existing capture and publication checks
remain authoritative.
- The all-member update must stay atomic. A database fault or
cancellation rolls it back. The original cohort, operation and snapshot
IDs remain fixed.
- The change adds one bounded transaction only when a fully saved
cohort's scope proof changes. It adds no recurring work, schema,
provider, archive format, concurrency or deployment changes.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The runtime does not expose a more specific deployment identifier or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 19:16:34 -06:00
DottaandClaude Opus 5.5 347d85fef3 perf(chat): index GitHub review lookups and share review-wait reads (#20400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub chat endpoints let agents review pull requests. The server
reads review state when it lists issues and when it updates GitHub
checks.
> - On a hosted instance, two GitHub review queries used the most total
database time. One query ran about 660,000 times at about 33 ms. The
other query ran about 10,000 times at about 1 s.
> - The first query finds a source comment's chat link. No index covers
`chat_message_links.comment_id`, so each call reads the full table.
Issue lists also read the same review waits two times per request.
> - The second query joins deliveries, links and runs on a JSON key. The
planner walks the agent's run history in creation order and reads each
run's context.
> - This pull request adds four concurrent indexes, splits the run
lookup into two small reads, and shares review-wait reads in one
issue-list request.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
GitHub review state reads. These are the inherited review-wait lookup by
source comment, and the review-check lookup of the run that a delivery's
comment woke.

**Subsystem affected**
Server chat channels (GitHub review waits and checks), issue list
attention maps, and the database schema.

**Current behavior**
- `readInheritedGitHubReviewWait` filters `chat_message_links` by
`company_id`, `comment_id` and `direction`. No index has `comment_id`,
so PostgreSQL reads the full table on each call. The run-origin check
does the same lookup first.
- An issue list calls the external review-wait read from the review
attention map and again from the blocked-inbox attention map. The two
maps read the same in-review issues.
- `githubReviewCheckState` joins deliveries, links and runs with
`context_snapshot->>'wakeCommentId' = comment_id::text`, sorts by
`created_at desc` and takes one row. The planner reads the agent's runs
in creation order and reads each run context. If no run matches yet, it
reads all of them. The delivery filter on `normalized_event` and the
link join on `delivery_id` also have no index.

**Proposed behavior**
The same rows come back from small index reads. Each issue's review wait
is read one time per issue-list request.

**Reason and benefit**
These two statement families used about 33,000 s of database time in one
cumulative statistics window. The database had about 50 concurrent
active queries, and request reads waited behind this background work.

**Breaking changes**
None. The migration adds indexes only.

## What Changed

- Migration `0456_chat_review_lookup_indexes` builds four indexes with
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It runs through the existing
top-level statement path, which repairs an invalid index on retry:
- `chat_message_links_company_comment_idx` on `(company_id, comment_id)`
where `comment_id is not null`.
- `chat_message_links_delivery_idx` on `(delivery_id)` where
`delivery_id is not null`.
- `chat_deliveries_github_automatic_delivery_idx` on `(endpoint_id,
normalized_event->'githubAutomatic'->'context'->>'deliveryId')` where
that value is not null.
- `heartbeat_runs_company_wake_comment_idx` on `(company_id,
context_snapshot->>'wakeCommentId')`. This index is not partial. The
planner does not use expression statistics from a partial index. Without
these statistics, it estimates many matches and keeps the creation-order
walk.
- `githubReviewCheckState` reads the delivery's comment IDs first. If
there is a comment, it reads the newest same-agent run with one of these
`wakeCommentId` values. If there is no comment, it skips the run read.
The comparison stays text equality on `comment_id::text`.
- `createExternalReviewWaitReader` shares one read per issue between the
review and blocked-inbox attention maps of one issue-list request. The
reader exists only for that request.
- `doc/DATABASE.md` describes the indexes and the new read order.
- `scripts/benchmark-review-lookup-indexes.ts` compares the pre-fix and
new code with and without the indexes.
`scripts/benchmark-review-lookup-index-migration.ts` times the migration
on a large synthetic database.

## Verification

Local SQL measurements on a synthetic fixture. They are not production
timings. Environment: embedded PostgreSQL 18.1, baseline revision
`02bfe71a6b`. Fixture: 50,000 links, 5,000 deliveries with about 2 KiB
events, and 4,000 runs with about 12 KiB incompressible contexts. The
wake run is in the middle of the run history. Each value is the median
server execution time from `EXPLAIN ANALYZE` (2 warmups, 9 samples). The
order was: no indexes, indexes, then no indexes again.

| Caller and statement | Before (no indexes, two passes) | After |
Shared buffers before → after |
| --- | --- | --- | --- |
| Inherited review wait, link by source comment (same SQL) | 2.72 / 2.36
ms | 0.076 ms | 929 → 12 |
| Review check, wake run found | 31.9 / 21.9 ms | 0.29 ms (two
statements) | 25,776 → 62 |
| Review check, no wake run yet | 71.0 / 47.5 ms | 0.06 ms (two
statements) | 34,861 → 12 |

- Indexes alone do not fix the review check. The old SQL with the new
indexes took 8.6 ms (run found) and 29.7 ms (no run), because it still
walked the run history. The new SQL without the new indexes was neutral
(23.5 and 35.1 ms).
- Service wall times on this shared machine were noisy, so the table
reports server execution time.
- Correctness: the benchmark fails unless the inherited statement
returns the same positional rows in each index state. The check service
result and the final wake-run rows of the new code must also match the
pre-fix module for both cases.
- Migration timing, normal migration runner, synthetic data: 200,000
runs (1.8 GB with 8 KiB incompressible contexts), 500,000 links and
100,000 deliveries. First run 4.2 s; rerun 0.38 s. Separate builds:
links by comment 1.5 s, run wake comment 1.1 s, deliveries 0.7 s, links
by delivery 0.3 s. All indexes were valid. The run index was 8.3 MB.
- New tests:
- `server/src/__tests__/github-review-check-state.test.ts` checks that
the newest same-agent run is selected. It also checks that other agents,
other companies, uppercase IDs, non-string values, list-only
`wakeCommentIds`, comment-less links and unknown deliveries do not
match. It checks that the generated statements can use the new indexes.
- `server/src/services/native-runtime/external-review-wait.test.ts`
checks that the reader reads each issue one time, keeps concurrent
callers correct, and rejects all waiting callers when a read fails.
- Red/green: with the pre-fix `review-check-state.ts`, the behavior test
passes and the index test fails. With this change, both pass.
- Focused runs pass: `issue-blocker-attention`,
`heartbeat-issue-liveness-escalation`, review-wait, review-check-state,
external-review-wait, review-repair (106 tests), and
`external-chat-wait.integration` (285 tests). `pnpm --filter
@paperclipai/server typecheck` and `pnpm --filter @paperclipai/db
check:migrations` pass.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-indexes.ts --baseline-ref=02bfe71a6b
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-index-migration.ts
```

## Risks

- The migration builds four indexes concurrently. It does not block
writes. On a large database, the run index reads every run context one
time. The local run took 1.1 s for 1.8 GB from a warm cache. Cold
storage on smaller compute can be slower.
- The run index adds one entry for each run insert and for each run
update that is not a heap-only update. Many runs have no
`wakeCommentId`, so these entries hold a null key.
- The review check now sends two statements instead of one when a
delivery has a comment. Each statement reads only a few index entries.
- When two matching runs have the same `created_at`, either can be
selected. The previous query had the same tie behavior.
- This change does not change the number of in-review issues that one
issue list reads. The blocked-inbox map still reads all in-review issues
of the company. A smaller scope needs a separate change to the liveness
classifier.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:47:34 -06:00
DottaandClaude Opus 5.5 0116e45958 perf(db): index pending budget Stops instead of scanning all run results (#20391)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget enforcement stops agent runs when a budget hard limit is
reached. The stop intent is saved in each run's `result_json` until run
control delivers it.
> - The budget worker, run control, and the idle-sleep check find these
pending stops with a predicate on `result_json`. No index supports that
predicate.
> - Each check therefore reads every heartbeat run and its large result
JSON from TOAST. On a hosted database, this statement family was among
the most expensive: about 8,000 calls at about 1 s each, about 9,800 s
in total.
> - Pending stops are rare. The predicate is constant, so a partial
index can contain only those rows.
> - This pull request adds that partial index. The SQL that the server
sends does not change.
> - The benefit is that each check reads a few index entries instead of
the full table. The database has more capacity for request work.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The check for committed budget Stops that are not delivered yet. The
budget worker runs it each time it looks for work
(`hasPendingBudgetEnforcement`), and again for each delivery pass
(`listPendingBudgetCancellationScopes`). Run control and the idle-sleep
check use the same predicate.

**Subsystem affected**
Budget enforcement in the server
(`server/src/services/heartbeat/budget-cancellation.ts`,
`server/src/services/budgets.ts`,
`server/src/services/heartbeat/run-control.ts`,
`server/src/services/idle-sleep-safety.ts`) and the `heartbeat_runs`
schema in `packages/db`.

**Current behavior**
The query groups all rows of `heartbeat_runs` and reads
`result_json->'startupCancellation'->'budgetEnforcement'` for each row.
A limit of 1 does not help, because the group and sort occur before the
limit. Cost grows with the run history, not with the number of pending
stops.

**Proposed behavior**
A partial index, `heartbeat_runs_pending_budget_cancellation_idx`,
contains only runs with a pending budget Stop. The planner uses it for
all four callers. The results are the same.

**Reason and benefit**
Local SQL measurements show a 300x to 1,000x decrease in time and buffer
reads (see Verification).

**Breaking changes**
None.

## What Changed

- Added the partial index
`heartbeat_runs_pending_budget_cancellation_idx` on `heartbeat_runs
(company_id)`. Its predicate is identical to
`pendingBudgetRunCancellationPredicate`.
- Added migration `0455_pending_budget_cancellation_index.sql`. It uses
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`, and it is in the list of
migrations that run outside a transaction, as `0436` is.
- Moved the scoped run-control read and the idle-sleep check into
exported builders in `budget-cancellation.ts`, next to the scope list
query. The SQL does not change. The callers and the test now share the
same statements.
- Added a real PostgreSQL regression test. It checks that delivered,
malformed, and missing intents are not pending. It also EXPLAINs the
statements that the callers send (scope list with limit 1 and 100,
scoped read, `EXISTS` check) and requires the new index in each plan. If
a person changes a query shape, the server predicate, or the index
predicate, the test fails.
- Added `scripts/benchmark-budget-cancellation-scan.ts`. It captures the
SQL that the real service sends, then compares the plans, timings, and
complete rows with and without the index.
- Updated `doc/DATABASE.md`.

## Verification

Red/green regression test:

- Before the migration, the new test failed: the plan was `Seq Scan`
with the predicate as a filter.
- After the migration: `npx vitest run
src/services/heartbeat/budget-cancellation.test.ts` passes (17 tests).
- Also passed: `src/__tests__/budgets-service.test.ts`,
`src/__tests__/idle-sleep-diagnostics-postgres.test.ts`,
`src/services/budget-resume-continuation.test.ts`.
- `pnpm --filter @paperclipai/db check:migrations`, and typecheck of
`@paperclipai/db` and `@paperclipai/server` pass.

Performance and result equality (local SQL measurements only):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-cancellation-scan.ts --runs=20000 --payload-kib=16 [--reverse]
```

- Fixture: embedded PostgreSQL 18.1. 20,241 runs in 2 companies, with
incompressible result JSON of about 16 KiB (table 379 MB). 40 pending
stops in 6 scopes, 200 delivered or malformed decoys, and 1
cross-company intent.
- The SQL is the same in both treatments. The script toggles only the
new index. 2 warmups and 7 samples per query. Rows are compared outside
the timed part. Ordered queries must match in order. The scoped
run-control read has no `ORDER BY`, so it is compared as a multiset.

| Query (caller) | Baseline median | With index median | Shared buffers
|
| --- | --- | --- | --- |
| `listPendingBudgetCancellationScopes(db, 1)` (worker `hasPending`) |
688 ms | 2.1 ms | 243,500 → 1,158 |
| `listPendingBudgetCancellationScopes(db)` (delivery) | 799 ms | 1.4 ms
| 243,500 → 1,158 |
| Scoped pending runs (run control) | 357 ms | 1.1 ms | 121,835 → 322 |
| `EXISTS` pending (idle-sleep check) | 440 ms | 0.37 ms | 240,560 → 2 |

- Reverse order (index first) gave the same result: 524 ms → 1.6 ms, 464
ms → 1.4 ms, 306 ms → 1.0 ms, 306 ms → 0.42 ms.
- Plans change from `Seq Scan` (or a broad bitmap scan) to `Index Scan`
or `Index Only Scan` on the new index.
- All result rows are equal in every run.

Migration duration:

- 20,241 runs (379 MB): index build 2.0 s to 3.9 s.
- 200,241 runs with 8 KiB result JSON (2.1 GB): index build 5.6 s. With
this table size, the four queries change from 1,148 / 879 / 514 / 4,392
ms to 1.1 / 1.2 / 0.85 / 8.3 ms.
- The build is concurrent, so it does not block run writes.

## Risks

- Migration: the build reads every run's result JSON one time. It is
concurrent and resumable (`IF NOT EXISTS`). The executor repairs an
invalid concurrent index on retry.
- Write cost: run writes evaluate the index predicate. The index
contains very few rows, so the extra cost is small.
- Predicate drift: if a later change edits only one of the two
predicates, the planner cannot use the index. The new regression test
detects this.
- The hosted timings are not measured yet. The local numbers come from a
synthetic fixture.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:14:52 -06:00
DottaandClaude Opus 5.5 ab80fea62f perf(recovery): probe run bindings per comment in the lost-wake check (#20412)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciler finds user comments whose wake a restart
lost, and it saves a wake for them (plan invariant 4)
> - Its lost-wake check matched a task's runs with one OR across the
legacy context ID and the native issue column
> - With that OR, PostgreSQL can join runs only by company. It then
tests each recent comment against every run of its company and reads
each run's context
> - On a hosted staging database, this statement used about 23 s per
call on average (384 calls, about 8,900 s in total). One call ran for 35
s. Other requests waited behind it
> - This pull request reads the check in three stages and probes the two
run bindings separately with existing indexes
> - The benefit is the same result with bounded, index-driven work for
each recent comment

## Linked Issues or Issue Description

**What happened?**

The reconciler's user-comment lost-wake check was one of the most
expensive statements on a hosted staging database. Query statistics
showed 384 calls with a mean of about 23 s. One active call was seen at
35 s. The database compute was busy, and short authentication reads
waited behind long statements.

The statement matched runs with `(context_snapshot->>'issueId' = issue
OR native_issue_id = issue)` in three subqueries. The OR gives
PostgreSQL no join key except `company_id`. A local plan on synthetic
data showed the cost: a hash join of 273 comments against all 38,002
runs, which took 1.0 s of a 1.3 s statement. The work grows with recent
comments multiplied by the company's run count, and each test reads a
run context.

**Expected behavior**

Each recent comment uses a small number of indexed probes. The check
returns the same comments in the same order, with the same limit and
grace logic.

**Steps to reproduce**

1. Seed a company with a deep run history and a few hundred recent user
comments.
2. Let some recent comments have no wake (for example, after a restart
lost the wake of a steered comment).
3. Run the check. PostgreSQL can pick a hash join on `company_id` with
the OR as a join filter. With nested loops off, the old statement takes
seconds. The local benchmark below shows this.

**Paperclip version or commit**

Baseline: `02bfe71a6b`.

**Deployment mode**

Server with PostgreSQL. Local proof uses disposable PostgreSQL 18.1.
Production and staging use hosted PostgreSQL.

Related: the earlier change that split native and legacy bindings for
other ownership reads (see the "Execution recovery query performance"
section in `doc/DATABASE.md`). A search found no open pull request for
this check.

## What Changed

- `checkUserCommentsWithoutWake` reads in three materialized stages:
1. Recent user comments on assigned, non-conversation tasks that no wake
request carries. This stage keeps the same issue, status and
agent-closed rules.
2. Of those, comments with no live or later run and with at least one
earlier run.
3. Of those, comments that no live run acknowledged in an item event
(the steer check). Only this stage reads event payloads.
- Each run lookup is two probes: one on the legacy context ID and one on
the native issue column. Both use existing indexes. `exists (A or B)`
becomes `exists (A) or exists (B)`, which is logically the same.
- Each probe ends with `offset 0`. This keeps it a per-row index lookup.
Without it, PostgreSQL can build a hashed subplan over all runs, or a
hash join over all item events of the company. The repository already
uses this fence in `execution-blocker.ts`.
- The final `order by c.created_at limit N`, the projection, the grace
window and the reported reason do not change.
- Add `scripts/benchmark-comment-wake-check.ts`. It loads the check from
an explicit baseline commit and from the working tree, and it compares
the complete service result and the complete positional SQL rows before
it records time.
- Add a regression test for each run binding on its own: native only,
legacy only, live run, later run, steered run, an acknowledgement before
the comment, a log-only mention, a run that ended before the comment,
and a run in another company.
- Document the check and the benchmark command in `doc/DATABASE.md`.
- No schema change, index, migration, or new startup work.

## Verification

- `pnpm exec vitest run
server/src/__tests__/ownership-reconciliation-report.test.ts
--maxWorkers=1`: 34 passed. This includes the existing lost-wake,
restart, steered and agent-closed tests and the new binding test.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-comment-wake-check.ts --baseline-ref=02bfe71a6b
--samples=7`

Benchmark setup: PostgreSQL 18.1 (embedded, local), `shared_buffers` 128
MB, `work_mem` 4 MB. All data is synthetic, with md5-hex filler so
contexts and payloads do not compress well. There are two companies per
fixture. Two warmup rounds, then 7 timed samples for each arm, in
alternating order. Each sample runs the real check function. Before any
timing, each fixture asserts that the complete check result and every
positional SQL row are equal for baseline and treatment, and that the
fixture reports at least one owed comment. Every timed sample also
asserts the same result.

| Fixture | Runs / recent comments / owed | Baseline median | Treatment
median | Speedup | Hash-join plan, baseline → treatment |
| --- | --- | --- | --- | --- | --- |
| small | 2,541 / 50 / 6 | 8.5 ms | 2.6 ms | 3.2x | 19 ms → 3 ms |
| representative | 75,502 / 750 / 75 | 26.4 ms | 23.3 ms | 1.1x | 2,262
ms → 17 ms |
| lost steer wakes | 38,002 / 750 / 75 | 454.7 ms | 66.9 ms | 6.8x |
2,717 ms → 68 ms |
| deep history | 150,750 / 1,125 / 114 | 26.9 ms | 18.8 ms | 1.4x |
4,006 ms → 34 ms |
| busy hour | 128,750 / 5,625 / 200 (limit) | 79.2 ms | 47.4 ms | 1.7x |
20,047 ms → 75 ms |
| long steered runs (stress) | 19,100 / 600 / 60 | 626.4 ms | 457.4 ms |
1.4x | 5,871 ms → 502 ms |

- The table above used baseline `02bfe71a6b`, the branch base at that
time. After the rebase, a repeat run with baseline `8004ad069e` (the new
merge base, with the helper guard fixed to run from the repository root)
gave the same picture on a busier machine: small 15.4 → 9.1 ms, lost
steer wakes 1,733 → 139 ms, busy hour 103 → 65 ms, all with equal
results.
- The "lost steer wakes" fixture shows the slow plan with default
planner settings. PostgreSQL hashed all runs by company and tested 273
comments against 38,002 runs.
- The last column runs each statement once with `enable_nestloop = off`.
This makes the company-only hash join that the old statement allows
visible, and it shows that the new statement has no such plan. The
result is checked as equal in this mode too. Staging's 23 s mean matches
this growth: recent comments multiplied by the company's run count.
- Shared buffers fell in every fixture except "small" with default
settings (for example, busy hour 176,416 → 89,803).
- Neutral path: in the "long steered runs" stress fixture, every steered
comment lost its wake. These comments correctly reach the steer check.
That check must read the payload of each item event after the comment in
the run that was live, so it is the remaining cost. In normal operation
a comment sent to a live run has a deferred wake that carries it, so
stage one removes it.

## Risks

- Low risk. The predicate is the same conjunction of conditions. Only
the evaluation order and the probe shape change, and the benchmark
asserts complete-row equality on six fixtures and in both planner modes.
- Rows with the same `created_at` have no defined order, as before. The
benchmark uses distinct times so positional comparison is meaningful.
- `offset 0` stops PostgreSQL from turning these probes into joins. If a
task someday has a very large number of runs, each probe still uses the
company and issue indexes.
- Upgrade safety: this change reads saved run, wake and event state
only. It changes no persisted shape, run identity, or saved session, so
state saved by the previous release carries forward unchanged. A cutover
test is not applicable.
- Not verified on the hosted database. The local proof uses synthetic
data, and I did not capture a hosted plan. Confirm with a fresh query
statistics window after deployment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing, local PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:12:02 -06:00
DottaandClaude Opus 5.5 c985541a57 perf(db): index retry-exhaustion receipts for the attention scan (#20386)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The attention feed shows runs that failed after automatic retries
stopped
> - To find these runs, the feed reads every `heartbeat_run_events` row
of the company and filters the rows by a message prefix
> - Run output makes this table very large, so each feed request does a
large scan
> - On a hosted test database, this one statement used the most total
database time: 923 calls at a mean of 5 s (4,709 s in total)
> - Request authentication waits behind this background work, so all
users feel the load
> - This pull request adds a small partial index that holds only the
retry-exhaustion receipts
> - The benefit is that the same query reads a few index entries instead
of all company events

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The attention feed query that finds retry-exhausted runs
(`listAttentionExhaustedRuns`).

**Subsystem affected**
Server attention feed and the `heartbeat_run_events` table schema.

**Current behavior**
The query reads all company rows in `heartbeat_run_events`, filters them
by `event_type = 'lifecycle'` and the `Bounded retry exhausted` message
prefix, and then joins the few matches to failed or timed-out runs. The
cost grows with all run output. It does not grow with the number of
exhausted runs.

**Proposed behavior**
A partial index holds only the matching receipts. The query plan reads
the index and joins a few rows. The query text and the result do not
change.

**Reason and benefit**
The scan was the most expensive statement by total time on a loaded
hosted database. A local reproduction shows a 54x faster query and about
116x fewer buffer reads.

**Breaking changes**
None. The API, the feed rows, and their order do not change.

## What Changed

- Add the partial index `heartbeat_run_events_retry_exhausted_idx` on
`(company_id, run_id, id DESC)`. The index covers only rows where
`event_type = 'lifecycle'` and `message like 'Bounded retry
exhausted%'`.
- Add migration `0454`. It uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`. The migration runner runs it as a top-level statement, the same
as migrations 0432, 0436 and 0450.
- Add a regression test. It pins the row order (newest run first, then
the newest receipt on a tie) and the latest receipt per run. It also
checks that the query plan can use the new index. If the query predicate
changes so that the index no longer applies, the test fails.
- Add `scripts/benchmark-exhausted-run-attention.ts`. It measures the
same query with and without the index, and the time to build the index.
- Document the index in `doc/DATABASE.md`.

## Verification

Cause and thesis. The scan cost comes from reading all company events,
not from the few exhausted runs. Thesis: an index that holds only the
receipts makes the query cost depend on the number of receipts, and
keeps the result identical. The benchmark below confirms the thesis.

Rejected alternative without a migration. I first tried a LATERAL lookup
that starts from the failed and timed-out runs. On the same fixture it
was about 2x slower (median 379 ms against 194 ms). It reads the full
event history of each failed run with random I/O, and that costs more
than one sequential scan.

Local SQL benchmark (the SQL that the service generates; same SQL with
and without the index):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-exhausted-run-attention.ts --runs=30000 --events-per-run=100 --samples=9
```

Fixture (synthetic): one company with 30,000 runs (3,819 failed or timed
out) and 3.0 M events, plus a decoy company with 0.5 M events. The table
is 1.4 GB. About 5% of events are `lifecycle` events. Events of 20
concurrent runs are interleaved. 1 in 23 runs has a receipt, some runs
have two receipts, and output decoys contain the prefix. Two agents are
terminated. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon with different compute, so the absolute times are local
SQL measurements only.

The script runs two cycles (without index, then with index; then with
index, then without index) to expose cache bias. Each measurement has 2
warmups and 9 samples.

| Query | Without index | With index | Samples and spread | Output check
|
| --- | --- | --- | --- | --- |
| `listAttentionExhaustedRuns` | median 157 ms (96–1,271 ms) | median
3.3 ms (2.2–5.7 ms) | 18 each | 133 rows, positional rows equal in all 4
measurements |
| Buffers (EXPLAIN ANALYZE) | 127,743 blocks | 5,580 blocks | | plan
uses the new index (asserted) |

Speedup: 48x (median), on head `ae6037b2a0` (the same SQL and index as
the final head). The machine was under heavy load from other work, so
the spread without the index is wide. An earlier run on the first head
gave 248 ms against 4.6 ms (54x). Index build (`CREATE INDEX
CONCURRENTLY` from the migration file): 1.7 s and 2.5 s for the 1.4 GB
table. Index size: 152 KiB.

Review fix: the query now writes `event_type = 'lifecycle'` as a SQL
literal, not a bound parameter. With prepared statements, PostgreSQL can
reuse a generic plan. A generic plan cannot prove that a bound parameter
matches the index predicate, so it would not use the index.

Tests:

- `cd server && npx vitest run src/__tests__/attention-service.test.ts`:
44 passed. The new test checks a fresh plan and a forced generic plan
(`plan_cache_mode = force_generic_plan` with `PREPARE`). It failed with
the bound `event_type` parameter and with a changed message prefix
(confirmed red), and it passes with the fix. Locally, the file's
`afterAll` database cleanup hook timed out on a heavily loaded machine
(load average about 28). The timeout also occurs when the new test is
excluded.
- `pnpm --filter @paperclipai/db check:migrations`: passed.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`: passed.
- `pnpm verify`: see the CI results for this head.

## Risks

- Migration time. A concurrent index build reads the full
`heartbeat_run_events` table and does not block writes. Locally, the
build took 1.2–2.5 s for 1.4 GB. The time grows linearly with the table
size. I did not time the build on a production-size clone. On a database
with a 5 s full company scan, I estimate the build at tens of seconds,
which is in the 1–2 minute budget. This estimate is not proven.
- If the build stops, PostgreSQL can leave an invalid index. The
migration runner repairs invalid concurrent indexes when it runs again,
and `IF NOT EXISTS` makes a rerun safe.
- The index predicate must stay textually equal to the query predicate.
If someone changes the message prefix or binds a predicate value as a
parameter, the planner cannot use the index. The new test catches both
cases, for fresh and generic plans.
- Each new receipt writes one more small index entry. Receipts are rare,
so the write cost is very small.
- The hosted improvement is not verified until deployment. After
deployment, compare a new query-statistics window for this statement.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL). Standard
context window and reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:37:20 -06:00
DottaandClaude Opus 5.5 c57cd1bdb9 perf(server): read plugin lifecycle inbox heads in one pass (#20405)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins get agent, project and agent runtime changes from the
`resource_lifecycle_events` journal. Each plugin acknowledges events one
resource at a time.
> - The inbox page query checked each unacknowledged event against all
older events of the same resource. Each check walked the acknowledged
history of that resource.
> - Agent runtime streams get a new event for each configuration change,
so one resource can have thousands of acknowledged events. The cost grew
as pending events times acknowledged history.
> - On a hosted instance, three copies of this read ran for 601 seconds
each, and one earlier copy ran for 19 minutes. The delivery worker runs
it again after each acknowledgment.
> - This pull request selects the head of each resource in one pass and
keeps the same result.
> - The benefit is that the page read is linear in the company journal
size. In local tests it is 250x to 660x faster, and the result rows are
equal.

## Linked Issues or Issue Description

**What happened?**
The plugin lifecycle inbox page query (`select "id", "company_id",
"resource_type", "resource_id", "action", "created_at" from
"resource_lifecycle_events" where ...`) ran for 601 seconds on a hosted
database. Three copies were active at the same time.

**Expected behavior**
A lifecycle page read must stay fast when a resource has a long
acknowledged history and a pending backlog.

**Steps to reproduce**
1. Give a company 40 agents with 1,500 `agent_runtime` events each.
2. Acknowledge the first 1,200 events of each agent for one plugin.
3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100,
undefined, ["agent", "project", "agent_runtime"])`.
4. The read takes about 10 seconds for 60,000 events of that company. At
200,000 events it takes about 6 minutes.

**Paperclip version or commit**
`02bfe71a6b` (master).

**Deployment mode**
Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18.

## What Changed

- `server/src/services/plugin-lifecycle-inbox.ts`: `list` now selects
one head per resource with `DISTINCT ON (resource_type, resource_id)`.
The head is the unacknowledged creation, else the oldest unacknowledged
event. Then the query applies `afterId`, id order and the limit. This is
the same set that the old `firstForResource` predicate admits.
`acknowledge` keeps the old predicate for its single-event check.
- `plugin-lifecycle-inbox.ts`: new `hasPending(companyId, types)`. It
returns whether any unacknowledged event exists. Each resource with an
unacknowledged event has a head, so this gives the same answer as
`list(companyId, 1).length > 0`.
- `server/src/services/plugin-lifecycle-delivery.ts`: the idle-sleep
journal check uses `hasPending`. This keeps its early exit. A one-row
`list` page would read the full journal.
- `server/src/__tests__/resource-lifecycle-events.test.ts`: new database
test for heads across acknowledged history, a creation recorded after
transitions, acknowledged gaps, shared ids across resource types,
per-plugin progress, another company, `afterId` pages, limits,
`hasPending`, and the next head after an acknowledgment. The test also
passes on the old query, so it locks the contract.
- `scripts/benchmark-plugin-lifecycle-inbox.ts`: new benchmark. It loads
the baseline service from git, runs both on the same fixture, alternates
order, asserts equal rows, and prints plans.
- `doc/DATABASE.md`: describes the read cost and the benchmark command.
- No schema change, index or migration.

## Verification

Thesis: the old read cost is (unacknowledged candidates) x (acknowledged
older events per resource), because each candidate walks the
acknowledged prefix of its resource. One pass that keeps the
per-resource head removes that product and returns equal rows.

Baseline plan at 220,080 events: 40,020 candidates each walked about
3,999 older rows, which gave 160,039,980 acknowledgment probes and
562,140,833 buffer hits. The treatment plan reads the journal once with
a hash anti join and sorts the 40,020 unacknowledged rows (3,925 buffer
hits).

Local SQL timings (embedded PostgreSQL 18, Apple Silicon; event counts
include a 20,000-event second company, synthetic data, warm cache, one
warmup, order alternates):

| Fixture (events / acknowledged) | Case | Before median | After median
| Samples (after) | Speedup |
| --- | --- | --- | --- | --- | --- |
| 40,060 / 16,040 | page of 100, all types | 3,263 ms | 13.1 ms | 3
(12.0-14.1) | 250x |
| 80,080 / 48,060 | page of 100, all types | 10,137 ms | 23.9 ms | 5
(22.7-32.6) | 424x |
| 220,080 / 160,060 | page of 100, all types | 355,718 ms | 538 ms | 3
(72-785) | 662x |
| 1,020,080 / 960,060 | page of 100, all types | not run (est. > 30 min)
| 1,682 ms | 7 (1,203-2,384) | - |
| 80,080 / 48,060 | page of 100, agent and project only | 1.5 ms | 1.6
ms | 5 | neutral |
| 80,080 / 48,060 | idle-sleep check | 14.5 ms | 13.8 ms | 5 | neutral |
| 1,020,080 / 960,060 | idle-sleep check | not run | 406 ms | 7 | - |

- Result rows are equal (`assert.deepEqual` on all mapped rows, in
order) for each case where both versions ran.
- The agent runtime revision read in `plugin-agent-runtime-state.ts` and
`agent-file-store.ts` (`select id ... order by id desc limit 1`) uses a
backward index-only scan and reads 5 buffers at 1M events. It is not
changed.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=02bfe71a6b --agents=40 --per-agent=1500 --acked=1200 --samples=5
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=02bfe71a6b --agents=40 --per-agent=25000 --acked=24000 --samples=7 --treatment-only
cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/agent-runtime-hooks.test.ts
```

- `resource-lifecycle-events.test.ts`: 28 passed.
`plugin-lifecycle-delivery.test.ts`: 40 passed (includes idle-sleep
"none" and "present"). `agent-runtime-hooks.test.ts`: 22 passed.
- `tsc --noEmit -p server`: pass.

## Risks

- The page read is still linear in the company journal size, because
acknowledgments are per plugin and the journal has no retention yet. At
1M events a page takes about 1.7 s locally. Each acknowledgment wakes
another full pass. A per-plugin cursor or retention can remove this
later.
- The planner can choose a different plan on the hosted database. The
local plans use a hash anti join and one sort. The hosted improvement is
not yet measured.
- `acknowledge` still checks one event against older events of its
resource. That check is linear in that resource's history per call, as
before.
- Low behavior risk: the new test passes on both the old and new
queries.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:37:14 -06:00
DottaandPaperclip 199670ba99 fix(accounting): prevent usage receipt stalls during streamed run updates (#20385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner commits usage receipts before it acknowledges retained
events.
> - Usage receipts and streamed run logs update the same run row.
> - A repeatable-read snapshot can become stale before the receipt
writer locks that row.
> - Continuous log writes can exhaust all accounting retries and stall
the retained event.
> - This pull request uses the existing run lock with READ COMMITTED for
standalone receipt writes.
> - Other accounting operations keep their consistent snapshot. Stall
diagnostics retain safe nested error codes.

## Linked Issues or Issue Description

Related: #20235.

**What happened?**

A hosted native run stopped on a retained usage event. Its diagnostic
reported `Error at source seq N`. The saved outbox and accounting
records locate the failing delivery boundary. Traces show repeated
accounting transactions while streamed events still updated the run row.
The historical inner exception is unavailable.

A controlled PostgreSQL reproduction proves a matching defect: a run-log
append after company validation makes the receipt writer's
repeatable-read snapshot stale. Repeated appends exhaust all four
retries with SQLSTATE `40001`. This is a demonstrated failure mechanism;
attribution of the historical incident remains an inference.

**Expected behavior**

Commit the receipt under its existing run-row lock despite concurrent
run-log writes. Observe concurrent finalization and preserve
late-receipt debt. Keep duplicate receipts idempotent and reject
conflicting or cross-company receipts.

**Steps to reproduce**

Run `server/src/services/usage-receipts-read.test.ts`. The concurrency
regression uses real PostgreSQL and the production run-log append
function. It commits a log append after company validation and before
the receipt locks the run. Before the repair, the receipt rejects with a
wrapped `40001` after four attempts. After the repair, it commits once.
The finalization variant verifies that a newly settled run receives
late-receipt debt without replacing settled state.

## What Changed

- Allow aggregate-lock-protected accounting writers to select READ
COMMITTED. Keep repeatable read as the default.
- Use READ COMMITTED only for standalone usage receipt writes. Keep the
existing run lock, immutable receipt journal, replay checks and
transactional budget-work publication.
- Add PostgreSQL regressions for concurrent log appends, concurrent
finalization, duplicate delivery and the unchanged default snapshot.
- Read the outer event-commit error and at most three nested causes for
an existing safe error code. Exclude exception messages and SQL.
- Document receipt isolation and bounded stall diagnostics.

## Verification

- Red: the production receipt writer exhausted four retries with
PostgreSQL `40001` under controlled concurrent run-log appends.
- Red: wrapped database and transport stall diagnostics returned `Error`
instead of `40001` and `ECONNRESET`.
- Green: Runner control-plane tests passed all 207 cases. All 68 server
assertions passed across receipt reads, native usage delivery and cost
accounting. Cost-accounting teardown exceeded its 30-second limit twice
on this loaded host, after all 28 assertions passed. The workspace
project retained its configured 30-second hook limit despite the CLI
override. This is a local suite cleanup failure, not a passing suite;
the same revision passed the complete server suites in CI.
- Server layered TypeScript checking passed, including source and test
layers. Runner TypeScript checking passed. Server checks cover
`ccfe3bdeb1ab88537a03fd6af9044f66863939a5`; the unchanged Runner checks
cover its parent.
- Local environment: macOS arm64, Node 26.4.0, disposable embedded
PostgreSQL. Native usage-delivery tests use the real Rust receiver.
- CI at `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`:
[build](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568428),
[typecheck](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568367),
native Runner integration, browser shards and all nine serialized server
groups passed. All 12 general server shards passed in the [full
run](https://github.com/paperclipai/paperclip/actions/runs/42714760702).
Final rollup: 59 successful checks/statuses, three intentionally skipped
checks, no pending or failed checks.
- [Greptile reviewed this revision at 5/5 with no actionable
findings](https://github.com/paperclipai/paperclip/pull/20385#issuecomment-6753163397).
- Hosted evidence came from the task, run, saved sandbox outbox,
accounting receipts and traces. The run started fresh and did not cross
a deployment. No original agent turn was replayed.
- The hosted repair has not been deployed. The original task has not
been resumed. A fixed-revision remote-provider user journey remains
unverified.

## Risks

- READ COMMITTED changes snapshot timing for standalone receipt writes.
The run-row lock protects all mutable receipt/accounting reads and
serializes competing receipts and finalization. Other accounting
operations retain repeatable read.
- Receipt identity, replay conflict checks, company scoping, budget
publication, retry bounds and acknowledgement rules remain intact.
- No persisted format, provider identity or database schema changes.
Existing saved state needs no migration.
- Diagnostics inspect at most four objects and retain only the existing
identifier format, with an 80-character code limit.
- The reproduced race matches the incident boundary and timing, but the
original inner exception was not recorded. Rollout and live acceptance
remain necessary before incident resolution.

## Model Used

- OpenAI Codex, GPT-6 family, with code and shell tools. The exact model
ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 17:35:13 -06:00
Dotta 0eacad0d0c fix: refresh preserved harness receipt facts without copying archives (#20322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Volume-free deployment must preserve historical controller files
before removing the old disk.
> - A WEB preservation receipt binds an archive and source files to the
selected session and database facts.
> - Later metadata changes can make those facts stale while the
preserved files and archive remain unchanged.
> - The existing owner refuses every stale receipt, so preservation
cannot converge.
> - This pull request refreshes only facts freshness after it verifies
the same source bytes, checkpoints and archive.
> - Existing locks and an old-receipt digest comparison prevent a
concurrent replacement from being overwritten.

## Linked Issues or Issue Description

Refs #19814 for WEB evidence preservation without settling remote
recovery. Refs #19832 for bounded larger inventories. Related harness
receipt PRs were searched; no existing change provides this guarded
refresh.

**What happened?**

A valid WEB preservation receipt becomes stale after later run metadata
changes. Its original-volume owner refuses another preservation attempt
because an invalid receipt already exists. This leaves an eligibility
blocker even when the source and saved archive are unchanged.

**Expected behavior**

The original-volume owner can refresh stale facts only after it proves
every other persisted binding unchanged. It retains the prior receipt
and remote recovery state.

**Steps to reproduce**

1. Preserve a settled session's WEB files and controller evidence.
2. Add later run metadata so its database facts digest changes.
3. Preserve the same scope again. The previous implementation refuses at
the existing-receipt check.

**Paperclip version or commit**

The failure was reproduced on master
`12523fb677585fcd6da929f868014514a4e96e87`. This branch includes master
`1b7fe33432016b3b27ccdfbf7b8944da4b40726d`.

**Deployment mode**

Managed Linux server with PostgreSQL and durable object storage.

## What Changed

- Reuse the exact prior archive for existing WEB receipts. Recheck
current source bytes, controller history, checkpoint material and
archive bindings. Read back ciphertext and verify its size and hash.
- Permit only database/local-observation freshness changes. Selected
session, source run, controller, archive, WEB entries, and assets must
still match the prior proof. Local inode and timestamp observations may
change across attempts; preserved bytes, paths and modes must match.
- Recheck current facts and source files under the existing finalizer,
task and run fences. Accept a matching concurrent winner or require the
exact previous receipt digest before replacement.
- Retain the prior database and local facts digests, witness
fingerprint, observation time and receipt digest in the same
company-scoped activity transaction. Require those four fields to
reconstruct the exact prior proof from unchanged current bindings before
replacement. Normal audit redaction stays enabled.
- Preserve current-key checks and archive reads on ordinary idempotent
calls. Keep active-source, changed-evidence and cancellation refusals.
- Document the refresh contract. No schema, saved-run format or public
API change.

## Verification

- Genuine RED: the new real PostgreSQL/encrypted-archive case failed at
the old previous-invalid gate.
- Before the audit review follow-up, the full local owner suite passed
**94/94 integration cases in 50.01 seconds**. It covers same-archive
refresh with no PUT, audit history, current-key failures, unreadable
archives, cancellation, changed files, replaced volume, active sources,
matching concurrent winners and conflicting receipt replacement.
- The cases also cover a source becoming active during an idempotent
readback. Earlier 91- and 93-case runs preceded these final review
guards and are not the final result.
- Server typecheck, module boundaries and diff checks passed on the
final production/test bytes before the clean master merge. Required CI
supplies final-head breadth. Four narrow post-merge
refresh/current-key/idempotent regressions passed in 5.70 seconds on
`82e270f29fff98442c4184b96082f72f18a88721`.
- The fixtures use a real disposable PostgreSQL database and actual
local encrypted archives/storage. Controlled faults affect the stated
storage or concurrency boundary. They do not prove a hosted refresh or
archive plaintext cold restore.
- The existing integration file now exceeds 1,000 lines. Keeping these
cases with its private original-volume, controller, archive and
PostgreSQL fixture avoids duplicating that setup.
- Initial worktree preflight stalled on the global package-manager
wrapper. Preflight completed with the existing pinned toolchain. No
setup failure is counted as a product test pass.
- Review found that activity redaction could change a full prior-receipt
copy. A new regression enabled the real username censor and reproduced
that mismatch. The fix stores only the four changed scalar fields and
their original proof digest; a prepublication canonical reconstruction
guard rejects dropped or unknown fields.
- On the audit follow-up, 16 affected real PostgreSQL cases passed in
11.98 seconds; 80 unchanged cases were intentionally skipped. They
include real redaction with a username-containing WEB path,
reconstruction across two refreshes, unknown prior fields, CAS, active
sources, current-key failures, unavailable archives and cancellation.
Final server typecheck and boundaries passed. This was an intermediate
revision; the final-head results appear below.
- A second review found that comparing old inode/timestamp observations
also broke ordinary valid retries. Three real cases reproduced
valid-receipt touch/inode failures and stale-facts touch failure. The
fix permits metadata freshness changes across attempts while keeping
exact byte/path/mode/checkpoint/archive coverage and the existing locked
within-attempt metadata fence.
- Final local owner suite on the metadata follow-up: **102/102
integration cases passed in 44.04 seconds**, including valid
touch/new-inode retries, stale refresh with the same archive and no PUT,
four-scalar audit reconstruction, changed bytes/mode/entries, and a
touch during archive readback that must refuse. Server typecheck, module
boundaries and diff checks passed. The final-head CI and review results
appear below.
- Final head `c81475c927d0862dcb8d2c2356dc41ab4a27e63f`: [PR workflow
attempt
5](https://github.com/paperclipai/paperclip/actions/runs/42667287803/attempts/5)
completed successfully at 19:24:28 UTC on 2026-11-09. All 51 jobs are
accounted for: 49 passed and 2 were skipped. The actual [final ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129382061558)
and [E2E
aggregate](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129381486342)
passed. Typecheck, build and the selected test lanes passed. The
exact-head rollup is successful with no pending or failed checks.
- Attempt 5 resolved both trusted workflows to
`3a2f71db31aa99633610dc241f55f34ac65124de`, after #20339 merged. Its
routing gate passed and real jobs started. This verifies successful
startup and completion of this rerun; it does not exercise the
empty-runner fallback because routing succeeded.
- Earlier CI failures on `82e270f29fff98442c4184b96082f72f18a88721` and
attempts 1–4 of this final-head workflow remain nonpasses. Attempt 1 had
interrupted leaf jobs and failed aggregates. Attempts 2 and 3 ended in
workflow failure without a final verify job. Attempt 4 failed startup
with zero jobs and empty-runner errors. Those failures are not counted
as passes or labeled unrelated.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746395090)
and
[Ponytail](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746327827)
reviewed the unchanged final head at **5/5**. Both historical review
threads are resolved; no inline thread remains open.
- Hosted application to an actual stale scope remains pending normal
release delivery.

## Risks

- Refresh must not bless different preservation evidence. The
unchanged-binding checks, fresh file and checkpoint proof, exact archive
reuse, source capability and locked receipt comparison remain mandatory.
- A fresh capability has a new witness fingerprint and expiry. The audit
retains the old fingerprint, database/local facts digests and
observation time. All unchanged proof bindings remain on the current
receipt. Their canonical reconstruction must match the old digest. The
new capability is validated at entry and publication.
- Readback verifies ciphertext size/hash and current-key checkpoint
material. This change does not claim a new archive GCM/plaintext
restore.
- Archive identity and rows remain unchanged, so retention ownership
does not change. Failed refreshes retain the previous receipt, original
files and unresolved remote recovery obligations.
- Older receipt shapes are not rewritten automatically. Normal
validation and named preservation refusals remain in force.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 13:26:27 -06:00
DottaandPaperclip 4d5265fe7e Preserve fully covered historical workspace sources (#19872)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration must preserve local files while retaining
unfinished recovery obligations.
> - A shared Git source needs proof for its companion workspaces before
import can publish.
> - Recovery backfill can add a historical-source marker to an exhausted
prepared-export hold.
> - The hold owner rejected that marker even when every recorded
allocation already belonged to its proven prepared source.
> - This pull request accepts only that fully validated form through the
existing source owners and publication fences.
> - Unrelated work can continue while import saves the local files and
keeps the remote recovery hold intact.

## Linked Issues or Issue Description

Refs #19830. No separate public issue was found. A search for the marker
found no open duplicate PR.

**What happened?**

A companion workspace with an exhausted prepared-export hold blocked the
whole shared Git cohort after canonical recovery backfill added its
historical-source marker. The prepared and historical source readers
both accepted the same allocation.

**Expected behavior**

Local file preservation can proceed when all historical references are
already covered by the proven prepared source. The recovery hold and
remote obligations remain unchanged. Unknown or uncovered sources stay
blocked.

**Steps to reproduce**

1. Create a shared Git repository with a selected leaf and a companion
workspace.
2. Give the companion a settled failed export, accepted result,
exhausted recovery and blocked task.
3. Run canonical historical recovery backfill for its retained
allocation.
4. Import the selected leaf through the normal online migration owner.

**Paperclip version or commit**

Reproduced on `4185e308659abf0887930e20049ca6e7cbfc0e81`.

**Deployment mode**

A source installation with legacy local Git workspaces and object
storage.

## What Changed

- Validate the exact historical marker schema, run and unique lease
references against the already proven prepared source and declared
allocations.
- Reuse the canonical historical-source inspector and compare the same
provider allocation and remote working directory.
- Reject unknown fields, malformed markers and foreign, duplicate or
uncovered references.
- Keep the full marker in the existing hold fingerprint and retain the
existing source, task, lifecycle, finalizer and publication checks.
- Add real PostgreSQL/Git tests through shared-cohort inspection and the
encrypted import/restore owners.

## Verification

- The unchanged owner failed the actual selected-leaf importer
regression with `online_workspace_source_scope_unproven`. The canonical
prepared/historical reader controls passed before that refusal.
- Production head `3e678e10fa` passed 41 focused PostgreSQL/Git cases
across three files in 18.93 seconds after rebasing onto the new
finalizer schema migration. This includes 14 new cases; 80 unrelated
cases were filtered out. The final test-only review update passed both
additional multi-allocation controls plus the original selected-leaf
cold restore (3/3, 11.29 seconds); TypeScript passed again.
- The positive uses the actual historical backfill producer. It
cold-restores an encrypted selected-leaf snapshot with the original Git
tree unavailable and keeps the peer's task, finalizer, result,
allocation and recovery action unchanged.
- The tests retain an unrelated active object-backed run, prove the peer
finalizer lock is held, refuse bad bindings and reject late marker
changes without discarding the uploaded snapshot.
- Server TypeScript, module-boundary checks and `git diff --check`
passed. The canonical PostgreSQL wrapper exited and cleaned up.
- Final-head CI passed: 56 successful checks and four intentional skips,
with no failures or pending jobs
([run](https://github.com/paperclipai/paperclip/actions/runs/42397090541)).
Greptile reviewed `0cbf0a0255` at 5/5 with no open findings; GitHub
reports CLEAN/MERGEABLE.

## Risks

- The allowance applies only to the existing exhausted prepared-export
hold. Other recovery forms and unknown or uncovered references remain
blockers.
- A marked hold adds database reads proportional to its already declared
allocations during maintenance inspection. It adds no provider calls,
payload download or recurring background work.
- Import saves the current local files. It does not claim that unsaved
remote files were recovered, that a provider stopped or that execution
may resume. Existing holds and warnings remain authoritative.
- Local tests use the real encryption and restore owners with a local
object backend. Hosted backfill and normal-delivery qualification remain
separate.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools and code
execution. The exact serving snapshot and context-window limit are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 11:53:00 -06:00
DottaandPaperclip 4185e30865 Recover archived commits from original bare sources (#19855)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration must preserve saved files before the web server
releases its local volume.
> - An archived Git checkout can retain a trusted branch-deletion
receipt after its private clone and generated project cache are removed.
> - The original repository can still exist at its recorded file URL,
but the archive owner did not check that source.
> - This pull request permits a narrow, verified original
bare-repository source for the exact recorded commit.
> - The existing encrypted snapshot and publication owners then preserve
the workspace through their normal migration path.

## Linked Issues or Issue Description

No public issue exists. Searches for the archived-source error and
original bare Git source found no duplicate change.

**What happened?**

Migration reported `archived_git_source_unavailable` for an archived
checkout even though its original bare repository still contained the
exact saved commit. The private clone and generated cache were absent.

**Expected behavior**

The canonical owner can use the recorded original local repository when
its full commit identity and source ownership are proven. Uncertain or
changed sources remain blocked.

**Steps to reproduce**

1. Create an isolated checkout from a local bare repository.
2. Close it through the normal cleanup owner and retain the successful
branch-deletion receipt.
3. Remove the private clone and generated cache, while retaining the
recorded original repository and full base revision.
4. Inspect the archived workspace for migration.

**Paperclip version or commit**

Reproduced on `84908d176b2800f4c792c4249c610ec0baebdd28`.

**Deployment mode**

A source installation with existing local Git workspaces and shared
object storage.

## What Changed

- Consider the exact configured file-URL repository only after the
canonical primary source is definitively absent.
- Require a matching source record and successful cleanup receipt, the
exact full base/deleted commit, a contained bare SHA-1 repository,
bounded standard configuration and alias-free metadata.
- Restrict this source path to local Git transport and a minimal
environment. Reconstruct the tree only in private migration scratch.
- Retain existing source fingerprints, journal identity, encrypted
upload/readback and final publication checks. Lock and recheck the
configured source and cleanup record before publication.
- Add real PostgreSQL/Git coverage for encrypted cold restore,
lost-reply retry and refused source or publication changes.

## Verification

- The unchanged owner failed the real four-object original-source
regression with an ineligible inventory.
- The review regressions failed before each repair: four
returned-primary retry/publication cases and two actual production
metadata/reaper cases.
- Final head `8925519310` passed all 40 focused PostgreSQL/Git tests in
56.38 seconds. Four positive formats use canonical/legacy metadata and
ordinary/verified-tip cleanup. Each imports an encrypted snapshot,
removes the original file URI from reach, then authenticates and
cold-restores exact files, executable mode and Git history.
- The same file covers source/receipt/writer guards, changed sources
before publication, authenticated lost-reply reuse and 18
returned-primary directory/alias/unreadable cases. Refused publication
retains returned edits and uploaded ciphertext.
- Server TypeScript, module-boundary checks and `git diff --check`
passed on this revision. Canonical test wrappers exited and cleaned up.
- An earlier focused run also passed 22 existing
archived-Git/restore-point controls. `pnpm verify --explain` completed
without executing extra suites. Required CI supplies final broad
coverage.
- Final head passed all 60 CI/review contexts: 56 successful checks and
four expected skips. Greptile reviewed this exact head at 5/5, with no
open review threads.
- One unchanged archive segment-boundary test timed out on both
framework attempts in the initial server shard. The one job-only retry
passed with unchanged code and timeout. The test and its resolved local
import graph match a green control revision. The original failures
remain retained; the cause is unresolved.

## Risks

- The fallback is intentionally narrow. Original repositories with a
different full tip, unsupported configuration, aliases (including an
aliased instance root), external object storage or more than 10,000
metadata entries remain blocked.
- The first source check and final publication check add bounded local
filesystem and Git work to this maintenance path. They add no recurring
worker or network Git access.
- Original repository files remain intact. Importing a retained
workspace does not make its configured local file URL portable for
future fresh project work. That separate configuration dependency still
requires preservation and qualification before volume removal.
- The local regression uses the real encryption and restore owners with
a local object-storage backend. It does not claim a hosted backfill or a
deployed migration.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools and code
execution. The exact serving snapshot and context-window limit are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 11:05:25 -06:00
DottaandPaperclip b8e02fc2b4 fix(native): preserve bounded larger web inventories (#19832)
## Thinking Path

> - Paperclip manages AI agents and preserves their work across server
changes.
> - Native harness preservation stores retained web-server files in
authenticated archives.
> - A retained skill can contain hundreds of valid local files.
> - The WEB evidence owner rejected any directory or inventory above 128
entries.
> - This blocked preservation of two 462-file skills even though their
payload fit the existing byte limits.
> - This pull request shares finite inventory bounds between capture and
archive validation.
> - It preserves every file and keeps the execution, identity, writer,
and publication guards.

## Linked Issues or Issue Description

Refs: #19814

**What happened?**

WEB evidence preservation rejected a retained source with 927 local
skill files. The source had settled execution ownership. The files fit
the existing payload limits. The directory enumeration failed before
archive publication because its entry limit was 128.

**Expected behavior**

Preserve a valid finite inventory within explicit file, payload, path,
and metadata limits. Keep remote recovery obligations unchanged.

**Steps to reproduce**

1. Retain two 462-file skills and one three-file skill under a settled
native source.
2. Request canonical WEB evidence preservation.
3. The old owner rejects directory enumeration. The added integration
test reproduces this refusal with real files and PostgreSQL.

## What Changed

- Share the WEB limit through the existing controller archive owner:
10,000 total entries, 4 MiB of manifest metadata, and 4,096 bytes per
path.
- Apply the limits to filesystem capture, published binding validation,
and archive descriptor validation.
- Keep the existing 16 MiB per-file and 64 MiB payload bounds. Keep
authority history at 128 entries and 32 epochs.
- Add real archive and cold-restore coverage for 927 files, legacy
manifest compatibility, and bound refusals. Document the limits.

## Verification

- Genuine baseline RED: the new 927-file integration case reaches the
original `captureWebEvidence.directoryNames` limit and fails before
publication.
- The initial setup attempt failed before test collection because the
generated plugin SDK was absent. The required SDK build fixed setup; it
was not counted as a regression result.
- The first affected three-suite run passed 93 cases. The large
roundtrip exceeded the generic 15-second case deadline. That failure is
retained. The isolated large case passed with a 60-second case limit,
but full CI later exceeded that limit while the other 4,577 tests in its
shard passed. Both timeout failures are retained. The final fixture
keeps all 927 files and 18.5 MB, uses bounded 16-way independent
comparisons, and has a 120-second whole-case limit. The canonical
preservation signal remains 60 seconds. No production deadline or
assertion changed.
- The 927-file case then passed: encrypted archive, authenticated cold
restore, all bytes/hashes/modes, unchanged remote leases, and
database-only validation with source volume absent and storage reads
forbidden.
- The final metadata repair adds a genuine RED case: an entry list below
4 MiB previously emitted a larger complete manifest without rejection.
The owner now checks the complete serialized manifest. All ten focused
cases passed after that repair. The final bounded-I/O fixture passed the
same ten cases in 37.26 seconds, including the full 927-file roundtrip.
All 93 other affected cases passed in the preceding three-suite run.
Fresh required CI supplies broad final-head coverage.
- Server TypeScript, module boundaries, and whitespace checks pass.

Run the focused PostgreSQL checks from `server/` with the repository
wrapper:

```sh
node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts --testNamePattern 'preserves all 927|existing 128-entry WEB|WEB manifest beyond|oversized WEB filesystem' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache
```

## Risks

The owner can hold more entries in memory. The fixed 64 MiB payload, 4
MiB metadata, path, and entry bounds constrain that cost. The archive
format and old receipt bytes do not change. Older builds still have the
128-entry limit and cannot qualify a larger new receipt. This change
does not settle remote recovery, grant execution authority, alter
finalizer locks, or change publication CAS rules. No hosted preservation
or deployment was performed for this PR.

## Model Used

OpenAI `gpt-6.1-sol`, with code execution and repository tools. The
context window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 09:11:00 -06:00
DottaandPaperclip ae1f781cac fix(native): preserve web files without settling remote recovery (#19814)
## Thinking Path

> - Paperclip manages work that AI agents perform for users.
> - Native sessions retain files on the web server and state in a remote
provider.
> - Replacing the web server must preserve its files without changing
remote recovery authority.
> - The existing maintenance owner requires destroyed-provider receipts
even when all web writers have settled.
> - This can keep the old web volume required only because a remote
recovery obligation remains unresolved.
> - This pull request adds a closed web-byte evidence mode to the
existing preservation owner.
> - The benefit is durable web-file coverage with unchanged remote
references, failures and continuation policy.

## Linked Issues or Issue Description

Refs #19748, #19806. A search for native harness filesystem evidence and
open web/harness changes found no duplicate implementation.

**What happened?**

A settled successful native source can retain a raw controller, a
separate warm supervisor and provider-home files while its historical
external lease has no termination receipt. The existing
filesystem-independence owner refuses before it can preserve these web
files. One retained skill file can also remain physically present after
its version row was deleted.

**Expected behavior**

Preserve every nonexcluded web file as authenticated evidence. Keep
missing files absent and deleted skill versions deleted. Leave VM
references, recovery actions, failures and execution authority
unchanged. Validate the published web-byte receipt on an empty
replacement using database bindings only.

**Steps to reproduce**

Use the included successful-source fixture with a settled web owner, an
unresolved external lease, real controller/supervisor files and a
retained skill whose version row is deleted. The baseline rejects at its
destroyed-provider guard. The new explicit maintenance mode preserves
the files and validates on a replacement. A separate fixture uses a
prior successful source, a failed retained last run, a genuinely absent
controller, a warm supervisor and 267 durable checkpoint parts.

**Paperclip version/commit**

Baseline `384fa2b7be`; final branch rebased onto `585c885168`. Current
head: `990210d7e9`.

**Deployment mode**

Original-volume maintenance followed by an ephemeral replacement. No
hosted operation or provider request is part of this PR.

## What Changed

- Add explicit `web_evidence` maintenance mode and a separate
`paperclip.native-harness-web-evidence.v1` receipt. Keep the existing
terminated-provider mode and old manifest shapes.
- Archive the closed canonical session-root inventory, the separate warm
supervisor and exact admitted skill manifests through the existing
encrypted `original_volume_source_evidence` owner. Preserve bytes and
modes. Keep canonical credential exclusions.
- Preserve local skill edits and deleted-version files as forensic
evidence. Create no skill version, executable checkpoint, provider
backup or recovery authority.
- Require original-volume capability, settled web writers,
physical/finalizer locks, exact task/source/cohort/input bindings and
final byte/mode/membership comparisons. Keep genuinely absent raw
controllers absent.
- Bind allocation and stamp identity while allowing later external
status and cleanup observations. Validate replacements from durable
database and archive bindings without the original anchor, filesystem or
storage reads.

## Verification

- Node 24.22.2, PostgreSQL 18.1, macOS arm64. The unchanged baseline
fails at the real destroyed-provider guard. The final revision passes
all 85 owner, original-volume capability and unchanged
continuation-policy cases in 21.14 seconds.
- Tests perform real encryption, persistence through the S3 storage
interface backed by the owned local fixture, authenticated restore and
exact byte/mode comparisons. The 281-byte file remains present while its
version row remains absent. The evidence archive stays separate from the
provider-session backup scope.
- The supervisor-only case preserves 267 database checkpoint parts,
keeps the raw controller absent and validates without original-volume
identity. Existing strict-mode and old receipt controls pass.
- Negative controls cover active web writers, expired but owned
finalizers, physical owners, foreign controller/supervisor identities,
missing manifests, symlinks, changed bytes/modes/membership/source
during upload, changed durable bindings, incomplete receipt coverage and
invalid late publication. Concurrent attempts retain one valid winner
and one audit event.
- Server TypeScript, module boundaries, diff checks and private-data
scans pass. The verification planner's first invocation used a wrong
script path; its canonical invocation refused a plan because fixture
inputs changed during selection. Neither attempt counts as a pass.
Required CI will provide broader application coverage.
- Earlier fixture and treatment failures remain recorded. They exposed a
missing helper, JSONB property-order instability, missing
separate-supervisor mode coverage and fixture timing/source/count
errors. The final same owner suite passes without weakened assertions.
- A canonical cleanup audit found two additional observation fields: the
cleanup controller and manual-attempt ID. Their real publication updates
reproduced false receipt invalidation on the first PR head. The narrow
correction keeps WEB coverage valid through those observations while
changed allocation identity still fails. The exact same 85-case suite
passes afterward. The final `990210d7e9` head has fresh Greptile 5/5
with no comments or unresolved threads. Required CI run
[42370607882](https://github.com/paperclipai/paperclip/actions/runs/42370607882)
passes: 55 successful checks and four intended skips, plus successful
Snyk. The old head's CI was superseded by this correction and is not
counted as a pass.

From `server/`:

```sh
node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-original-volume.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache
```

## Risks

This changes a maintenance trust boundary. It proves web-byte
preservation, not provider recovery or termination. External recovery
obligations and existing resume failures remain. Unknown layouts,
missing manifests, active local owners and changed authority fail
closed. Invalid receipts are not automatically replaced.

The inventory has independent limits of 128 entries, 16 MiB per file and
64 MiB total. Larger valid data stays on the original volume and blocks
this mode. Canonical credential exclusions remain excluded. Evidence
uses the existing encrypted ledger and retention roots; it does not
become an executable backup. Old receipts remain byte-compatible.

Startup adds bounded per-retained-scope database reads. Measure the
affected cohort's actual serving startup cost before enrollment.
Validation adds no filesystem or object read, worker or recurring poll.
Live source qualification and publication remain separate after review,
merge and normal delivery.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools and
PostgreSQL/filesystem test execution. The runtime does not expose a more
specific serving model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 06:16:04 -06:00
DottaandPaperclip 585c885168 fix: preserve prepared sources held by provider artifact failures (#19810)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online workspace migration archives local files before it changes
their storage authority.
> - A failed provider session can leave an active board-owned recovery
hold over a prepared local source.
> - The existing migration owner accepts finalized sources and
workspace-export failures.
> - It rejects a settled provider-artifact failure even when the
prepared source and recovery allocation have complete bindings.
> - This PR permits that exact preservation case under the existing
source and finalizer locks.
> - The remote recovery hold remains active. The archive preserves local
bytes and the existing recovery recipe.

## Linked Issues or Issue Description

Refs #19642 and #19805 for the existing online migration and
source-closure owners.

**What happened?**

Online workspace migration rejects a prepared source held by a terminal
`runner_remote_provider_artifact_incompatible` recovery. This also
blocks another local workspace that shares the Git source. The provider
session had executed work. A pre-execution exemption would be unsafe.

**Expected behavior**

Preserve the local source through the existing immutable archive owner
when the board recovery, task version, terminal run, finalizer, and
declared allocation all match. Keep the remote recovery action and
authority unchanged.

**Steps to reproduce**

1. Create a prepared native source and a failed run with provider
session and tool events.
2. Set its finalizer to an unowned terminal artifact failure. Add the
matching active board recovery and task status version.
3. Remove the local descriptor and inspect or migrate a shared local Git
alias.
4. Before this change, the source proof is rejected. The regression
fixture reproduces this failure.

**Paperclip version or commit**

Reproduced on master `1674902b4f`. The final branch is based on
`5a92d53f7d`.

## What Changed

- Accept only the matching active board-owned artifact failure with a
current task block, no live execution authority, an exact producer
fingerprint, and a declared prepared-source allocation.
- Include the task version and complete existing source/recovery
evidence in publication revalidation.
- Lock the proven recovery tasks in sorted company scope before
publication checks. A held descendant task produces the existing busy
error.
- Add a real PostgreSQL regression with 124 tool executions, a missing
descriptor, shared Git aliases, immutable download, and cold restore of
the local archive.
- Add stale, foreign, owned, controlled, malformed, and publication-race
rejection cases.
- Document that this archive preserves local bytes and the recovery
recipe. It does not prove remote export or release the remote hold.

## Verification

- Genuine baseline RED: the positive migration regression rejects the
source before the production fix.
- Focused treatment: 12/12 PostgreSQL cases pass.
- Review-race RED: both a selected source and a Git companion published
while a descendant recovery task was locked. The corrected tests reject
both cases.
- Two affected owner suites after the lock repair: 91/91 pass in 23.18
seconds.
- Strict server TypeScript check passes after the lock repair.
- Module boundaries and `git diff --check` pass.
- [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/42367152081)
passes: 56 successful checks, four conditional skips, and one neutral
review check. No checks remain pending or failed.
- Greptile and Paperclip reviews are both 5/5 on `0a6e6c3ece`, with no
open findings.
- The first descendant fixture attempt hit the immutable task/privacy
guard before it reached the race. The corrected fixture creates that
task binding at admission and produces the two genuine REDs described
above.

## Risks

- A broad classification could let a migration move local authority
without a complete recovery recipe. The new branch requires the exact
failure, producer, task version, source allocation, and released
execution owners.
- Evidence can change during upload. Existing locks and full publication
revalidation reject those races.
- The remote allocation may contain changes absent from the local
source. This PR preserves only local bytes and the durable recovery
route. It does not mark remote work exported, finalized, or safe to
discard.
- No schema, provider lifecycle, admission, or polling behavior changes.

## Model Used

OpenAI `gpt-6.1-sol`, with reasoning, tool use, and local code
execution. The context window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 05:27:55 -06:00
DottaandPaperclip d2efe3392b fix(native): preserve retired authority evidence (#19806)
## Thinking Path

> - Paperclip manages work that AI agents perform for users.
> - Native sessions retain control-plane files when execution authority
moves to a later run.
> - Volume migration must preserve these historical bytes before a
server can stop depending on its old volume.
> - The existing preservation owner archives the current controller and
supervisor, but refuses a nonempty authority-epoch directory.
> - A settled four-run session can therefore remain blocked after its
current controller archive is complete.
> - This pull request adds a closed historical inventory to the existing
encrypted evidence archive.
> - The benefit is preserved source evidence and database-only
replacement validation without granting execution or recovery authority.

## Linked Issues or Issue Description

Refs #19748. A search of pull requests for filesystem independence and
issues for authority-epoch history found no duplicate change. This fixes
its uncovered historical-file case.

**What happened?**

A settled native session retained three prior controller files in
authority-epoch directories. Its three historical database checkpoint
headers were absent. Maintenance preserved the current controller, then
refused filesystem independence because those historical files remained
uncovered.

**Expected behavior**

Preserve the exact historical bytes and modes through the existing
encrypted evidence owner. Keep original files, prior archive rows, run
outcomes, warnings, stamps, session pointers, and recovery policy
unchanged. Validate the resulting receipt on a replacement without the
source volume.

**Steps to reproduce**

Use the included four-run fixture with a current-controller-only
archive, three prior authority epochs, destroyed provider receipts,
released ownership, and absent historical checkpoint headers. Call the
canonical filesystem-independence owner. The base rejects the nonempty
directory; the treatment preserves the complete evidence and validates
it after removing source-volume identity from the test process.

**Paperclip version/commit**

Baseline `ef83d566cacf1d5f740d701560597dad2bfa4619`; final branch
rebased onto `27170dca49`.

**Deployment mode**

Original-volume maintenance followed by an ephemeral replacement. This
PR performs no hosted migration or provider operation.

## What Changed

- Add optional closed authority-epoch entries to the existing
`original_volume_source_evidence` archive. Preserve nested file bytes
and modes through the existing encryption, readback, ledger, audit, and
retention owners.
- Bind each epoch to its retained run, canonical checkpoint identity
when present, or explicit header absence plus the exact admitted
authority binding. Keep current-source checkpoint and artifact checks
strict. Historical evidence never becomes an executable checkpoint.
- Recheck the complete historical inventory at archive and receipt
publication fences. Keep physical locks, all-run ownership and
termination checks, source capability checks, and final CAS.
- Bind the history inventory and its database facts into the
authenticated archive manifest. Replacement validation uses database
bindings only. Later header presence, removal, retirement, or changed
identity invalidates the receipt.
- Leave history-free manifest bytes and receipt shapes unchanged. Keep
an existing current-controller-only archive when a later complete
archive is added. Unknown children and other nonempty history
directories remain blockers.

## Verification

- Node 24.22.2, PostgreSQL 18.1, macOS arm64. The final four-run fixture
fails on the baseline owner at its nonempty-history guard. The same
final fixture and treatment pass all 51 focused PostgreSQL/filesystem
cases.
- The positive case performs actual encryption, object-store
persistence, authenticated restoration, and exact byte/mode comparisons.
It proves replacement database-only validation with absent source paths
and storage reads refused.
- Controls cover foreign identities, unknown children, owned prior runs,
changed bytes or new files after upload, missing or newly present
historical headers, malformed and retired headers, an unknown authority
override, receipt-identity tampering, and a late concurrent winner.
Existing current-controller evidence and history-free manifest bytes
remain intact.
- The baseline/treatment driver restored only its two owned source files
in `finally` and verified their exact hashes. Earlier setup and
treatment failures remain recorded: missing bundled PostgreSQL library
links, an overly narrow raw-schema check, and an incorrect synthetic
transient-workspace binding. The final baseline and treatment use the
corrected same fixture; those earlier attempts are not counted as
passes.
- Review follow-up `21183b75f4` closes the missing-history-field gap.
Deleting history from an actual published receipt reproduced an invalid
true result before the fix. Unconditional manifest reconstruction now
rejects it; all 46 owner cases pass. Genuine history-free manifests
retain their exact bytes.
- Review follow-up `f3d305dd36` adds suspended historical Runner-state
bytes and mode checks through authenticated restore, active/foreign
state refusals, exact 16 MiB and 64 MiB boundaries, and retention of an
oversized file without publication. All 51 cases and server TypeScript
pass; production behavior is unchanged.
- CI run `42362784772` on the previous head had one unrelated 30-second
timeout in `manual-private-cell-report-assets.test.ts`, with 6,864
support tests passing. The failure remains recorded. The prior head
`21183b75f4b223cd3ec28124cdc194f24b4f79ba` passes [required CI run
42363408227](https://github.com/paperclipai/paperclip/actions/runs/42363408227),
including Runner support: 55 successful checks and four intended skips,
plus successful Snyk. The final test/documentation head
`f3d305dd3690a7db38b57c4a77389c17206cb76b` passes [required CI run
42364958386](https://github.com/paperclipai/paperclip/actions/runs/42364958386):
55 successful checks, four intended skips, one neutral review check, and
successful Snyk. Fresh Greptile is 5/5; all four review threads are
resolved. No timeout or assertion was weakened.
- The count warning is resolved as independent bounds: the actual
descriptor accepts 32 control-plane-only epochs (97 entries) and 32
epochs with 31 Runner states (128 entries), and rejects 129 entries. The
32-epoch and 128-entry maxima both apply; the capture budget is
unchanged.
- Server TypeScript and module-boundary checks pass. The local
verification planner selects ten broad application lanes; required CI
supplies that wider coverage instead of duplicating it locally.
- Reproduce the focused owner suite from `server`:

```sh
node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache
```

## Risks

This changes a maintenance trust boundary. A source capture is limited
to 128 entries, 16 MiB per file, and 64 MiB total. These limits are
intentionally below the Runner's 256 MiB state maximum. Larger valid
epochs remain on the original volume and continue blocking filesystem
independence. It uses bounded directory reads and file descriptors,
rejects aliases and unknown paths, and rechecks bytes before
publication. Storage failures retain original files and ledger evidence;
they do not settle recovery or authorize a retry.

Startup adds bounded database reads for retained historical header
bindings or their absence. It adds no source filesystem or storage
reads. The selected cohort's serving startup cost still needs
measurement before enrollment. Other historical layouts require their
own canonical source qualification. No missing provider history is
declared recovered, and no automatic corruption repair is added.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, source editing, shell tools,
and PostgreSQL/filesystem test execution. The runtime does not expose a
more specific serving model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 04:55:18 -06:00
DottaandPaperclip f673bd3cb5 fix(native): verify legacy harness filesystem independence (#19748)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions retain provider checkpoints and recovery state
across turns.
> - Server replacement must preserve every remaining local data
dependency.
> - A legacy backup stamp can outlive a destroyed provider and an absent
local backup.
> - Surviving skill files can already have exact immutable package
copies.
> - Raw controller files also contain stream events that database
checkpoints omit.
> - This pull request records verified filesystem independence under the
original-volume maintenance owner.
> - Replacement startup checks durable bindings while existing resume
failures and policies remain intact.

## Linked Issues or Issue Description

Refs #19715, #19729, #19733.

**What happened?**

Startup treats every retained legacy backup stamp without a current
backup row as a local filesystem dependency. This also blocks sessions
whose provider was already destroyed and whose remaining local skill
bytes have exact durable copies.

**Expected behavior**

An authorized maintenance owner can prove that an exact retained source
has no uncovered local harness payload. Startup can use that proof after
replacement. The proof must preserve failed recovery state and must not
invent a provider backup.

**Steps to reproduce**

1. Retain a terminal native session, its checkpoint, and its legacy
backup stamp.
2. Record exact canonical destroyed-provider receipts. Keep only
admitted immutable skill files on the original host.
3. Publish the owner-verified independence receipt. The old startup
predicate still blocks; the updated predicate validates its durable
bindings.

## What Changed

- Add an internal original-volume capability. It requires a private
nonce anchor, independently reviewed attachment evidence, current
reserved runtime identity, and a witness that is at most 120 seconds
old.
- Verify retained and selected-source inputs, checkpoints, leases,
destroyed-provider receipts, released ownership, and exact immutable
skill references.
- Inspect canonical persistence roots and compare surviving skill bytes
and modes. Refuse any provider directory present outside the current
persistence profile, including an old Codex home after a
preconfigured-home switch. Unknown data, backups, history, owners, or
source changes still block publication.
- Preserve raw controller and supervisor bytes through the existing
encrypted evidence archive owner. Match recovery records to the database
checkpoint, authenticate the exact archive row, and compare the original
bytes again before publication. The archive grants no executable
authority.
- Publish one audited receipt with source locks and a final comparison.
Keep original files, stamps, warnings, statuses, and session pointers. A
late capture retains a valid published winner and refuses an invalid
receipt.
- Validate published receipts from current database bindings on
replacement hosts. Add no startup filesystem probe, object read,
provider call, or recurring worker.

## Verification

- Genuine baseline regression: restore the original startup predicate
while keeping the actual receipt owner. The positive
PostgreSQL/filesystem case fails with expected `false`, actual `true`
after valid publication. Restore the changed predicate; the case passes.
- Final rebased head `e78f4b4424`: 42 affected PostgreSQL/filesystem
cases pass. This includes 32 owner guards, three paired continuation
cases, six existing failed-source archive controls, and one existing
lease-verified backup migration control. The other 113 tests in the
selected existing files were explicitly unselected.
- Genuine controller baseline: the unchanged prior owner publishes no
raw archive and accepts missing checkpoint parts. Both new regressions
fail. The repair passes encrypted byte-for-byte controller/supervisor
restore, missing checkpoint, foreign source, active source, and
changed-after-upload controls.
- Genuine omitted-provider baseline: the unchanged owner accepts old
Codex, OpenCode, and ACPX history after a preconfigured-home switch. All
three refusal cases fail. The fix passes those cases and permits
preconfigured execution with no omitted directory.
- Genuine late-publication baseline: both concurrent winner tests fail
before the repair. The repair keeps the existing valid receipt, refuses
an invalid one, and keeps one audit event. A delayed raw upload also
retains the first authenticated archive and its receipt.
- Capability controls: 12 passed, including empty replacement refusal,
stale or future evidence, changed deployment, missing volume, path
aliases, changed anchor, and current remount identity.
- The actual server-to-Runner boundary preserves explicit recovery
policy on original and empty homes. The separate actual CREATED-sandbox
preparation callback preserves checkpoint identity and policy without a
backup. Provider I/O and final transport construction are controlled.
- Warm sessions retain the same canonical refusal on both homes, make no
backend/provider attempt, and retain the saved checkpoints.
- Server TypeScript, module boundary checks, diff checks, and a
private-data scan pass. The Runner TypeScript build was refreshed after
rebase to include two new upstream exports.
- Fresh Greptile review is 5/5 with no unresolved threads on
`e78f4b44240b0fc33c2fd7cae2d5b4a2578e3153`; check `128330103712`
completed successfully. Required CI remains pending. No live
preservation or enrollment action ran.

Focused command from `server/`:

```sh
node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts src/services/native-runtime/native-controller-migration.integration.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts --testNamePattern 'owner-verified|legacy harness continuation|preserves a failed source online|migrates a lease-verified' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache
```

## Risks

- This proves that no uncovered harness payload remains local. It does
not recover provider history or certify that no other copy exists.
Existing governed-wait and warm-session failures remain.
- Unknown providers, retained owners, extra payloads, missing
references, or changed source bindings fail closed. Invalid receipts
require investigation; this change does not repair or replace them
automatically.
- The maintenance capability requires independently reviewed Railway
attachment evidence. The original anchor remains stable across normal
releases; each new staging invocation needs current deployment and mount
evidence.
- Startup validation adds bounded per-scope database reads. It pages
candidate scopes and caps each source history at 500 rows. Measure the
affected cohort's actual startup latency on the qualified serving build
before enrollment.
- Raw archives use the existing backup ledger, encryption, retention,
and restore machinery. They preserve evidence only. They do not repair
missing provider history or grant a new resume policy.
- Local tests control provider and transport boundaries. Live staging
remains a separate step after review, merge, normal delivery, and a
fresh original-volume capability.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and local
code/test execution. The exact runtime variant and context-window limit
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 00:57:20 -06:00
DottaandClaude Opus 5.5 7e3caa7bac fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.

## Linked Issues or Issue Description

Refs #19394

**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.

**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.

**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.

**Paperclip version or commit**
master at 4d49a31286 (includes #19394).

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `issue-tree-control.ts`: add `ISSUE_PAUSED_ERROR_CODE` and
`pauseSuppressionResult()`. A gate records `treeHoldSuppression: {
holdId, rootIssueId }` on a run that it refused for a pause.
- `findOwedPauseResume` now treats the latest run as owed when it is an
`issue_paused` cancellation under a pause hold. It uses the recorded
hold. For an unmarked run (a scheduled retry cancelled before this
change), it uses a snapshot pause that was active at cancellation. The
operator Stop rule does not change.
- Scheduled-retry gate (`run-dispatch/adapters/postgres.ts`,
`cancelSuppressedRetryInTx`): locks the pause row `FOR SHARE` and
records the covering hold. A concurrent release either sees the refusal,
or the retry sees no pause and runs normally.
- Queued-run claim gate (`heartbeat/queue.ts`): cancels with
`issue_paused` and the hold marker, after the agent start lock is
released. Before, the cancel re-entered that lock from inside it, and
the agent's queue stalled until the 30-second stale-lock timeout. After
the refusal is saved, the gate re-checks the hold and resumes the task
itself if release already finished (`resumeIfPauseAlreadyReleased`).
- Release (`issue-tree-resume.ts`, route): revisits the snapshot members
and every task with a run refused for that hold, also when `wakeAgents`
is set. This covers tasks created under the tree after the snapshot and
tasks moved out before release.
- `packages/db`: partial index
`heartbeat_runs_tree_hold_suppression_idx` (migration 0436, `CREATE
INDEX CONCURRENTLY`, registered as a top-level migration) for that
lookup.
- Docs: `doc/execution-semantics.md` (gate-refused work, race handling,
non-assignee and pre-fix scope) and `doc/DATABASE.md` (index).

## Verification

- Red before the fix, green after: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/issue-tree-pause-resume-run.test.ts` (21
tests).
- "resumes a parked retry the pause suppressed exactly once after
release / resume / legacy": before, `expected [] to have a length of 1`
(no resumed run). After, one `issue_tree_resumed` run with
`interruptedRunId` = the suppressed retry, no execution blocker, and the
task finished. `reconcileStrandedAssignedIssues` added no run during the
hold or after release.
- "resumes a queued run the claim gate refused during the hold exactly
once": before, the test timed out at 15 s (start-lock re-entry). After,
the run is cancelled with `issue_paused` and the hold marker, and
release re-dispatches it once.
- "resumes a refused retry on a task moved out of the paused tree before
release" and "... on a task outside the snapshot when release also wakes
members": review findings, one resume each.
- "leaves a due retry parked when the pause is released while the gate
refuses it": the test waits until the gate blocks on an uncommitted
release. The retry is not cancelled and runs once. Without the `FOR
SHARE` lock the gate does not wait and the test fails.
- "resumes work a gate refused after the release that should have
resumed it had finished": the late-refusal re-check resumes once, and a
repeated check does not dispatch again.
- "does not resume a paused-code cancellation that no pause hold
covered": negative case.
- Related suites pass locally: `src/__tests__/issue-tree*`,
`src/modules/run-dispatch`, `src/services/heartbeat/queue.test.ts`,
`heartbeat-dependency-scheduling`, `heartbeat-workspace-busy`, and
`packages/db` `client.test.ts`.
- I did not run a live staging check here. Staging APIs are not
reachable from this environment.

## Risks

- Migration 0436 adds a partial index on `heartbeat_runs`. It is built
concurrently and covers only cancelled runs with the marker, so it is
small.
- The scheduled-retry gate now takes a `FOR SHARE` row lock on the pause
hold before it cancels. Release updates that row, so the two serialize
briefly. Release does not lock issues or runs, so the lock order cannot
deadlock with the gate's issue-then-run locks.
- The queued-run claim gate now records `issue_paused` in place of the
generic `cancelled`. Stranded recovery already classifies `issue_paused`
as non-retryable. The pause resume owns that work, so recovery does not
add a second run.
- Scope limits (documented): a non-assignee run refused by a pause is
not replayed, the same as non-assignee wakes that admission drops during
a hold. Among pre-fix rows, only scheduled retries carry `issue_paused`,
so only those fall back to the snapshot. A pre-fix row whose hold is
already released does not self-recover; the staging task that showed
this bug needs one manual wake, or a new pause and release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:03:15 -06:00
f2593b7905 feat: make Coordinator and Agent Chat standard and speed up navigation (#19360)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People enter that work through the workspace, agent conversations,
and tasks.
> - Coordinator and Agent Chat still require experimental enrollment.
> - Task link prefetches do not warm the compact queries used by the
default task view. Chat also waits for a request when its navigation
list already holds the conversation.
> - This pull request makes Coordinator and Agent Chat standard and
reuses the reads needed for navigation.
> - People can choose Classic workspace in Experimental settings. Cached
work can appear while fresh data loads.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Workspace selection, Agent Chat availability, and navigation between
Chat, Tasks, and task details.

**Subsystem affected**

Cross-cutting: UI, workspace preferences, instance settings, and the
database migration.

**Current behavior**

People must enable Coordinator and Agent Chat in Experimental settings.
Opening a task can repeat requests because prefetch warms the full
detail and comment queries, while the default view reads compact task
data and a recent comment window. Opening Chat waits for a detail
request even when the current person's conversation list contains it.

**Proposed behavior**

Coordinator is the standard workspace for everyone. Experimental
settings offer Classic workspace, off by default. Old Coordinator
experiment values do not select Classic. Agent Chat is available to
everyone, including instances with an old saved or managed off value.
Task link intent warms the queries used by the mounted view. Chat can
use the current person's company-scoped conversation list during
refresh.

**Reason and benefit**

People can use the standard workspace without setup. Repeat navigation
can show cached content while the server refreshes it.

**Breaking changes**

Existing Coordinator experiment and Agent Chat off values no longer
disable these surfaces. Classic requires an explicit new opt-in. The
workspace API keeps its existing response shape and audit checks. The
migration retains legacy values and adds a separate Classic preference.
Execution permissions, company access, and chat privacy checks still
apply.

**Additional context**

Roadmap check: this graduates existing surfaces and improves their
navigation. It does not add a new agent execution capability. Duplicate
searches found no open Coordinator workspace PR. Related server cache
work, Refs: #17828, addresses compact issue-list invalidation; this PR
addresses client navigation queries.

## What Changed

- Add a default-off Classic workspace preference. Derive the existing
workspace API response from that choice. Keep personal persistence,
account isolation, and audited writes in the workspace preference
service.
- Replace the Coordinator experiment control with Classic workspace. Use
Coordinator while preferences load. Update the existing settings
examples and navigation recipe.
- Retire the Agent Chat experiment control. Ignore old stored and
managed off values. Retain the compatibility key and the existing
execution and access checks.
- Reuse account- and company-scoped chat list data with its original
freshness timestamp.
- Share compact task and recent-comment query options between prefetch
and the task view. Keep full-history prefetch for Classic tasks and
expanded history.
- Add upgrade, replay, cache, account isolation, and browser switching
regressions. Update product, database, development, and roadmap
documentation.

## Verification

- Targeted preference, settings, migration, cache, and UI regression
tests passed during implementation. The cache tests hold refresh
requests pending and check that content still renders. They also check
account isolation and stale refresh behavior.
- A real browser test verified fresh Coordinator defaults, Classic
opt-in persistence, and return to Coordinator. It used an isolated
source checkout with fresh data on macOS.
- Token, migration numbering and safety, map, and Storybook catalog
checks pass after the rebase.
- Repository typecheck and build pass. Typecheck and token gates also
pass after review repairs. The broad local run passed all six chat
integration shards (1,931 tests). It then found the host-CLI-dependent
recovery fixture described below. Its remaining duplicate work was
stopped after the full current-head CI matrix passed; this is not a
claim that the broad local command completed green.
- Review repairs passed 419 UI/migration fixture tests, 19 Coordinator
navigation tests, and all 214 tests in the five affected server suites.
The 24 Chat/workspace browser tests pass, including standard Coordinator
switching and explicit Classic persistence.
- The broader browser repair run passed 85 tests. It retained two failed
assertions and skipped five later serial cases. The affected files then
passed all 11 tests after the fixes: reopen the desktop panel after
reload and open More for Audit and Connectors. This includes the five
serial cases that had not run.
- The broad local suite exposed a recovery fixture that executed the
host Codex CLI during a simulated remote probe. On a host with Codex
0.154.0, model fallback broke its fixed-model assertion. The fake
sandbox now reports Codex 0.160.0. All 20 selected recovery/provider
compatibility cases pass after the repair.
- A CLI company import/export suite timed out during server startup in
the second CI run. The port fallback is intentional in that fixture. No
root cause was established. That suite passes on the final commit in CI.
This records recovery, not a claimed product fix for the earlier
timeout.
- The first CI run exposed remaining assumptions about the retired flags
and Classic UI. The repair commit updates those fixtures and navigation
steps. It also restores the missing enabled Cases destination in
Coordinator. Earlier local recovery attempts used the wrong fixture
notification and advanced fake time during live execution; the corrected
tests use the subscribed recovery queue and wall time during execution.
- Greptile reviewed commit `98972b030d4505d838427a2fca6b196b8cad6d47` at
**5/5** with no new comments. The original finding is resolved.
- Current-head CI is green: **56 successful checks**, with three checks
skipped by workflow filters. This includes typecheck, build, the
complete Vitest matrix, all eight browser shards, Runner verification,
release checks, and security checks. [Final CI
run](https://github.com/paperclipai/paperclip/actions/runs/42132651414).
- Typecheck caught missing `pid` and `startedAt` fields in the simulated
command result. The fixture now returns the complete process contract,
and the server test-layer typecheck passes.
- Storybook catalog alignment was checked. Storybook play tests were not
run. Hosted latency and cold bundle loading were not measured.

## Risks

- The new default changes the workspace for people who previously left
the Coordinator experiment off. They can opt into Classic in
Experimental settings.
- The additive migration must run before the new server code. It is safe
to replay and preserves explicit Classic choices.
- Cached content can be stale during background refresh. Existing query
invalidation and source timestamps still control freshness.
- These changes reduce avoidable loading waits. They do not establish a
measured improvement in hosted request latency or initial bundle
download time.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution, and browser automation. The exact serving revision and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 16:28:25 -08:00
DottaandClaude Opus 5.5 bf10bc695a feat(task): show the agent's context window in task properties (#19273)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run through a harness (Codex app-server, or Claude through
ACPX), and the runner turns harness events into Paperclip events
> - Both harnesses report how full the model's context window is. Codex
sends `modelContextWindow` with its token usage. Claude sends `used` and
`size` in ACP `usage_update`
> - Paperclip dropped both values. The task view could not show how
close an agent is to compaction, and a compaction left no visible mark
in the task chat
> - This pull request carries the context-window occupancy through the
runner, stores it with the existing usage events, and shows it as a
"Context" row in the task Properties panel
> - It also shows a "Context compacted · 230k → 38k tokens" divider in
the task chat when the agent compacts its context
> - The benefit is that operators can see when a long task is near its
context limit and when the agent lost detail to a compaction

## Linked Issues or Issue Description

Refs #19252 (same pattern: a canonical harness event turned into
task-chat UI). Refs #19267 (status notices; it turns Claude's compaction
banner into a notice, and this PR reads that notice too).

**What existing behavior does this improve?**
The task view shows no context-window information, and a compaction is
invisible in the task chat.

**Subsystem affected**
Runner usage normalization (Rust and TypeScript transport), the PRP
usage schema, a read-only issue API, the task Properties panel and the
task-chat transcript.

**Current behavior**
Codex `modelContextWindow` and ACP `used`/`size` are dropped in the
runner. `context.compacted` always has null sizes, Claude compactions
emit no `context.compacted` event at all, and the live runner turn hides
every marker except "interrupted".

**Proposed behavior**
`usage.reported` carries an optional `context {usedTokens,
windowTokens}`. The task Properties panel shows a Context row ("10% of
258k") with a hover breakdown (tokens used, input and cached input,
output, cost, compactions). The transcript shows a compaction divider,
with sizes from the usage before and after the compaction.

**Reason and benefit**
Operators can see when a long task is close to its context limit, and
they can see where the agent's earlier context was summarized.

**Breaking changes**
None. The new schema field is optional, and the new endpoint is
read-only.

## What Changed

- PRP `usage.schema.json`: add an optional `context {usedTokens,
windowTokens}` object. Regenerate the schema bundle and the protocol
manifest.
- Rust `provider_events.rs`: add `codex_context_usage` (last request
`totalTokens` and `modelContextWindow`) and `acpx_context_usage` (ACP
`used`/`size`). ACPX `usage_update` now sets `context`.
- Rust `provider_backend.rs`: Codex `thread/tokenUsage/updated` sets
`context` on live (non-historical) usage.
- Rust `provider_events.rs`: emit `context.compacted` for the completed
`context_compaction` provider notice that #19267 produces from Claude's
compaction banner. Only this path marks a Claude compaction, so the
server counts each compaction once.
- `runnerd-codex-transport.ts`: `rehydrateRunnerdUsageNotification`
keeps a valid `context` on `tokenUsage`, so it reaches the stored
`item.completed` usage item.
- Server: new `GET /api/issues/:id/context-usage` (issue read access is
required) and `issueContextUsage` service. It reads only the assignee's
current session (the session of its newest usage report), and returns
that session's latest context reading, non-zero totals, cost, and the
count of compactions across all of the session's runs. Added to OpenAPI.
Shared type `IssueContextUsage`.
- DB: migration 0432 adds two partial indexes on `heartbeat_run_events`,
built concurrently. One indexes usage items by run and time. The other
indexes `context.compacted` events by company and session. `EXPLAIN`
shows that both lookups use them.
- UI: new `ContextUsageMeter` (bar, percentage, hover card) and
`IssueContextUsageRow` in the Execution section of Properties. The row
refreshes every 8 s while a run is active, and once more when the run
ends.
- UI transcript: `native-run-events.ts` maps `context.compacted` to a
`context` activity entry with pre/post sizes from neighbouring usage
items. `transcript-adapter.ts` turns it into a new `context_compacted`
marker variant. The live runner turn keeps that marker as its own
timeline row.
- Storybook: `task-context-window.stories.tsx` (Properties panel in a
task, all states, hover breakdowns, divider).

## Verification

- `cargo test -p paperclip-runner-core --lib provider_events` (new
tests: ACPX context, Codex context, banner compaction, notice
compaction)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport.test.ts`
- `cd ui && npx vitest run src/components/ContextUsageMeter.test.tsx
src/components/transcript src/components/task-chat
src/components/issue-properties`
- `cd server && npx vitest run
src/__tests__/issue-context-usage.test.ts` (embedded Postgres)
- `node scripts/typecheck-low-memory.mjs`, `pnpm check:token-gates`,
`pnpm --filter @paperclipai/paperclip-runner run check:protocol-types`
- Live check on a local worktree instance through the runnerd path:
- A Codex `gpt-5.5` task showed "10% of 258k" (27k of 258k, input 77k
with 49k cached, output 1.1k).
  - A Claude Sonnet 4.6 task showed "17% of 200k" (34k of 200k).
- Storybook: Tasks and conversations / … / Task chat / Context window.
- Not verified live: a real compaction. The divider is covered by unit
tests through the same projector path the task chat uses.

## Risks

- The schema field is optional, and older runners omit it. The UI
renders nothing until a harness reports a window.
- Migration 0432 builds two partial indexes concurrently on
`heartbeat_run_events`, which is a large table. The indexes cover only
usage items and compaction events, so they stay small. They add a little
write overhead to those events.
- Claude in-progress usage rows still record zero tokens with `requests:
1`. The final turn usage is correct. This PR does not change accounting.
- A "Compact now" action is out of scope. It needs a compaction-only run
that does not produce a run result. It will come in a follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, in-app browser for Storybook and
the live instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:54:27 -06:00