Commit Graph
545 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 95a6ab31e6 feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.

## Linked Issues or Issue Description

No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).

**Current behavior**

A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.

**Proposed behavior**

An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.

**Reason and benefit**

Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.

## What Changed

- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
  - `signal` in `signalDatabaseWork`;
  - `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
  - `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
  - each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
  - `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
    - the process instance is disabled without the env;
    - allowlist filtering, and spec parsing including colon topics;
    - per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
    - real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
    - suppressed records never capture a stack;
    - buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
    - no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.

## Verification

- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:

```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```

## Risks

- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:54:00 -06:00
DottaandClaude Opus 5.5 d25667f370 perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.

The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.

**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.

**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.

## What Changed

- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:

  ```diff
       remove: (x) => {
  -      const index = xs.indexOf(x)
  -      return index === -1
  +      const at = xs.indexOf(x, index)
  +      return at === -1
           ? null
  -        : (xs.splice(index, 1), x)
  +        : (xs.splice(at, 1), x)
       },
  ...
         if (index === xs.length) {
           index = 0
           xs = []
  +      } else if (index > 1024 && index * 2 > xs.length) {
  +        xs = xs.slice(index)
  +        index = 0
         } else {
           xs[index - 1] = undefined
         }
  ```

The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:

  | Measurement | Before | After |
  | --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |

## Risks

- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:46:41 -06:00
DottaandClaude Opus 5.5 f452b449fd fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:29:23 -06:00
DottaandClaude Opus 5.5 3ae7aba643 perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server uses `withDedicatedDbConnection` (packages/db) when an
action must hold one database session outside the main pool: advisory
transaction locks, lock keepalive, and `pg_settings` reads for workspace
lifecycle, finalization, source locks, skill caches and status cards
> - Each call built a new postgres.js client, opened a new physical
connection (TCP, TLS and startup through the pooler), built a new
drizzle client with the full schema, and then ended all of it
> - On staging the dedicated application name shows about 21 connection
setups per second, and the main thread is pinned (event-loop utilization
0.965)
> - This pull request keeps a small set of idle single-connection
clients per owner and reuses them
> - The benefit is that a dedicated action costs about 0.2 ms of
main-thread time instead of 4 to 8 ms, with the same exclusive-session
guarantee

## Linked Issues or Issue Description

**What existing behavior does this improve?**

`withDedicatedDbConnection` opens and closes one new database client and
one new physical connection for every call.

**Subsystem affected**

Database client (`packages/db`) and the server services that take
dedicated advisory-lock sessions.

**Current behavior**

Each call builds a new postgres.js client, opens a new connection (TCP,
TLS and startup through the pooler), and builds a new drizzle client
with the full schema. On staging (Neon statement statistics, 326 s
window) `SET application_name='paperclip-workspace-finalization-lock'`
ran 21.4 times per second. Only this factory uses that application name.
In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2%
inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that
is this per-call setup.

**Proposed behavior**

A caller still gets exclusive use of one session for the whole action.
The session and its drizzle client are reused by later actions, so the
per-call cost is the action's own queries, not a full connection setup.

**Reason and benefit**

The server main thread is saturated on staging (event-loop utilization
0.965). Reuse removes about 5 ms of main-thread work per call.

**Breaking changes**

None. The exported function keeps its signature.
`closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are
new exports.

## What Changed

- `packages/db/src/client.ts`: each `createDb` owner now has one set of
dedicated single-connection clients, shared with every private client
derived from that owner (so nested dedicated actions use the same set).
- An action takes an idle client (last in, first out) or opens a new
one. It never waits. Callers hold advisory locks and nest dedicated
actions, so a waiting queue could deadlock.
- On success the client returns to the idle set if fewer than
`DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed.
- When the action throws, the client is closed and never reused. This
covers a failed or cancelled transaction, a broken socket, and an
interrupted lock wait.
- When an action ends the client itself (the source-lock cancel path
calls `$client.end`), the client is never reused.
- Idle clients keep the owner's idle timeout and max lifetime, so
postgres.js still closes idle physical connections after
`DATABASE_IDLE_TIMEOUT_SECONDS`.
- New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle
set and stops retention. Ending the owner `$client` does the same. A
later action still works with a connection that closes on release.
- `server/src/index.ts`: shutdown closes the dedicated set next to the
other private pools.
- `doc/DATABASE.md`: documents the idle cap, the "not an admission
limit" rule, and the transaction-scoped session state contract.
- Session state audit: every current caller uses `pg_advisory_xact_lock`
/ `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller
uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or
temporary tables. A reset round trip per call is therefore not added.

## Verification

- New `packages/db/src/dedicated-connection-pool.test.ts` (embedded
PostgreSQL, 10 tests).
- Red on `origin/master`: 200 sequential calls used 200 distinct
sessions (`expected 200 to be 1`). All 10 tests failed.
- Green with this change, 3 runs in a row: 200 sequential calls use 1
session; 7 concurrent callers each hold a different session for the
whole transaction, and afterwards at most 4 stay open and later calls
reuse them; a failed transaction's connection is closed; a terminated
socket during the action is not reused; an idle connection terminated by
the server reconnects; a client the action ended is not reused;
transaction-local settings do not leak; shutdown and owner `$client.end`
close every retained connection; nested actions do not block.
- Focused suites pass: packages/db `work-signals`,
`prepared-queries-postgres`, `response-capture-postgres`,
`checkpoint-pool`, `client`, `client-options`,
`client-teardown-registry`, `postgres-connection-recovery`,
`client-disconnect-replay` (132 tests). Server suites that use dedicated
connections: `skill-package-store`, `status-card-evaluation-work`
(updated: it now proves advisory locks are released, since sessions are
reused), `agent-instruction-working-copies`,
`workspace-recovery-retry-copyback`, `native-execution-lease-batch`,
`company-skills-empty-runtime`, `decisions-service`,
`legacy-controller-lease`, `server-startup-feedback-export` (467 tests).
- `pnpm verify --explain` selects 2779 files for a
`packages/db/src/client.ts` change. Per AGENTS.md I ran the focused
checks above and let CI give the broad coverage.
- Benchmark: 200 calls of `withDedicatedDbConnection(db, d =>
d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))`
against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating
before/after process pairs per mode. Main-thread time is
`eventLoopUtilization().active`.

| Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call
(median, range) |
| --- | --- | --- | --- |
| TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) |
| TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) |
| Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) |
| Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) |

At the staging rate of about 21 calls per second, this removes roughly
100 ms of main-thread work per second (TLS figures; staging CPUs may be
slower).

## Risks

- A retained connection that the server or network drops while idle:
postgres.js reconnects on the next query when it has seen the close
(tested). If the drop and the reuse race, the first query of that action
fails like a main-pool query would. Before this change, a dedicated
action always started on a fresh connection.
- Through a transaction-mode pooler, `SET application_name=...` is a
server-side parameter switch, so its count may not drop as much as the
client-side connection count. Measure the deployed effect with pooler
client connections and main-thread CPU, not only that statement.
- Concurrency above 4 opens extra connections exactly as before; they
close on release. Total retained connections per owner grow by at most 4
idle clients, which still close after the idle timeout.
- An action that sets session-level state would now leak it to the next
action. The contract is documented on `withDedicatedDbConnection` and in
`doc/DATABASE.md`; no current caller does this.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:37:04 -06:00
DottaandClaude Opus 5.5 9824277f65 perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Most runner events that are not text deltas
(`tool.execution.started/progressed/completed`, `item.completed`) wake
the chat progress projection
> - That wake is registered inside the append's transaction, so each
such event takes 4 statements (BEGIN, the run lock read, the append,
COMMIT), where an event without a wake takes 1 (#21511)
> - The chat progress projection rebuilds itself from the stored events;
the wake only tells it when to look
> - This pull request writes an event whose only wake is chat progress
with the one-statement append, and sends the wake after that statement
commits
> - The benefit is 1 statement instead of 4 for most non-delta runner
events, with response, reassessment, retention and recovery wakes
unchanged

## Linked Issues or Issue Description

Refs #21511 (one-statement append). Refs #21528 (lock read returns the
transaction id). Refs #21549 (load model at recorded rates).

**What existing behavior does this improve?**
`appendHeartbeatRunEvent` for native source events whose only wake is
chat progress.

**Subsystem affected**
Server run event ingest (`heartbeat-run-events.ts`), the chat progress
notification helper, and two additive exports in
`packages/db/src/work-signals.ts`.

**Current behavior**
A progress event opens a transaction, reads the run lock with
`runtime_mode`, the bound task and the transaction id, probes for a
replay, registers the chat-progress wake as a tracked intent, appends,
and commits.

**Proposed behavior**
When chat progress is the event's only wake, the append takes the
one-statement path of #21511. That statement also returns the run's
`runtime_mode` and bound task id from the row it locks, and, where
commit notifications are on, calls `pg_notify` inside the same statement
for runs that are native and bound to a task, so other server processes
get the wake at commit. After the statement returns, this process
publishes the wake as settled to its own listeners. A replay, a gap, a
changed run or any other fallback takes the existing transaction, which
registers the wake as before.

**Reason and benefit**
At recorded rates these events are about 61% of the non-delta events
(#21549), so this cuts about 3 statements per such event.

**Breaking changes**
None. No schema change.

## What Changed

- `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other
wake clear) also qualifies for the one-statement path.
- `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement
append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`,
rendered once) from the locked row, plus a conditional `pg_notify`.
- `chat-native-progress-notifications.ts`:
`chatNativeProgressCommitNotification` and
`publishChatNativeProgressSettled`.
- `work-signals.ts` (additive): `databaseWorkCommitNotification(owner,
topic)` returns the cross-process payload when notifications are on;
`publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled
wake to this process's listeners.

## Verification

- Red/green on embedded PostgreSQL. Each new test failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "appends an event whose
only wake is chat progress in one statement and wakes progress after it
commits": 1 statement (before: a transaction), the wake is published
once as settled with no intent, and a replay publishes nothing.
- "keeps the transaction for the last events of a turn and a run, and
one statement for progress-only events":
`tool.execution.started/progressed/completed` and a non-final
`item.completed` take one statement; a final `item.completed`,
`run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled`
and `turn.interrupted` keep the transaction.
- "sends the progress wake to other server processes with the one
statement's commit": with commit notifications on, another connection
that listens on the work channel receives exactly one chat-progress
notification, and a replay sends none.
- `chat-native-progress-work.test.ts` "wakes the projection from a
one-statement progress event after the statement commits": the real chat
progress worker publishes the progress for an event appended in one
statement.
- Mutation check: without the settled publish after the statement, that
test fails.
- Lost-wake repair (condition b): an event row stored without any wake
models a lost hint.
- "repairs a lost progress wake with the run's next progress event":
nothing is published for the stored event; the run's next progress event
publishes the progress.
- "repairs a lost progress wake with the startup reconcile": a worker
that starts after the stored event publishes it.
- The existing response-wake test now uses a final reply and still shows
the intent before the write, and the race test of #21528 uses a final
reply so it still covers the transaction path.
- Other focused suites, one file at a time: the port suite with
conformance (105), `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51), `work-signals` (25), atomic-append (31),
`chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run
typecheck` and `pnpm check:module-boundaries` pass.

- CPU per operation (a local script, not committed: 800
`tool.execution.progressed` appends to one native task run on embedded
PostgreSQL, one connection, three rounds, `origin/master` code against
this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms
after in the warm rounds; wall time per event 0.84–1.04 ms before and
0.52–0.53 ms after.
- Inside a caller's transaction: new test "keeps a progress-only event
inside a caller's transaction on the locked path, and settles its wake
only when that transaction ends". `publishDatabaseWorkSettled` publishes
at once only outside a transaction; inside one it waits for the
outermost commit and drops the wake on rollback (new
`work-signals.test.ts` test, which failed with the unguarded publish).
The one-statement path is only taken when the owner is not a transaction
(`!is(db, PgTransaction)`, #21511), so the two guards agree.

## Risks

- **Lost wake.** If the statement's outcome is unknown (the connection
drops after the server ran it), this process sends no wake for that
event. In a chat channel (Slack, Telegram) the progress line for that
task can then stay one step behind (for example it does not move from
"working" to the latest tool step) until the run's next runner event, a
run milestone, or a server restart. A turn normally ends with an event
that keeps the transaction
(`turn.completed/failed/cancelled/interrupted`), and the run's
completion notifies chat run milestones, which include chat progress
(`notifyChatRunMilestones` in `run-completion.ts` and
`native-run-finalizer.ts`). So progress cannot stay stale after a run
ends; in the worst case it lags one step for the rest of a turn. The
projection rebuilds from stored events, so nothing is lost or
duplicated.
- The chat-progress wake no longer publishes an intent before the write.
Intents let the coordinator fence a pass against an in-flight write; the
settled wake after the commit makes the projection run again after the
row is visible.
- `packages/db/src/work-signals.ts` gets two additive exports and no
change to existing behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:37:35 -06:00
DottaandClaude Fable 5.2 689f3b190e fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic recovery continues interrupted and failed tasks without a
person, with a per-task limit of 6 automatic attempts per hour
> - There is no company-level limit. When the sandbox provider refuses
to create sandboxes, every task fails the same way, and recovery keeps
creating new runs for each task
> - On a hosted instance, one hour of a provider outage made 510
automatic runs, 587 failures and 0 successes across 289 tasks. No task
had more than 4 runs, so the per-task limit did not engage. Each start
asked a refusing provider for a sandbox and used server memory, which
made the outage longer
> - Recovery must still continue every task by itself. The answer is to
wait while the infrastructure fails, not to give the task to a person
> - This pull request adds a company-scoped pause for automatic work: it
opens when recent automatic runs fail before the agent starts, the
recovery passes stop creating new automatic runs while it is open, and
it closes by itself when runs start normally again
> - The benefit is that an infrastructure outage no longer multiplies
itself through recovery, and recovery resumes with no person acting

## Linked Issues or Issue Description

**What happened?**
During a sandbox provider outage, automatic recovery created hundreds of
new runs per hour across hundreds of tasks. All of them failed before
the agent did any work. The per-task limit did not engage because no
single task reached it.

**Expected behavior**
When most recent automatic runs of a company fail before they start,
recovery waits instead of creating more runs. It tries again at a small
rate. It resumes at full rate when runs start normally again. A person's
work is never held.

**Steps to reproduce**
1. Make the sandbox provider refuse lease creation (HTTP 503) for a
company with many assigned tasks.
2. Let recovery continue the failed tasks.
3. Count automatic runs created per hour: it grows with the number of
tasks, and every run fails at sandbox creation.

Refs #21394 (continuation classification this change will reuse when it
lands), #21433 (start-pass rewrite that the start hook will build on).

## What Changed

- New table `automatic_run_brakes` (one row per company, cascade delete
with the company; migration `0480`). Migration `0481` builds
`heartbeat_runs_company_finished_idx (company_id, finished_at desc)
where finished_at is not null` concurrently, so the signal reads the
runs that finished in the window directly. It stores the state (closed,
open, half_open), the reason with the window numbers, and the
compare-and-set fields that two servers share. A company with no row is
closed.
- New module `server/src/services/recovery/automatic-run-brake.ts`:
- Signal: on the database clock, the automatic runs of the company that
finished in the last 10 minutes (and since the last state change),
newest 500 by finish time, plus up to 200 running runs. A run counts as
a failure only when it ended in its own start phase: no provider work
(`executionRecovery.providerWorkStarted = false` or the native
`nativePreDispatchFailure` record), a legacy run still preparing, or a
transient setup code. A run interrupted by a server stop, restart or
lease loss is neutral, so a release swap does not open the pause. A run
that succeeded, or ended for a person-owned or task reason, counts as
healthy.
- Opens at 12 or more counted runs with 80% or more failures, or on 5
sandbox refusals (`environment_provider_unavailable`) in a row.
- Each outcome counts once, at its own time: a finished run at its
finish time, a running run that got past setup at its start time. Read
time never orders evidence.
- Closes in two steps: open moves to half_open after at least 2 minutes
when the newest 2 counted runs are healthy or the newest one succeeded;
half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64)
and then closes. It reopens from half_open if the signal trips again. It
closes by itself after 15 minutes with no automatic activity.
- Reads are one indexed query per company, cached for 5 seconds in a
bounded in-process map (256 companies). Each state change writes one
structured log line
(`automatic_run_brake.opened|half_open|closed|reopened`) with the
numbers.
- It also contains the start gate `holdAutomaticRunStart` (one probe at
a time, at most one new probe per 60 s, release budget per window by
compare-and-set, no I/O while the cached "held until" time has not
passed, person origin taken as an argument with a cached fallback). This
pull request does not call it from the start pass; see Risks.
- Recovery passes defer creating runs while the pause is open. The gate
is only where a run is created:
- `deliverReconciledExecutions`: after the agent-room check and right
before the wake, the delivery asks for a creation slot; when the pause
holds it, the continuation stays pending. The held backlog
(`continueHeldExecutions`) is not gated: recording a pending
continuation creates no run and costs nothing, and its delivery passes
this gate once.
- Assigned-work sweep `act()`: after the process rate limit and before
it claims a continuation, it asks for a creation slot; when held, it
returns `deferred_rate_limit` with the next try time, so nothing is
written to the per-task attempt ledger. New assignments with no earlier
run are not deferred.
- A reserved slot is given back (compare-and-set on its own timestamp)
when no run is created: the wake is refused or throws, the task is
capped, the claim changed, or the task went back to review. The
give-back is conditional in SQL on no run for that task having been
created since the reservation, so a wake that commits its run and then
throws keeps the slot.
- The sweep's overdue saved-work recovery path returns before the slot
is asked for: it starts a recovery attempt, not a run, and is never
held.
- Person work is never deferred. The gate is called only at the delivery
(`agentHasRoom`) and the sweep (`act()`), and each call passes the task
and the run it continues. While the pause is open, the gate does not
hold the record when any of these is true; it holds only when none is.
With no task or no source run it does not hold (fails open).

| Shape | How it is recognised |
| --- | --- |
| A Retry or "move forward" waiting for the task
(`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an
unstarted wake request from a user for the task |
| A Retry folded (`coalesced`) into the continuation, or a Retry while
the delivery is pending | a wake request from a user for the task,
requested after the source run finished (any status) |
| A person's message or a new assignment by a person after the last run
| the same: a wake request from a user for the task after the source run
finished |
| A held task selected because a person pressed Retry (the refused Retry
run) | a run for the task created after the source run finished whose
wake is a person reason (`task_retry`, chat message), or that has
`taskRetryRequestId` or `explicitUserContinuation` |
| A continuation of a person's Retry (any depth) | the source run's
chain reaches a person's run (`listPersonOriginSourceRunIds`, the
dispatch rule) |
| A Retry during pending delivery that becomes the continuation itself
(`deliverContinuationAsRun`) | not gated: that run is created by the
person's own wake, never by these passes |
| The held backlog's saved-input path | not gated: recording or
releasing a held continuation creates no run here |
- To prevent a deadlock, a pass may still create one automatic run per
minute per company when no automatic run of the company started in the
last 2 minutes, however many runs wait (compare-and-set, shared by
servers).
- `GET /api/companies/:id/assigned-work-invariant` now returns
`automaticWork` with the state, the time it paused, the window numbers
and a plain sentence, for example "Automatic work is paused for this
company since T because 47 of 50 recent automatic runs failed before
starting."
- `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is
now exported, unchanged, so the gate uses the same person-origin rule as
dispatch.
- The outcome rules (person-owned codes, restart codes, transient setup
codes, native pre-dispatch record) come from their owning modules
through one memoised dynamic import. A static import of
`heartbeat/retries.ts` made a module cycle at server start.

### What the pause consumes, and how waste is bounded

| Token | Taken | Given back or bounded |
| --- | --- | --- |
| Creation slot (one per minute per company while open) | right before
the run is created, after every other check | released only when no run
for the task was committed after the reservation |
| Probe slot (start gate) | when the start gate lets one automatic run
through | another run may take it when the reserved run has not started
within 10 seconds (claim declined, agent paused or full) or was
cancelled, deleted or finished |
| Gradual-release start (start gate) | one per run the start gate lets
through | a declined claim spends one start of that window; the window
ends within 60 seconds and the next one doubles |
| Probe success count | once per probe run, by compare-and-set on its id
| a probe run counts once; cancelled or deleted probes count as neither
success nor failure |

### When the pause opens, stays open and closes

| State | Leaves when | Bound |
| --- | --- | --- |
| closed | opens when, in the last 10 minutes since the last change, 12
or more counted automatic runs finished and 80% or more failed before
starting, or the newest 5 counted runs were sandbox refusals | none
needed |
| open | moves to half_open when it has been open 2 minutes or more and
the newest 2 counted runs are healthy, or the newest one succeeded |
evidence arrives at least once a minute: when no automatic run started
in the last 2 minutes, a pass may create one per minute |
| open | closes when no automatic run started or finished for 15 minutes
(queued runs do not count) | 15 minutes after the last activity |
| open | stays open only while new automatic runs keep failing before
starting | each new try is spaced by the existing passes and by one
creation per minute when nothing starts |
| half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6
minutes |
| half_open | reopens when the signal trips again, counted only since
half_open began | none needed |

Restart-class endings (server stop, restart, lease loss, process loss,
interrupted native session) are neutral in every state, so a release
swap does not open the pause.

## Verification

- `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests,
pass): classification; opens on 20 failures before start; 50 queued
automatic runs give exactly one probe and no failure or attempt; a
person's Retry, a new assignment, a run a person joined and the
continuation of a person's Retry are not held; mixed and person-owned
outcomes stay closed; old data is ignored; 40 working runs interrupted
by a restart stay closed, then the same continuations failing at sandbox
acquisition open it; 5 refusals in a row open it; failed probe waits for
the interval, two probes past setup move to half_open, the release gives
exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen
from half_open; probe slot frees after cancel, delete or no start; quiet
close after 15 minutes; two instances on one database agree and start
one probe; one creation per minute while open across servers; 250 held
runs over 100 passes use at most 5 queries (at most 255 with the
person-origin fallback, one per run, once); caches stay at 256 companies
and 1024 runs.
- `npx vitest run
src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one
test per person shape above at the delivery and the sweep, with the
pause open and no creation slot, plus a control that still holds a
continuation no person asked for): includes "no pause record: every pass
behaves as before", "the waiting run cannot start: a pass continues a
task on another agent, then the pause closes", "one held task spends one
creation slot", and "a person's Retry or message is never paused at the
delivery, the sweep or the held backlog".: sweep, delivery and held
backlog create nothing while open, write no attempts, and continue by
themselves after recovery; the sweep still continues one task a minute
when nothing automatic waits. With the three deferral lines removed, all
4 fail (red), and they pass with them (green).
- The pause reads as closed and logs once when its run rules cannot load
(test in `automatic-run-brake.test.ts`).
- `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the
module loads after the heartbeat service, in the server's import order.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify`: see the comment below for the result. CI supplies the
full suite.

## Risks

- This pull request does not hold starts. Already queued runs and
`transient_failure_retry` runs still start while the pause is open; they
act as the natural tries. The start gate goes in as a follow-up on top
of #21433, at a point before that pass counts a claim attempt and reads
the full run row, so a held run costs no attempt and no read.
- The sweep reuses the `deferred_rate_limit` outcome for a deferral, to
keep its outcome type unchanged while #21394 edits that file.
- The automatic-run predicate is built from master exports
(`RECOVERY_WAKE_REASONS`, retry markers). It will switch to
`isAutomaticContinuationRun` when #21394 lands.
- Migration `0480` adds one small table.
- Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`)
runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement:
it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in
`packages/db/src/client.ts`, like 0470 and 0476, and the runner drops
and rebuilds an invalid index after an interrupted build. The index is
declared in the Drizzle schema, so `pnpm db:generate` reports no
changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On
a large `heartbeat_runs` table the build is online (no write lock) but
takes time proportional to the table. After it exists, it is one more
index on a hot table: each run that finishes writes one more index
entry.
- No user-visible text changes outside the operator report.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell,
file edits, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:20:09 -06:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00
DottaandClaude Opus 5.5 a726dce4f0 feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools from connected apps through the Paperclip MCP
gateway
> - Some tool calls take a long time: exports, builds, large searches
> - The MCP progress utility lets an app report how far a call is, but
Paperclip ignored it
> - So the task chat showed only a spinner until the call finished, with
no sign of life
> - This pull request reads the app's progress and shows it on the
tool's row while the call runs
> - The benefit is that people can see that a long call is working, how
far it is, and how long it has run

## Linked Issues or Issue Description

**Subsystem affected**
Connected apps (MCP gateway) and the task chat.

**Problem or motivation**
A long tool call from a connected MCP app shows only a spinner in the
task chat. Paperclip does not send a `progressToken`, and it drops
`notifications/progress` from the app.

**Proposed solution**
Send a `progressToken` on each `tools/call` that an agent run makes.
Read the app's progress notifications from the call's response stream.
Show a progress bar (or an indeterminate sweep), the latest message, and
elapsed time on the running tool row. Keep the latest state so that a
reload mid-call shows it.

**Alternatives considered**
A new transport for progress. The existing company live-events socket
already carries per-run events with run read checks, so this change uses
it.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress

## What Changed

- `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`.
`readMcpHttpResponse` gives each notification on the response stream to
an observer before the result. The observer cannot fail or delay the
call. New `mcpProgressNotificationOf` matches a notification to its
token.
- `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an
agent run send a per-call token and forward progress to a reporter.
Calls without a run send no token.
- New `server/src/services/tool-call-progress.ts`: checks each update
(finite, increasing, plain single-line text of at most 200 characters).
It publishes a `heartbeat.run.tool_progress` live event at most every
250 ms (latest wins) and saves the latest update on the invocation at
most every 1 s. Nothing is awaited by the call. A final `done` event
goes out when the call ends.
- DB: nullable `tool_invocations.progress` jsonb column (migration
`0479`, `ADD COLUMN` only). A save never writes onto a finished call.
- Live delivery: the new event follows its run like transcript output. A
client gets it only for runs it follows, after the run read check, in
the run's company only.
- New route `GET /api/issues/:issueId/tool-call-progress` (task read
check) returns the latest progress of the task's running calls, for a
reload mid-call. Documented in `docs/api/issues.md`.
- UI: `ToolCallProgressProvider` (one per chat) keeps the latest update
per call. A running tool row subscribes and shows a bar or sweep, the
message, the percent and elapsed time (design tokens, reduced motion
honored). Finished rows drop it and never fetch or re-render for
progress. `LiveUpdatesProvider` ignores the event for query
invalidation.
- `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool
(`steps`, `stepMs`, `withTotal`) in its own file.
- Storybook: `Tasks/Tool progress` stories (running with total, running
without total, done).
- Docs: new `doc/connections/MCP-PROGRESS.md`;
`MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported.

## Verification

- Red/green: on the original source, 5 of the new server tests fail
(token in `_meta`, SSE with interleaved progress and result,
notification matching, run-interest delivery, gateway end-to-end). With
this change they pass.
- `server/src/__tests__/mcp-http.test.ts` (13),
`tool-call-progress.test.ts` (9: throttling, latest wins, bounded text,
out-of-order drop, finish, failures never throw, company/run/tool
scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres:
company and task scoping, no write onto a finished call),
`realtime/live-event-delivery.test.ts` (run interest),
`tool-gateway.test.ts` end-to-end against the real SDK kv demo server
over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7
throttled events, all for the right company, run, task and tool, the
last one `done`; no event reaches another company; the invocation keeps
the progress; the result is unchanged.
- UI: `tool-call-progress.test.tsx` (10: name matching, row states,
plain-text rendering, drop on complete, restore after reload, no fetch
for finished rows). All 67 task-chat test files pass.
- `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP
with and without a total.
- Storybook: checked the three stories in a browser. The bar fills to
40%, the sweep animates, the elapsed time ticks, and the done row shows
no progress.
- `pnpm verify`: see the CI checks for the final revision.

## Risks

- Low. The migration adds one nullable column.
- An SSE response is now bounded per message, not per stream, so handled
notifications (such as progress) do not use up the result's size limit.
Each message is still bounded.
- Progress code runs only while a run's tool call is in flight. It adds
a few bytes to the request and throttled, fire-and-forget work. Agent
boot and time to first token do not change.
- A row finds its call by run and tool name. If one run calls the same
tool twice at the same time, both rows show the newer call's progress.
- Progress from a call that became an MCP Task is not shown.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:14:22 -06:00
DottaandClaude Opus 5.5 a4c570aede feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected MCP apps through the tool gateway.
> - An MCP app can ask the person a question in the middle of a tool
call (MCP elicitation). There are two kinds: a form, or a link to open.
> - Today a form question is a dead end:
> - A 2026-07-28 server's form `input_required` stops with "needs
information Paperclip can't collect yet".
> - An older server's `elicitation/create` makes a card, but the answer
is never sent back, and the server waits on an open stream.
> - Paperclip does not declare form support, so compliant servers never
ask.
> - This pull request shows the question as a card in the task chat. The
card names the app, and only a person can answer it. When the person
answers, the agent wakes. The agent's repeated call sends the answer to
the app exactly once, for both protocol eras.
> - The benefit: apps that need a choice or a detail from the person now
finish their work. Restarts lose nothing, and no call is left waiting.

## Linked Issues or Issue Description

Refs #21140 (stateless MCP client; this PR closes its "form elicitation"
follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs
#21389 (MCP Tasks follow-ups: the card answer validation item).

**Is your feature request related to a problem?**
When a connected MCP app asks the person a form question during a tool
call, the call stops and cannot continue. A 2026-07-28 server gets a
"needs input" error. An older server gets a card whose answer never
reaches it.

**Describe the solution you'd like**
Show the question in the task chat. Let a person answer, skip, or leave
it. Wake the agent when the question is resolved. Send the answer on the
agent's repeated call, once, with the server's exact state.

**Describe alternatives you've considered**
Holding the tool call open until the person answers. We rejected this:
it holds a stream and a run slot, and it does not survive deploys or
restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records
nine design decisions with their alternatives.

## What Changed

- **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec,
the inventory, the decisions, the journey, tests, and risks.
- **New module `server/src/services/mcp-elicitation.ts`.** It is the
single mapping for direct calls and MCP Tasks:
- It turns each spec form field into a real card field: text with
defaults, number, yes/no, single choice (`enum` and `oneOf`), and
multiple choice (`items.enum` and `items.anyOf`).
  - It turns card answers back into typed `content`.
- Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or
expired sends `cancel`.
- It refuses forms that ask for passwords, keys, tokens, card numbers,
or one-time codes.
  - It keeps app text plain, so links in it are not clickable.
- It seals `requestState` with AES-256-GCM, bound to the company and
connection.
- **New table `tool_mcp_elicitations`** (migration 0479). It is
company-scoped, with one row per question round. It holds a keyed call
identity, the sealed state, the questions and their cards, a claim time,
and a 24-hour expiry.
- **Gateway, stateless servers:**
  - Paperclip declares `elicitation: {form, url}` on task tool calls.
- A form-only `input_required` saves the round, makes the cards, and
stops with a wait message.
- A repeated call before the person answers stops again without
contacting the app.
- After the answer, the repeated call claims the round and sends
`inputResponses` and the exact `requestState`, once.
- A failed send releases the claim. A crash mid-send lets the claim
lapse.
- **Gateway, older servers:**
  - Paperclip declares form support at `initialize`.
  - The first question gets `cancel` at once, so the server never waits.
- When the same question (same words and schema) arrives on the repeated
call, Paperclip answers it from the card.
- **Gateway, mixed rounds and retries:**
  - A form beside a URL step shows the link first.
- A repeated call that stopped for a question runs again in the same
run. It is no longer replayed by the per-run idempotency key.
- **Cards:** the title is "*App* needs your input". The resolver policy
is `human_only`. "Send to *App*" sends the answer. A user comment does
not supersede the card.
- **Activity log:** new events `tool_gateway.elicitation_sent`,
`tool_gateway.elicitation_refused`, and
`tool_gateway.elicitation_expired`. They record the action only, never
the values.
- **MCP Tasks:** task cards use the same mapping and gain the app name,
real fields, and `human_only`. A secret or an unsupported form stays
unsupported.
- **Storybook:** `Tasks and conversations/Questions and approvals/MCP
App Questions` has five states: Asking, Answered, Skipped, Expired, and
TwoQuestions. The fixtures are generated from the server mapper's
output.
- **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the
README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are
updated.

## Verification

- `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass
on the branch.
- **Red/green:** 9 of the 9 gateway tests fail with master's
`tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new
module.
- The tests use a fake 2026-07-28 server and a fake 2025-06-18 server
that asks over the open stream and waits for the reply.
- Covered: answer, skip (`decline`), withdraw (`cancel`), two questions
at once, a restart while waiting, a crash mid-send, another company with
the same app and arguments, expiry, a refused password field, an agent
that tries to answer (rejected), and capabilities.
- Existing suites: `mcp-tasks`, `mcp-protocol-versions`,
`tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See
the results comment below.
- Storybook: the five stories render in headless Chromium, and the play
tests pass. The only console errors are the 404s from the backend-less
API calls that every story makes.
- `tsc --noEmit` is clean for `server` and `ui`. `node
scripts/storybook-catalog.mjs` passes.

## Risks

- **Older servers may start asking questions.** Declaring form support
at `initialize` can make older servers ask where they did not before.
Mitigation: the first question always gets `cancel` at once, so no call
hangs, and the agent gets a clear next step.
- **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one
indexed lookup for a waiting round. Nothing is added before the first
token or to agent boot.
- **Migration.** One additive table with no backfill, so it is safe to
apply online.
- **Changed Tasks behavior.** A task card that is withdrawn or closed
now sends `cancel` instead of `decline`, as the spec defines. A skipped
card still sends `decline`.
- **Stale cards.** Expiry is checked only when the agent calls again. A
card can sit past 24 hours until the next call. Then it is withdrawn and
the app asks again.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, file edits, headless browser), with extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:56 -06:00
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
DottaandClaude Opus 5.5 969cf246ae perf(run-events): keep each native source's cursor on the run row (#21435)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Each native source event append counted all retained events of its
source (`count(*)`, `max(source_seq)`) to compute the acknowledgement
cursor
> - So the cost of each append grew with the length of the run, and long
runs made every text delta slower
> - This pull request keeps the gap-free cursor of each source on the
run row and moves it in the same UPDATE that reserves the event sequence
> - The benefit is that an in-order append stops reading other events,
so its cost does not grow with run length

## Linked Issues or Issue Description

Refs #21131 (earlier reduction of statements per event). Item 1b of the
agent scaling plan in #21418.

**What existing behavior does this improve?**
The append of a native source event (`appendNativeSourceEvent` in
`server/src/services/heartbeat-run-events.ts`). It ran a `retained` CTE
that counted every event of the source on every append.

**Subsystem affected**
Server run event ingest (`heartbeat_run_events`, `heartbeat_runs`).

**Current behavior**
On a run that already holds 50,000 events, one in-order append takes
about 10–13 ms (p50) on a local PostgreSQL. On a run with 400 events it
takes about 3.3 ms.

**Proposed behavior**
The append reads the stored cursor in the run lock statement that it
already sends. An in-order event moves the cursor and does not count
events. On the same 50,000-event run an append takes about 1.7–3.3 ms
(p50).

**Reason and benefit**
Text deltas are about 85% of the server's statements in the load model.
Their cost must not grow with run length.

**Breaking changes**
None. The new column is nullable. The previous release ignores it.

## What Changed

- Add the nullable column `heartbeat_runs.native_source_cursors`
(`jsonb`, `{"<source instance id>": <seq>}`). Migration `0479` is `ADD
COLUMN` only and classifies as `online_compatible`.
- The run lock read of a native append also reads the cursor of its
source.
- When the event sequence is at most cursor + 1, the append statement
has no `retained` CTE. It moves the cursor in the UPDATE that reserves
the sequence, and the receipt uses the cursor.
- Every other case (no cursor, a gap, or a cursor behind the stored
events) runs the earlier recount. In the same UPDATE it stores the
cursor when the source has no gap, and removes it when the source has a
gap. So the event that fills a gap recounts.
- The cursor is a lower bound. A writer that does not keep it (the
previous release during a deploy overlap, or
`NativeRunCoordinatorStore`) can only make it lag. The next out-of-order
event then recounts and corrects it.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts
src/__tests__/heartbeat-run-event-sequencing.test.ts` passes (19 tests).
- Red/green: before the change, 3 tests failed: the two new tests and
the updated lock-read statement test. After the change, all pass.
- New: "keeps the source cursor on the run so an in-order append does
not count the run's events" checks that the append statement has no
`count(` and that a replay returns the cursor.
- New: "recounts a source whose run has no cursor or a cursor behind its
stored events" covers rows saved before the change, rows from a writer
that does not keep the cursor, a gap, and the event that fills the gap.
- Focused related tests: `paperclip-control-plane-port.test.ts`,
`native-observer-replay`, `native-event-boundary-golden`,
`chat-native-progress-work`, `live-recovery-work`,
`native-interaction-bridge`.
- `pnpm check:module-boundaries` passes. `pnpm verify` returned exit 2
(broader CI coverage required) because of local benchmark scripts that
are not in this pull request. Required CI supplies the broad coverage.
- Append cost on a run with many events (local embedded PostgreSQL,
1,000 in-order appends, busy machine, load average about 20–40):

  | Events already in run | Before p50 / p90 | After p50 / p90 |
  | --- | --- | --- |
  | 50,000 (run 1) | 10.4 / 15.8 ms | 1.7 / 3.7 ms |
  | 50,000 (run 2) | 12.9 / 21.6 ms | 3.3 / 5.9 ms |
  | 400 | 3.3 / 6.2 ms | 3.7 / 7.7 ms (noise) |

- Statements per event do not change: BEGIN, run lock read, one append
statement, COMMIT (4). The append statement no longer scans the source's
events.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`): the machine was too busy for a stable result.
Deltas only: before 20,000 of 20,000 done at 3,999 statements/s. After:
13,796 and 14,817 of 20,000 in two runs. All sources: before 6,346 of
20,000 deltas done. After: 20,000 and 2,083 in two runs. A 20-second
window gives only 400 events per run, so it cannot show this change. The
long-run figures above are the evidence.

## Risks

- Low. The column is nullable and the previous release does not read it.
Runs that are in flight during a deploy have no cursor. Their first
append recounts once and stores the cursor.
- A stale cursor can only make the acknowledgement lag, never pass the
stored events. Event rows are deleted only with their company or agent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:08 -06:00
DottaandClaude Opus 5.5 8aaf983151 perf(db): send a transaction's work notifications in one statement at commit (#21436)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server signals queue work with database work signals, and it
sends a PostgreSQL notification for each signal so that another server
can wake during a deploy overlap
> - Each signal sends its own `select pg_notify($1, $2)`, inside the
open transaction, and waits for it
> - On staging this is the most-called statement in the database: 31.8
per second, 15.1 million calls in total against 7.5 million `begin`,
about 3.7 awaits per transaction
> - Every await inside an open transaction holds its locks and its
connection for one more turn of a saturated event loop
> - This pull request collects a transaction's notifications and sends
them in one statement just before commit
> - The benefit is at most one notification round trip per transaction,
with the same payloads delivered at the same commit

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Database work signals send one `pg_notify` statement per signal, inside
the transaction.

**Subsystem affected**
`packages/db` work signals, used by every delivery queue in the server.

**Current behavior**
`signalDatabaseWork` and `notifyDatabaseWorkOnCommit` each run `select
pg_notify(channel, payload)` on the transaction and wait for it.
`notifyIssueRecovery` alone signals 7 queues per call, and a transaction
often calls it several times. A local legacy run (wake, start, 20 output
lines, finish) sends 424 notification statements. Staging measures 31.8
per second, 8.5 `begin` per second.

**Proposed behavior**
A signal inside a transaction records its payload on that transaction.
Before the outermost commit, the transaction sends every distinct
payload in one statement. Payloads and the channel are byte-identical to
today's, and PostgreSQL still delivers them only on commit.

**Reason and benefit**
At most one notification statement per transaction instead of one per
signal. Fewer awaits while a transaction holds its locks and its
connection.

**Breaking changes**
None. The listener protocol and the payload format do not change.

## What Changed

- `packages/db/src/work-signals.ts`: a signal records `(channel,
payload)` on the current transaction level. A released savepoint merges
its records into its parent; a rolled-back savepoint drops them. The
outermost transaction sends all distinct payloads in one statement
(`select pg_notify(channel, payload) from
jsonb_array_elements_text($2::jsonb)`) after its callback returns and
before commit. A transaction that signals nothing sends no extra
statement.
- The record lives on the transaction object only. A retried callback
gets a new transaction object and starts empty.
- `publishDatabaseWorkHint` (outside a transaction) is unchanged and
still sends at once.
- `packages/db/src/work-signals.test.ts`: tests on real PostgreSQL with
a LISTEN client.

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/work-signals.test.ts`: 22 pass. 6 of the new tests fail on master.
- New tests (real PostgreSQL, LISTEN client, asserting what is delivered
as well as what is sent):
- Distinct topics in one transaction: one statement at commit. The
payloads delivered are identical to the format on master (`{version,
origin, topic}`), in first-signal order.
  - A transaction that signals nothing: no notification statement.
- Savepoints: a released savepoint two levels deep keeps its topic. A
rolled-back savepoint drops its topic. A topic from a rolled-back
savepoint that the outer level signals again is still delivered.
  - The callback throws: nothing is sent or delivered.
- The commit-time send fails: the transaction fails and rolls back its
write, and nothing is delivered.
  - 500 distinct topics: one statement, all 500 delivered.
  - Sequential, concurrent and retried transactions each start clean.
- Server suites that use notifications pass: `deployment-notifications`,
`run-start-handover`, `heartbeat-run-start-handover`,
`agent-runtime-hooks`, `legacy-controller-lease`,
`heartbeat-run-dispatch-lifecycle` (99 tests). Two of them hit statement
timeouts once on a loaded machine and then passed 41/41 when run alone.
- Before and after, one local legacy run (embedded PostgreSQL, process
adapter, 20 output lines 50 ms apart, uninstrumented pass, alternating
arms):
`TSX_DISABLE_CACHE=1 UNITS=legacy TASKS=50 AGENTS=3
NODE_OPTIONS=--expose-gc pnpm --filter @paperclipai/server exec tsx
scripts/query-cost/drive.ts` (harness from #21413)

  | Per legacy run | Before (2 runs) | After (2 runs) |
  | --- | --- | --- |
  | Statements | 883, 883 | 474, 491 |
  | `pg_notify` statements | 424 | 21 (one per signalling transaction) |
  | Notification round trips per signalling transaction | about 20 | 1 |
  | Transactions (BEGIN plus COMMIT) | 72 | 72 |
  | Allocated | 81.1, 79.3 MB | 76.3, 74.3 MB |
  | Wall time | 5.0, 5.2 s | 4.1, 7.7 s (noisy: loaded machine) |

Statement counts are exact and stable. Allocation fell by about 5 MB per
run (about 6%). Wall time varies too much on this machine to compare.
- Staging, measured by the coordinator (Neon statistics, 64 seconds):
31.8 `pg_notify` per second, 8.5 `begin` per second. At about 44 KB of
allocation per statement, notifications cost about 1.4 MB/s, about 2.4%
of the 57.6 MB/s. The allocation saving is part of that, about 2%, and
it is not the reason for this change. The reason is round trips inside
open transactions: at most one notification statement per transaction
instead of about 3.7. The per-second effect on staging is an estimate
until it is measured after deploy.

## Risks

- A failure of the notification send now surfaces at commit time, not at
the signal call. No caller catches errors around a signal call and
continues: the only `.catch` near a signal (`notifyBudgetStopsSettled`)
wraps a root-level call that is not in a transaction and already fails
before this change.
- Order within a transaction: notifications are sent in first-signal
order. A savepoint's records join its parent when the savepoint is
released, which matches the order of sequential code. Listeners do not
depend on order: each payload is an independent hint, and PostgreSQL
already removes duplicate payloads within a transaction.
- The notification is sent after the transaction's last statement
instead of in the middle. Delivery still happens only at commit, as
before.

Remaining work, not in this pull request: sending stops being useful
when no other server listens. The listener is open only during a startup
handoff, while this process owns a native execution, or during a
run-start handover, and a server drops its own notifications. A later
change could send only while an overlap is possible. It would have to
prove this interleaving: a new server announces itself, the old server
is still outside its own window, and the old server must start sending
before the new server relies on a hint. Otherwise the new server waits
for its next reconciliation pass.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation describes the send timing)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:39:36 -06:00
DottaandClaude Opus 5.5 2de4787f41 fix(db): release a postgres.js connection's receive buffer once it is fully read (#21432)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - A hosted server keeps about 0.5 GB of `external` memory (Buffers)
for its whole life, with 340 open database connections
> - postgres.js keeps a zero-length view of the buffer of each
connection's last read. The view keeps the whole buffer alive until that
connection reads again
> - An idle connection therefore pins the socket chunk or the whole
buffer of its largest recent message, which can be one MB or more
> - This pull request replaces a fully read buffer with an empty one in
the driver patch
> - The benefit is that idle connections hold no receive buffers. At 340
connections that each read a 1 MB row, external memory drops from about
320 MB to 0

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server holds 455 to 520 MB of `external` memory for
its whole life. It has about 340 open database connections. Each idle
postgres.js connection keeps the buffer of its last read.

**Steps to reproduce**
1. Open 80 or 340 pool connections.
2. Read a 1 MB row on each connection.
3. Drop the results, let the connections go idle, and force garbage
collection.
4. Measure `process.memoryUsage().external`. Each idle connection still
holds about 0.95 MB.

**Expected behavior**
An idle connection holds no buffer from a result that it has completely
read.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18.

## What Changed

- `patches/postgres@3.4.9.patch`: in `data()` of `src/connection.js` and
`cjs/src/connection.js`, the driver now replaces the receive buffer with
an empty buffer when it has handled every message in it. A partly read
buffer is not changed. The other hunks of the patch are unchanged. I
regenerated the patch with `pnpm patch postgres@3.4.9`, edited the two
files, and ran `pnpm patch-commit`. I did not commit the lockfile change
from that command.
- `packages/db/src/postgres-receive-buffer.test.ts` and
`packages/db/src/__fixtures__/postgres-receive-buffer.mjs`: a regression
test for ESM and CommonJS on real PostgreSQL.
- It checks that results are reassembled correctly for a 3 MB row, for
161 row sizes around one 64 KB socket chunk (to land message boundaries
at the end of a chunk), and for 20,000 small rows that arrive many to a
chunk.
- It records a weak reference to each large receive buffer and checks
that none of them stays alive after the connection becomes idle.
- `packages/db/scripts/benchmark-postgres-receive-buffer.ts`: one
command that reports heap, external and rss per connection, before and
after large rows.
- `doc/DATABASE.md`: documents the new patch behavior, the test and the
benchmark.

## Verification

- Red/green: with the new line removed from the installed driver, the
fixture fails for ESM and CommonJS ("an idle connection must not keep
the receive buffer of its last result"). With the line in place it
passes, repeated twice for each format.
- `pnpm --filter @paperclipai/db exec vitest run
src/postgres-receive-buffer.test.ts
src/postgres-connection-recovery.test.ts`: 20 passed.
- `pnpm --filter @paperclipai/db exec vitest run src/backup-lib.test.ts
src/backup-lib-binary.test.ts src/client-disconnect-replay.test.ts
src/client-teardown-registry.test.ts src/work-signals.test.ts`: 36
passed. In 1 of 4 runs, the JavaScript backup test (COPY and cursor
reads) timed out on a machine with load average 20 to 34. It took 39 s
and 25 s with the change and 21 s without it in passing runs. Read
throughput is the same with and without the change: a 200,000-row select
plus a COPY TO STDOUT has a median of 319 and 348 ms with it, and 329 ms
without it.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/prepared-statements-driver.test.ts
src/__tests__/prepared-statements.test.ts`: 5 passed.
- `pnpm --filter @paperclipai/db typecheck`: passed.
- Benchmark: `node --expose-gc --import tsx
packages/db/scripts/benchmark-postgres-receive-buffer.ts` with a 1 MB
row on each connection, numbers after garbage collection:

| Connections | External per idle connection, before | After | External
held in total, before | After |
| --- | --- | --- | --- | --- |
| 80 | 917 KB | 1 KB | 72 MB | 0 MB |
| 340 | 960 KB | 0 KB | 319 MB | 0 MB |

The rss per connection also went down (80 connections: 1,555 to 1,375
KB; 340 connections: 1,308 to 714 KB). The allocator keeps some freed
memory, so rss changes less than external memory.

## Risks

- Low risk. The change runs only when the buffer has no unread bytes. A
message split across chunks keeps the same reassembly path. The test
covers split messages, exact chunk boundaries and many messages in one
chunk.
- This saves external memory (Buffers outside the JavaScript heap). It
does not reduce `heapUsed`. It does not fix the out-of-memory crashes of
the hosted server by itself.
- The patch must be removed or ported when Paperclip upgrades
postgres.js. The test fails if a driver version keeps the buffer again.

## Model Used

- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), used in
Claude Code with tool use (shell, file editing, local PostgreSQL), heap
snapshots and allocation measurement.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:38:32 -06:00
DottaandClaude Opus 5.5 944b1c64d7 feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use connected apps through Paperclip's MCP client in the tool
gateway
> - Some provider actions run for minutes: CI runs, deploys, batch jobs
> - Today such a call is lost if it outlives the tool timeout or if
Paperclip restarts or deploys mid-call, and a retry can start the work a
second time
> - Recovery must just work, and the target rate of runs that need a
manual retry is 0%
> - This pull request implements the MCP Tasks extension
(`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client
> - The benefit is that a long provider call becomes a durable task: it
survives timeouts, restarts, and deploys, and it is never started twice

## Linked Issues or Issue Description

**Problem**
A long provider tool call holds an HTTP request open. When it passes the
tool timeout, or when Paperclip restarts or deploys during the call, the
result is lost. The agent then often repeats the call, and the provider
does the work again.

**Proposed solution**
Implement the client side of the MCP Tasks extension. Declare it on
stateless tool calls. Store each returned task durably, wait on it
during the call, keep polling it in the background after the call ends,
and let the same call attach to the stored task instead of sending
`tools/call` again.

**Why it matters**
CI, deploy, and batch actions are where agents lose the most work today.
A durable task handle makes these actions safe across timeouts,
restarts, and deploys, without a person.

Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP
client, and MCP Skills work this builds on).

## What Changed

- **Off by default:** MCP Tasks are behind the new instance experimental
setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default
off. When it is off, Paperclip never declares the extension, nothing
attaches to or polls tasks, pending rows do not block idle sleep, and
behavior matches master. A test covers the off state.

- `mcp-http.ts`: the tasks extension is declared per request, only on
HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers
do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`.
- New `mcp-tasks.ts`: parses task results, bounds the server's timing
hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals
task IDs at rest (AES-256-GCM, bound to company and connection; a keyed
hash gives uniqueness).
- New company-scoped table `tool_mcp_tasks` (migration `0475`,
online-compatible). It links the task to the call, run, task,
connection, and grant. It stores no provider result.
- `tool-gateway.ts`:
- A returned task is recorded, then polled up to the normal tool
timeout. Its result goes through the same validation, redaction, and
recording as a direct result.
- Past the timeout, or when the caller disconnects, the agent gets
`provider_task_pending` ("call again with the same arguments"). The task
stays durable.
- A call with the same endpoint, tool, full arguments, grant, agent,
task, and actor attaches to the undelivered task. A retry with the same
idempotency key also attaches, and it can never start different work
under that key.
- Input: URL requests show the provider link (Paperclip never answers
them). Form requests become a question card; only the person's answer is
sent with `tasks/update`. Other requests stop with a clear "needs input"
error.
- Stopping the run sends `tasks/cancel` and marks the task cancelled. A
caller that only disconnects keeps the task.
- `failed` becomes a tool error with the provider's message. An unknown
task or a passed TTL becomes `provider_task_expired`. Every task stops
being tracked at its TTL, or after 7 days at most.
- Audit entries: `tool_gateway.task_started`, `task_input_sent`,
`task_cancelled`.
- New `mcp-task-work.ts`, registered at startup with the shared deadline
scheduler: it resumes polling pending tasks after a restart, cancels
tasks whose task closed or run was stopped, and wakes the assignee once
when a task settles after its run ended.
- Docs: `doc/connections/MCP-TASKS.md`, linked from
`doc/connections/README.md`.

Background work (per `rules/sleep-aware-background-work.mdc`): trigger
is a task row write or release (`mcp-tasks` queue) and one startup pass.
The deadline is the earliest `next_poll_at`. Cadence is the server's
`pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting
for a person. It is quiet when no row has a next poll. This is the
narrow external-source exception: providers give no wake Paperclip can
receive, and polling stops with the task.

## Verification

- `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake
2026-07-28 server that returns tasks only to a client that declares the
extension and checks `Mcp-Name`. They cover:
- polling to completion, with the result identical to a synchronous
call;
  - timeout, then an attaching call with no second `tools/call`;
  - a keyed retry, and a reused key with different arguments (refused);
- restart mid-task: a fresh gateway with no in-memory state polls the
row, wakes the agent once, and the recovered run gets the result once
with no second `tools/call`;
  - no wake while the run is still going;
- URL input (link, never answered), and form input (question card, then
`tasks/update` with the answer);
- cancellation by the run, by closing the task, and no cancellation when
only the caller leaves;
- failure, an unknown task, and TTL expiry (ends in under 3 s, no hang);
  - another agent never attaches;
- synchronous calls unchanged and older servers never see the extension.
- Red/green: with attach disabled, the timeout, keyed-retry, and restart
tests fail (3). With `tasks/cancel` disabled, both cancellation tests
fail (2). Before `tools/call` declared the extension, 13 end-to-end
tests failed. All pass with the change.
- Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`,
`mcp-skills`, `server-startup-feedback-export`, `tool-gateway`,
`tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests).
- `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db
check:migrations` passes (the migration is classified
online-compatible). Snapshots pruned to five.
- Agent boot and time-to-first-token: no change in the boot or
first-turn path. The only new per-call cost is one indexed lookup on
stateless tool calls.
- Not run: a live provider that returns tasks. No public MCP server that
returns tasks was available without new credentials.

## Risks

- Shipped off by default. Review findings that only affect the
setting-on path after this merge are tracked as follow-ups.

- Migration: a new table with indexes and foreign keys to `companies`,
`tool_invocations`, and `tool_connections`. Online-compatible.
- Behaviour change only for servers that return tasks, which they may do
only to clients that declare the extension. Older servers and stdio
servers are unchanged.
- A task that settles while nobody is waiting stays attachable for 1
hour, so a deliberate repeat of the same call within that hour collects
the earlier result instead of starting new work.
- The wake after a background task settles uses the generic task wake;
the agent must call the tool again to collect the result.
- Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling
and roots input requests; showing a long action's progress in the task
chat.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:32:06 -06:00
DottaandClaude Opus 5.5 8af889f612 feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP Apps (#21119) show interactive widgets from MCP servers, such as
Circleback meeting cards, with the agent's reply in the task chat.
> - The first version deferred three parts of the MCP Apps spec. A
widget could not propose a chat message (`ui/message`). It could not
give the agent context (`ui/update-model-context`). It could not go full
screen.
> - Paperclip also ignored the `icons` that MCP servers declare on tools
(SEP-973).
> - The security review of #21119 showed what these features need from
day one: explicit consent, untrusted-data labels and strict icon
sanitizing.
> - This pull request adds the three host features and provider tool
icons, with those protections built in.
> - The benefit is that widgets can now work with the person and the
agent, not only display data.

## Linked Issues or Issue Description

Refs #21119

**Subsystem affected**
MCP Apps host (task chat widgets), tool catalog, run wake prompt.

**Problem or motivation**
The MCP Apps host refused `ui/message` (-32601), dropped
`ui/update-model-context` and kept every view inline. MCP tool `icons`
were not parsed or shown. So widgets could not ask the agent a
follow-up, the agent could not see what the person selected in a widget,
large widgets were cramped, and tool rows used only generic icons.

**Proposed solution**
Add consent-gated view messages, per-view app context delivered to the
task's next run as untrusted data, a full-screen dialog, and sanitized
provider icons. Details are under "What Changed".

**Alternatives considered**
- Send view messages with a new server route that creates the comment
itself. Rejected: comment creation has many rules (agent chat ownership,
paused conversations, queued delivery, wakeups, receipts). The widget's
Send button uses the same send path as the composer. A small route then
records which view the comment came from.
- Keep the iframe in place for full screen with CSS. Rejected: chat
containers can create containing blocks that break `position: fixed`. A
moved iframe always reloads, so the dialog hosts a new frame instead.

**Roadmap alignment**
Continues the MCP feature work after #21119, #21140 and #21219. It does
not touch the MCP Tasks work (gateway `tools/call` polling and
elicitation).

## What Changed

- **`ui/message`.** The host shows "<App> wants to send: …" with
**Send** and **Dismiss** under the widget (and inside the full-screen
dialog). Nothing is sent before the click. Send posts the text as the
viewer's own comment through the chat's normal send path
(`handleThreadAdd`). `POST
/api/issues/:issueId/mcp-app-views/:viewId/messages` then logs
`issue.mcp_app_message_sent`. It accepts only the caller's own comment
on the same task, matched by `clientRequestId`, from the last 10
minutes. One proposal per view can wait at a time. The view gets `{}`
after the send, or -32000 "Message sending denied" after Dismiss. Only
`user` text up to 8,000 characters is accepted.
- **`ui/update-model-context`.** `PUT
/api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the
latest context per view in `tool_mcp_app_views.model_context` (text
blocks and `structuredContent`, 16 KB at most). Updates are applied in
order and deduped in the browser and on the server. A change logs
`issue.mcp_app_context_updated`, and an empty update clears the context.
The task's next run gets the three most recently updated contexts as an
`## App context (untrusted)` wake-prompt section. The section quotes the
data as JSON and says it is third-party data, not instructions. The read
runs in parallel with the other run preparation and never fails the run.
CLI adapters and the native runner render it through the same
`renderPaperclipWakePrompt`.
- **Who may do this.** Messages and context need a signed-in person with
`issue:comment` on the task. Agents and API keys get 403. Other tasks
and companies get 404. `message` and `updateModelContext` are advertised
in `hostCapabilities` only where the viewer can send messages.
- **Full screen.** The host advertises `availableDisplayModes:
["inline","fullscreen"]` and handles `ui/request-display-mode`. It
honors the view's declared modes and answers with the resulting mode. It
sends `host-context-changed` with `displayMode` and the container (fixed
`height`/`width` in full screen), and again when that container resizes.
A new **Open full screen** control sits next to "Widget from …". Full
screen is a Paperclip dialog that closes with Exit, Escape or the view's
own request.
- **Provider tool icons.** `icons` on MCP tool descriptors are sanitized
into the new `tool_catalog_entries.icons` column, outside the version
hash, so a new icon never quarantines a reviewed tool. Only `https:`
URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A
`data:` icon is at most 32 KB, a URL at most 2,048 characters, with at
most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists
them keyed by gateway tool name, in one cached request per chat. Task
chat tool rows and the app's actions list show them through
`ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a
fixed size and never inlines markup. It checks the source again before
rendering and falls back to the taxonomy icon.
- **Schema.** Migration `0477` adds `tool_catalog_entries.icons`,
`tool_mcp_app_views.model_context` and `model_context_updated_at`, all
nullable.
- **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new
behavior, and the finished items are removed from "Limits and
follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a
message", "Full screen" and "Tool rows with provider icons". The fixture
view now uses `ui/message`, `ui/request-display-mode` and
`ui/update-model-context`.

## Verification

- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/openapi-routes.test.ts
packages/shared/src/mcp-icons.test.ts
packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These
tests cover:
- Context: stored, deduped, logged once, 413 when too large, cleared,
scoped to the task and company, refused for agents (403), and rendered
into the wake prompt as untrusted data.
- Messages: attributed only to the viewer's own recent comment on the
same task. A teammate's comment, an unknown request id or an old comment
gets 404. An agent gets 403.
- Icons: stored sanitized, and the version hash and status do not
change. Another company sees nothing.
- `cd ui && pnpm exec vitest run src/components/mcp-app/
src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69
pass. These tests cover:
- The host protocol for each new method: capabilities, consent wait, one
proposal at a time, invalid input, ordered and deduped context, the size
limit, and the display-mode rules.
  - The consent card (nothing sent without Send).
  - The full-screen dialog (one frame at a time).
  - Icon rendering (no-referrer `img`, no inline SVG, fallback).
- `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for
the server: clean.
- Red/green: the new route and host tests fail on `master`, where the
routes do not exist and the host answers -32601 or drops updates. The
icon descriptor test failed when the publication write was removed.
- Not run locally: the full suite, and
`packages/adapter-utils/src/server-utils.test.ts › runChildProcess ›
does not crash when the child closes stdin before a delayed write`. That
test times out on this Mac with or without this change, so it is a local
platform issue. CI covers both.

## Risks

- Migration risk is low: three nullable columns, no backfill.
- Every run on a task now does one extra indexed read, in parallel with
the other preparation. Errors are logged and the run continues without
app context.
- If the attribution request fails after the comment is sent, the
comment stays without the `mcp_app_message_sent` activity row. This is
documented.
- Switching between inline and full screen restarts the view, so state
inside the widget resets. This is documented as a limit.
- Icon names shortened with a hash by the gateway (very long names) do
not match, and those rows keep the taxonomy icon.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. One subagent (the same model) built the provider-icon
part.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:32:48 -06:00
DottaandClaude Opus 5.5 49f0030f06 fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:17:57 -06:00
Nicky LeachandClaude Opus 5.5 717aae060a fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute on environment leases, and `createOwnedSandbox`
records each sandbox creation so that a lost reply never creates a
second sandbox
> - The record key is company, environment, run and request key. The
plugin request key is the environment fingerprint, so it is the same for
every attempt of one run
> - A same-run recovery can need a new environment after the first lease
is closed with successful cleanup and a termination receipt
> - The old record is still `attached` to the closed lease, so the
recovery fails with "Sandbox creation has already transferred ownership"
before the provider is called
> - This pull request lets the creation fence accept the same stop proof
that recovery already requires. A creation record whose sandbox is
proven stopped becomes resolved, and a resolved record no longer holds
its key
> - The benefit is that same-run recovery gets one replacement sandbox,
while unconfirmed cleanup still blocks a second sandbox

## Linked Issues or Issue Description

Fixes #21214

Refs #20755. That pull request fixes the same failure with a different
rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when
the recorded lease has ended, and it accepts any ended status, including
`failed`, without a termination receipt. A lease whose cleanup is not
confirmed can then produce a second sandbox while the first one can
still run. This pull request keeps the original key and requires the
same stop proof that Core uses elsewhere
(`hasRemoteTerminationReceipt`). It does not add a key generation.

## What Changed

- `server/src/services/sandbox-creation-operations.ts`: when
`createOwnedSandbox` finds the key held by an `attached` operation whose
lease is not active, it checks the same stop proof that same-run
recovery requires before it acquires:
`hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for
the sandbox that this operation created (`creationProviderLeaseId`, or
the lease's `providerLeaseId`).
- With that proof, it runs the same admission transaction one more time.
That transaction first marks the old operation `resolved` (`WHERE id AND
state = 'attached' AND lease_id`), and then inserts the new operation.
  - Without that proof, it throws exactly as before.
- The check covers every source of a stop receipt: `releaseLease`, the
Runner `complete` operation (which writes receipts with direct SQL for
reusable leases), stop-and-retain settlement, and the receipt that
`resolveRemoteTerminationLease` derives for an expired reusable lease.
- A task replacement can leave an earlier operation attached to a lease
that now holds another sandbox. That operation is not proven and keeps
its key.
- Concurrent recoveries serialize on the row update. The second one
updates nothing, finds the first one's unresolved operation through the
partial index, and replays or refuses as before. There is one provider
call.
- Migration `0475_sandbox_creation_owner_unresolved.sql` recreates
`sandbox_creation_operations_owner_idx` under the same name as a partial
unique index (`state <> 'resolved'`). The drop and the create are in one
migration transaction. The Drizzle schema matches.
- The conflict lookup in `createOwnedSandbox` ignores resolved rows. An
active attached lease is still replayed. Pending and
`needs_reconciliation` rows still refuse. `releaseLease` does not
change.
- `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the
operation lifecycle.

## Verification

- `pnpm --filter @paperclipai/db generate`
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`
- `vitest run src/migration-snapshot-drift.test.ts` (db)
- `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48.
With the service change reverted, the 5 replacement tests fail.
- The 9 other server test files that read `sandbox_creation_operations`,
plus `environment-runtime.test.ts`: 564/564
- `pnpm check:migrations` passes

New tests in `sandbox-creation-operations.test.ts`:

- A stop receipt from `releaseLease` (sandbox driver, and the plugin
driver through `acquireRunLease` in the #21214 sequence), from the
`complete` operation's direct write on a retained reusable lease, and
derived only from the destroyed successor of an expired reusable lease
each let the next creation resolve the old operation and create exactly
one replacement. A retry replays that replacement. These tests fail
before the fix.
- Failed cleanup, success without a receipt, a receipt for another lease
or another provider task, a retained lease, and pending cleanup each
still throw "transferred ownership" with no provider call.
- An earlier task's operation that a task replacement left attached
keeps its key.
- Two concurrent recoveries make one provider call. This test fails
before the fix.
- Reapplying migration 0418 keeps the partial index.

I did not run `pnpm verify` or the full suite locally. CI runs them.

## Risks

- Migration: the drop and the rebuild of the index take an exclusive
lock on `sandbox_creation_operations`. The table has one small row for
each sandbox creation, so the lock is short.
- Rolling deploy: the migration is marked `online_compatible`. The
previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also
matches a partial unique index. Its conflict lookup has no state filter,
so it can find a resolved row and throw. That fails closed and never
creates a second sandbox.
- A legacy runner-task operation without `creationProviderLeaseId`
matches any lease it is attached to. Runner-task operations record this
field before they attach, so this is only a theoretical gap.
- An operation stays `attached` until the next creation with the same
key checks it. This is the same as before when no recovery comes.
- Stop-and-retain settlement is not tested directly. The shared receipt
check covers it.
- The fence is unchanged for every case without a stop receipt.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:24:34 -08:00
DottaandClaude Opus 5.5 500bc694b5 fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap runs two servers for a short overlap. The old server
drains, and the new server takes over the runs.
> - Each server proves that it is alive with a controller-instance
lease. Recovery uses that lease to decide whether a run's controller is
gone.
> - In local swap cells, a healthy old server exited during the overlap,
because its lease renewal waited for a connection on the busy main pool.
Every run it drove was interrupted.
> - When a controller crashed, its runs waited for another server boot,
or for a timer that restarted on every look. Tasks stood still for 21 to
28 minutes.
> - This pull request renews the lease on its own connection, and lets
the live server end a crashed controller's runs within a bounded time.
> - The benefit is that a swap does not kill a healthy server, and a
crashed server's runs end in about 2 minutes, so their tasks continue by
themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. During a two-server release swap (local cells, load 14 to 34), the
outgoing server exited with "controller instance lease lost while
holding work". Five lease renewals in a row timed out after 10 s. The 10
s timeout included the wait for a connection on the shared main pool,
which the traffic switch and the replacement's startup pass kept busy.
2. When the old server was killed in the middle of the overlap, its runs
stayed in progress for 21 to 28 minutes. The live server did not
re-check a crashed controller's runs: only a new boot did. The
unverified-controller wait started when a server first looked at the
run, not when the controller left, and the successor's own wait record
counted as the dead owner's sign of life.

**Expected behavior**
A healthy server keeps its lease during a swap. When a controller is
gone (its lease lapsed and it is fenced or stopped), the live server
ends that controller's runs within a bounded time, without another
server boot, so restart recovery continues the tasks.

**Steps to reproduce**
1. Run two servers against one database with a release swap and a 200 s
overlap.
2. Kill the old server 40 s into the overlap (SIGKILL).
3. See that its runs stay in progress for more than 20 minutes, until
another server starts.

## What Changed

- Lease renewal on its own connection (`packages/db/src/client.ts`,
`packages/db/src/index.ts`,
`server/src/services/controller-instance-lease.ts`):
- New `controllerInstanceLeaseDb(db)`: one private 1-connection pool
(`application_name` `paperclip-controller-instance-lease`), made the
same way as the existing `controllerLeaseDb` and closed by
`closeLivenessDb`.
  - The renewal is one statement on that connection, as before.
- The connection sends `statement_timeout` = 8 s (below the 10 s client
timeout) as a startup parameter (new
`DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The
server cancels a slow renewal, so it never holds the single connection
that the next renewal needs, and there is no extra round trip per
renewal. (An earlier revision set it per transaction; that added three
round trips to every renewal and broke the idle-server receipt test.)
- If a connection pooler refuses the startup parameter ("unsupported
startup parameter"), the lease logs a warning and renews on a
replacement connection without it.
- A crashed controller's runs
(`server/src/services/recovery/ownership-registry.ts`,
`server/src/index.ts`,
`server/src/services/native-runtime/native-restart-recovery.ts`):
- The ownership reconciler on the live server hands runs whose
controller is proven gone to restart recovery. It does not wait for
another server boot.
- The unverified-controller wait is bounded from the moment the
controller's lease lapsed, not from the first look.
- When the controller boot is `stopped` or `fenced`, the run ends at
once.
- A successor's wait record no longer refreshes the dead owner's sign of
life.

Safety of ending a run when its controller boot is fenced or stopped:
- Only `stopped` (written by the boot's own graceful stop) and `fenced`
(a newer epoch fenced it after its lease expired plus 30 s grace) count.
`live` and `grace` never count, so a controller that is alive and
renewing is never treated as gone. `expired` alone does not count
either.
- A boot that cannot renew stops itself at its self-fence window (about
its lease expiry), and a boot that finds itself fenced exits at once. So
when another server can fence it, the old process has already stopped
driving runs. The lease change above removes the main cause of healthy
boots failing to renew.
- If the old process is still alive (partition or stall) and its Runner
continues: the old run is fenced. Its agent token is refused for writes
(`agent_run_superseded`), its remote workspace epoch is abandoned or the
continuation gets a new workspace, and the native coordinator records
`replacementDenied`. There are never two writers in one workspace.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/controller-instance-lease.test.ts` "own
connection": red (10004 ms timeout while the main pool is held) to
green. Full file green (26).
- `server/src/__tests__/controller-instance-lease.test.ts` "has the
server cancel a renewal blocked on a row lock, and renews on the same
connection next time": red without the connection-level timeout (the
renewal runs to the 10 s client timeout and stays active), green with it
(cancelled at about 8 s, no active lease statement left, the next
renewal succeeds in under 2 s on the same backend).
- `packages/db/src/client-options.test.ts`: the statement timeout is
sent as a startup parameter.
- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`:
"lapsed longer ago" / "end of the bound" (wait counted from the lease
lapse), the live-server re-check of a crashed controller, the
fenced/stopped boot ending the run at once, and the successor's wait
record. Full file green.

Changed existing expectation (it asserted the old wait for a stopped or
fenced controller):
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
"keeps foreign local Runner and provider ownership after its controller
is stopped/fenced": the disposition is now `blocked`
(`controller_process_space_unverified`) instead of `awaiting_evidence`,
the run is `failed` with `native_restart_recovery_blocked`, and the
finalization is `terminal_failure` with `executorStopEvidence:
"unverified"`. The safety checks stay: no local PID probe, no claim, no
stamp, no local stop proof, the run keeps its process identity and
session checkpoint, and no result is written.

Local commands, on this branch based on current master:
- `cd server && npx vitest run
src/__tests__/controller-instance-lease.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140
tests passed.
- `npx vitest run src/__tests__/controller-instance-lease.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
225 passed, 14 skipped (real-process cases that need CI).
- `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts
-t "releases an ordinary held task Retry"`: passed (idle receipt: one
lease statement after the work ends).

Local swap cells (two servers, embedded Postgres, real agents, old
server SIGKILLed 40 s into a 200 s overlap):
- Before: runs of the killed server stayed in progress 21 to 28 minutes.
- N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the
successor's wait write still moved the timer; fixed by the last commit).
- R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run
end 3 min 39 s.
- N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min
01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the
lease TTL (90 s) plus the fence grace (30 s).
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds this fix and the related recovery fixes: 11/11 tasks finished
hands-off (master: 9/11). The lease and crashed-controller fixes were
visibly working in it.

## Risks

- A pooled database URL whose pooler refuses `statement_timeout` as a
startup parameter: the first renewal fails over to a connection without
it (logged), so the lease keeps working without the server-side cancel.
- One extra database connection per server, held while the server runs.
It is idle except for one renewal every few seconds.
- A run whose controller boot is fenced or stopped now ends at once
instead of after the bound. See the safety notes above: only a boot that
is not alive can be in these states.
- No migration. No schema change. Nothing changes before `turn.start` on
the normal path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:11:27 -06:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
DottaandClaude Fable 5.2 31c3412bbb perf(db): build tx.query on first read instead of for every transaction (#21148)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every server module reaches PostgreSQL through one client,
`createDb` in `packages/db`. The server has about 1,260 `transaction(`
call sites. Run events, runner checkpoints and the run queue use a
transaction for each write.
> - Drizzle builds one relational query builder for every schema table
when it creates a database object. It creates a database object for each
transaction and for each savepoint.
> - The Paperclip schema has 321 tables. One `BEGIN` therefore allocates
about 100 KB and uses about 80 microseconds of CPU before the first
statement. One `SAVEPOINT` costs the same again. Five places read the
relational API inside a transaction.
> - On a hosted instance the server main thread was near one full core,
and garbage collection used a quarter of it. A fixed 100 KB for each
transaction is garbage that no caller asked for, and it grows with every
new table.
> - This pull request makes `createDb` build `tx.query` the first time a
transaction reads it.
> - The benefit is 82% less allocation and 7 times fewer garbage
collections for an empty transaction, and 45% less allocation for one
native run event. Transaction behavior does not change.

## Linked Issues or Issue Description

Related: #21131 (one transaction of two statements for a native run
event). That pull request reported this cost as a follow-up. No issue
exists for it.

**What existing behavior does this improve?**
The cost of `db.transaction()` and of a nested `tx.transaction()`
(savepoint) on the client that `createDb` returns.

**Subsystem affected**
Database client (`packages/db`).

**Current behavior**
`createDb` calls `drizzle(sql, { schema })`. Drizzle's `PgDatabase`
constructor then runs this loop for the root client, for every
transaction and for every savepoint:

```js
for (const [tableName, columns] of Object.entries(this._.schema)) {
  this.query[tableName] = new RelationalQueryBuilder(...);
}
```

A transaction is a `PgTransaction`, which extends `PgDatabase`
(`drizzle-orm/pg-core/session.js`). `PostgresJsSession.transaction` and
`PostgresJsTransaction.transaction` construct one for each `BEGIN` and
each `SAVEPOINT` (`drizzle-orm/postgres-js/session.js`).

Measured on `master` at 4e932fd178, Node 26, local PostgreSQL, 3,000
iterations, median of 5 passes:

| Scenario | Allocated | CPU | Collections per 1,000 |
| --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 27 µs | 0.3 |
| Empty transaction | 125.8 KB | 131 µs | 5.0 |
| Transaction with one statement | 136.6 KB | 153 µs | 5.3 |
| Transaction with an empty savepoint | 231.6 KB | 235 µs | 9.7 |
| Empty transaction, Drizzle without a schema | 18.5 KB | 53 µs | 0.7 |

The allocation profile of an empty transaction attributes 96.8 KB of
115.6 KB sampled to the `PgDatabase` constructor (`pg-core/db.js:17`),
reached from `PgTransaction` and `postgres-js/session.js:109`. About 50
KB is the builders and the `query` object. About 46 KB is
`Object.entries` of the schema.

The relational API is in use, always as a single-table `findFirst({
where[, columns] })`. It has 22 call sites. Five of them run on a
transaction: `services/announcements.ts:19` and `:25`,
`services/decision-training.ts:331`, and
`services/resource-memberships.ts:361` and `:369` (the service is built
on a transaction in `updateAgent` and in
`services/primary-agent.ts:60`). The other 17 run on the root client.

**Proposed behavior**
A transaction or savepoint that does not read `tx.query` builds no
relational query builder. A transaction that reads `tx.query` gets the
same builders as before, bound to that transaction.

**Reason and benefit**
Less garbage and less CPU on the server main thread for every write
path. Hosted statement statistics showed 13.4 native run events per
second and 5.3 checkpoint inserts per second. Each is one transaction,
so those two paths alone discarded about 2 MB per second. The total
transaction rate of the hosted instance was not measured.

**Breaking changes**
None. No schema change, no API change, and the `Db` type is identical.

## What Changed

- `packages/db/src/lazy-transaction-queries.ts` (new):
`drizzleWithLazyTransactionQueries(client, schema)`.
- It creates the root client with the schema, as before. `db.query` and
`db._` do not change.
- It creates a second Drizzle client on the same postgres.js client,
without the schema, and `db.transaction` opens transactions through it.
`BEGIN`, `SAVEPOINT`, commit, rollback and connection reservation are
still Drizzle's and the driver's own code. No Drizzle internals are
patched.
- Each transaction and savepoint gets a `query` accessor. The first read
builds one `RelationalQueryBuilder` for each table from the root
client's schema description and the transaction's session, then stores
the result as a plain property. `tx._.schema`, `tx._.fullSchema` and
`tx._.tableNamesMap` keep their values.
- `packages/db/src/client.ts`: `createDbClient` calls the new function.
The work-signal and response-capture wrappers install on top of it in
the same order as before.
- `packages/db/scripts/benchmark-transaction-allocation.ts` (new): the
benchmark in this description.
- `doc/DATABASE.md`: new section "Relational queries inside
transactions".

## Verification

Benchmark, same script before and after (`createDb` client):

```sh
cd packages/db
TSX_DISABLE_CACHE=1 node --expose-gc --import tsx scripts/benchmark-transaction-allocation.ts 3000 5
```

| Scenario | Allocated before | Allocated after | CPU before | CPU after
| Collections per 1,000 before | after |
| --- | --- | --- | --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 11.3 KB | 27 µs | 31 µs |
0.3 | 0.3 |
| Empty transaction | 125.8 KB | 22.4 KB | 131 µs | 54 µs | 5.0 | 0.7 |
| Transaction with one statement | 136.6 KB | 35.6 KB | 153 µs | 82 µs |
5.3 | 1.0 |
| Transaction with an empty savepoint | 231.6 KB | 35.6 KB | 235 µs |
111 µs | 9.7 | 1.3 |
| Transaction with a savepoint of one statement | 242.7 KB | 44.8 KB |
331 µs | 217 µs | 10.7 | 2.0 |

After the change, no Drizzle frame is in the allocation profile of an
empty transaction (19.8 KB sampled). The script also runs Drizzle with
the schema and Drizzle without a schema in the same process, so the
comparison can be repeated on any later revision.

The machine was shared and under load (load average 6 to 35 on 18 CPUs).
Allocation and collection counts were stable across four runs. CPU
values moved by a factor of two between runs. The CPU columns come from
the two runs with the lowest load.

One native run event, through the real append path:

```sh
TSX_DISABLE_CACHE=1 EVENTS=1500 PASSES=3 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-run-event-append.ts
```

`native-stream`: 220.4 -> 120.9 KB allocated for each event, 12.0 -> 5.3
collection pauses per 1,000 events. `native-burst`: 221.1 -> 121.0 KB,
10.6 -> 5.3 pauses. Statements for each event do not change (4.6).

Tests, all on real PostgreSQL
(`packages/db/src/lazy-transaction-queries.test.ts`, 15 tests):

- Red and green for the mechanism. On `master` these fail, and with the
change they pass:
- `builds none for a transaction or savepoint that never reads
tx.query`: 321 builders for the transaction and 321 for the savepoint on
`master`, 0 with the change.
- `allocates the same for an empty transaction as a client without the
schema`: 101 KB more than a schema-less client on `master`, below the 16
KB limit with the change. It reads V8's allocation counter, not a timer,
and it skips on a Node.js without that counter.
- Semantics. These pass on `master` and with the change, which shows the
behavior is the same:
  - `commits and returns the callback's value`
- `rolls back and rejects with the thrown error` (thrown error,
`tx.rollback()`, SQL error)
- `keeps the outer transaction usable after a savepoint rolls back`
(three levels)
- `discards a released savepoint when the outer transaction rolls back`
  - `applies the isolation level and access mode`
- `runs concurrent transactions on separate reserved connections and
releases them` (12 transactions on a pool of 3)
- `returns the only connection to the pool after a commit, a rollback
and a failed savepoint` (pool of 1)
  - `behaves the same with prepared statements enabled`
- Instrumentation. These pass on `master` and with the change:
- `registers each transaction and savepoint with the client's work
signals`: the transaction and the savepoint have the client's
work-signal scope, the registration statement runs on the transaction's
own connection, listeners get `intent` at once and `settled` only after
the outer commit, and a rollback reconciles. When transactions bypass
the work-signal wrapper, this test and five tests in
`work-signals.test.ts` fail.
- `response capture counts every statement of a transaction, a savepoint
and tx.query`: six statements, counted for each operation and table,
including two relational reads.
- Every `tx.query` pattern in the codebase:
- `reads tx.query through the transaction, as the announcement and
decision-training services do`
  - `reads tx.query inside a savepoint and follows its rollback`
- `gives tx.query one builder per schema table, the same object on every
read`

Commands run locally on the final revision:

- `pnpm exec vitest run src/lazy-transaction-queries.test.ts
src/work-signals.test.ts src/response-capture-postgres.test.ts` in
`packages/db` on the final revision: 43 passed.
`prepared-queries-postgres.test.ts` passed on the first commit.
- Before the rebase onto current `master`, also
`response-capture.test.ts`, `response-capture-bundle.test.ts`,
`client-options.test.ts`, `client-teardown-registry.test.ts`,
`checkpoint-pool.test.ts`, `client-disconnect-replay.test.ts`: 111
passed in 10 files.
- On the first commit, server callers of `tx.query` and `db.query`, and
hot transaction paths: `announcements-routes`, `decision-training`,
`heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`,
`primary-agent`, `resource-memberships-routes`, `workspace-preferences`,
`sidebar-preferences-routes`, `paperclip-control-plane-port`,
`native-runner-checkpoint-authority.integration`, `dispatch-work`,
`decision-target-work`, `issue-thread-interactions-service`: all passed.
`workspace-preferences` timed out once in its database start hook while
eight suites ran in parallel, and passed alone.
- `pnpm exec tsc --noEmit` in `packages/db`: passed. A type probe showed
that `Db` is the same type as before.
- `pnpm check:module-boundaries`: passed.
- `pnpm verify` exits 2: `packages/db` is a broad input. CI supplies the
full suites. I did not run the full local suite, repo-wide typecheck or
build.

Not verified: the hosted transaction rate, and the effect on a hosted
instance. CI uses Node 24, which has no V8 allocation counter. On CI the
allocation test skips itself, and the builder-count test covers the
mechanism. The benchmark needs Node.js 25 or later and stops with that
message on Node 24. I did not run that error path, because this machine
has Node 26 only.

## Risks

- Every module uses this client. The mitigation is that the transaction
path is still Drizzle's own `transaction()` on the same postgres.js
client. Only the schema argument is absent. The semantics tests pass
unchanged on `master` and on this branch.
- The change relies on Drizzle surface that is not documented API. The
lazy `tx.query` calls the `RelationalQueryBuilder` constructor from
`drizzle-orm/pg-core/query-builders/query`. That subpath is in Drizzle's
export map and the constructor is typed, but `drizzle-orm/pg-core` does
not re-export it. The change also writes the three schema fields of
`tx._`, which the types mark `readonly`. In `drizzle-orm@1.0.0-beta.22`
the same path exports a different class. An upgrade that changes the
constructor fails the typecheck and the `tx.query` tests. An upgrade
that renames `db._.schema` makes the function return the plain eager
client: behavior stays correct, the saving is lost, and the
builder-count test fails. The repository owner decides whether this
dependency on Drizzle is acceptable. A version that uses only documented
API removes the 22 relational reads from the server and creates the
client without the schema.
- `tx.query` is now an accessor until its first read. Code that reads
the property descriptor of `query` sees the difference. No code in the
repository does that.
- The second client and the builders use Drizzle's default dialect
options, as the root client does. If `createDb` ever passes `casing`,
`logger` or `cache` to Drizzle, the same option must go to all three.
The source comment says so.
- A transaction that reads `tx.query` pays the old cost once, on first
read.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use
and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 14:31:21 -06:00
DottaandClaude Opus 5.5 8ff45c0fa9 feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.

## Linked Issues or Issue Description

**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.

**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.

**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.

**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.

**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).

**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.

## What Changed

- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
  - App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
  - the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
  - warnings in the sign-in step;
  - `doc/connections/CIRCLEBACK.md`.
  
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.

## Verification

- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
  - `ui_meta` is stored;
  - an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
  - a call without a view records nothing;
  - the routes serve the view's HTML and CSP;
  - the view's tool call works and a model-only tool is refused;
  - another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
  - Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
  - Discovery stored the real view declarations:
    - `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
    - `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
  - A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
    - The 1 MB tool-result limit blocked the view document.
    - Views were hidden inside the folded tool history.

## Risks

- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:28:05 -06:00
DottaandClaude Opus 5.5 b8207c4ea9 fix(runner): evict finished API snapshots instead of blocking reads at the company quota (#21133)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs read the Paperclip API through `call_api`. Large
responses are saved as "API snapshot" assets so the agent can read them
in pages.
> - Each company has a 20 GiB snapshot quota. Before a large read, the
server reserves 1 GiB of worst-case space against that quota.
> - Snapshot assets were never removed. Reservations from interrupted
captures (crash, restart, ambiguous upload) stayed at 1 GiB each until
"an operator reconciles" them. Nothing did that.
> - After a morning of high retry volume, the quota filled. Every large
read then failed with `api_response_company_storage_limit`, and agents
blocked their own tasks and asked an operator to free storage.
> - This pull request makes company admission reclaim space by itself:
it releases stale reservations of finished runs and evicts the oldest
snapshots of finished runs, oldest first, only as much as the new read
needs.
> - The benefit is that a read never fails because old snapshots filled
the quota, and no person must clear storage by hand.

## Linked Issues or Issue Description

**What happened?**

On staging, about 97 tasks of one agent had successful runs but ended in
`blocked`. In each run, `GET` calls through `call_api` for large records
(for example heartbeat runs and their events) failed with
`api_response_company_storage_limit`: "Company API snapshot storage
quota is exhausted. Read an existing asset or ask the operator to remove
old snapshots or increase the quota." The agent then reported a
task-wide blocker owned by "the instance storage operator". At least 15
tasks had a final comment such as "Blocked: company API snapshot storage
is exhausted". The error message itself told the agent that an operator
must act.

**Expected behavior**

A read must not fail because old snapshots filled the company quota. The
server must reclaim the space automatically. Only snapshots that active
runs still hold, or that users attached to issues, cases or logos, may
stay.

**Steps to reproduce**

1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB.
2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or
leave two 1 GiB unattached reservations from finished runs that are
older than 20 minutes.
3. In a running native run, call `call_api` for a response larger than
the inline limit.
4. Before this change, the call fails with
`api_response_company_storage_limit`. After this change, the call
succeeds and the oldest finished snapshot is evicted.

**Paperclip version or commit**

Staging deploy `704b3f4d63`. Fix is based on `origin/master` at
`8bdd53c11f`.

**Deployment mode**

Hosted (staging cloud), native runtime with Daytona sandboxes.

**Agent adapter(s) involved**

Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is
in the server-side `call_api` capture admission, so it affects every
native adapter.

## What Changed

- Added
`server/src/services/native-runtime/runner-api-capture-retention.ts`. It
measures company snapshot usage and reclaims space under the existing
per-company advisory lock.
- Reclaim step 1: release unattached reservations older than twice the
capture deadline (20 minutes) whose run is missing or not `queued`,
`running` or `scheduled_retry`.
- Reclaim step 2: evict snapshot assets oldest first until the new
reservation fits. A snapshot is eligible only when it is older than 1
hour, every run that captured it has finished (succeeded, interrupted,
failed, cancelled or timed out), and no issue attachment, case
attachment or company logo links to it. Each eviction writes an
`asset.evicted` activity-log row with the asset id and the owning run
id.
- A capture upload that finishes after its reservation was reclaimed is
rejected in the publish transaction. The same transaction records the
object for deletion, and the bounded cleanup removes it. The upload has
a 10-minute abort signal.
- Failed deletes are logged per company and counted with the
`runner_api_capture_delete_failed` metric. When cleanup keeps failing,
the read error says so in plain words.
- Eviction credits deletes that are already pending before it chooses
more snapshots. Eviction is durable. In the same transaction that
deletes an asset row, it inserts a reservation row with
`pending_delete_object_key`, so the bytes stay counted. Admission then
deletes the stored objects in pages, under a 30-second abort signal, and
removes each row only after its delete succeeds. A failed or interrupted
delete is retried at the next over-quota admission. No background poller
is added.
- Added the `pending_delete_object_key` column and an index on
`runner_api_response_reservations.asset_id` (migration 0472).
- `#reserveApiResponseCapture` now runs reclaim only when the quota
would be exceeded, so reads below the quota do no extra work. After
marking evictions it finishes the deletes and retries admission once.
- When no space can be reclaimed, the error message now names what holds
the space (runs still working, files attached to tasks, deletes in
progress). It no longer tells the agent to ask an operator.
- Replaced the test that asserted old snapshots block reads with tests
for eviction, multi-page eviction, durable delete retry, a bounded
stalled delete, stale-reservation release, and protection of active and
attached snapshots.

## Verification

- Red: the three new tests in `runner-api.integration.test.ts` fail on
`origin/master` (two get `api_response_company_storage_limit`, one gets
the old "ask the operator" message).
- Green: `cd server && npx vitest run
src/services/native-runtime/runner-api.integration.test.ts` passes 70/70
with the fix.
- The multi-page test failed first with
`api_response_company_storage_limit` because pending deletes ran in one
200-row batch. The paged delete loop fixed it.
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- An agent that reads a saved snapshot of a finished run after it was
evicted gets a not-found response. It can read the API again. Eviction
happens only under quota pressure, oldest first.
- A released stale reservation can leave an unknown storage object from
an ambiguous upload. These bytes were never linked to an asset, so they
were already unreachable. This is the same storage the old code kept
counted forever.
- Eviction runs inside the company capture lock. The candidate query is
bounded to pages of 200 rows and uses the new `asset_id` index.
- Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`)
and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It
is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and
safe for old and new servers during a rolling deploy.
- During a storage outage, an over-quota read still fails until deletes
succeed. The bytes stay counted, so the quota is never exceeded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:38 -06:00
DottaandClaude Opus 5.5 dfc510b887 fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud replaces the server process on each release. The old
server keeps serving until the new one passes health checks and takes
the traffic.
> - During that overlap the old server keeps starting new runs right up
to SIGTERM, although the new server can already start them.
> - SIGTERM then cuts off exactly those runs while they still prepare.
In one audited swap the old server started 30 runs after the new server
booted, and 23 of them ended `process_lost`.
> - Admission must never pause, so the fix must hand over who starts new
runs, not stop admission.
> - This pull request makes the new server record when it starts runs.
The old server then leaves new run starts to it, and starts them again
if the new server stops or dies.
> - The benefit is that a deploy no longer starts runs on a server that
is about to stop, and the company always has one server that starts
runs.

## Linked Issues or Issue Description

Refs #20852 (preserves preparing and launching turns that a deploy
interrupts; this PR reduces how many such turns the old server starts).
Related: #21046, #21107 (shared controller-instance owner proof), #20986
(queued-run dispatch signal).

**What existing behavior does this improve?**

The start of new runs while two servers overlap during a Cloud deploy.

**Subsystem affected**

Server run admission (`server/src/services/heartbeat/queue.ts`), the
controller instance table, and startup and shutdown in
`server/src/index.ts`.

**Current behavior**

The old server starts queued runs until it receives SIGTERM. Runs that
it starts in the last minutes of the overlap are interrupted while they
prepare. Nothing tells the old server that its replacement already
starts runs: `controller_instances.epoch` is written at process start,
and that happens minutes before the new server can start runs.

**Proposed behavior**

The new server records `dispatching_at` and a short
`dispatch_lease_expires_at` on its `controller_instances` row right
after its startup queued-run pass. The old server reads this through an
in-memory flag at run admission and leaves new run starts to the new
server. Runs it already started continue and are handed over at SIGTERM
as today. If the new server stops, drains or dies, the old server starts
runs again within seconds.

**Reason and benefit**

Fewer runs are cut off by each swap. There is never a moment without a
server that starts runs.

**Breaking changes**

None. The migration adds two nullable columns. Releases without this
code ignore the columns, and their rows never carry the marker, so a
mixed-version overlap behaves as today.

## What Changed

- `controller_instances`: new nullable columns `dispatching_at` and
`dispatch_lease_expires_at` (migration
`0471_controller_instances_dispatch_marker`, two `ADD COLUMN`
statements, no lock beyond the catalog update).
- New `server/src/services/run-start-handover.ts`. It is one worker on
the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`).
Its renewal and recheck are coordinator deadlines, and a drain change or
expiry wakes it.
- Replacement side: `begin()` records the intent and writes the marker
(it creates the boot's row if the row is missing). If the server drains
at that moment, the worker writes the marker when the drain ends. The
worker then publishes a `run-start-handover` hint and renews the 20 s
lease every 5 s. It renews only while an older boot that is not stopped
or fenced may still serve, and for at most 30 min. All marker writes run
in one serialized chain. `release()` at shutdown lands after any renewal
already in flight. A clear that fails is retried, and shutdown waits for
the release. While the server cannot dispatch (drain, quiesce, admission
closed), the worker clears the lease.
- Old-server side: `isYielding()` is an in-memory read. The hint
refreshes it, and so does one read that a dispatch pass waits for (at
most 1 s) when the last read is more than 2 s old. A read that gets no
answer within 1 s never keeps the server yielding. The server admits,
and the yield also ends when the replacement's last seen lease runs out
on the old server's own clock. Reads are a single flight on the
dedicated connection. While it yields, the server rechecks every 3 s,
calls `onResume` when the marker is gone, and sends a coalesced
queued-run-dispatch hint for skipped starts.
- It yields only to a newer epoch with a live marker. It never yields to
an older boot, to itself, or to a row without the marker, and it does
not yield without a row of its own.
- **Database invariant:** every statement of the hand-over runs in a
small transaction on the dedicated one-connection controller-lease pool
(`controllerLeaseDb`) with a 1 s statement timeout. That covers reads,
marker writes, the `run-start-handover` notification (commit-bound, in
the marker write's own transaction) and the skipped-start hint. None of
it touches the shared pool or registers tracked database work. The pool
is shared only with legacy controller-lease renewals, and never with the
liveness pool that native lease renewals use. A hand-over statement
waits at most one legacy batch (10 s), inside the 15 s between a marker
renewal and the marker's expiry. It delays a legacy renewal (60 s lease,
renewed every 10 s) by at most 1 s. I did not piggyback on the
controller-instance lease renewal, because that renewal runs only while
the server holds work and only every 30 s.
- **Import audit of `run-start-handover.ts`:**
  - `drizzle-orm` `sql`: builds statements, no I/O.
- `controllerLeaseDb`: the dedicated connection, used by every statement
here.
- `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs
`pg_notify` only on the transaction it is given and registers no intent,
so the coordinator's `reconcileDatabaseWork` has nothing of ours to
check.
  - `logger`: no I/O to the database.
  - `controller-process-identity`: in-memory boot id and host hash.
- `delivery-work-coordinator`: type only. The worker is registered for
lifecycle and deadlines; its `run` awaits only the serialized marker
chain, and reads are not awaited.
  - `DELIVERY_QUEUES`: constants.
  - `task-admission` drain subscriptions: in-memory events.
- `heartbeat.ts`: run admission is also closed while the server yields.
A `runStartHandover` service option (default: the process's registered
hand-over) lets one process host two assembled services in tests.
`queue.ts`: `startNextQueuedRunForAgent` and
`resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That
call returns `null` (nothing to await) unless the hand-over read is
stale.
- `index.ts`: creates the hand-over only when the instance lease is on,
the deployment notification channel exists, and the scheduler is
enabled. It calls `begin()` after the RunDispatch startup pass and wakes
that queue. It releases the hand-over when run admission closes at
shutdown, and stops it before the lease stops.
- `deployment-notifications.ts`: keeps the LISTEN connection open while
a hand-over involves this server, so skipped starts and lease changes
reach the other server at once.
- `doc/execution-semantics.md`: new section "Run starts during a deploy
overlap". It covers the marker, who yields, failure handling, what
counts as starting a new run, and what a post-deploy audit should show.

**Marker contract for other components (for example the deployer):** a
replacement starts runs when its `controller_instances` row has an
`epoch` greater than the old boot's, and `dispatching_at is not null and
dispatch_lease_expires_at > now() and stopped_at is null and fenced_at
is null`. `dispatching_at` stays set after the lease lapses, as the time
the boot began starting runs.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test
with two `heartbeatService`s, each with its own pool, listener,
coordinator and boot. A wake that the old server accepts stays queued
there. The replacement's real queued-run dispatch starts it, and claims
are observed per pool. After a replacement release, the old server
starts the run itself. Red: with the yield removed from
`isRunAdmissionClosed`, the old server claims the run.
- `cd server && npx vitest run
src/__tests__/run-start-handover.test.ts`: 19 tests pass against
embedded PostgreSQL. Each test uses two servers, each with its own pool,
boot id, LISTEN connection and delivery coordinator. The tests cover:
- the old server is ready and the new server starts; the new server sets
the marker; the old server yields; the queued run is claimed only by the
new server, and a skipped start wakes the new server's dispatch
- the old server learns from its own dispatch path without a
notification: a read older than 2 s is refreshed before the start
- the new server dies after it sets the marker: the old server resumes
and runs its queued-run pass
  - the new server shuts down: the old server resumes at once
- the new server drains: it releases the marker and takes it back
afterwards
- a replacement that starts up during a drain takes over when the drain
ends
  - a shutdown release during an in-flight renewal stays released
- a clear that fails at shutdown (statement timeout behind a row lock)
is retried by a later release
- an older server that outlives the lease TTL keeps yielding while the
replacement renews
- a yield that the old server cannot confirm within 1 s (its only
connection is busy) does not stop starts (red/green)
- a yield ends when the replacement's lease runs out, even while every
read waits for a connection (red/green)
- renewals blocked past the TTL hand starts back once, then the
replacement takes them again (bounded flapping)
- a clear that fails when the replacement starts draining is retried
well inside the lease (red/green)
- with both servers' shared pools blocked past the TTL, the marker is
written, renewed and released, and the old server yields and resumes,
all before the block ends (red when hand-over statements use the shared
pool)
  - an older-release row without the marker changes nothing
- a server never yields to an older boot or to itself, and does not
yield without a row of its own
  - a lone server writes once, never renews, and admits from memory
  - without a hand-over (self-hosted) there is nothing to await
- Red: on `origin/master` the test file cannot run, because the module
and the columns do not exist. The old server has no signal and keeps
starting the handed-over run.
- `cd server && npx vitest run
src/__tests__/deployment-notifications.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts
src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`:
all pass.
- `cd packages/db && pnpm check:migrations`: safety and compatibility
checks pass. `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify` asks for broad verification because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Run starts move from the old server to the new server during the
overlap. If the new server cannot dispatch, the old server resumes on
the lease clear (about 3 s) or the lease lapse (up to about 23 s after a
crash). If a hand-over read fails, the server keeps admitting.
- If the old server was not listening for notifications, it can start
runs for up to about 2 s after the marker (the age of its last read). On
a dispatching server, the pre-dispatch read adds one small bounded query
at most every 2 s.
- An older boot that has been idle for more than 30 min counts as gone.
The new server then does not renew, and the old server admits again as
today.
- The migration adds nullable columns only. The deployer can apply it
while the previous release serves.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:09:53 -06:00
DottaandClaude Opus 5.5 148d2b629c fix(heartbeat): renew legacy controller leases on their own connection, timed from the statement (#21042)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy run keeps a controller lease on its `heartbeat_runs` row.
The server renews every lease in the process with one batched statement
every 10 seconds, and stops a run when its local lease deadline passes.
> - The batched renewal ran on the 2-connection liveness pool. Native
runner lease renewals use the same pool. Each native renewal holds a
connection for a row lock, about seven notifications, and an update.
> - With 18 native runner runs renewing every 20 seconds on a slow
database, the legacy batch waited 30 to 50 seconds for a connection. The
local deadline also counted that wait, so it fired 20 to 40 seconds
before the database lease could expire.
> - Runs then stopped with `controller_lease_lost` while no other
controller owned them, and their issues looped through retries.
> - This pull request gives legacy renewal its own connection and
measures the deadline from the time the statement starts on that
connection.
> - The benefit is that a slow server no longer stops runs that still
own their lease. A real takeover still stops the run.

## Linked Issues or Issue Description

**What happened?**

On staging, after the 05:37Z deploy, 63 of about 175 new runs failed
with `controller_lease_lost` ("Legacy controller lease lost") at about 3
per minute. All were legacy runs in preparation
(`executionRecovery.kind: bootstrap`, `providerWorkStarted: false`). One
example had `controllerLeaseLoss: {cause: "renewal_deadline",
renewalCount: 7, sinceRenewalMs: 42537, renewalPendingMs: 38798,
deadlineLateMs: 134}`. Several runs that failed at the same moment had
identical timing values, so they were members of the same renewal batch.
The previous successful renewal took 17 to 44 seconds, so the deadline
that it set was only 16 to 43 seconds away. At that time 27 legacy runs
were preparing and 18 native runner runs were renewing on the same
liveness pool. The issues looped: `issue_assigned`, lease lost,
`issue_continuation_needed`, lease lost, `transient_failure_retry`,
lease lost.

**Expected behavior**

A slow server or database must not cause a lease loss while the
controller still owns the lease. Only a real expiry or a takeover by
another controller must stop the run.

**Steps to reproduce**

1. Start legacy runs and native runner runs on one server.
2. Make the database slow, so that each native lease renewal holds a
liveness connection for several seconds.
3. The legacy renewal batch waits for a liveness connection, and its
next local deadline is shortened by that wait.
4. The next renewal also waits, and the local deadline fires first. The
run fails with `controller_lease_lost` (`renewal_deadline`).

**Paperclip version or commit**

Staging at 8cefbe9977. The fix is based on master at aa94939988.

**Deployment mode**

Hosted cloud (staging).

**Agent adapter(s) involved**

Legacy controller runs (Codex and Claude local adapters). Native runner
runs share the liveness pool.

## What Changed

- `packages/db`: add `controllerLeaseDb(db)`, a private 1-connection
pool (`paperclip-legacy-controller-lease`) for legacy controller
renewal. `closeLivenessDb` also ends it, so shutdown does not change.
- `legacy-controller-lease.ts`: the batched renewal runs on
`controllerLeaseDb` instead of the liveness pool that native lease
renewals use.
- `legacy-controller-lease.ts`: each batch records the time when its
transaction starts on the connection. The watcher measures the next
deadline from that time, not from the time the request was queued. The
database sets the new expiry at UPDATE time, which is later, so the
local deadline never passes the database expiry.
- The renewal statement, its boot-id and expiry conditions, and
revocation do not change. A takeover or an expired lease still rejects
the renewal and stops the run.

## Verification

- Red/green regression tests in
`server/src/services/legacy-controller-lease.test.ts`:
- `renews without queueing behind native lease transactions that hold
every liveness connection` (embedded Postgres). Before the fix, the
renewal waited 9.8 s behind four 5-second liveness transactions. After
the fix, it completes in less than 2 s.
- `keeps a controller owned when every renewal waits thirty-five seconds
for a connection before its statement runs`. Before the fix, the lease
was lost with `{sinceRenewalMs: 25000, renewalPendingMs: 20000,
renewalCount: 1}`, the same pattern as on staging. After the fix, the
controller stays owned for four lease periods.
- Updated the private-pool tests to check the new pool and its
application name.
- `npx vitest run src/services/legacy-controller-lease.test.ts`: 40
passed.
- `pnpm verify` (affected tests), plus `typecheck` for `packages/db` and
`server`.

## Risks

- Each server process opens one more database connection, only when it
has a legacy run.
- The local deadline can now be later than before, by the time that the
batch waited for a connection. It stays at or before the database
expiry, because the database sets the expiry after the recorded start
time. Readers that act on an expired lease are not affected.
- Low risk otherwise. There is no schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20924, #20936

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:52:06 -06:00
DottaandClaude Opus 5.5 5a7378af56 fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued runs. The run-dispatch worker
re-runs the queue only when a write sends a "queued run dispatch"
signal.
> - A follow-up run can queue while its predecessor still holds its
sandbox lease. The claim then declines the run without a log line, and
the run stays `queued`.
> - After a controller restart, nothing sent a new signal once that
lease was settled. The run stayed `queued` forever. The issue had no
Retry, and the reconciler counted the queued run as a valid next action.
> - This pull request sends a dispatch signal from the native
finalization writes that can clear a predecessor's cleanup gate, logs
why a claim was declined, and bounds the wait. The dispatch worker
re-drives an undispatched queued run after 60 s as a backstop, and the
ownership reconciler reports and re-drives one still queued after 5
minutes.
> - The benefit is that a follow-up starts within seconds of its
predecessor settling, a lost signal delays it by about a minute instead
of forever, and a run that still cannot start is visible.

## Linked Issues or Issue Description

Refs #20909, #20935, #20953, #20970

**What happened?**

Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` +
#20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold,
`--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the
outage with an accepted result. The next controller finalized it at
22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z
(`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source
`native_status_decision`). The run stayed `queued` with `claimed_at =
null` until the wave ended 19 minutes later. The agent stayed `running`
with no running run. The task had no Retry. The grade was
`R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`.

**Root cause**

`claimQueuedRun` (`server/src/services/heartbeat/queue.ts`,
`lockIssueExecutionClaim`) declines a queued run while
`hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is
true for its predecessor. After the restart, the predecessor's warm
`reuse_by_environment` lease was still `active` when the continuation
queued, so the first claim was declined. The decline is correct, but
nothing owned the retry. `registerRunDispatchWork`
(`server/src/services/heartbeat/dispatch-work.ts`) has `next: async ()
=> null`: the queue is only re-run by a later dispatch signal. At wave
end the lease was `retained` with `cleanupStatus: success`, which does
not block a claim. A dispatch pass in that state starts the run
(verified locally). So no pass ran after the gate cleared. The ownership
reconciler listed the run as `queued_run` with owner `run_dispatch`
(always "alive"), and I7 treats any queued run as a next action, so
nothing reported it.

This is not the #20935 throttle. The wave base predates #20935, and a
throttled intent keeps its wake intent `queued` with a `nextAllowedAt`
deadline. Here the run row existed and its claim was never retried. On
current master #20935 parks this specific `response_wake` self-wake, but
any continuation, comment wake or Retry that queues behind a settling
predecessor after a restart has the same gap. The red test uses a
`same_agent` continuation, which master still enqueues.

**Steps to reproduce**

`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
"starts a continuation that queued behind its predecessor's sandbox
release after a graceful restart".

## What Changed

- Event-driven dispatch. A lease release already signals queued-run
dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`,
stop-and-retain settlement). I enumerated every `environment_leases`
status writer: each one that moves a lease out of a blocking state
already signals. The writers that do not signal only acquire a lease or
set `pendingCleanupInFlight`, so they add a block and do not clear one.
The native coordinator side had no signal, so:
- `notifyNativeFinalizationWork` (the shared notifier of coordinator
commits and terminal transitions) now also signals queued-run dispatch
for the company.
- The coordinator writes that clear `leaseOwner` or end the phase
outside that notifier now signal in the same transaction:
`releaseLapsedCoordinatorOwner`, native startup cleanup,
observed-workspace preservation, `record_finalization_error` in the
interaction bridge, the wake-queue terminal recovery, and the
reconciler's ambiguous-state terminal failure.
- Not changed: three non-transactional pre-dispatch closes in
`run-completion.ts` and `agent-execution-stop.ts`. Their callers queue
or settle the successor after the write, so the successor's own enqueue
starts the queue.
- `claimQueuedRun` now logs the reason when it declines a queued run at
the issue execution claim (`predecessor_cleanup_pending`,
`predecessor_still_executing`, `issue_execution_owned_by_another_run`,
case handoff, task chat cleanup), when a predecessor is still settling
in this process, and when a temporary budget hold keeps it queued.
- New `server/src/services/queued-run-dispatch-liveness.ts`:
`listUndispatchedQueuedRuns` reads queued runs in active companies whose
ready, non-paused agent has no running run. It reads every page with a
keyset cursor until the set is exhausted, so older held runs cannot hide
a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each
such run 60 s after it was created, then every 2, 4, 8 minutes, capped
at 15 minutes. Run ids that leave the set are dropped.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch worker's `next` now plans those re-drives. It re-reads only the
companies the pass just dispatched, keeps the deadlines of every other
company, and runs no query when the pass dispatched nothing. Only an
instance-wide pass (startup or explicit wake) reads every company. It
adds each due company to the dispatch scopes and returns the next due
time.
- `packages/db`: new partial index `heartbeat_runs_queued_created_idx
(company_id, created_at, id) WHERE status = 'queued'` (migration
`0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in
`client.ts`). It returns `null` (no deadline, no SQL until a signal)
when no run is undispatched or scheduling is suppressed. Every re-drive
goes through the normal `startNextQueuedRunForAgent` and
`claimQueuedRun` gates, so it never bypasses a hold, budget, pause or
cleanup gate.
- `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns`
into the worker (two lines).
- `server/src/services/recovery/ownership-invariant-report.ts`: new
check. A queued run still undispatched 5 minutes after creation is an I7
violation (`kind: queued_run`, `reason: queued_run_not_dispatched`),
written to the activity log once. A younger one sets the report's next
due time.
-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
`applyOwnershipReportBackstops` sends a queued-run dispatch signal once,
for each company with a newly found violation. Repeat passes do not
signal again, so a held run does not wake the report every 30 s.
- `tests/restart-chaos/README.md`: records the wave and the fix in the
CR results table. The existing CR1 cell (#20909) and the enforced U1
check already cover this class in the harness.

## Verification

- RED on master (`23909077ad`):
- `heartbeat-native-restart-suspension.test.ts` "starts a continuation
that queued behind its predecessor's sandbox release after a graceful
restart": the continuation is `queued` behind the predecessor's `active`
warm lease. The run-dispatch worker is registered, the lease then
settles to `retained`/`success` with no dispatch signal, and the run
stays queued: `AssertionError: expected null not to be null` after 75 s.
- `ownership-reconciliation-report.test.ts` "reports and re-drives a
queued run that no dispatch started while its agent had nothing
running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no
violation for a run queued 19 minutes).
- `ownership-reconciliation-report.test.ts` "signals queued-run dispatch
when finalization work settles or a finished coordinator's lapsed owner
is released": `expected [] to include 'settled'`.
- GREEN on this branch: all pass. The restart test has two cases. When
the lease is released through `releaseLease`, the continuation starts
within 5 s. When the lease is settled by a write with no signal, the
backstop starts it within 75 s. A new report test puts three older held
runs ahead of a stranded run with a page size of 2, finds the stranded
run, and checks that the backstop signals once.
- `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap,
and going quiet.
- Focused suites pass (22 files, 495 tests):
recovery-deadline-acceptance, live-recovery-work,
heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work,
deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the
ownership kinds suites, and others.
`heartbeat-native-restart-suspension` and
`ownership-reconciliation-report` pass in full. `tsc --noEmit` for the
server passes.

## Risks

- A queued run that is held on purpose (for example a temporary budget
hold or a case handoff waiting for release) is now re-checked at most
every 15 minutes after the first minute. Each re-check is one indexed
query and the normal claim gates. Such a run is also reported once as an
I7 violation.
- An instance with an undispatched queued run keeps a dispatch deadline,
so it does not idle while that run waits. An instance with no such run
is unchanged.
- Schema: one additive partial index, built concurrently. No data
change. State saved by an older version is read as is. Nothing runs
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:04:58 -06:00
8818402e7a feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - The inbox collects requests that need a human decision.
> - Teams need to know who owns each request and who may act on it.
> - A shared close operation must differ from hiding a card for one
person.
> - This pull request adds personal routing and durable outcomes to the
existing attention service.
> - People can claim or transfer work and inspect what another person
did for them.
> - Domain permissions still decide who may resolve each request.

## Linked Issues or Issue Description

**Subsystem affected**

Server, shared contracts, and database. This is the routing foundation
for a stacked inbox UI change.

**Problem or motivation**

A company-wide attention feed does not give teams a clear personal
queue. Archive, withdrawal, transfer, and decisions by another person
need distinct scopes and records.

**Proposed solution**

Resolve a request audience from its addressee, task participants, and
configured owners. Provide personal, unassigned, company, all-assignee,
and archived views. Add claim, transfer, personal archive and snooze,
shared withdrawal, receipts, and personal history.

**Alternatives considered**

A single shared inbox leaves ownership and personal hiding ambiguous.
The service keeps one canonical request and projects its audience
instead of copying requests for each person.

**Roadmap alignment**

This extends the existing Multiple Human Users and Agent Reviews and
Approvals capabilities in ROADMAP.md.

**Additional context**

Refs #19363 and #19520 for the current inbox navigation and confirmation
replacement rules. Refs #20268 for related actionable-request work. That
PR filters actionability; this stack adds audience routing and personal
outcomes.

This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this
branch.

## What Changed

- Add nullable company and project decision owners and a company budget
owner.
- Add audience views and personal badge counts to the canonical
attention projections.
- Add claim, transfer, withdrawal, personal archive, snooze, and restore
operations with permission checks and activity records. Lock addressed
transfers to the current addressee or routing administrators, and
validate the recipient against canonical resolution policies.
- Add receipts and scoped history for work handled by another person.
- Keep closed-task requests and replaced confirmations out of current
cards.
- Add an idempotent migration and a concurrent receipt-history index.
- Update typed fixtures so the existing UI remains compatible with the
new contracts.

## Verification

- Server layered typechecking passes, including all three test-tree
partitions.
- Seven routing and interaction-resolution suites pass all 249 tests
against a dedicated external PostgreSQL test database on `4caecc21d8`.
Six new regressions failed before the repair; they cover takeover
followed by approval, legitimate delegation, excluded creators/review
requesters, and unauthorized claims.
- Prior focused verification covers private approval history and
receipts, transfer visibility, audience escalation, withdrawal
governance, hidden-card counts, per-recipient confirmation replacement,
and SDK source contracts.
- A synthetic three-person company passed 37 routing and lifecycle
acceptance checks. The UI stack adds browser acceptance for the shared
controls.
- The complete [GitHub Actions CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331)
passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24:
typecheck, build, general and serialized tests, Runner support, browser
shards, and aggregate gates). Greptile is 5/5 on that exact head with no
open findings. The fresh security scan passes after explicitly
rechecking the addressee-transfer repair.
- `pnpm verify` reports exit 2 because a CI-global script changed; full
CI supplies that broader coverage. Embedded PostgreSQL could not start
on this Mac because shared-memory slots were exhausted, so local
database checks use a separate external test database. Linux CI covers
the embedded setup.

## Risks

- Personal badge counts change to the viewer's assigned requests.
Accessible feed totals and task membership keep their existing scope.
- The migration adds nullable owner fields and a dismissal fingerprint.
Its index runs outside a transaction. The migration is classified and
rerunnable.
- Routing changes the addressee used by resolution authorization. The
current addressee or a routing administrator must authorize an addressed
transfer; the recipient must satisfy writer access, canonical resolver
exclusions, and task review policy under row locks.
- The shared validator export barrel crosses 1,000 lines because it
exports the new attention schemas. Schema implementation stays in the
dedicated attention validator module.

## Model Used

OpenAI GPT-6 through Codex. Used repository tools, code execution, and
browser checks. This session does not expose a more specific runtime
model ID or its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
(OpenAPI contracts and docs/api/attention.md)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 17:49:39 -08:00
DottaandClaude Fable 5.2 e8e548e856 feat(db): classify migrations for application under the running release (#20913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time

## Linked Issues or Issue Description

**Subsystem affected**

Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.

**Problem or motivation**

Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.

**Proposed solution**

Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.

**Alternatives considered**

- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.

**Roadmap alignment**

This is release infrastructure. It adds no product feature.

## What Changed

- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".

Published shape:

```json
"migrationCompatibility": {
  "version": 1,
  "migrations": [
    {
      "idx": 468,
      "tag": "0468_busy_wendell_rand",
      "sqlSha256": "56a6e272…",
      "compatibility": "online_compatible",
      "classified": "online_compatible",
      "override": false,
      "reasons": [
        { "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
      ]
    }
  ]
}
```

`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).

### Distribution over the existing migrations

No existing migration is rewritten or annotated.

| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |

Migrations per blocking reason (one migration can have several):

| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.

## Risks

- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.

## Model Used

- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 14:14:40 -06:00
DottaandClaude Opus 5.5 c02de922af fix(chat): deliver the person's time zone to chat agents (#20898)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to agents in chat, and agents create routines and other
scheduled work from that chat
> - Paperclip does not record the person's time zone, so a chat agent
cannot read "every weekday at nine" or "today" in the person's zone
> - The agent then saves schedules in UTC, and the person gets the work
at the wrong hour
> - This pull request saves the browser time zone with the person's
personal preferences and gives it to chat agents in the task prompt
> - The benefit is that schedules and dates from chat match the person's
local time

## Linked Issues or Issue Description

**What happened**
A person in America/Chicago wrote in chat: "Every weekday at nine in the
morning, summarize Launch for me." The agent saved the routine schedule
`0 9 * * 1-5` with the UTC time zone. It replied that it used UTC
because the person's time zone is not recorded.

**Expected behavior**
The agent knows the person's time zone. It saves one enabled `0 9 * *
1-5` schedule with the time zone America/Chicago. Its reply names the
zone.

**Steps to reproduce**
1. Open the board in a browser with the time zone America/Chicago.
2. Open a chat with an agent.
3. Send "Every weekday at nine in the morning, summarize Launch for me."
4. Look at the routine trigger that the agent creates. Its time zone is
UTC.

## What Changed

- Database: add the nullable `time_zone` column to
`user_workspace_preferences` (migration `0468`). This is an
instance-wide personal preference by user, as AGENTS.md section 5
permits.
- Shared: `WorkspacePreference` has `timeZone`.
`updateWorkspacePreferenceSchema` accepts an optional IANA `timeZone`
(max 64 characters, checked with `Intl.DateTimeFormat`).
`coordinatorWorkspaceEnabled` is now optional. The request must contain
at least one field.
- Server: the preference service returns `timeZone` and updates only the
fields in the request. The activity log entry stays the same. The
OpenAPI entry uses the shared schema.
- UI: a new `useBrowserTimeZoneSync` hook in `Layout` saves the browser
time zone one time when it is different from the saved zone. It is
silent and best-effort. It uses the selected company as the audit
context.
- Prompt: for a conversation (any adapter), the task context reads the
chat owner's saved zone. Recovery keeps the zone that the first turn
admitted. The chat block adds one line only when the zone is known:
`The person's time zone is America/Chicago (from their browser). Read
times and dates they give, such as "nine in the morning" or "today", in
that zone, and save schedules such as routine triggers with that
timezone; say the zone when you report a schedule.`

## Verification

- `cd server && npx vitest run
src/__tests__/workspace-preferences.test.ts
src/__tests__/workspace-preferences-routes.test.ts
src/services/heartbeat/task-markdown.test.ts
src/__tests__/heartbeat-task-context.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts`
- `cd ui && npx vitest run src/hooks/useBrowserTimeZoneSync.test.tsx
src/hooks/useCoordinatorWorkspaceEnabled.test.tsx
src/components/Layout.test.tsx`
- Red/green: without the task-context and task-markdown changes, the two
new prompt tests fail. With the changes, they pass.
- `pnpm check:token-gates` is clean. `tsc --noEmit` passes for
`packages/shared` and `packages/db`.
- Manual: open the board in a browser with a non-UTC zone. Then ask a
chat agent to create a weekday routine at nine in the morning. The
trigger uses the browser zone.

## Risks

- Low risk. The migration adds one nullable column.
- The PUT body is now more flexible: `coordinatorWorkspaceEnabled` is
optional. An empty body still gets a 400 response.
- The prompt grows by one line only for conversations with a known zone.
- The saved zone changes when the person uses a browser in a different
zone. The latest browser wins.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:11:57 -06:00
DottaandPaperclip cda23f08ac fix(db): preserve legacy project reads during rolling upgrades (#20860)
## Thinking Path

> - Paperclip controls agent work while web servers change versions.
> - The old and new servers can share a database during that change.
> - Older servers still read and write the project environment column.
> - Migration 0466 removes that column before those servers stop
serving.
> - A later repair alone leaves a committed missing-column window and
loses values.
> - This pull request applies the exact drop and repair in one
transaction.
> - Old reads and saved values survive while current placement behavior
stays unchanged.

## Linked Issues or Issue Description

Refs #20835 and #20852.

**What happened?**

Migration 0466 drops `projects.environment_id`. The previous server
still selects that column and reads it during admission. Applying a
separate repair afterward leaves a window in which those requests fail.
The drop also removes saved values.

**Expected behavior**

A database that has not yet applied 0466 must keep the column and its
values visible to concurrent old servers throughout the upgrade. Both
published migration entries must be recorded truthfully. Current project
responses must keep the new workspace placement contract.

**Steps to reproduce**

1. Use a database before migration 0466 with nonempty project
environment values.
2. Keep an old project writer transaction open and start the pending
migrator.
3. Start an old project read, then commit the writer.
4. Without this repair, the saved values disappear. With this repair,
the reader sees the preserved values and both migrations commit
together.

## What Changed

- Add migration 0467 to restore the legacy nullable UUID column and its
original foreign key. Leave the published 0466 SQL unchanged.
- Apply only the exact SHA-bound 0466/0467 pair in one transaction. Lock
the project table, retain nonempty values in a temporary table, execute
both real SQL files, restore values, and record both real journal
entries before commit.
- Apply the additive 0467 repair normally when 0466 was already
committed.
- Keep a deprecated database-only field and omit it from current project
API responses. Current placement logic stays unchanged.
- Add PostgreSQL regressions for concurrent old reads and writes,
idempotence, rollback, and an already-applied drop. Document the overlap
contract and retain the five newest generated snapshots.

## Verification

- Node 24 cached agent preflight passed.
- `pnpm exec vitest run
packages/db/src/project-environment-overlap-migration.test.ts
server/src/__tests__/project-environments.test.ts -t 'project
environment rolling migration|saves storage, not'`: 4 passed in 32.57
seconds against real disposable PostgreSQL.
- Before the owner fix, the regression had two failures: saved values
disappeared and the restore-failure case committed. The fixed case
proves that both DDL and journal entries roll back together.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/db check:migrations`: passed.
- `git diff --check`: passed. Migration 0466 bytes are unchanged. The
generated snapshot set is 0463 through 0467.
- Review found a raw DELETE response that still exposed the
database-only field. The delete projection now omits it; a real
authorized HTTP DELETE passed with successful cleanup. The concurrency
test now observes the old reader waiting inside PostgreSQL before
releasing the old writer. All three migration cases passed with that
barrier. Earlier HTTP fixture setup/cleanup failures remain recorded as
failed attempts; they were not product-regression passes.
- Candidate `047178538c0df3a9e382259a51635a6e46da5646` has fresh
Greptile 5/5 with no open findings. [Required final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/43062354612)
passed, including build, typecheck, migration checks, all test shards
and the aggregate verification gate. The separate Superagent security
scan ended inconclusive/neutral; it did not produce a passed scan.
Deployed acceptance remains pending. No hosted database or app
deployment was changed by this branch.

## Risks

The pair takes a table lock for its transaction. Operators must retain
their normal lock and statement timeouts. Both SQL files and their exact
hashes must match before the pair runs. A failed restore rolls back the
drop and both journal entries.

If 0466 was already committed, 0467 restores the column and foreign key.
It cannot recover values that were already deleted by the earlier
release. This repair does not claim to recover those values.

## Model Used

OpenAI Codex (GPT-6), with tool-assisted implementation and real
PostgreSQL qualification. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 11:08:30 -06:00
Nicky LeachandClaude Opus 5.5 369e238295 fix(projects): make workspace storage the project's only placement choice (#20835)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A repository project gets its working files from core staging or
from a workspace storage plugin
> - #20459 reported that an installed storage plugin attached every new
repository project, so sandbox agents could not use those projects
> - #20587 fixed creation with an **Available in** choice that selects
storage
> - #20651 then saved an environment on every new project and used it
before the agent default to place runs. That solved a problem that did
not exist: the project needs storage, not a run environment
> - Creation saves the first environment of the chosen group, so a
project silently sends every agent's runs to that environment. Project
Configuration also shows two controls (Environment and Workspace
storage) for one decision
> - This pull request removes the project run environment and makes
storage the only placement choice a project owns. Storage is fixed at
creation
> - The benefit is that runs follow the agent again, and project
settings show one clear, read-only fact about storage

## Linked Issues or Issue Description

Refs #20459. Follow-up to #20587 and #20651.

Refs #11644. That issue asks for a project-level run environment
override. This pull request removes the override that #20651 added for
it, because the override is not necessary to fix #20459. #11644 stays a
separate feature request.

Related: #18176 (open) repairs placement through the legacy
workspace-policy field. This pull request does not restore project
placement, so it conflicts with the direction of #18176.

## What Changed

- **Remove the project run environment.** Migration `0466` drops
`projects.environment_id`. Run placement no longer reads the project.
Runs use the agent default, then the instance default, then local
execution. The low-trust task sandbox and the enforced sandbox policies
keep their precedence.
- **Keep the #20651 recovery and binding guards.** A recovered run still
uses its admitted environment lease (source `run`). Company-binding
checks before admission and during lease creation stay.
- **Keep the creation choice.** **Available in** in New Project still
selects storage. It still shows only when the environments need more
than one storage owner. That can only occur when a storage plugin is
installed. `POST /projects` with `environmentId` selects the storage of
that environment and does not save the environment.
- **Fix storage at creation.** `PATCH /projects/:id` rejects any storage
change with `409 project_storage_locked`, from core to a plugin and from
a plugin to core. Before, a core project could move to a plugin.
- **Drop legacy writes.** Older clients can still send
`executionWorkspacePolicy.environmentId`. The service removes it on
create and update.
- **Simplify Project Configuration.** Remove the Environment and
Workspace storage selectors. Show an advisory **Available in** list of
the environments that can use the project's storage. Show it when a
storage plugin is installed or when the project uses one. When the
project's plugin is not installed, show **Storage plugin unavailable**.
Without a storage plugin, show nothing.
- **Issue workspace card.** Remove the "project default" environment
label.
- **Shared contract.** Remove `Project.environmentId`, the
`environmentId` project field, and `expectedStorageProviderPluginId`
from the update schema. The create schema keeps `environmentId` and
`expectedStorageProviderPluginId` from #20587.
- **Docs and maps.** Update `doc/project-environment-compatibility.md`,
`docs/api/goals-and-projects.md`, and the project feature and
agent-access maps.
- **SDK source pins.** Refresh
`tests/runner-e2e/source-sdk-public-inputs.json` for the two shared
files. The shared file blobs return to their values before #20651. The
refresh also corrects the `pnpm-lock.yaml` pin and `lockSha256`:
`master` pins an older lockfile blob than the one it contains.

## Verification

- Targeted server suites pass (227 tests): `project-environments`,
`project-workspace-provider.integration`,
`heartbeat/environment-selection`, `execution-workspace-policy`,
`environment-selection-route-guards`, `environment-run-orchestrator`,
`openapi-routes`.
- Related server suites pass (349 tests): `environment-service`,
`environment-routes`, `plugin-workspaces.integration`, `issues-service`.
- Targeted UI suites pass: `ProjectProperties.concurrency`,
`IssueWorkspaceCard`, `NewProjectDialog`,
`NewProjectDialog.managed-sandbox`, `ProjectDetail`.
- New tests:
- Creation saves storage, not an environment. Core and plugin projects
both reject a storage change with `project_storage_locked`.
  - A legacy policy `environmentId` is dropped on create and update.
- A new run follows the agent default for any project. A recovered run
keeps its admitted lease after the agent default changes.
- Project Configuration has no storage or environment selector. The note
is hidden without a plugin. It lists the core or plugin environments for
the project's storage. It shows **Storage plugin unavailable** for a
removed plugin.
- `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter
@paperclipai/ui typecheck`, and `pnpm --filter @paperclipai/shared
typecheck` pass.
- `pnpm --filter @paperclipai/db check:migrations`, `pnpm
check:token-gates`, `node scripts/check-maps.mjs`, and `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` pass.
- Not run locally: the full `pnpm test:run`, `pnpm build`, and browser
suites. CI supplies that coverage.
- Manual check: open Project Configuration on an instance with a storage
plugin. Confirm that there is no Environment or Workspace storage
selector, and that **Available in** lists the environments.

## Risks

- **Migration.** `0466` drops `projects.environment_id` and its foreign
key. The migration deletes the saved project environments. Apply it
before you serve the new revision. An old server revision that reads
`projects` fails until it stops. This is the same pattern as earlier
drop-column migrations (`0229`, `0289`).
- **Behavior change for existing projects.** Projects created after
#20651 shipped have a saved environment that overrides the agent. After
this change, their runs follow the agent default again. This is the
intended fix. Before #20651, a project environment selection was not
applied to runs (#11644), so no older behavior is lost.
- **No path from core to a plugin.** A core project, including one
created before a plugin was installed, cannot move to the plugin. To use
plugin storage, create a new project.
- **API.** A `PATCH` that sends `environmentId` or
`expectedStorageProviderPluginId` no longer has an effect. The schema
strips these fields.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with
tool use (shell, file edits, test runs). Extended thinking was
available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:00:21 -08:00
DottaandPaperclip 81c9143fe4 fix: upgrade Better Auth to 1.7.16 with account migration (#18906)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The board uses Better Auth for credential accounts and sessions.
> - The server pins Better Auth 1.7.2.
> - The November security release fixes 12 issues in Better Auth.
> - The new version stops writing the old required account issuer field.
> - This change updates the packages and the database contract together.
> - New users can sign up, and existing users can still sign in.

## Linked Issues or Issue Description

**What happened?**
The server uses Better Auth 1.7.2. The [security
release](https://better-auth.com/blog/security-release-2026-11-03) fixes
12 issues in 1.7.16. Earlier update PRs #17909 and #17911 target 1.7.7,
which is below this security release.

**Expected behavior**
Use the patched stable release. Keep new and existing credential
accounts usable.

**Steps to reproduce**
Read the server dependency pin. Upgrade it without a schema change.
Better Auth then rejects the required account issuer column. See the
[upgrade guide](https://better-auth.com/docs/guides/1-7-upgrade-guide).

## What Changed

- Pin Better Auth to 1.7.16. The repository refresh bot owns the
committed Core lockfile.
- Make the old issuer column nullable. Keep existing issuer values.
- Replace the issuer index with a unique provider/account index.
- Generate the migration and retain the five newest schema snapshots.
- Test new signup, new-account sign-in, legacy issuer sign-in, duplicate
rejection, and separate provider identities with real PostgreSQL.
- Update the database schema test. Test the historical issuer backfill
separately. Verify upgrades over existing rows and rollback on duplicate
provider/account keys.
- Document the duplicate check and migration order.
- Add the approved two-line dependency resolution step before the backup
job’s frozen install. Leave the frozen check and backup tests intact.

## Verification

- Core head: `af70282c7e24335b8688c8f3cc88fdf65045632c`; parent
`0f1481e4462100357a7caa496b875cf0bc500d95` includes master `84fc67317`.
- Nine changed files. The only workflow edit is the approved two-line
backup dependency resolution step; no managed lockfile edits.
- Migration `0465_better_auth_account_identity.sql` is three SQL
statements; snapshot changes only `public.account`, five newest
snapshots retained.
- DB typecheck and 23 focused tests passed on the parent. Both real
PostgreSQL migration/signup suites were rerun November 11: all 4 tests
passed. Application source and tests are unchanged in this workflow-only
commit.
- The parent’s sandbox-provider lane passed all 205 file-sync tests but
reported an unhandled EPIPE during the symlink rejection case; its
aggregate verify therefore failed. No unrelated source change was made.
The new head reruns the full applicable CI.
- Reproduced the old backup install failure locally. Resolving manifests
followed by a frozen install passed locally; restored the bot-owned
committed lock afterward. Hosted backup qualification remains required.
- Current-head CI and review are running; no final-head green claim or
merge yet.

## Risks

Apply the migration before the new server starts. Check for duplicate
provider/account pairs first. Duplicate rows stop index creation. Keep
separate users separate when resolving duplicates. The migration keeps
issuer data. Do not restore its NOT NULL constraint while new rows
contain null. No deployment is included.

## Model Used

OpenAI GPT-6 Codex, with reasoning and tool use. The runtime did not
expose a more specific model ID or context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 08:26:58 -06:00
Nicky LeachandPaperclip f30ac226d6 Keep project environment mappings independent of shared defaults (#20651)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A project supplies work context and repository storage to task runs.
> - Creation selected a storage owner but did not retain the selected
environment on the project.
> - Configuration stored an environment inside checkout policy, but
policy reads and run selection dropped it.
> - Projects therefore continued to follow shared agent and instance
defaults.
> - This pull request saves placement on each project and uses it for
new runs.
> - The benefit is an independent Project to Environment association
with the existing storage, trust, and recovery safeguards.

## Linked Issues or Issue Description

Refs #20459 and #11644. Follow-up to #20587.

The reported journey is Project > Configuration: changing one project's
environment must not change another project's mapping or shared
defaults.

Related open PR #18176 repairs placement through the legacy
workspace-policy field. This follow-up introduces a separate project
association, saves creation defaults, and migrates saved legacy
references. It overlaps that PR's placement, permission, and recovery
safeguards. The changes should be reviewed together. #9588 concerns
environment-variable response privacy and does not address placement.
ROADMAP.md has no separate project-association work planned.

## What Changed

- Add nullable `projects.environment_id` and expose
`Project.environmentId`. Save the effective creation default once on the
project. Keep environment placement separate from checkout/isolation
policy.
- Read and write Environment directly in Project Configuration. Keep the
control available with isolated workspaces disabled. Each form reads its
project's value, even when environment choices share a query cache.
- Offer environments compatible with the current storage owner. Change
the target and storage together when the existing storage selector
attaches a core project to a plugin. Preserve retained-data restrictions
and validate the observed storage owner again on save.
- Resolve new task runs from their organizational project's target
before agent and instance defaults. File sources from another project do
not change placement. Explicit low-trust sandbox choices and enforced
sandbox policies retain precedence. External provider placement remains
separate.
- Restore admitted native runs from their owned environment lease. Later
project edits apply to new admissions. Require workspace-write
permission for environment changes and clears. Check company bindings
before runtime admission and during lease creation.
- Migrate valid saved policy references into the project column.
Normalize legacy API writes. Count and clear matching project references
when an environment is deleted. Update API and behavior documentation.
- Refresh the maintained SDK source registry for the two changed shared
files. Runtime pins are unchanged.

## Verification

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/42908045146)
is green on `8b3b7d5b5aed51fc4c8e9c2e3506508bbaf9e00c`: **60 successful
checks**, three skipped gates, no failures, and `ci / verify` passed.
GitHub reports no merge conflicts. Standard Greptile scored the same
commit **5/5** with both review threads resolved.
- Source checkout on macOS arm64, Node 26.12.0, with fresh isolated
PostgreSQL. No production data or live provider calls.
- The two-project component test changes the first form with isolation
disabled and a shared query cache. The second form keeps its target.
Storage changes update only the selected project. A saved target remains
selected while the list loads, and an explicit clear remains possible.
- API integration creates two projects from the same default, patches
one, reads both, and checks that agent and instance defaults remain
unchanged. A later instance-default change does not rewrite either saved
project target. Tests cover unavailable/incompatible choices, anonymous
and cross-company denials, and storage preconditions.
- The migration test runs the actual SQL against an existing project row
and verifies its environment and disabled checkout policy survive.
Deletion accounting and clearing affect only matching project
references.
- The final guard set passes **53 tests across four files**:
`project-environments.test.ts`,
`environment-selection-route-guards.test.ts`,
`heartbeat/environment-selection.test.ts`, and
`ProjectProperties.environment.test.tsx`. All **64
environment-run-orchestrator tests** pass, including direct lease
acquisition without an extra caller argument. The broader affected set
passed **109 tests across eight files** before the additional admission
guards. These sets overlap.
- UI typecheck, migration numbering/safety, token gates, module
boundaries, query-error rendering, and diff checks pass. `pnpm verify
--explain` returns exit 2 and requires broad CI because shared
schema/runtime paths and behavior documentation changed. Broad coverage
is delegated to current-head CI.
- Early iterations failed due to a legacy parser that drops the
environment field, a fixture without real company membership, and
incorrect expectations for privacy-preserving 404 denials. The service
now reads legacy references before normalization, the API fixture has
real membership, and denial assertions match the existing privacy
contract. The orchestrator mock also needed the reused company-binding
guard in its service contract. The focused suites above pass after those
corrections.
- Local server typecheck remains blocked by installed ACPX, Slack, AI
SDK, and Runner type mismatches in unchanged dependency-facing code. It
reports no errors in the changed production paths. This is not a full
local typecheck or suite pass. Hosted CI supplies full current-head
coverage.
- SDK source-proof and privacy qualification passes **56 tests across
three files**, including the real built SDK. The maintained source
registry matches the committed revision; runtime pins are unchanged.
- Review reproduced two failures: an incoming policy without a storage
owner could be validated before the existing owner was restored, and
uppercase legacy UUIDs missed migration matching. Preserve the owner
before validation and compare normalized UUID text. Both regressions now
pass. The post-fix project/storage/orchestrator batch passes **107
tests**: `node node_modules/vitest/vitest.mjs run
server/src/__tests__/project-environments.test.ts
server/src/__tests__/project-workspace-provider.integration.test.ts
server/src/__tests__/environment-run-orchestrator.test.ts --silent`.
- The first hosted run failed build, typecheck, and canary at the same
`run-completion.test.ts` call because lease acquisition required a new
argument. The lease method now enforces binding internally. Its existing
caller compiles without a new field, and the orchestrator tests prove
the binding flag reaches the runtime. The full server check no longer
reports that error; the local dependency errors above remain.

## Risks

- Apply the normal database migration before serving the new revision.
The migration adds a nullable foreign key and copies valid legacy
project selections. It does not move repository data or resize provider
storage. The five newest schema snapshots are retained.
- New projects retain their creation target when shared defaults change.
Existing projects without a saved selection continue to inherit
defaults. A null target explicitly opts into inheritance.
- Existing saved project selections now affect placement. Missing,
archived, incompatible, and foreign-company targets fail validation;
they do not silently choose another storage owner. A running admitted
process keeps its original owned lease.
- Local dependency limitations are listed above. No live-provider,
deployed-browser, or production rollout qualification is claimed.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact serving snapshot and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 14:31:48 -08:00
DottaandClaude Opus 5.5 353945c86d fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output

## Linked Issues or Issue Description

Refs #17737
Refs #20581

**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.

**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.

**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.

## What Changed

- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.

## Verification

- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on 39f38a7f6c (before the last three commits): both FAILED, for
agent-behavior reasons the later commits address. Not rerun yet.
- rotate-token: the agent created the replacement (pending reveal, no
value), then revoked the old token in the same turn, despite the create
response saying to wait. Its reasoning cited the eval-looking fixture
note. The grader rejected the premature revocation. The reply linked the
Tokens page correctly.
- create-gateway: the agent created the gateway and one pending token,
then asked a question card about which apps to add to the empty profile,
with no link to the gateway. The grader rejected the missing link.
- Token values: a scan of all evidence files of both cells found no full
`pcgw_<id>.<secret>` value and no `pcgw_<uuid>` form. Agent `call_api`
payloads in run events are redacted; the integration test proves the
agent tool result has no value.

## Risks

- Behavior change for agents: token create returns metadata and a Tokens
page path, not a value. Agents that expect `token` in the response must
send the person to the Tokens page.
- Migration adds one boolean column with a default. Existing tokens get
`false` and keep working.
- A pending token that is never revealed stays as an unusable row until
someone revokes it. It cannot authenticate.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:13:34 -06:00
DottaandClaude Opus 5.5 3bf7af6c77 perf(plugins): read the lifecycle inbox from per-resource cursors (#20576)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins get agent, project and agent runtime changes from the
`resource_lifecycle_events` journal. Each plugin acknowledges events one
resource at a time.
> - #20405 made the inbox page read linear in the company journal size.
The journal has no retention, so each page still reads every event of
the company to find the unacknowledged ones: about 1.7 s per 1M events.
> - The delivery worker reads a page again after each acknowledgment.
The idle-sleep check also reads the whole journal when all events are
acknowledged. `acknowledge` checks every older event of the resource.
> - On a hosted instance, the page read ran 181 times in one window at
about 1 s each.
> - This pull request adds a per-plugin, per-resource cursor. Reads
start at the cursor, so they do not read acknowledged history again.
> - The benefit is that page, idle-check and acknowledgment cost follow
the number of resources and pending events, not the journal size.
Delivery order and results stay the same.

## Linked Issues or Issue Description

Refs #20405

**What happened?**
The plugin lifecycle inbox page query (`select ... from
"resource_lifecycle_events" ... DISTINCT ON ...`) ran 181 times at about
1 s each on a hosted database in one window. Each acknowledgment causes
another full read of the company journal. The idle-sleep check
(`hasPending`) reads the full journal when all events are acknowledged.
`acknowledge` reads the full history of the resource.

**Expected behavior**
A page read, the idle-sleep check and an acknowledgment must not read
acknowledged history again. Their cost must not grow with the journal
size.

**Steps to reproduce**
1. Give a company 400 agents with 2,500 `agent_runtime` events each (1M
events).
2. Acknowledge the first 2,000 events of each agent for one plugin.
3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100,
undefined, ["agent", "project", "agent_runtime"])`.
4. The read takes about 1.7 s. `hasPending` takes about 240 ms.

**Paperclip version or commit**
`8d2bf78323` (master).

**Deployment mode**
Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18.

## What Changed

- `packages/db/src/schema/plugin_lifecycle_resource_cursors.ts`: new
table `plugin_lifecycle_resource_cursors`. It has one row per plugin,
company, resource type and resource id. `acknowledged_through_id` means:
every event of that resource at or below this id is acknowledged by the
plugin. Plugin and company deletion cascade.
- Migration `0462_plugin_lifecycle_resource_cursors`: creates the table
and carries existing acknowledgments forward (see Risks for the
previous-release story).
- `server/src/services/plugin-lifecycle-inbox.ts`:
- `list` and `hasPending` use a backward loose index scan on
`resource_lifecycle_events_resource_order_idx`. It visits each resource
stream once and gets its latest event id. A stream whose latest event is
at or below its cursor is settled and is not read again. For other
streams, the read gets one head: the unacknowledged creation, else the
first unacknowledged transition above the cursor. Then it applies
`afterId`, id order and the limit, as before.
- `acknowledge` reads only the events between the cursor and the
acknowledged event. It still writes the per-event
`plugin_lifecycle_acknowledgments` row. It moves the cursor only over a
contiguous acknowledged prefix. A creation recorded after a pending
transition does not move the cursor.
- An event counts as acknowledged when it is at or below the cursor, or
when it has an acknowledgment row. Thus rows written by an earlier
release, and gaps in old history, keep their current behavior.
- `server/src/__tests__/resource-lifecycle-events.test.ts`: new database
tests for the migration carry-forward, gaps, late creations, cursor
reads, the rollback rows and the plugin-delete cascade. All tests from
#20405 still pass without change.
- `scripts/benchmark-plugin-lifecycle-inbox.ts`: runs the migration
backfill on the fixture and times it. Adds an acknowledgment case. Runs
`VACUUM ANALYZE`, because hosted tables are autovacuumed.
- `doc/DATABASE.md`: describes the cursor, its read cost and the
migration.
- The delivery worker (`plugin-lifecycle-delivery.ts`) is not changed.

## Verification

Thesis: the cost is a read of the whole company journal (page, idle
check) or the whole resource history (acknowledgment). If a per-resource
cursor marks the acknowledged prefix, each read can skip that prefix.
Then the cost follows the number of resource streams and pending events.
The rows must stay equal.

Local SQL timings: embedded PostgreSQL 18.1 on Apple Silicon, synthetic
data, warm cache, one warmup, 7 samples, baseline and treatment order
alternate per sample. The machine was shared with other jobs (load
average about 35), so spread is wide. Values are median (min to max) in
ms. Rows are equal (`assert.deepEqual`) for every list and probe case.

Fixture A: 400 agents x 2,500 runtime events, 2,000 acknowledged per
agent. 1,020,440 events, 800,420 acknowledgments.

| Case | Before | After | Speedup |
| --- | --- | --- | --- |
| page of 100, all types | 1,700 (1,633-2,438) | 11.5 (9.5-12.2) | 148x
|
| page of 100, agent and project | 3.3 (2.5-5.4) | 4.2 (3.6-6.7) | 0.8x
(slower) |
| idle-sleep check | 238 (174-367) | 2.7 (2.1-3.4) | 88x |
| acknowledge one head | 15.3 (9.4-30.1) | 20.3 (12.9-29.9) | neutral
(noise) |
| migration backfill | - | 9.3 s | - |

Fixture B: 40 agents x 25,000 runtime events, 24,000 acknowledged per
agent (long history). 1,020,080 events.

| Case | Before | After | Speedup |
| --- | --- | --- | --- |
| page of 100, all types | 1,943 (1,642-4,979) | 4.4 (2.9-5.6) | 444x |
| page of 100, agent and project | 3.5 (2.1-5.7) | 4.6 (2.8-12.0) | 0.8x
(slower) |
| idle-sleep check | 234 (220-243) | 2.8 (1.9-6.5) | 85x |
| acknowledge one head | 64.6 (48.5-71.6) | 5.0 (2.6-7.1) | 13x |
| migration backfill | - | 4.6 s | - |

Fixture C: 5,000 agents x 200 runtime events, 190 acknowledged per agent
(many resources). 1,025,040 events.

| Case | Before | After | Speedup |
| --- | --- | --- | --- |
| page of 100, all types | 1,867 (1,649-1,975) | 76.9 (66.2-497) | 24x |
| page of 100, agent and project | 10.3 (8.7-12.4) | 21.6 (20.2-23.2) |
0.5x (slower) |
| idle-sleep check | 220 (201-298) | 2.8 (2.0-5.3) | 78x |
| acknowledge one head | 4.9 (3.4-8.6) | 6.0 (5.2-7.6) | neutral |
| migration backfill | - | 15.6 s | - |

`EXPLAIN (ANALYZE, BUFFERS)` for the all-types page on fixture A: before
2,488 ms, 13,597 shared hits plus temp files; after 7.0 ms, 7,926 shared
hits.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=400 --per-agent=2500 --acked=2000 --samples=7
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=40 --per-agent=25000 --acked=24000 --samples=7
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=5000 --per-agent=200 --acked=190 --samples=7
cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/project-instance-secrets.test.ts src/__tests__/agent-runtime-hooks.test.ts
pnpm --filter @paperclipai/db check:migrations
```

- `resource-lifecycle-events.test.ts`,
`plugin-lifecycle-delivery.test.ts` and
`project-instance-secrets.test.ts`: 86 passed.
`agent-runtime-hooks.test.ts`: 22 passed.
- `tsc --noEmit -p server`: pass. `check:migrations`: pass.

## Risks

- **Schema change.** Migration `0462` adds one table and two foreign
keys to `plugins` and `companies`. It adds no index on an existing
table. The backfill reads the acknowledgment table once. It took 4.6 s
to 15.6 s on 1M-event fixtures with 0.8M to 1M acknowledgments.
- **Migration order.** The number `0462` is reserved after open PRs that
use `0459` to `0461` (#20571, #20568, #20573). Merge this PR after them.
Before merge, rebase onto master and chain the snapshot after the latest
migration.
- **Previous-release story.**
- Upgrade: the backfill stores, per plugin and resource, the end of the
contiguous acknowledged prefix. A cursor never passes an unacknowledged
event, so no event is lost. Acknowledged events above a gap keep their
rows, so no event is delivered again. A test runs the migration
statement on rows that only an earlier release wrote and checks that
`list` returns the same rows before and after.
- Rolling deploy: an earlier release that still runs writes only
acknowledgment rows. The new reads honor those rows.
- Rollback: the new code still writes every acknowledgment row, so an
earlier release sees the same progress.
- **Ordering assumption.** The cursor is safe because writers record a
resource's events while they hold that resource's row lock
(`paperclip_record_agent_runtime` locks the agent row; agent status and
project writers document the lock). Thus a later commit for the same
resource always has a higher id. A writer that breaks this contract
could commit an event below the cursor, and that event would not be
delivered. A lower id from a different resource is still delivered
(existing test).
- **Slower path.** A page that asks only for `agent` and `project`
streams is 1 to 11 ms slower. These streams are short, so the old full
read was cheap, and the new read visits each stream once. The cost is
bounded by the number of resources.
- **Many resources.** The page cost grows with the number of resource
streams of the requested types. At 5,000 agents a page is about 77 ms,
compared with about 1.9 s before.
- The hosted planner can choose a different plan. The hosted improvement
is not yet measured.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:58:32 -06:00
DottaandClaude Opus 5.5 8e4c5e81b9 perf(issues): read created-from tasks by index and inbox review waits by page (#20573)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task page shows "Created from this task". The inbox shows
blocked-inbox attention for a page of tasks.
> - On a hosted database, the created-from list ran about 4,000 times in
one afternoon at about 499 ms each. The task page sends it each time a
task opens.
> - Its filter was an `OR` of three provenance paths over the list's own
rows. PostgreSQL could not use an index for it. It read every task of
the company and tested each one, with the list's other per-row
conditions (such as the chat conversation state subquery).
> - The blocked-inbox attention map also read external review waits for
every in-review task of the company, not only for the tasks of the page.
> - This pull request reads created-from tasks from three index probes,
and reads review waits only for the in-review tasks that can change a
finding of the page.
> - The benefit is less database work on each task open and each inbox
page. The responses stay the same.

## Linked Issues or Issue Description

Refs #20400 (shares review-wait reads between the attention maps of one
request).
Refs #20551 (authorizes the review waits of a page with one read; this
change narrows the waits that the page asks for, and adds a test that
pins the batched read).
Refs #13241 (adds the created-from list).

**What happened**

- `GET /api/companies/:id/issues?createdFromIssueId=…` used
`createdFromIssueCondition`. Its filter was `origin_issue_id = source OR
(origin_issue_id IS NULL AND origin_run_id IN source runs) OR (… AND
EXISTS creation activity of a source run)`. The `OR` stops index use, so
the statement read every task of the company. The list's other
conditions, which include the chat conversation state subquery, ran for
each task before the task was excluded.
- `listIssueBlockedInboxAttentionMap` read review waits for every
in-review task of the company graph. The caller uses the findings of the
page rows only.

**Expected behavior**

The created-from list reads only the created tasks. The blocked-inbox
map reads review waits only for in-review tasks that the liveness
classifier can read for a page row. The rows, the order, and the
attention values do not change.

**Steps to reproduce**

1. Use a company with tens of thousands of tasks, runs, and activity
rows.
2. Open a task page. Look at the created-from list statement: it reads
the full task set of the company.
3. Open an inbox page with blocked-inbox attention. Look at the
review-wait read: it gets the ID of every in-review task of the company.

## What Changed

- `server/src/services/issue-creation-origin.ts`:
`createdFromIssueCondition` is now `issues.id <> source AND issues.id IN
(three branches)`. Each branch starts from an index:
  - recorded origin task: `issues_company_origin_issue_idx`;
- legacy origin run: the source task's runs
(`heartbeat_runs_company_origin_issue_idx`), then the new partial index
on `issues (company_id, origin_run_id)`;
- creation activity: `activity_log_run_id_idx` for the source runs, then
the task by primary key.
The precedence does not change: a recorded origin task wins, the run
path applies only without one, and the activity path applies only
without both. The activity path matches only an `entity_id` that is the
canonical text of the task ID, as before (it also compares `id::text =
entity_id`).
- `packages/db`: add `issues_company_legacy_origin_run_idx` on `issues
(company_id, origin_run_id) WHERE origin_issue_id IS NULL AND
origin_run_id IS NOT NULL`. Migration `0461` creates it with `CREATE
INDEX CONCURRENTLY IF NOT EXISTS` and is registered with the other
concurrent migrations. `doc/DATABASE.md` documents the index and the
benchmark command.
- `server/src/services/issues.ts`: add `blockedInboxReviewWaitIssueIds`.
It returns the in-review tasks in the blocker closure of the page rows.
The blocked-inbox map reads review waits only for these tasks. The
liveness classifier reads a wait only for a row or for a blocker that it
reaches from a row, and the caller reads only the findings of the rows.
The classifier does not change.
- Tests:
- `issue-created-from-routes.test.ts`: new cases for precedence in both
directions, a source-run comment, an activity row whose task has another
origin run, a non-canonical and a non-UUID activity entity ID, the
source's own creation activity, and other-company rows.
- `blocked-inbox-review-wait-scope.test.ts`: the narrowed set for
chains, cycles, shared blockers and off-page tasks. The rows get the
same findings with the narrowed waits as with the waits of every
in-review task, and each narrowed wait is needed.
- `issue-list-review-wait-authorization.integration.test.ts`: on real
PostgreSQL with two authorized waits, `listExternalReviewWaits` and one
issue-list request with blocked-inbox attention each send one bound-run
message-link statement. With the per-wait authorization from before
#20551, both tests fail (2 statements).
- `scripts/benchmark-created-from-issues.ts`: local benchmark. It loads
the baseline `issues.ts` and `issue-creation-origin.ts` from a pinned
commit, asserts identical complete row payloads, and reports alternating
timings and `EXPLAIN (ANALYZE, BUFFERS)`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-created-from-routes.test.ts
src/__tests__/blocked-inbox-review-wait-scope.test.ts
src/__tests__/issue-list-review-wait-authorization.integration.test.ts
src/__tests__/agent-task-hierarchy-depth.test.ts
src/__tests__/issue-blocker-attention.test.ts
src/__tests__/issue-list-assignee-filter-routes.test.ts
src/__tests__/issues-parent-id-alias.test.ts
src/__tests__/batched-chat-authorization.integration.test.ts
src/__tests__/issue-liveness.test.ts` passes (126 tests). One timing
test in `issue-list-assignee-filter-routes` failed one time under
machine load and passed on rerun.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/db check:migrations` pass.
- `pnpm verify` asks for broader verification because the change adds a
file under `scripts/`. Required CI supplies that coverage.
- Benchmark: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-created-from-issues.ts --baseline-ref=bd30ef8510`

Fixture (synthetic, local PostgreSQL 18.1): one company with 30,000
tasks (10,628 with a recorded origin task, 10,445 with only a legacy
origin run), 60,030 runs with wide context snapshots (145 MB), and
261,079 activity rows; a second company with 10,000 tasks and 20,000
runs. The busy source task has 63 created tasks through all three paths,
plus decoys. The quiet source task has 3. The UI's page: `sortField=id`,
`asc`, `limit=500`, routine executions included. Both versions return
identical complete payloads. 9 alternating pairs after 2 warm-up pairs:

| Case | List statement before | List statement after | List call
before, median (min to max) | List call after, median (min to max) |
| --- | --- | --- | --- | --- |
| Busy source (63 tasks) | 234.9 ms, 47,437 buffers | 0.90 ms, 437
buffers | 156.0 ms (114.5 to 480.7) | 31.6 ms (19.9 to 62.7) |
| Quiet source (3 tasks) | 90.7 ms, 47,413 buffers | 0.25 ms, 46 buffers
| 126.1 ms (90.6 to 300.4) | 10.5 ms (7.2 to 26.1) |

The list call time includes the enrichment reads (labels, runs,
attention), which do not change.

Inbox page (50 open tasks, blocked-inbox attention): the review-wait
read goes from 2 statements with 1,215 parameters (every in-review task
of the company) to 1 statement with 31 parameters. The page time does
not change in this fixture (median 801 ms before, 873 ms after, both
with a wide spread), because the fixture has no review waits and the
other graph reads of the map are most of the time. With real waits, the
server does an inherited GitHub review read for each wait it finds, so
the gain grows with the number of off-page waits.

Migration timing (local PostgreSQL 18.1): `0461` (measured as the same
statement under its earlier number) on 500,000 tasks (846 MB, 20
companies, one third with a legacy origin run) takes 1.6 s. The index is
12 MB.

## Risks

- The new index adds a small write cost to tasks with only an origin
run. New tasks with a source task record `origin_issue_id`, so the index
holds mostly older tasks. The migration does not lock writes.
- The created-from rewrite must keep the same rows. The new tests and
the benchmark payload check cover precedence, company scope, and ID text
forms. Visibility and read filters are the list's other conditions and
do not change.
- The review-wait narrowing depends on the liveness classifier reading
waits only for rows and their blocker chains. A unit test compares the
findings of the rows with full and narrowed waits. A future classifier
change that reads waits of other tasks must update
`blockedInboxReviewWaitIssueIds`. The function comment says this.
- The local timings are synthetic. Hosted acceptance needs a fresh
query-statistics window after deployment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, local PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:38:51 -06:00
DottaandClaude Opus 5.5 1805a1d247 perf(workspaces): probe workspace run bindings through indexes (#20568)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep checks each Git workspace for recent
activity and for live consumers before it expires or moves the workspace
> - These checks find the runs of a workspace with one SQL scope,
`expirationScope`. The scope used one `OR` over all run bindings, and
one branch decoded the context JSON of each run
> - PostgreSQL could not use an index for that `OR`. Each check read
every run of the company and its large context from TOAST. On a hosted
database, each call took 2 to 5 seconds, about 5,200 s in total in one
window
> - This pull request splits the scope into one index probe per binding
and adds one concurrent expression index for the legacy
`executionWorkspaceId` context key
> - The benefit is that each check reads a few index entries instead of
all company runs. The cleanup decisions do not change

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The workspace activity and cleanup-safety checks:
`gitWorkspaceLastActivityAt`, `isGitWorkspaceExpirationEligible`,
`workspaceMoveEligible` and the orphan-folder consumer check. All of
them use `expirationScope` in
`server/src/services/git-workspace-expiration.ts`.

**Subsystem affected**
Execution workspace cleanup in the server
(`git-workspace-expiration.ts`, `isolated-repository-rehome.ts`,
`isolated-task-directory-retention.ts` through the shared scope) and the
`heartbeat_runs` schema in `packages/db`.

**Current behavior**
The `relevant_runs` CTE selects company runs where any of these is true:
`issue_id` or `native_issue_id` is in the issue tree or the bound
issues, the run is a bound issue's checkout or execution run, or the
context JSON has a matching `issueId`, `taskId` or
`executionWorkspaceId`. The planner does one sequential scan of the
company runs and reads each context from TOAST. The cost grows with the
run history of the company, not with the size of the workspace.

**Proposed behavior**
Each binding is a separate index probe. The `UNION` of run IDs goes to a
primary-key probe. The SQL gives the same rows.

**Reason and benefit**
Local SQL measurements show a 34x to 169x decrease in time. The decrease
grows with the run history (see Verification).

## What Changed

- `expirationScope` builds `relevant_runs` from a `UNION` of index
probes, one for each binding. The key sets go in as arrays (`=
ANY(ARRAY(...))`). With `IN (subquery)`, the planner chose a hash join
over a full scan, because it does not count the cost of reading TOASTed
contexts.
- The legacy context keys use `context_snapshot ->> 'key'`. This
expression matches the existing indexes
`heartbeat_runs_company_ctx_issue_created_idx` and
`heartbeat_runs_company_ctx_task_created_idx`. For a non-object context,
`->>` gives `NULL`, as the strict JSONPath decode did. Such a run can
still match through a native binding.
- New partial index `heartbeat_runs_company_ctx_execution_workspace_idx`
on `(company_id, context_snapshot ->> 'executionWorkspaceId') WHERE
context_snapshot ? 'executionWorkspaceId'`. The query repeats the
predicate. For an array context, `?` can be true, but `->>` gives
`NULL`, so the result does not change. The index is partial so that it
serves only this probe. A full index that starts with `company_id` made
the planner choose it for an unrelated company-scoped query. Migration
0460 builds it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The
migration is in the list of migrations that run outside a transaction.
- New tests: a legacy `issueId` or `executionWorkspaceId` binding on a
run that is bound to a different issue still blocks cleanup. A plan test
with `enable_seqscan = off` checks that no binding scans
`heartbeat_runs` and that the three context indexes are used.
- The benchmark script `scripts/benchmark-workspace-activity.ts` gets a
`--runs-per-workspace` option. Unrelated fixture runs now have distinct
context IDs, as real history does.
- `doc/DATABASE.md` describes the probes and the new index.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/git-workspace-activity.test.ts
src/__tests__/native-harness-backup-expiration.test.ts`: 132 of 132
tests pass. The second file checks that the native session index stays
the planner's choice for its query. These include the existing route
matrix (root, child, bound, native, context issue, context task, context
workspace, checkout, execution run), exactly-once counts, malformed and
uppercase contexts, company isolation, controller leases and lease
vetoes.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/db check:migrations` pass.
- Benchmark (local SQL measurement, PostgreSQL 18.1, synthetic fixture,
8 workspaces per sample, before and after alternate in each iteration).
The script compares the complete ordered caller results and the complete
`relevant_runs` multisets of the baseline and treatment. They are equal
in every run.

`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-workspace-activity.ts --baseline-ref=origin/master
--runs-per-workspace=<N>`

| Fixture | Check | Before median (min–max) | After median (min–max) |
Speedup |
  | --- | --- | --- | --- | --- |
| 4,800 runs, 330 MB | last activity | 523 ms (460–598) | 12 ms (8–15) |
44x |
| 4,800 runs, 330 MB | eligibility | 452 ms (432–473) | 13 ms (11–15) |
34x |
| 4,800 runs, 330 MB | idle eligibility | 465 ms (442–573) | 12 ms
(10–13) | 40x |
| 24,000 runs, 1.7 GB | last activity | 2,240 ms (2,214–2,454) | 13 ms
(11–14) | 169x |
| 24,000 runs, 1.7 GB | eligibility | 2,245 ms (2,097–2,377) | 17 ms
(14–19) | 133x |
| 24,000 runs, 1.7 GB | idle eligibility | 2,196 ms (2,179–2,381) | 15
ms (13–18) | 143x |

Before: one `Seq Scan` on `heartbeat_runs`. After:
`heartbeat_runs_company_issue_created_idx`,
`heartbeat_runs_company_native_issue_contract_id_uq`, the three context
indexes and `heartbeat_runs_pkey`.
- Index build time: on a synthetic table with 200,000 runs and 8 KiB
contexts (1.8 GB), the full form of this index took 3.7 s to build
concurrently and used 15 MB. The partial index reads the same rows and
holds only runs that have the key.

## Risks

- Cleanup safety: a wrong "allowed" result could delete an active
workspace. The scope keeps every binding of the old query. The benchmark
checks exact equality of the run sets, and the existing tests check each
binding route and each veto.
- Migration: the index build reads every run context once. It runs
concurrently and does not block writes. Until migration 0460 builds the
index, the new query is still correct but the `executionWorkspaceId`
probe scans.
- Plan choice: the array form makes index probes the cheapest plan for a
small tree. A very large issue tree gives a large array, but each
element is still one index probe.
- The sweep frequency does not change. This pull request reduces only
the cost of each check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:17:46 -06:00
DottaandClaude Opus 5.5 d5c1f48180 perf(native): index the backup lease lookup by scope digest (#20571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native harness backup expiration removes old session backups. Before
it removes a backup, it finds the environment leases that reference the
backup root
> - This lookup reads leases by run ID, by the
`metadata.nativeHarnessBackup.sessionScopeSha256` digest, or by the
identity's own lease ID, in one `OR` query
> - The metadata arm has no index. So PostgreSQL reads every lease of
the company, and the metadata of each lease, for each lookup
> - On a hosted test database, this statement ran 21,736 times at a mean
of 158 ms in one window. After other fixes, it still ran 2,364 times at
a mean of 121 ms
> - This pull request adds a small partial expression index on the
digest
> - The benefit is that each lookup reads a few index entries instead of
all company leases. The query text and the result do not change

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The lease lookup in `backupDependencies` in
`server/src/services/native-runtime/native-harness-backup-expiration.ts`.
`reconcileBackup` calls it for each backup root when the expiration
worker loads or checks a root.

**Subsystem affected**
Native harness backup expiration and the `environment_leases` table
schema.

**Current behavior**
The query is `company_id = $1 and (heartbeat_run_id in (...) or
metadata->'nativeHarnessBackup'->>'sessionScopeSha256' = $n or id =
$m)`. The run arm and the ID arm have indexes. The digest arm has no
index. A `BitmapOr` plan needs an index for every arm, so the planner
reads all company leases and evaluates the JSON expression on each row.
Lease metadata holds sandbox details, so these reads are large.

**Proposed behavior**
The partial index `environment_leases_native_backup_scope_idx` keys
leases by the digest. It holds only leases that have a backup stamp.
Every arm of the `OR` now has an index, and the planner combines three
index probes. The SQL and the returned lease IDs do not change.

**Reason and benefit**
This statement was in the top ten by total database time on a loaded
hosted database. A local reproduction shows a query that is more than
6,000x faster and reads 9 buffers instead of about 436,000.

**Breaking changes**
None. The API, the lease-matching rules and the results do not change.

## What Changed

- Add the partial expression index
`environment_leases_native_backup_scope_idx` on
`(metadata->'nativeHarnessBackup'->>'sessionScopeSha256')`, where that
value is not null.
- Add migration `0459`. It uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`. The migration runner runs it as a top-level statement, the same
as migrations 0454–0458.
- Add a regression test. It checks the lease-matching rules: run leases,
a lease with only the backup stamp, and the identity's own lease match.
Another digest, a non-string stamp, a top-level key with the same name,
and the same digest in another company do not match. The test also
checks that the custom plan and a forced generic plan of the generated
SQL use `BitmapOr` with the new index and no sequential scan.
- Add `scripts/benchmark-environment-lease-backup-lookup.ts`. It
captures the SQL that the real service generates and measures it with
and without the index.
- Document the index in `doc/DATABASE.md`.

## Verification

Cause and thesis. The cost comes from the one `OR` arm without an index,
which forces a scan of all company leases and their metadata. Thesis: an
index on the digest lets the planner use a `BitmapOr` of three index
probes, so the cost no longer depends on the company's lease count. The
results stay identical. The benchmark confirms the thesis.

I did not change the SQL. With the index, the existing `OR` query
already gets the index plan. A `UNION` rewrite would not help without
the index, because the digest arm would still scan the company.

Local SQL benchmark:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-environment-lease-backup-lookup.ts
```

Fixture (synthetic): one company with 100,000 leases with 4 KiB of
incompressible metadata each. 30% of the leases have a backup stamp for
another session. A second company has 20,000 leases. The table is 678
MB. The backup root has 3 run leases, 1 lease with only the stamp, and 1
identity lease. The same digest is also in the other company. Embedded
PostgreSQL 18 on an Apple laptop, which other work loaded heavily. The
hosted database is Neon with different compute, so absolute times are
local SQL measurements only.

The script measures in the order: no index, index, no index, index. Each
measurement has 2 warmups and 11 samples of `EXPLAIN ANALYZE` execution
time and of the full `reconcileBackup` call.

| Measurement | Without index | With index |
| --- | --- | --- |
| Lease query execution, run 1 | median 196 ms (138–562 ms) | median
0.010 ms (0.007–0.017 ms) |
| Lease query execution, run 2 | median 185 ms (144–295 ms) | median
0.030 ms (0.020–0.045 ms) |
| `reconcileBackup` wall time (includes file reads), runs 1 and 2 |
median 188 ms and 216 ms | median 1.4 ms and 4.2 ms |
| Plan | `Gather > Seq Scan` | `Bitmap Heap Scan > BitmapOr` of
`heartbeat_run_idx`, the new index and `pkey` |
| Buffers | 365,880 hit + 70,050 read | 9 hit |

These numbers are from head `65bd9d7ac6`. The final head only renumbers
the migration to 0459 after rebases; the SQL, the index and the service
code are the same. Two earlier runs on the same fixture, SQL and index
gave 236–425 ms without the index and 0.023–0.033 ms with the index. The
machine had heavy load from other work, so the spread without the index
is wide. Output check: in all four measurements, the service returned
the same 5 lease IDs, and the complete query rows were equal.

Index build (`CREATE INDEX CONCURRENTLY` from the migration file):
0.8–2.4 s for the 678 MB table over all runs. Index size: 3.4 MB with
30% of leases stamped.

Tests:

- `cd server && npx vitest run
src/__tests__/native-harness-backup-expiration.test.ts`: 109 passed. The
new test failed when I dropped the index before the plan check
(confirmed red), and it passes with the index.
- `pnpm --filter @paperclipai/db check:migrations`: passed.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`: passed.
- `pnpm verify`: it asks for broader coverage (exit 2) because of the
new script. See the CI results for this head.

## Risks

- Migration time. A concurrent index build reads the full
`environment_leases` table and does not block writes. Locally, the build
took 0.8–2.4 s for 678 MB. The time grows linearly with the table size.
I did not time it on a production-size clone. I expect a few seconds to
tens of seconds on a large instance, which is in the 1–2 minute budget.
This estimate is not proven.
- If the build stops, PostgreSQL can leave an invalid index. The
migration runner repairs invalid concurrent indexes when it runs again,
and `IF NOT EXISTS` makes a rerun safe.
- The index expression must stay textually equal to the query
expression. If someone changes the JSON path, the planner cannot use the
index. The new test catches this for custom and generic plans.
- The index is partial, so ANALYZE does not collect statistics for the
expression. The planner uses the default equality estimate. On the large
fixture and in the small test, the planner still selects the index plan.
A company with very few leases can still get a sequential scan, which is
cheap.
- Each lease write with a backup stamp writes one more small index
entry. The write cost is small.
- This change does not reduce how often the worker calls the lookup. It
makes each call cheap. The other digest lookups in backup retention
(`native-harness-backup-retention.ts`) and session retention checks can
also use the index.
- Previous-release story: no persisted state changes. The index is
derived from existing lease metadata.
- The hosted improvement is not verified until deployment. After
deployment, compare a new query-statistics window for this statement.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:30:05 -06:00
DottaandClaude Opus 5.5 765ccd8691 perf(chat): index chat-admission checks by company and payload issue id (#20562)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connections save each external message as a `chat_actions` row
before the server admits it as an agent wakeup.
> - The server checks if an issue holds chat input that it did not
admit. Recovery skips such an issue. Manual task start refuses it.
> - On a hosted instance, the recovery read with this check used the
most total database time: 488 calls at about 1 s each. The manual-start
check ran 155 times at about 0.9 s.
> - The check matches `chat_actions.payload->>'issueId'` against the
issue. No index covers this value, so each call reads the full
`chat_actions` table and its payloads.
> - This pull request adds one concurrent partial index for the check.
The check SQL does not change.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The chat-admission check `unadmittedChatWakeupCondition`. Recovery uses
it in the per-issue candidate read. Manual task start uses it in the
held-input check.

**Subsystem affected**
Server chat admission, issue recovery, manual task start, and the
database schema.

**Current behavior**
The check finds `inbound_wakeup` actions by `company_id` and
`payload->>'issueId'`. Two nested checks find `failed_run_retry` actions
and later `inbound_wakeup` actions on the same company and payload issue
ID. `chat_actions` has no index on `company_id` or on the payload issue
ID. Each call reads the full table and reads each payload. On a hosted
instance, the two statements used about 630 s of database time in one
statistics window.

**Proposed behavior**
The check and its nested checks read a small set of rows from one index.
The admission result does not change.

**Reason and benefit**
The hosted database had many concurrent active queries. Request reads
waited behind background work such as this check.

**Breaking changes**
None. The migration adds an index only.

## What Changed

- Migration `0458_chat_admission_issue_index` builds
`chat_actions_company_issue_kind_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. It runs through the existing top-level statement path,
which repairs an invalid index on retry.
  - Keys: `(company_id, payload->>'issueId', kind, status)`.
  - Partial: `kind IN ('inbound_wakeup', 'failed_run_retry')`.
- One index serves the outer input lookup, the failed-run retry lookup,
and the later-input lookup. All three match on the same company and
payload issue ID.
- The Drizzle schema declares the index. The snapshot chain moves to
`0458` and keeps five snapshots.
- New regression test
`server/src/__tests__/chat-admission-condition.test.ts` pins the
admission semantics on real PostgreSQL:
- Pending and failed input without a receipt holds the issue. Received,
processed and cancelled input does not. Other action kinds do not.
- A skipped receipt, a receipt for another agent, or a receipt in
another company does not admit input. A failed or cancelled run still
counts as admission.
- The issue ID is compared as company-scoped text. Uppercase, array,
null, other-issue and other-company values do not match.
- Only an admitted retry in the same conversation and issue that lists
the action releases skipped input.
- Only later admitted input in the same conversation, issue and session
generation releases a failed input. Pending input stays held.
  - The check reads `chat_actions` only through the new index.
- `scripts/benchmark-chat-admission-index.ts` measures both caller
statements with and without the index and fails if the rows differ.
- `scripts/benchmark-chat-admission-index-migration.ts` times the
migration through the normal migration runner.
- `doc/DATABASE.md` documents the index and the benchmark commands.

## Verification

Local SQL measurements on synthetic data. PostgreSQL 18.1 (embedded),
default settings. The fixture has 100,995 `chat_actions` rows (662 MB):
60,000 GitHub actions with 6 KiB incompressible payloads, 2,000 issues
with 20 admitted inbound messages each (3 KiB payloads), and 1,000
failed-run retries. Each case has 2 warmups and 9 samples of `EXPLAIN
ANALYZE` execution time. Runs alternate: no index, index, no index,
index.

| Caller statement | Case | No index, median (range) | Index, median
(range) |
| --- | --- | --- | --- |
| Manual-start held check | quiet issue | 3,093 ms (2,150–3,519) / 3,586
ms (2,375–11,235) | 0.036 ms (0.034–0.052) / 0.040 ms (0.033–0.047) |
| Manual-start held check | admitted input | 3,555 ms (3,310–8,080) /
5,205 ms (3,353–9,525) | 0.049 ms / 0.052 ms |
| Manual-start held check | unadmitted input | 3,002 ms (2,864–3,221) /
4,179 ms (3,200–6,585) | 0.051 ms / 0.044 ms |
| Manual-start held check | failed, later admitted | 3,586 ms / 3,722 ms
| 0.066 ms / 0.054 ms |
| Manual-start held check | admitted retry | 2,733 ms / 4,953 ms | 0.049
ms / 0.053 ms |
| Recovery candidate read | quiet issue | 43 ms (41–48) / 96 ms (44–219)
| 0.047 ms / 0.055 ms |
| Recovery candidate read | admitted input | 203 ms (48–575) / 174 ms
(44–314) | 0.055 ms / 0.058 ms |
| Recovery candidate read | unadmitted input | 66 ms / 144 ms | 0.053 ms
/ 0.054 ms |

- Buffers per call fall from about 40,000 shared hits to 5–23.
- Locally, the recovery read without the index uses a hashed full scan
of `chat_actions`. The held check uses a full scan per call. On the
hosted instance both took about 1 s per call. The local fixture is
smaller than the hosted table.
- Correctness: complete positional rows match across all four runs for
all six cases (quiet, admitted, unadmitted, failed then admitted,
retried, other-company decoy). The script asserts the expected held
result for each case.
- Migration timing: 600,001 `chat_actions` rows (1,759 MB), 300,000 of
them in the index scope. First run through `applyPendingMigrations`: 4.2
s. Rerun after the history row is removed: 0.5 s. Concurrent build
alone: 3.4 s. Index size: 30 MB.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index.ts
OTHER_PAYLOAD_KIB=2 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index-migration.ts
cd server && npx vitest run src/__tests__/chat-admission-condition.test.ts   # 6 passed
pnpm --filter @paperclipai/db check:migrations                               # passed
pnpm --filter @paperclipai/server typecheck                                  # passed
```

## Risks

- The migration builds one index concurrently. It does not block writes.
On the 1.76 GB local fixture it took about 4 s. A larger hosted table
takes longer, but the build does not lock out reads or writes.
- Writes to `inbound_wakeup` and `failed_run_retry` actions maintain one
more index. Status updates on these rows were already not HOT updates,
because other partial indexes use `status`.
- The planner has no expression statistics for a partial index. The new
test pins that the check uses the index on an analyzed table with
unrelated rows.
- The check SQL does not change, so admission behavior does not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:44:51 -06:00
DottaandClaude Opus 5.5 56aa160a4e perf(native): index heartbeat runs by company and native session (#20560)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs share a native session. Retention, recovery, resume and
archive code reads all runs of one session from `heartbeat_runs`.
> - On a hosted instance, the native retention lookup ran about 67,000
times in one day at about 22 ms, and about 10,000 times at about 11 ms
in a later window.
> - No index has `native_session_id`. Each read walks the agent's or the
company's run history and filters the rows.
> - This pull request adds one concurrent partial index on `(company_id,
native_session_id)`. The SQL does not change.
> - The benefit is less background database load. Request reads wait
less, and the results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The native session-run lookups. The largest caller is
`sessionRetentionBlocker` in native harness backup and checkpoint
retention. It selects `id, status, finished_at`, an incomplete-recovery
flag and the issue ID for one company, agent and native session.

**Subsystem affected**
Server native runtime (retention, recovery, resume and archive reads)
and the database schema.

**Current behavior**
`heartbeat_runs` has no index on `native_session_id`. PostgreSQL reads
the whole run history of the company (or of the agent through
`heartbeat_runs_company_agent_started_idx`) and filters by session. The
cost grows with the run history, not with the size of the session.

**Proposed behavior**
The index finds the few runs of one session directly. Every caller
already filters by `company_id` and `native_session_id`, so one index
serves the agent-scoped and the issue-scoped reads.

**Reason and benefit**
This statement family used about 1,500 s of database time in one day on
a busy hosted database. The local proof below shows 12.3 ms to 0.04 ms
per call.

## What Changed

- `packages/db`: add `heartbeat_runs_company_native_session_idx` on
`(company_id, native_session_id) WHERE native_session_id IS NOT NULL`.
Migration `0457_native_session_run_index` builds it with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. The migration is registered as a
top-level-statement migration, the same way as 0454–0456.
- Add a regression test in `native-harness-backup-expiration.test.ts`.
It captures the real retention lookup and proves that its plan can use
the index. It also pins the company and session scope: a live run of
another company with the same session ID does not protect a checkpoint.
A live run of the same agent in another session does not protect it. A
live run of the same session does protect it.
- Add `scripts/benchmark-native-session-run-index.ts`. It runs the real
checkpoint retention pass (dry run), captures the exact SQL, and replays
it with and without the index. It also times the migration through the
normal migration runner.
- `doc/DATABASE.md`: describe the index and the benchmark command.

## Verification

Local SQL measurements, synthetic data. PostgreSQL 18.1 (embedded test
server). Fixture: 100,000 runs (50,000 of the measured agent, 50,000 of
a second agent), 4 runs per session, 4 KiB incompressible
`runner_profile_json` per run, 638 MB table. Order: no index, index, no
index, index. 15 `EXPLAIN (ANALYZE, BUFFERS)` samples after 3 warmups.

| Pass | Plan | Median execution | Min / max | Shared buffers |
| --- | --- | --- | --- | --- |
| No index (1) | Seq Scan | 12.30 ms | 10.33 / 35.27 ms | 3,607 |
| Index (2) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.036 ms | 0.021 / 0.115
ms | 38 |
| No index (3) | Seq Scan | 12.38 ms | 10.24 / 26.14 ms | 3,607 |
| Index (4) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.043 ms | 0.022 / 0.098
ms | 38 |

- Correctness: the service SQL is the same in every pass. The returned
rows (all columns) and the retention result are equal in every pass. The
script fails on any difference.
- Migration: the first run through the migration runner took 525 ms on
the 638 MB fixture. The rerun took 273 ms and skipped the existing
index. A direct concurrent build took 145–221 ms. Index size: 1.9 MB.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-native-session-run-index.ts`
- `npx vitest run
src/__tests__/native-harness-backup-expiration.test.ts` (server): 108
passed.
- `pnpm --filter @paperclipai/db check:migrations`: passed. Five
snapshots remain.
- `pnpm verify` exits 2 (a new script under `scripts/` is a global CI
input). Required CI supplies the broad coverage.

## Risks

- Low risk. The migration adds one index concurrently and does not lock
writes. The SQL of every caller is unchanged.
- The index adds a small write cost to `heartbeat_runs` inserts and to
updates of `native_session_id`. Rows with a null session are not
indexed.
- Persisted run state: no saved run, session or checkpoint shape
changes. State saved by the previous release carries forward unchanged.
- Hosted latency is not measured yet. The numbers above are local SQL
measurements.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:04:22 -06:00
DottaandClaude Opus 5.5 347d85fef3 perf(chat): index GitHub review lookups and share review-wait reads (#20400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub chat endpoints let agents review pull requests. The server
reads review state when it lists issues and when it updates GitHub
checks.
> - On a hosted instance, two GitHub review queries used the most total
database time. One query ran about 660,000 times at about 33 ms. The
other query ran about 10,000 times at about 1 s.
> - The first query finds a source comment's chat link. No index covers
`chat_message_links.comment_id`, so each call reads the full table.
Issue lists also read the same review waits two times per request.
> - The second query joins deliveries, links and runs on a JSON key. The
planner walks the agent's run history in creation order and reads each
run's context.
> - This pull request adds four concurrent indexes, splits the run
lookup into two small reads, and shares review-wait reads in one
issue-list request.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
GitHub review state reads. These are the inherited review-wait lookup by
source comment, and the review-check lookup of the run that a delivery's
comment woke.

**Subsystem affected**
Server chat channels (GitHub review waits and checks), issue list
attention maps, and the database schema.

**Current behavior**
- `readInheritedGitHubReviewWait` filters `chat_message_links` by
`company_id`, `comment_id` and `direction`. No index has `comment_id`,
so PostgreSQL reads the full table on each call. The run-origin check
does the same lookup first.
- An issue list calls the external review-wait read from the review
attention map and again from the blocked-inbox attention map. The two
maps read the same in-review issues.
- `githubReviewCheckState` joins deliveries, links and runs with
`context_snapshot->>'wakeCommentId' = comment_id::text`, sorts by
`created_at desc` and takes one row. The planner reads the agent's runs
in creation order and reads each run context. If no run matches yet, it
reads all of them. The delivery filter on `normalized_event` and the
link join on `delivery_id` also have no index.

**Proposed behavior**
The same rows come back from small index reads. Each issue's review wait
is read one time per issue-list request.

**Reason and benefit**
These two statement families used about 33,000 s of database time in one
cumulative statistics window. The database had about 50 concurrent
active queries, and request reads waited behind this background work.

**Breaking changes**
None. The migration adds indexes only.

## What Changed

- Migration `0456_chat_review_lookup_indexes` builds four indexes with
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It runs through the existing
top-level statement path, which repairs an invalid index on retry:
- `chat_message_links_company_comment_idx` on `(company_id, comment_id)`
where `comment_id is not null`.
- `chat_message_links_delivery_idx` on `(delivery_id)` where
`delivery_id is not null`.
- `chat_deliveries_github_automatic_delivery_idx` on `(endpoint_id,
normalized_event->'githubAutomatic'->'context'->>'deliveryId')` where
that value is not null.
- `heartbeat_runs_company_wake_comment_idx` on `(company_id,
context_snapshot->>'wakeCommentId')`. This index is not partial. The
planner does not use expression statistics from a partial index. Without
these statistics, it estimates many matches and keeps the creation-order
walk.
- `githubReviewCheckState` reads the delivery's comment IDs first. If
there is a comment, it reads the newest same-agent run with one of these
`wakeCommentId` values. If there is no comment, it skips the run read.
The comparison stays text equality on `comment_id::text`.
- `createExternalReviewWaitReader` shares one read per issue between the
review and blocked-inbox attention maps of one issue-list request. The
reader exists only for that request.
- `doc/DATABASE.md` describes the indexes and the new read order.
- `scripts/benchmark-review-lookup-indexes.ts` compares the pre-fix and
new code with and without the indexes.
`scripts/benchmark-review-lookup-index-migration.ts` times the migration
on a large synthetic database.

## Verification

Local SQL measurements on a synthetic fixture. They are not production
timings. Environment: embedded PostgreSQL 18.1, baseline revision
`02bfe71a6b`. Fixture: 50,000 links, 5,000 deliveries with about 2 KiB
events, and 4,000 runs with about 12 KiB incompressible contexts. The
wake run is in the middle of the run history. Each value is the median
server execution time from `EXPLAIN ANALYZE` (2 warmups, 9 samples). The
order was: no indexes, indexes, then no indexes again.

| Caller and statement | Before (no indexes, two passes) | After |
Shared buffers before → after |
| --- | --- | --- | --- |
| Inherited review wait, link by source comment (same SQL) | 2.72 / 2.36
ms | 0.076 ms | 929 → 12 |
| Review check, wake run found | 31.9 / 21.9 ms | 0.29 ms (two
statements) | 25,776 → 62 |
| Review check, no wake run yet | 71.0 / 47.5 ms | 0.06 ms (two
statements) | 34,861 → 12 |

- Indexes alone do not fix the review check. The old SQL with the new
indexes took 8.6 ms (run found) and 29.7 ms (no run), because it still
walked the run history. The new SQL without the new indexes was neutral
(23.5 and 35.1 ms).
- Service wall times on this shared machine were noisy, so the table
reports server execution time.
- Correctness: the benchmark fails unless the inherited statement
returns the same positional rows in each index state. The check service
result and the final wake-run rows of the new code must also match the
pre-fix module for both cases.
- Migration timing, normal migration runner, synthetic data: 200,000
runs (1.8 GB with 8 KiB incompressible contexts), 500,000 links and
100,000 deliveries. First run 4.2 s; rerun 0.38 s. Separate builds:
links by comment 1.5 s, run wake comment 1.1 s, deliveries 0.7 s, links
by delivery 0.3 s. All indexes were valid. The run index was 8.3 MB.
- New tests:
- `server/src/__tests__/github-review-check-state.test.ts` checks that
the newest same-agent run is selected. It also checks that other agents,
other companies, uppercase IDs, non-string values, list-only
`wakeCommentIds`, comment-less links and unknown deliveries do not
match. It checks that the generated statements can use the new indexes.
- `server/src/services/native-runtime/external-review-wait.test.ts`
checks that the reader reads each issue one time, keeps concurrent
callers correct, and rejects all waiting callers when a read fails.
- Red/green: with the pre-fix `review-check-state.ts`, the behavior test
passes and the index test fails. With this change, both pass.
- Focused runs pass: `issue-blocker-attention`,
`heartbeat-issue-liveness-escalation`, review-wait, review-check-state,
external-review-wait, review-repair (106 tests), and
`external-chat-wait.integration` (285 tests). `pnpm --filter
@paperclipai/server typecheck` and `pnpm --filter @paperclipai/db
check:migrations` pass.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-indexes.ts --baseline-ref=02bfe71a6b
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-index-migration.ts
```

## Risks

- The migration builds four indexes concurrently. It does not block
writes. On a large database, the run index reads every run context one
time. The local run took 1.1 s for 1.8 GB from a warm cache. Cold
storage on smaller compute can be slower.
- The run index adds one entry for each run insert and for each run
update that is not a heap-only update. Many runs have no
`wakeCommentId`, so these entries hold a null key.
- The review check now sends two statements instead of one when a
delivery has a comment. Each statement reads only a few index entries.
- When two matching runs have the same `created_at`, either can be
selected. The previous query had the same tie behavior.
- This change does not change the number of in-review issues that one
issue list reads. The blocked-inbox map still reads all in-review issues
of the company. A smaller scope needs a separate change to the liveness
classifier.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:47:34 -06:00
DottaandClaude Opus 5.5 0116e45958 perf(db): index pending budget Stops instead of scanning all run results (#20391)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget enforcement stops agent runs when a budget hard limit is
reached. The stop intent is saved in each run's `result_json` until run
control delivers it.
> - The budget worker, run control, and the idle-sleep check find these
pending stops with a predicate on `result_json`. No index supports that
predicate.
> - Each check therefore reads every heartbeat run and its large result
JSON from TOAST. On a hosted database, this statement family was among
the most expensive: about 8,000 calls at about 1 s each, about 9,800 s
in total.
> - Pending stops are rare. The predicate is constant, so a partial
index can contain only those rows.
> - This pull request adds that partial index. The SQL that the server
sends does not change.
> - The benefit is that each check reads a few index entries instead of
the full table. The database has more capacity for request work.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The check for committed budget Stops that are not delivered yet. The
budget worker runs it each time it looks for work
(`hasPendingBudgetEnforcement`), and again for each delivery pass
(`listPendingBudgetCancellationScopes`). Run control and the idle-sleep
check use the same predicate.

**Subsystem affected**
Budget enforcement in the server
(`server/src/services/heartbeat/budget-cancellation.ts`,
`server/src/services/budgets.ts`,
`server/src/services/heartbeat/run-control.ts`,
`server/src/services/idle-sleep-safety.ts`) and the `heartbeat_runs`
schema in `packages/db`.

**Current behavior**
The query groups all rows of `heartbeat_runs` and reads
`result_json->'startupCancellation'->'budgetEnforcement'` for each row.
A limit of 1 does not help, because the group and sort occur before the
limit. Cost grows with the run history, not with the number of pending
stops.

**Proposed behavior**
A partial index, `heartbeat_runs_pending_budget_cancellation_idx`,
contains only runs with a pending budget Stop. The planner uses it for
all four callers. The results are the same.

**Reason and benefit**
Local SQL measurements show a 300x to 1,000x decrease in time and buffer
reads (see Verification).

**Breaking changes**
None.

## What Changed

- Added the partial index
`heartbeat_runs_pending_budget_cancellation_idx` on `heartbeat_runs
(company_id)`. Its predicate is identical to
`pendingBudgetRunCancellationPredicate`.
- Added migration `0455_pending_budget_cancellation_index.sql`. It uses
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`, and it is in the list of
migrations that run outside a transaction, as `0436` is.
- Moved the scoped run-control read and the idle-sleep check into
exported builders in `budget-cancellation.ts`, next to the scope list
query. The SQL does not change. The callers and the test now share the
same statements.
- Added a real PostgreSQL regression test. It checks that delivered,
malformed, and missing intents are not pending. It also EXPLAINs the
statements that the callers send (scope list with limit 1 and 100,
scoped read, `EXISTS` check) and requires the new index in each plan. If
a person changes a query shape, the server predicate, or the index
predicate, the test fails.
- Added `scripts/benchmark-budget-cancellation-scan.ts`. It captures the
SQL that the real service sends, then compares the plans, timings, and
complete rows with and without the index.
- Updated `doc/DATABASE.md`.

## Verification

Red/green regression test:

- Before the migration, the new test failed: the plan was `Seq Scan`
with the predicate as a filter.
- After the migration: `npx vitest run
src/services/heartbeat/budget-cancellation.test.ts` passes (17 tests).
- Also passed: `src/__tests__/budgets-service.test.ts`,
`src/__tests__/idle-sleep-diagnostics-postgres.test.ts`,
`src/services/budget-resume-continuation.test.ts`.
- `pnpm --filter @paperclipai/db check:migrations`, and typecheck of
`@paperclipai/db` and `@paperclipai/server` pass.

Performance and result equality (local SQL measurements only):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-cancellation-scan.ts --runs=20000 --payload-kib=16 [--reverse]
```

- Fixture: embedded PostgreSQL 18.1. 20,241 runs in 2 companies, with
incompressible result JSON of about 16 KiB (table 379 MB). 40 pending
stops in 6 scopes, 200 delivered or malformed decoys, and 1
cross-company intent.
- The SQL is the same in both treatments. The script toggles only the
new index. 2 warmups and 7 samples per query. Rows are compared outside
the timed part. Ordered queries must match in order. The scoped
run-control read has no `ORDER BY`, so it is compared as a multiset.

| Query (caller) | Baseline median | With index median | Shared buffers
|
| --- | --- | --- | --- |
| `listPendingBudgetCancellationScopes(db, 1)` (worker `hasPending`) |
688 ms | 2.1 ms | 243,500 → 1,158 |
| `listPendingBudgetCancellationScopes(db)` (delivery) | 799 ms | 1.4 ms
| 243,500 → 1,158 |
| Scoped pending runs (run control) | 357 ms | 1.1 ms | 121,835 → 322 |
| `EXISTS` pending (idle-sleep check) | 440 ms | 0.37 ms | 240,560 → 2 |

- Reverse order (index first) gave the same result: 524 ms → 1.6 ms, 464
ms → 1.4 ms, 306 ms → 1.0 ms, 306 ms → 0.42 ms.
- Plans change from `Seq Scan` (or a broad bitmap scan) to `Index Scan`
or `Index Only Scan` on the new index.
- All result rows are equal in every run.

Migration duration:

- 20,241 runs (379 MB): index build 2.0 s to 3.9 s.
- 200,241 runs with 8 KiB result JSON (2.1 GB): index build 5.6 s. With
this table size, the four queries change from 1,148 / 879 / 514 / 4,392
ms to 1.1 / 1.2 / 0.85 / 8.3 ms.
- The build is concurrent, so it does not block run writes.

## Risks

- Migration: the build reads every run's result JSON one time. It is
concurrent and resumable (`IF NOT EXISTS`). The executor repairs an
invalid concurrent index on retry.
- Write cost: run writes evaluate the index predicate. The index
contains very few rows, so the extra cost is small.
- Predicate drift: if a later change edits only one of the two
predicates, the planner cannot use the index. The new regression test
detects this.
- The hosted timings are not measured yet. The local numbers come from a
synthetic fixture.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:14:52 -06:00
DottaandClaude Opus 5.5 c985541a57 perf(db): index retry-exhaustion receipts for the attention scan (#20386)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The attention feed shows runs that failed after automatic retries
stopped
> - To find these runs, the feed reads every `heartbeat_run_events` row
of the company and filters the rows by a message prefix
> - Run output makes this table very large, so each feed request does a
large scan
> - On a hosted test database, this one statement used the most total
database time: 923 calls at a mean of 5 s (4,709 s in total)
> - Request authentication waits behind this background work, so all
users feel the load
> - This pull request adds a small partial index that holds only the
retry-exhaustion receipts
> - The benefit is that the same query reads a few index entries instead
of all company events

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The attention feed query that finds retry-exhausted runs
(`listAttentionExhaustedRuns`).

**Subsystem affected**
Server attention feed and the `heartbeat_run_events` table schema.

**Current behavior**
The query reads all company rows in `heartbeat_run_events`, filters them
by `event_type = 'lifecycle'` and the `Bounded retry exhausted` message
prefix, and then joins the few matches to failed or timed-out runs. The
cost grows with all run output. It does not grow with the number of
exhausted runs.

**Proposed behavior**
A partial index holds only the matching receipts. The query plan reads
the index and joins a few rows. The query text and the result do not
change.

**Reason and benefit**
The scan was the most expensive statement by total time on a loaded
hosted database. A local reproduction shows a 54x faster query and about
116x fewer buffer reads.

**Breaking changes**
None. The API, the feed rows, and their order do not change.

## What Changed

- Add the partial index `heartbeat_run_events_retry_exhausted_idx` on
`(company_id, run_id, id DESC)`. The index covers only rows where
`event_type = 'lifecycle'` and `message like 'Bounded retry
exhausted%'`.
- Add migration `0454`. It uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`. The migration runner runs it as a top-level statement, the same
as migrations 0432, 0436 and 0450.
- Add a regression test. It pins the row order (newest run first, then
the newest receipt on a tie) and the latest receipt per run. It also
checks that the query plan can use the new index. If the query predicate
changes so that the index no longer applies, the test fails.
- Add `scripts/benchmark-exhausted-run-attention.ts`. It measures the
same query with and without the index, and the time to build the index.
- Document the index in `doc/DATABASE.md`.

## Verification

Cause and thesis. The scan cost comes from reading all company events,
not from the few exhausted runs. Thesis: an index that holds only the
receipts makes the query cost depend on the number of receipts, and
keeps the result identical. The benchmark below confirms the thesis.

Rejected alternative without a migration. I first tried a LATERAL lookup
that starts from the failed and timed-out runs. On the same fixture it
was about 2x slower (median 379 ms against 194 ms). It reads the full
event history of each failed run with random I/O, and that costs more
than one sequential scan.

Local SQL benchmark (the SQL that the service generates; same SQL with
and without the index):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-exhausted-run-attention.ts --runs=30000 --events-per-run=100 --samples=9
```

Fixture (synthetic): one company with 30,000 runs (3,819 failed or timed
out) and 3.0 M events, plus a decoy company with 0.5 M events. The table
is 1.4 GB. About 5% of events are `lifecycle` events. Events of 20
concurrent runs are interleaved. 1 in 23 runs has a receipt, some runs
have two receipts, and output decoys contain the prefix. Two agents are
terminated. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon with different compute, so the absolute times are local
SQL measurements only.

The script runs two cycles (without index, then with index; then with
index, then without index) to expose cache bias. Each measurement has 2
warmups and 9 samples.

| Query | Without index | With index | Samples and spread | Output check
|
| --- | --- | --- | --- | --- |
| `listAttentionExhaustedRuns` | median 157 ms (96–1,271 ms) | median
3.3 ms (2.2–5.7 ms) | 18 each | 133 rows, positional rows equal in all 4
measurements |
| Buffers (EXPLAIN ANALYZE) | 127,743 blocks | 5,580 blocks | | plan
uses the new index (asserted) |

Speedup: 48x (median), on head `ae6037b2a0` (the same SQL and index as
the final head). The machine was under heavy load from other work, so
the spread without the index is wide. An earlier run on the first head
gave 248 ms against 4.6 ms (54x). Index build (`CREATE INDEX
CONCURRENTLY` from the migration file): 1.7 s and 2.5 s for the 1.4 GB
table. Index size: 152 KiB.

Review fix: the query now writes `event_type = 'lifecycle'` as a SQL
literal, not a bound parameter. With prepared statements, PostgreSQL can
reuse a generic plan. A generic plan cannot prove that a bound parameter
matches the index predicate, so it would not use the index.

Tests:

- `cd server && npx vitest run src/__tests__/attention-service.test.ts`:
44 passed. The new test checks a fresh plan and a forced generic plan
(`plan_cache_mode = force_generic_plan` with `PREPARE`). It failed with
the bound `event_type` parameter and with a changed message prefix
(confirmed red), and it passes with the fix. Locally, the file's
`afterAll` database cleanup hook timed out on a heavily loaded machine
(load average about 28). The timeout also occurs when the new test is
excluded.
- `pnpm --filter @paperclipai/db check:migrations`: passed.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`: passed.
- `pnpm verify`: see the CI results for this head.

## Risks

- Migration time. A concurrent index build reads the full
`heartbeat_run_events` table and does not block writes. Locally, the
build took 1.2–2.5 s for 1.4 GB. The time grows linearly with the table
size. I did not time the build on a production-size clone. On a database
with a 5 s full company scan, I estimate the build at tens of seconds,
which is in the 1–2 minute budget. This estimate is not proven.
- If the build stops, PostgreSQL can leave an invalid index. The
migration runner repairs invalid concurrent indexes when it runs again,
and `IF NOT EXISTS` makes a rerun safe.
- The index predicate must stay textually equal to the query predicate.
If someone changes the message prefix or binds a predicate value as a
parameter, the planner cannot use the index. The new test catches both
cases, for fresh and generic plans.
- Each new receipt writes one more small index entry. Receipts are rare,
so the write cost is very small.
- The hosted improvement is not verified until deployment. After
deployment, compare a new query-statistics window for this statement.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL). Standard
context window and reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:37:20 -06:00
DottaandPaperclip facd409073 perf(workspaces): index legacy workspace companion lookups (#20372)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online workspace migration must retain prepared sources and recovery
holds.
> - Companion discovery reads saved workspace references from run
profiles.
> - The selector can scan large saved profiles even when it returns few
rows.
> - This pull request adds one company and workspace expression index.
> - Companion discovery uses the same expression as existing migration
lookups.
> - The result preserves each source check and reduces history scan
work.

## Linked Issues or Issue Description

**What happened?**

The companion selector used a JSON path expression with no matching
index. A read-only plan on about 22,000 run rows showed a sequential
scan over a table with about 1.3 GB of table and TOAST storage. A
repeated inspection reached its statement timeout. Existing migration
lookups use a different JSON operator form, so one expression index must
match both callers.

**Expected behavior**

Use a company-scoped indexed lookup. Keep prepared and finalized
references, the exact text comparison, native issue references, and run
order. Keep all source and recovery validation.

**Steps to reproduce**

1. Retain many run profiles with saved workspace references and wide
JSON fields.
2. Inspect a workspace with few matching companion runs.
3. Explain the company and saved-workspace selector. Without the new
index, it can scan run history and read unrelated JSON values.

**Paperclip version or commit**

Baseline: `0e3fdebea8257bdff11573081c76b306f12db230`.

**Deployment mode**

Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1.
The observed server plan used PostgreSQL 17.11.

Related: [batched companion discovery in
#19735](https://github.com/paperclipai/paperclip/pull/19735). The
existing batching remains in place. The duplicate search found no open
workspace index change. This branch also carries the isolated API
route-summary correction shared with
[#20370](https://github.com/paperclipai/paperclip/pull/20370).

## What Changed

- Add one concurrent index on company ID and the saved native workspace
ID expression. The index has no state, runtime, or status predicate.
- Use nested JSON operators in companion discovery. Existing migration
lookups already use this expression. Keep both accepted states and the
same ordering.
- Register the migration on the existing top-level migration path.
Retain its advisory lock, interrupted index repair, and completion
journal behavior.
- Add a real PostgreSQL migration and ordered-result test. Update the
existing companion query counter for the equivalent SQL form.
- Generate the schema snapshot and retain the five newest snapshots.
- Describe the task case-list route as “Find the cases for a task.” This
keeps the route on the first discovery search page. Search logic and
test assertions are unchanged.

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/heartbeat-workspace-companion-index.test.ts --maxWorkers=1`: **1
passed**. It builds the concurrent index through the canonical migrator,
checks index validity, preserves prepared/finalized and
cross-issue/null-issue results, excludes foreign and malformed
references, and checks one journal entry after a second migration call.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/remote-workspace-online-source.test.ts -t 'discovers all
Git companions' --maxWorkers=1`: **1 passed; 86 not selected**. This
uses the real source owner and retains finalized recovery holds, company
isolation, and a fresh obligation between inspections.
- `pnpm --filter @paperclipai/db typecheck`, migration numbering/safety
checks, and `git diff --check`: **passed**.
- One small local SQL comparison used the captured baseline selector,
the changed selector, 3,012 synthetic rows, about 37.8 MB of table
storage, and the new expression index. Ordered results were identical.
One client timing changed from **8.70 ms to 0.86 ms**. This is one local
warm-cache comparison, not a server latency claim. Hosted improvement is
**unverified**.
- Initial setup attempts failed before the affected test ran: registry
access, local tsx IPC, and a missing SDK build. The canonical Node
24.22.2/pnpm 9.15.4 setup and SDK prerequisite resolved these setup
failures. The local SQL timing client used Node 26.4.0; the focused
application tests used Node 24.22.2.
- [The first CI
run](https://github.com/paperclipai/paperclip/actions/runs/42705129724/job/129460702073)
failed the inherited case-history API first-page search assertion. The
same failure was reproduced on the unchanged base: the route ranked
sixth for “case for task.” The isolated route-summary correction from
#20370 passed the same three assertions. The search algorithm and
assertions were not changed.
- No public command changes require a documentation update. The schema
and migration record the added index. No hosted DDL or deployment was
performed for this PR.

- Final head `c50101dabf46ade42b545d2051d724d93f413ccd`: [full
CI](https://github.com/paperclipai/paperclip/actions/runs/42707623920)
passed with 60 successful checks and three configured skips. [Fresh
Greptile
review](https://github.com/paperclipai/paperclip/pull/20372#issuecomment-6752305627)
is 5/5 on this head with no open threads.

## Risks

- The index adds storage and write cost when a run profile changes.
- A concurrent build reads the table and can wait for old transactions.
The existing migration path runs it outside a transaction and can repair
an interrupted invalid index.
- Existing nested-operator lookups can use the index. The companion
selector needs this code change to use it. Other JSON path queries are
unchanged.
- The small local result does not establish end-to-end import speed or
PostgreSQL 17 hosted latency. Fresh deployed measurement remains
necessary.

## Model Used

OpenAI Codex, GPT-6 family. The exact runtime model ID and context
window are not exposed in this session. Used reasoning, code editing,
shell tools, and local PostgreSQL tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:17:13 -06:00
DottaandClaude Opus 5.5 d2fe85e57c fix(secrets): delete a secret that a binding proposal referenced (#20193)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents can propose a secret binding. A person approves or rejects it
on a review card.
> - Each binding proposal row refers to one source: a secret or a
pending secret proposal. A check constraint enforces this.
> - The secret reference used `ON DELETE SET NULL`. When a person
deletes the secret, Postgres sets the reference to null. The check
constraint then rejects the row.
> - So a person cannot delete any secret that a binding proposal ever
referenced. The API returns 500.
> - This pull request deletes the proposal rows together with the
secret, as the other source reference already does. Before the delete,
it rejects pending proposals so their cards close with a reason.
> - The benefit is that secret deletion works again and no review card
points at a removed proposal.

## Linked Issues or Issue Description

Refs #15644 (related: block secret deletion while bindings still exist).

**What happened**
`DELETE /api/secrets/:id` returned 500 for a secret that a binding
proposal referenced. The server log showed `new row for relation
"company_secret_proposals" violates check constraint
"company_secret_proposals_shape_check"`. This happened for pending,
rejected and approved proposals. It also happened after the person
removed the agent's binding.

**Expected behavior**
The person can delete the secret. Proposals that referenced it go with
it. A pending review card closes as rejected and tells why.

**Steps to reproduce**
1. An agent proposes a binding for an existing company secret (`POST
/api/agents/me/secret-proposals` with `secretId`).
2. Approve or reject the proposal, or leave it pending.
3. Delete the secret with `DELETE /api/secrets/:id`.
4. The response is 500 and the secret stays.

## What Changed

- `packages/db`: `company_secret_proposals.secret_id` now uses `ON
DELETE CASCADE`, the same as `secret_proposal_id`. Migration
`0452_secret_proposal_secret_cascade.sql` replaces the foreign key
(`DROP CONSTRAINT IF EXISTS`, then `ADD CONSTRAINT`).
- `server/src/services/secrets.ts`: `remove` accepts an optional
`beforeDelete` hook. The hook runs after the secret is marked deleted
and before its row is removed.
- `server/src/services/secret-proposals.ts`: new
`rejectPendingBindingsForSecret`. It rejects pending binding proposals
for a secret with a reason and returns them. This closes their review
cards and writes the normal activity entries. `createBinding` now holds
the secret row `FOR SHARE` in its insert transaction and refuses a
deleted secret. So no new pending proposal can start after the delete
marks the secret deleted.
- `server/src/routes/secrets.ts`: the delete route rejects pending
proposals in `beforeDelete`. For each one it sends the normal rejection
notice and wake (`notifySecretProposalResolution`), so the proposing
agent continues.
- `server/src/__tests__/secret-proposals-routes.test.ts`: a regression
test deletes a secret with a pending, rejected and approved binding
proposal through the route. It checks the secret and proposal are gone,
the card status is correct, and only the pending case sends one
rejection notice and one wake. A second test checks that a binding
proposal for a secret that the delete already marked deleted fails with
404.

## Verification

- Red first: the new test failed on `origin/master` with the shape-check
violation for all three cases.
- `cd server && vitest run --config vitest.config.ts
src/__tests__/secret-proposals-routes.test.ts`: 47 passed.
- `vitest run src/__tests__/chat-run-milestone-governed-work.test.ts
src/__tests__/agent-secrets-routes.test.ts
src/__tests__/secrets-service.test.ts`: 112 passed.
- `tsc --noEmit` for `server` and `packages/db`: pass. `pnpm --filter
@paperclipai/db run check:migrations`: pass.
- Found by the `secrets.secret-proposal` chat evaluation. Its cleanup
deletes the fixture secret and failed on this error in runs 42563093640
and 42571127911.

## Risks

- The migration only replaces one foreign key. It does not rewrite rows.
Existing rows already have a non-null `secret_id` or
`secret_proposal_id`, so nothing else changes.
- Deleting a secret now also deletes its binding proposal rows. The
activity log keeps the proposal history (`secret.proposal.*` and
`secret.deleted`). Secret-kind proposal rows that created the secret
keep `created_secret_id ON DELETE SET NULL`, so their history stays.
- Other paths that delete secrets (company deletion, user secret
cleanup) now cascade instead of failing on the same constraint. They do
not reject pending cards first.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:54:07 -06:00
DottaandClaude Opus 5.5 5a51a7685d fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop cancels the scope's running work; the board
resolves it with Raise budget and resume
> - In the restart-chaos H1 scenario, the cancelled tasks did not
continue after resume and each needed a manual Retry
> - Resume is the board's explicit decision to continue, so the work it
stopped should continue once, as if the board clicked Retry
> - This pull request files that Retry automatically, keyed by the
cancelled run, once the run's stop has settled
> - The benefit is that a budget raise restores the work without manual
Retries, without repeating completed work, and without charging the
run's attempt budget

## Linked Issues or Issue Description

Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101
(superseded: split into #20143 and this PR)

**What happened?**
Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802
(H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run
(enforcement stop) and CHA-1's run (agent lifecycle pause from the
budget hold) stayed cancelled until someone clicked Retry.

**Expected behavior**
Resuming continues each task its budget stopped, exactly once, with no
attempt charged, an activity entry, and no repeat of work that already
continued. A run a person stopped stays stopped.

**Steps to reproduce**
Set a budget below spend while a task runs; let the stop cancel the run;
resolve with Raise budget and resume. Before this change, no new run is
queued.

## What Changed

- New table `budget_resume_claims` (migration 0451). One row per
cancelled run, outside the run's result JSON, so a Stop that settles
during the resume cannot erase it.
- `markBudgetCancelledWorkForResume` (called by `resolveIncident` before
the incident closes): claims every task run its budget stopped since the
incident opened. That covers runs budget enforcement fenced, including
stops still in flight, and runs the agent lifecycle pause stopped for
this budget (`pauseStop`, carried from the lifecycle operation's
`pauseCause` so a resume before the Stop keeps the link). Runs a person
stopped are excluded. Two indexed passes led by `(company_id, status)`.
- `deliverBudgetResumeContinuations`: once a claimed run's stop settles,
files the resolving board member's Retry (`requestTaskRetry`,
idempotency key `budget-resume:<run>`, with the member's session source
and the chat Retry callback). The Retry path checks, under the task
lock, that the run is still the task's latest and the task is open, so
continued or completed work is never repeated. Claim settlement and its
activity entry commit together; retryable refusals and errors retry.
- Delivery runs in the always-on budget-enforcement worker after each
pass's Stop delivery. Its schedule includes deliverable claims and the
earliest stale-claim expiry. The execution-continuation worker is a
backstop.
- `agent.paused` / `agent.resumed` logging is in #20113.
`budget.cancelled_work_resume_requested` (incident) and
`budget.cancelled_work_continued` / `_not_continued` (task) are recorded
here.

## Verification

- New `budget-resume-continuation.test.ts` (Postgres, real
`resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry`
into a queued run):
- each task continues once, including a blocked task; repeat delivery or
resolve saves nothing new;
- a stop still in flight at resume, whose terminal write replaces the
result JSON, is still continued;
  - a 30-task scope is fully continued;
- a lifecycle-pause stop caused by the budget is continued, and one
caused by a manual pause is not;
- continued work, done tasks, a run that finished before its stop, and
person-stopped runs (including ones enforcement also fenced) are not
continued.
- `lifecycle.test.ts`: the pause records its cause on the operation and
on stopped runs, including when the budget is resumed before the pause's
Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery,
and a wake at claim expiry.
- 287 tests across the affected suites passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun (Phase 3 runs it after merge).

## Risks

- Resume now continues budget-stopped tasks automatically. A board that
wants a task stopped can Stop it again; person-stopped runs are never
continued.
- A new table and one index (migration 0451).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:56:28 -06:00
DottaandClaude Opus 5.5 c49ab6b9ee feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task delegates to children and waits on blockers. Its agent
learns of progress only through wakes
> - Today only completion (`issue_children_completed`) and resolution
(`issue_blockers_resolved`) wake a waiting issue
> - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on
a recovery action that needs the board, and its parent sat silently
blocked for 50+ minutes
> - This pull request wakes the waiting agent once per such stall, tells
it what it can do, and shows the stall on the parent's page with the
child's Retry
> - The benefit is that a stuck child or blocker no longer silently
strands its parent

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan; restart chaos campaign). Restart
chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude).

**Problem**

From the kept database:
- CHA-8 `blocks` CHA-5.
- CHA-8's board-owned recovery action (`native_session_retry_exhausted`)
was created at 18:44:02.909 and resolved with outcome `blocked` at
18:44:08, when CHA-8 moved to `blocked` for the board.
- CHA-5's only wake in the whole wave was its `issue_assigned` at
18:20:48. Its agent never heard about CHA-8.

**Expected behavior**
- When a child or blocker stalls on the board, the waiting agent is
woken once. It can then re-delegate, escalate, or block with a reason.
- The board sees the stall on the waiting task, with the child's Retry.

## What Changed

- **Shared:**
- `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`),
`ISSUE_BLOCKER_STALL_RELATIONS`, and the types
`IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`;
  - `Issue.stalledDependencies`.
- **Server, `services/recovery/blocker-stall-wakeups.ts`:**
- A stall is an issue that is not done or cancelled, whose latest
recovery action needs the board. That means either open
(`active`/`escalated`) and board-owned or escalated, or board-owned and
resolved as `blocked` while the issue is still `blocked`. The second
case is what t05 needed: recovery resolves its action that way when it
hands the issue to the board.
- Waiters are the parent, and every issue the stalled one `blocks`, when
they are open, visible and agent-assigned.
- Each waiter gets one wake per stall. The idempotency key
`issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked
through the indexed `(company_id, payload->>'issueId')` path. A later
stall has a new action id, so it wakes again.
  - Each wake writes activity `issue.blocker_stalled_wake`.
- The pass runs from the recovery-action sweep
(`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep
(`reconcileResolvedDependencyWakeBackstop`, which visits each blocked
issue). No new worker.
- A waiter's agent that cannot read the stalled issue (`issue:read`,
which honors private visibility) is woken without its title, identifier
or recovery details; the activity entry withholds them too.
- **Database:** migration `0450` adds the partial unique index
`agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id,
idempotency_key)` for `issue_blocker_stalled:` keys in delivered or
pending statuses, so two concurrent sweeps cannot both deliver a stall.
It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed
in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`).
- **Queue:** `allowsDependencyBlockedWake` (in
`run-dispatch/domain/wake-context.ts`) lets a stall wake run while the
waiter's blockers are unresolved, at both dependency gates. Without it
the wake was parked until the stalled blocker resolved, which is never.
Pause holds are unchanged.
- **Agent guidance:** the wake payload carries `blockerStall`, and
`renderPaperclipWakePrompt` adds a "Stalled dependency" section. It
names the task and its recovery action and the board's next action, then
the three choices: re-delegate, escalate with a comment, or block with a
reason naming the stalled task.
- **Other reason lists:** `issue_blocker_stalled` joins the
responsible-user wake reasons, the conversation execution-wake reasons
(it never drives a chat turn) and the wake diagnostic's known reasons.
- **Board surface:**
- `GET /issues/:id` returns `stalledDependencies`, readable issues only
(`readableIssueEdgeIds`).
- `StalledDependencyNotice` shows on the issue's chat notices, next to
`IssueBlockedNotice`: each stalled child or blocker, its board next
action, a link, and the child's own Retry (`useTaskRetry`), offered only
when the child's interruption allows it.

## Verification

Red/green:
- `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests).
  - On master: no `issue_blocker_stalled` wake is raised.
- With this PR: the parent is woken once per stall, and re-woken after a
new stall. A blocker's dependent is woken.
- The t05 shape (an action resolved as `blocked`) wakes once, and
nothing more after a Retry.
- No wake for an agent-owned action, a done, unassigned or hidden
waiter, or an action resolved another way.
  - `listStalledDependencies` lists children and blockers.
- `heartbeat-dependency-scheduling.test.ts` "admits an
issue_blocker_stalled wake while a blocker is unresolved". On master the
wake was parked (`null`); once admitted, the run-start gate cancelled
it. With this PR it runs with `dependencyBlockedInteraction`. The file:
14 passed.
- `packages/adapter-utils` `server-utils.test.ts`: the new prompt test
failed on master (no `blockerStall`). The file: 141 passed.
- `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed.
`IssueChatThread.test.tsx`: 109 passed.

Regression and static checks:
- Nearby server suites (recovery durable wait, dependency wake routes,
recovery actions, run-preparation): 149 passed.
- `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`.
- `pnpm check:token-gates` is clean.

Review round 1 (Greptile, Superagent), each with a regression:
- A backlog waiter is never woken. The waiter's status and assignee are
guarded at enqueue.
- One stall per run: no coalescing, so a merged run cannot keep only the
last stall's context.
- Only a delivered or pending wake ends a stall. A refused one
(wake-on-demand off) is retried after 10 minutes, and an agent that
takes no wakes is skipped without writing a row.
- Company-wide passes resume after the last stall a capped pass reached.
The per-issue sweep also checks the issue's own children and blockers,
which covers an unassigned stalled blocker.
- The dependency gates no longer trust the reason string.
`verifyBlockerStallWake` requires a system request, the stalled issue's
current stall action, and a real child or blocker edge. An
agent-requested wake with the same context is parked (tested).
- The notice shows a refused Retry's reason, and a lost response as
"Taking longer than usual…", with Retry still available (tested).

Review round 2 (Greptile), each with a regression:
- Race: two passes claiming one stall leave one delivered row (unique
index; the loser's insert is skipped).
- The company recovery sweep selects companies whose only stall is a
resolved-`blocked` board action, so an unassigned stalled blocker with a
todo waiter is reached.
- Private stalled issue: the waiter's wake and activity carry no title,
identifier or recovery details.
- `GET /issues/:id` always returns `stalledDependencies`, so a cleared
stall replaces the cached list.
- The notice shows a failed child read with a Reload button (tested).
- `pnpm --filter @paperclipai/db check:migrations` passes.

Review round 3 (Greptile), each with a regression:
- A stall wake that fails to enqueue is retried: the recovery-action
pass reports `blockerStallWakeFailures` and the dependency pass counts
it in `enqueueFailed`, so the worker throws and retries at its backoff
(`issue-recovery-work-lifecycle.test.ts`).
- The notice drops a stall once the child's refreshed status has moved
on (`StalledDependencyNotice.test.tsx`, 7 tests).

## Risks

- A deploy wakes the waiters of every issue that is stalled today, once
each. That is intended (they are stranded now), and it's bounded at 200
stalls per pass.
- A stall wake may run while blockers are unresolved, like a comment
wake. It is bounded to one per stall, and pause holds still apply.
- The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet
list the new reason. The wake prompt carries the guidance.
- One additive index, built concurrently; no persisted run state change.
Existing rows cannot violate it before deploy, since no
`issue_blocker_stalled:` keys exist yet.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 18:26:07 -06:00
DottaandClaude Opus 5.5 e8a3f1f6f2 feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, the control plane settles in-progress records
(leases, claims, intents) only once it proves their owner process is
gone
> - The proof compares host and PID space, so a stamp from a replaced
container or a rebooted machine stays unprovable forever, and its
records never settle
> - Self-hosted users on Docker, Kubernetes and Fly hit this on every
redeploy; a Railway deployment with an exclusive volume is covered by
`deployment_replaced` (#20026)
> - This pull request adds a fenced controller epoch: each boot holds a
database lease, and an expired boot is fenced so its records can be
proven gone
> - The benefit is that work stranded by a replaced container is
recovered, without keeping an idle database awake

## Linked Issues or Issue Description

Follow-up to #20026 and the Phase 2 durable-ownership data. Related:
#19965 (owner stamps), #19883 (process space). No duplicate PR was
found.

**What happened?**

A record stamped by a server process in another container reports
`different_process_space`. A dead PID elsewhere is never proof, so the
record stays `unprovable` and is never settled. Every redeploy of a
self-hosted container leaves such records behind.

**Expected behavior**

A record owned by a server boot that was replaced (stopped cleanly, or
lost its lease and was fenced) is proven gone and settled exactly once.
A live boot's records stay with it. An idle server issues no recurring
queries.

**Steps to reproduce**

1. Run Paperclip in a container and start work, so an environment lease
is stamped by that boot.
2. Replace the container. The new container has a new PID namespace.
3. The ownership reconciler reports `different_process_space`
(unprovable) for the old lease, forever.

**Paperclip version or commit**

`master` before this PR.

## What Changed

- New `controller_instances` table (migration 0449): boot id, epoch
(bigserial), process space, pid, a lease on the database clock,
`fenced_at` / `fenced_by_epoch`, `stopped_at`.
- `controller-instance-lease.ts`, registered with the shared
delivery-work coordinator (`controller-instance-lease` queue, runs
during drain):
- **Holds work → renews; idle → silent.** One upsert creates the row on
first use and extends the lease only if the boot holds work: a stamp
within the last 90 s, or a stamp whose ownership write may still be
pending (held until the server's tracked work settles, capped at 10
minutes; a renewal that sees some other held record does not clear it,
so a slow ownership write cannot be outlived), or an in-progress record
held by its boot id. Those records are active environment leases,
in-flight cleanup claims, sandbox creation intents, workspace finalize
attempts, agent lifecycle steps, native finalizer and cleanup leases
(`lease_holder` bound to the token), open native coordinator claims, and
running legacy runs. Renewal runs every 30 s while the boot holds work.
When it finds none, the lease lapses and the worker sleeps until the
next stamp (`onControllerStamp`).
- **No idle trailing statements.** When the server's last tracked work
settles, the worker re-checks holdings at once, at most every 5 s (the
rate limit also absorbs the worker's own run). So the last check happens
with the work, not 30 s later.
- **Quiesce, don't crash.** Each renewal has a 10 s bound. A failed or
timed-out renewal applies an internal task drain, so no new work is
dispatched. A recovered renewal resumes. The boot exits only after 60 s
of failure (counted from the first failed attempt's start) while it
still holds work, or at once when it finds itself fenced.
- **Fence on demand, by a newer epoch.** For a stamp from another host
or PID space, the prover reads that boot's row:
    - `stopped` or `fenced`: gone.
    - live lease: alive.
- otherwise, the boot's last sign of life is the later of its lease
expiry and the record's last write plus the TTL. Within 30 s of that,
the owner stays `alive` (`controller_lease_grace`), so the settle watch
keeps rechecking. After that, the prover fences the boot, but only with
a lease row of its own (created on demand) and only an older epoch.
  - A graceful shutdown records `stopped_at`.
- A reconciliation pass that settles nothing (`record: false` or
`settle: false`) never fences and never creates the boot's row.
- `owner-proof.ts`: these proofs apply only to server-held records. A
stamp with a recorded process group stays unprovable, because agent
processes can outlive their server. The reconciler passes each record's
`updatedAt` to the proof.
- `task-admission.ts`:
- the quiesce drain never replaces an operator's drain, and has no idle
owner, so HTTP stays open;
- a `subscribeIdleWorkSettled` hook fires when the last tracked work
finishes.
- Off on a Railway deployment with an exclusive volume (it has its own
replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on|off`
overrides this. A stateless Railway service keeps the lease. A server
without its own lease row never fences.
- `maintenance-server-routes.test.ts`: in the post-run quiet window, the
lease may check once more that the run's records were released. The test
now excludes those statements, then asserts that the lease issues
nothing in a further window.
- Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the
two known gaps.

## Verification

- `controller-instance-lease.test.ts` (15 cases):
- a replaced container with all Railway variables empty is fenced and
settled exactly once over two passes;
  - a live lease is alive;
- an owner in grace, and work written after an idle lapse, stay alive
and watched;
  - an older epoch never fences a newer one;
- no fencing without its own lease row (off or unregistered), and an
idle prover creates its row on demand;
  - a stopped boot is gone;
  - process-group stamps stay unprovable;
- idle issues zero statements, and one upsert renews only while records
are held;
  - **a 45 s outage:** quiesced, no exit, resumes;
- **a renewal that never answers:** bounded, and the boot exits on time;
- **a 70 s outage:** exits at 60–65 s, and the successor recovers the
work exactly once;
  - fenced: stops;
  - the settle re-check;
- graceful stop; off on a volume-exclusive Railway deployment, on for a
stateless one, and the on/off override;
  - a running legacy run keeps the lease live.
- Red, with the proof's instance check removed: the replaced-container,
live, stopped and successor cases fail.
- Wider: task-drain, idle-sleep, idle-local-work, owner-proof and
ownership suites, plus the maintenance quiet-window tests. Server
low-memory typecheck is clean.

## Risks

- **Clock skew.** The proof compares a record's `updatedAt` (often the
writer's clock) with the lease (database clock). The 30 s grace absorbs
ordinary skew.
- **A new exit path exists.** A server whose database is unreachable for
more than 60 s while it holds work now exits, after first quiescing.
This is the intended safety backstop.
- **Bounded trailing activity.** A lease that holds work issues one
statement every 30 s, plus at most one check when its work settles. An
idle server issues none.
- **Boots from before this release** have no row, and their records stay
unprovable. The board "mark owner gone" action for those follows in a
separate PR.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:06:32 -06:00
DottaandClaude Opus 5.5 96badeaa42 fix(lifecycle): never repeat a passed harness verification; settle a dead verifier's lease at once (#20030)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A new or reconfigured agent runs a harness verification (a hello
probe in a test sandbox) before it is ready, and its runs wait until it
passes
> - In the final Daytona restart chaos wave, a Codex agent's runs waited
about 417 s for their first sandbox, while boot took only about 32 s
> - Each control-plane restart killed the verifier. The next process
waited out the 120 s lifecycle lease and then booted a fresh sandbox.
Nothing recorded a configuration that had already passed.
> - This pull request records a passed verification by what makes it
valid and reuses it, and settles a dead verifier's lease on the shared
ownership path at once
> - The benefit is that a restart never repeats a passed verification,
and never leaves runs waiting out a dead process's lease

## Linked Issues or Issue Description

**What happened?**
In the final Daytona chaos wave, the Codex agent was created at 09:55:46
UTC and its three runs were queued. Verification ran four times, in four
sandboxes, because faults K1, K4 and K2 each killed the verifying
server. After K2 the server was ready at 10:00:20, but verification
restarted only at 10:02:13: it waited out the 120 s lifecycle lease. The
first run sandboxes were created at 10:02:43, 417 s after launch.

**Steps to reproduce**
1. Create a Codex agent and assign it a task at once, so the task waits
for harness verification.
2. Kill the control plane (SIGKILL) while the agent is verifying.
3. The restarted server waits out the dead verifier's 120 s lifecycle
lease, then verifies again in a new test target. The task's first token
arrives about 130 s after the fault.

**Expected behavior**
- A passed verification of the same effective configuration in the same
runner image is never repeated.
- A verification is never restarted from scratch while its holder
process is alive. When the holder is proven gone, its lease is settled
at once.
- Runs gate only on a configuration that has never passed.

**Root cause**
`verify()` (`server/src/services/agent-harness-verification.ts`) leased
a fresh sandbox on every attempt and recorded nothing on a pass. The
lifecycle lease (`modules/agent-lifecycle/adapters/postgres.ts`) named
no holder process, so a restart could only wait for it to expire.

## What Changed

- Restart chaos TTFT driver: a `verify` phase, reported in its own rows,
and a `--verify-fault K1|K2|K6` option. The fault fires 1 s after
`verifying` is observed, so it interrupts a held lease. The seed keeps
each agent's creation body, so the driver can hire an identical agent.

- **Persisted pass:** new columns `agents.harness_verified_digest` and
`harness_verified_at` (migration 0448). The digest is an HMAC, keyed
with the instance's decision-signing secret, over:
- the fully resolved effective configuration: every value the test used,
including `cwd`, resolved credentials, and agent-over-environment
precedence;
- the selected AI account and its credential generation (connection and
grant ids and `updatedAt`, never the credential);
  - the environment and the runner image.

The key means the stored value cannot confirm a guessed credential
offline. For a sandbox, the runner image is an immutable image digest or
a qualified Daytona snapshot ID (`sandboxBinding`, now exported). For
local runs it is the server version. A mutable image or snapshot name,
or a local environment redirected to the managed sandbox, never reuses a
pass.
- **Server-only:** `normalizeAgentBaseRow` and
`redactAgentRowForResponse` strip the digest and its timestamp from
agent API responses.
- `verify()` still runs every account, policy and secret check, then
returns complete on a matching digest without booting a test target. It
records the digest on a pass before the lifecycle step completes, so a
restart in between reuses it.
- **Lease holder:** the lifecycle `claim()` stamps the claiming
controller process on the lease (`lifecycleOperation.leaseHolder`,
supplied through the lifecycle effects port) and clears the stamp with
the lease. The compare-and-swap release lives in the lifecycle module
(`releaseAgentLifecycleLeaseOfHolderInTransaction`).
- **Shared ownership path:** a new `agent_lifecycle_lease` kind with
`settleAgentLifecycleLease`.
- When the reconciler proves the holder gone, settle clears exactly that
lease (compare-and-swap on the holder), notifies lifecycle work, and
logs `agent.lifecycle_lease_reconciled`.
- A live holder is never taken over. The lifecycle state, operation and
progress are untouched.

## Verification

- `restart-verification.test.ts` (embedded Postgres, real lifecycle
store and worker, real reconciler pass):
- A verification held by a dead process is claimed and completed at once
after the startup pass, with the agent ready in under 10 s instead of
waiting out the 120 s lease. It fails without the new kind.
  - A live holder is never taken over.
  - The claim stamps the holder, and `defer` clears it.
- `agent-harness-verification-preflight.test.ts`:
- A second `verify()` of the same configuration allocates no target. A
revoked account still fails before reuse. A changed model, `cwd` or
literal credential tests again. It fails without the reuse.
  - A failed verification records nothing.
- Ownership report, all lifecycle module tests, remote, provider-API and
unit harness verification tests, and crash-point call sites: 13 files,
220 tests pass.
- Targeted local run: the TTFT driver's new `verify` phase (`--phases
verify --verify-fault K1`, Codex, 2 samples) hires a new agent, assigns
it a task at once, kills the control plane (K1) while it verifies, and
times fault to first agent token:
- Before (master server code): 129.8 s and 129.8 s. The restarted server
waited out the dead verifier's 120 s lease. Run-log TTFT from run
creation was only 4.1 s.
- After (this branch): 15.8 s and 17.2 s. The startup reconciler pass
settled `agent_lifecycle_lease` and verification re-ran at once.

## Risks

- **A stale pass:** the digest covers every resolved value, the selected
account generation, the environment and an immutable runner image, so
any change tests again. Rotating the instance signing secret invalidates
every digest, so each agent verifies once more.
- **Migration:** two nullable columns, no backfill. Existing agents
verify once more, then reuse.
- **Settle:** it runs only on a `gone` proof, compare-and-swap on the
exact holder. It never changes the lifecycle state.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:10:40 -06:00
DottaandClaude Opus 5.5 576ba29fff feat(recovery): settle the legacy controller lease on the shared ownership path (#19953)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Legacy (non-native) runs hold a controller lease on `heartbeat_runs`
while the server prepares and dispatches the adapter. The shared
ownership reconciler now settles proven-gone owners (Phase 2 of the
durable ownership plan)
> - The lease stamped only a boot id. The reconciler could never prove
its owner gone (`missing_identity` in the Phase 2 data), and recovery
released it by age: lease expiry (60 s), then a 5-minute age guard on
dispatched runs
> - A hard-killed server's legacy run therefore stayed `running` for up
to 6 minutes after the next boot
> - This pull request records the controller process with the claim,
registers a compare-and-swap settle, and makes live-run recovery use the
shared prover instead of the age guard when the controller is proven
gone
> - The benefit is that a dead controller's run is settled at once,
while a live, unprovable, or previous-release controller keeps today's
behavior

## Linked Issues or Issue Description

Refs #19933 (the settle hook, merged). Stacked on #19950 (migration
0446, registry) and targets its branch until it merges; then it
retargets to master.

**What happened?**
After a hard kill, a dispatched legacy run stayed `running` until its
controller lease expired and its last update was 5 minutes old.

**What did you expect?**
The next boot proves the old controller gone and fails the run as a lost
process at once.

**Steps to reproduce**
1. Start a legacy (non-native) run and let it dispatch the adapter.
2. Kill the server (SIGKILL).
3. Start the server. The run stays `running` for up to 6 minutes.

## What Changed

- `heartbeat_runs.controller_process` (jsonb, migration 0447): `{bootId,
pid, hostHash, processStartedAt}` of the claiming server process. It
names the controller only while its `bootId` equals
`controller_boot_id`. Revocation swaps the boot id, so a revoked lease
never keeps it.
- `legacyControllerClaim` writes it in the same UPDATE (from
`currentControllerProcessStamp()`, no I/O).
- `legacy-controller-lease.ts`: `legacyControllerOwnerStamp`,
`legacyControllerProvenGone` (calls `proveOwnerGone`), and
`settleLegacyControllerLease` (compare-and-swap expiry of the proven
controller's outstanding lease after it wakes live-run recovery).
- `ownership-registry.ts`: `legacy_controller_lease` registers `settle`
and `listForRuns`. The owner is the bound process; other rows are
boot-id fences. The listing covers outstanding leases only.
- `heartbeat/recovery.ts` (3 small hunks): a controller of another boot
that the shared prover proves gone has its live lease settled through
`settleOwnableKind` and skips the age guard. The local child-process
check after it is unchanged, so a live child still keeps the run
running.
- Migration 0447 builds the partial index with `CREATE INDEX
CONCURRENTLY` (heartbeat_runs is a known-large table) and is added to
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`.

Bespoke code deleted: none. The kind had no liveness proof; the age
guard stays for controllers that are not proven gone.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts`: 33 passed.
- `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed.
- Red with `recovery.ts` and `ownership-registry.ts` from the base
commit (tests, schema and lease module from this branch): 4 failed, 29
passed. Red: the crash-point reaper test, the expired-lease age-guard
test, the reconciler settle test, the exact-controller CAS test. The
tests were written after the code.
- New tests:
  - the claim stamps this process, and a changed boot id unbinds it;
- crash point: a controller dispatched the adapter and was killed with
its lease live; the startup reaper (default 5-minute age guard) fails
the run as `process_lost` at once;
  - a dead controller's expired lease skips the age guard;
- counter-cases keep the run and lease through the reaper and a
reconciler pass: a live earlier-boot PID, a dead PID in another PID
space, and a previous-release row with no `controller_process`; an
expired lease of a live controller stays behind the age guard;
- a read-only pass (`record: false`) never settles; a recording pass
expires the lease and leaves the run to live-run recovery;
  - settle refuses a controller that changed after the proof.
- Not run locally: typecheck (CI runs it) and the wide suites.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved
by the previous release have no `controller_process`. They stay boot-id
fences with today's release (expiry, revocation, age guard). A rollback
ignores the column.
- Unprovable controllers (another host or PID space, unknown space, a
live PID with an unreadable start time) never release early. Their
handoff stays the lease expiry plus revocation, and the age guard. A
different host is never proof.
- Behavior change: a dispatched run whose controller is proven gone is
failed as `process_lost` without the 5-minute wait. The retry decision
is unchanged.
- The settle expires the lease only. Live-run recovery still settles the
run under its own checks. Accounting and workspace obligations are not
touched.
- The index build is concurrent. A cancelled build is retried by the
migrator's invalid-index handling.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:27:54 -06:00
DottaandClaude Opus 5.5 bf3a067076 feat(recovery): settle the finalizer and cleanup leases on the shared ownership path (#19950)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep their ownership in `native_run_finalizations`, and
the shared ownership reconciler now settles proven-gone owners (Phase 2
of the durable ownership plan)
> - The finalizer lease (`native-finalizer:*`, 5 min) and the
retained-session cleanup lease (`native-cleanup:*`, 60 s) recorded only
a random token. They named no holder process
> - A server killed while it finalized or cleaned up kept the run
waiting for the whole lease. The lease was released by age alone. A
graceful shutdown also keeps an in-progress finalizer lease for the next
boot to wait out
> - This pull request records the holder process with each claim and
moves both kinds onto the shared prover and a compare-and-swap settle
> - The benefit is that a dead holder's run continues in the first pass
after boot, and a live or unprovable holder is never released

## Linked Issues or Issue Description

Refs #19933 (the settle hook and the first moved kind, merged).

**What happened?**
A finalizer or cleanup lease whose server process died stayed held until
the lease expired (5 min or 60 s). The run stayed `running` and its
follow-up work waited.

**What did you expect?**
The next boot proves the dead holder gone and frees the lease at once. A
live holder, or one that cannot be proven gone, keeps its lease.

**Steps to reproduce**
1. Accept a native result and let the finalizer claim it.
2. Kill the server (SIGKILL) before the finalizer commits.
3. Start the server. The run stays `running` for up to 5 minutes.

## What Changed

- `native_run_finalizations.lease_holder` (jsonb, migration 0446):
`{token, bootId, pid, hostHash, processStartedAt}` of the process that
took a token-only lease. It names the holder only while `token` equals
`lease_owner`, so renewals keep it and a later lease never inherits it.
The executor's `controller_*` columns are not changed.
- The finalizer claim (`native-run-finalizer.ts`) and the cleanup claim
(`native-session-executor.ts`) write it in the same UPDATE as the lease,
from `currentControllerProcessStamp()` (no I/O).
- New `native-token-lease-settlement.ts`: compare-and-swap settle that
expires exactly the proven-gone holder's outstanding lease, after it
notifies the finalization or cleanup work that the expiry frees.
- Both claims also wake the ownership worker queue in their transaction,
so a quiet worker watches the new holder (Greptile round 1).
- `ownership-registry.ts`: `native_finalizer_lease` and
`native_cleanup_lease` register `settle`. Their owner is the holder
process; rows without a holder keep the token fence. Listings cover
outstanding leases only.
- `heartbeat/recovery.ts`: restart recovery settles both kinds before it
finalizes and schedules cleanup. A failure is logged and the reconciler
pass retries it.
- Two sparse partial indexes on `lease_expires_at` for the two listings,
so the settle watch re-check stays indexed.

Bespoke code deleted: none. These kinds had no liveness check; their
only release was lease age (the takeover in the claim), which stays as
the fence for unprovable holders.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts -t
"finalizer|cleanup|holder|exact lease"`: 12 passed (rerun after the
rebase on master). The crash-point test also asserts the ownership wake
for its run; it fails without the signal.
- `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed.
- Red on the base branch (server code at `bea3bfb20b`, tests and schema
from this branch): 7 new tests fail, 5 pass (the 5 are existing tests).
Red: the crash-point restart test, the two reconciler settle tests, the
three live/unprovable counter-cases (owner was a token fence), and the
exact-lease CAS test. The tests were written after the code.
- New tests:
- crash point: a finalizer claim was written and its holder killed
before commit; the next restart finalizes the run, and this boot's claim
stamps itself as holder;
- a dead finalizer or cleanup holder is settled by a recording pass,
never by a read-only pass (`record: false`); the accepted result and
phase are untouched;
- live counter-cases: a live earlier-boot PID (`pid_alive`), the current
boot (`current_process`), a dead PID in another PID space
(`different_process_space`, unprovable) keep the lease through a pass
and a restart;
- a holder written for another token, and a holderless row of the
previous release, stay token fences, also with
`PAPERCLIP_EPHEMERAL_SERVER=true`;
- settle refuses a lease taken over after the proof
(`owner_changed_since_proof`).
- Not run locally: typecheck (CI runs it) and the wide suites.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved
by the previous release have no `lease_holder`. They stay token fences
and the existing claim takes them over at expiry. Neither lease outlives
5 min, so this covers at most 5 min after the upgrade. A rollback
ignores the new column.
- Unprovable holders (other host or PID space, unknown space): no
operator action. The lease is short and self-expiring; at expiry the
next claim takes it over and the old holder's next token
compare-and-swap fails. That fence is the handoff. A different host is
never proof.
- The settle only expires a lease. Finalization and cleanup still run
under their own checks (cleanup keeps its maintenance-history proofs).
Settling ownership does not settle the accepted result, workspace
preservation, or accounting.
- Migration adds a nullable column and two partial indexes on a sparse
predicate. Low risk.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 22:16:34 -06:00