Commit Graph
4169 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 533d02ff1a perf(recovery): match scoped deadline entries only against scopes of their own company (#21467)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live-run recovery uses the shared scoped deadline worker
(`scoped-deadline-work.ts`). Each queue keeps its entries (one for each
running run) and the scopes that writes marked dirty (for example one
company and one run)
> - To schedule and to pick due work, the worker tested every entry
against every dirty scope. It also decoded each scope hint with
`JSON.parse` again inside those loops, and each reconstructed scope
walked all entries to drop vanished ones
> - With many runs and many dirty run scopes, this work grows with
entries × scopes on the event loop of the one server process that
controls all runs
> - This pull request lets an owner declare a partition key for its
scopes. Live-run recovery uses the company. The worker then tests an
entry only against the scopes of its own company, and decodes each hint
once
> - The benefit is that matching cost grows with entries + scopes, with
the same results

## Linked Issues or Issue Description

Refs #21444 (the lease renewal change from the same load model).

**What existing behavior does this improve?**
The scoped deadline worker in
`server/src/services/scoped-deadline-work.ts`, which live-run recovery
(`server/src/services/heartbeat/live-recovery-work.ts`) uses for its
seven queues.

**Subsystem affected**
Server: background work scheduling for live-run and resource recovery.

**Current behavior**
For 400 entries in 40 companies and 40 dirty run scopes, one schedule
computation and one pass call `matches` 27,830 times and decode each
hint many times.

**Proposed behavior**
The same case calls `matches` fewer than 3,200 times and decodes each
hint at most once. Owners that do not declare a key behave exactly as
before.

**Reason and benefit**
The plan for 1,000 concurrent agents needs per-run background work that
does not grow with the product of runs and changes. The work runs on the
main thread of the controlling process.

**Breaking changes**
None. `scopeKey` is optional.

## What Changed

- `scoped-deadline-work.ts`: new optional `scopeKey` option
(`entry(entry)` and `scope(scope)`). The contract: a scope with a
defined key matches only entries with the same key. A scope without a
key (for example `*`) is tested against every entry, as before.
- The worker indexes the dirty scopes by key, so `next()` and the due
filter test each entry only against the scopes of its key and the
unkeyed scopes.
- When a reconstructed keyed scope drops vanished entries, it visits
only the entries of its key (an index built once for each pass).
- Scope hints are decoded once and the decoded value is reused while the
hint is dirty or scanning.
- `live-recovery-work.ts`: declares the company as the key. Its
`matches` already requires the same company for every defined scope.

## Verification

- New test in `scoped-deadline-work.test.ts`: 400 entries in 40
companies, then 40 run-scoped invalidations. It requires fewer than
3,200 `matches` calls and at most 40 decodes, and the next deadline is
unchanged.
- Red first: with `origin/master` `scoped-deadline-work.ts`, the test
failed with 27,830 `matches` calls.
- `pnpm exec vitest run src/__tests__/scoped-deadline-work.test.ts`: 22
passed.
- The live-run recovery matrix (`live-recovery-work.test.ts`) is covered
by CI.

## Risks

- If an owner declares a key that its `matches` does not respect, the
worker can miss a match. Only live-run recovery declares a key, and its
`matches` returns false for any defined scope of another company.
- Low risk otherwise: owners without a key keep the old code path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:21 -06:00
DottaandClaude Opus 5.5 754b1f36d3 perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running native agent. Each run
holds a native execution lease that the process must renew before it
expires (60 seconds for a remote run, 20 minutes for a local run)
> - Each run renewed its lease on its own timer, in its own transaction,
on a private pool of two connections. At 1,000 remote runs that is about
50 transactions each second through two connections
> - When renewals queue behind each other, a renewal can miss its 60
second deadline, and the run is lost although the server is healthy
> - Each renewal also woke six live-run recovery queues for its run. The
recovery owner then re-checked a healthy run about every 20 seconds,
although a renewal only moves the deadline later
> - This pull request renews all leases of one process with one shared
timer. All leases that are due go into one transaction of fixed size,
the same model that the legacy controller lease already uses. A renewal
no longer wakes recovery, as #20924 did for the legacy lease
> - The benefit is that the renewal cost no longer grows with the number
of runs, and all safety properties of a single renewal stay the same

## Linked Issues or Issue Description

Refs #20924 (the same change for the legacy controller lease), #21242.

**What existing behavior does this improve?**
Native execution lease renewal in
`server/src/services/native-runtime/native-execution-lease.ts`. Each run
had one `setInterval` and one transaction per renewal: statement
timeout, row lock, recovery notification, update, commit.

**Subsystem affected**
Server: native runtime, controller lease renewal on the liveness
connection pool.

**Current behavior**
The renewal cost grows with the number of runs. A local load model with
1,000 remote runs sends 35.7 lease transactions and 212.8 lease
statements each second, and the event-loop utilization is 0.34. All of
them use a pool of two connections.

**Proposed behavior**
One timer for each database client renews all due leases in one
transaction for each 200 runs. With 1,000 remote runs the same model
sends 0.29 lease transactions and 1.7 lease statements each second, and
the event-loop utilization is 0.07.

**Reason and benefit**
The plan for 1,000 concurrent agents needs renewals that do not scale
with the run count (item 1d). A renewal that misses its deadline loses
the run.

**Breaking changes**
None. The `NativeExecutionLeaseRenewal` contract (`stop()`,
`remainingMs()`) and `renewNativeSessionExecutionLease` do not change.

## What Changed

- `native-execution-lease.ts`: a renewal (batched or single) no longer
calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline
later. Recovery already holds a deadline at the earlier expiry. At that
deadline it re-reads the row and moves to the new expiry. A lapsed lease
is still found at its expiry. This change does not touch the signals of
the writers that release, revoke or take over a lease, or that change
the run status. This also removes six notifications for each renewal,
which was 1,200 notifications for a batch of 200 runs during a deploy
overlap.
- `native-execution-lease.ts`: one shared schedule for each database
client replaces the timer of each run. When the first lease is due, all
leases that are due within a quarter of their own interval renew with
it. Thus the leases of a process converge on one batch for each
interval.
- A batch is one transaction on the liveness pool for each 200 runs:
`statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the
coordinator rows, and one `UPDATE … FROM (VALUES …)`.
- The batch keeps the guarantees of a single renewal:
- It locks each coordinator row before it reads the expiry, as before.
Thus an owner whose lease expired while another writer held the row is
never revived.
- It extends a row only for the exact owner: run, company, task, lease
owner, attempt, controller boot id, pid, process start, and a live
lease.
- The confirmed deadline is sampled after the locks and before the
update, as before.
- A run whose row was locked and not extended fails with
`native_session_lease_lost`. The other runs in the batch succeed.
- These rows do not renew in the batch. They renew alone through the
unchanged single-run path:
- A row that another writer holds (`SKIP LOCKED`). Thus one busy row
cannot hold the batch until the statement timeout.
- A row that the lock did not find (another owner, or no row). The
single path gives the same answer as before.
- A local lease that lapsed (host sleep). Only the single path may take
it back, under the run row lock.
- A batch that is cancelled (statement timeout `57014` or deadlock
`40P01`) extends nothing. Its runs retry after 1 second, and each run
keeps its own deadline timer. Before this change, a statement timeout
waited for the next 20 second tick.
- Any other batch failure, also an error thrown while the statement is
built, may come from one bad row. Each run of that batch then renews
alone at once through the single path, so a bad row can cost only its
own run. A run that fails alone while others in its batch renew keeps
renewing alone (out of later batches) until a single renewal succeeds.
During a database outage the single renewals also fail; those runs
return to the batch at their next interval, not after a 1 second retry,
so an outage causes at most one queued single transaction for each run
and interval.
- Runs that become due while a batch is in flight wait for that batch. A
batch that runs for more than 15 seconds no longer holds back the next
batch.
- A tick that passed while a renewal was in flight is skipped, as a
fixed interval skips it. Thus a slow database gets no extra load.
- The shared timer replaces the interval timer of each run. It exists
only while this process holds a lease, and it fires at the next renewal
deadline. With no held lease there is no timer and no SQL.
- New export `renewNativeExecutionLeasesNow(db)` renews all leases of
the process at once. Tests use it.

## Verification

- New PostgreSQL tests in `native-execution-lease-batch.test.ts`:
- 3 runs and 60 runs each renew in 1 transaction with the same number of
statements.
  - 450 runs renew in 3 transactions.
- In one batch, the runs whose lease was taken over, whose controller
boot changed, or whose lease expired fail with
`native_session_lease_lost`. Their rows do not change. The healthy runs
renew. A newer attempt under the same owner is not lost, as before.
- Interleaving: another transaction holds a coordinator row from before
its lease expires until after it expired. The healthy run in the same
batch renews at once. After the lock is released, the expired owner is
lost and its row stays expired.
- A takeover that commits before the batch: the batch extends nothing
and the run is lost.
- A local and a remote lease renew in one transaction, each for its own
lease length.
- A poison binding (an attempt outside the integer range) makes the
shared UPDATE fail with `22003`. The other 4 runs still renew in the
same tick, only the poison run is lost, and the next tick renews the 4
runs in 1 transaction again.
- An invalid date throws while the statement is built. The other runs
renew, the broken run is lost alone, and no lease stays busy (the next
call renews again in 1 transaction).
- New PostgreSQL test in `live-recovery-work.test.ts` with the real
registered recovery owners: another server's native run renews its
lease. No recovery pass runs after the renewal. At the old expiry
recovery re-reads the row, keeps the run running and moves its deadline
to the new expiry. When renewals stop, recovery acts on the run within
10 seconds after the end of the last lease.
- Red first: with a shim on `origin/master` that renews each lease
alone, the transaction test failed (`expected 3 to be 1`) and the
450-run test failed.
- The unit tests in `native-execution-lease.test.ts` now drive the
batch. New tests show that a non-cancelled batch failure renews the run
alone at once, and that an outage (batch and single renewal both fail)
does not retry every second and loses no run.
- `native-execution-lease-reacquire.test.ts`: the host-sleep tests now
go through the batch, which hands the lapsed local lease to the single
path.
- Load model (`startNativeSessionExecutionLeaseRenewal` for N remote
runs that start at random times over 20 seconds, 70 second window, local
PostgreSQL, machine load average about 50):

| Runs | Lease transactions/s before → after | Lease statements/s before
→ after | Event-loop utilization before → after | Smallest margin to the
60 s deadline | Lost |
| --- | --- | --- | --- | --- | --- |
| 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0
s | 0 → 0 |
| 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s
| 0 → 0 |

The window includes the 20 second ramp, so the steady rate before is
higher (one transaction each 20 seconds for each run, 50/s at 1,000
runs). After the change, one batch carries up to 200 runs.

## Risks

- **Behavior change: batch-wide error handling.** One shared transaction
now carries up to 200 runs, so one database error reaches all of them. A
cancelled batch only skips the extension and retries after 1 second. Any
other batch error sends each run to the single path at once, so one bad
row (a bad value, a cast error, an error while the statement is built)
costs only its own run. In a database outage, up to one single
transaction for each run waits on the two-connection pool at each
interval. Each run's own deadline is unchanged.
- **Behavior change: no recovery signal on renewal.** See below.
- During a deploy overlap the old server renews its runs one by one and
the new server renews in batches. Both use the same row lock and the
same conditions, so they do not conflict. Rows written by the old
version have the same columns.
- `SKIP LOCKED` sends a busy row to the single path, which waits for the
lock as before.
- **No recovery signal on renewal:** recovery of a healthy run of
another server now runs once for each lease period at the old expiry,
not after each renewal. A dead run is found at the same time as before:
at the expiry of its last lease.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:16 -06:00
DottaandClaude Opus 5.5 969cf246ae perf(run-events): keep each native source's cursor on the run row (#21435)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Each native source event append counted all retained events of its
source (`count(*)`, `max(source_seq)`) to compute the acknowledgement
cursor
> - So the cost of each append grew with the length of the run, and long
runs made every text delta slower
> - This pull request keeps the gap-free cursor of each source on the
run row and moves it in the same UPDATE that reserves the event sequence
> - The benefit is that an in-order append stops reading other events,
so its cost does not grow with run length

## Linked Issues or Issue Description

Refs #21131 (earlier reduction of statements per event). Item 1b of the
agent scaling plan in #21418.

**What existing behavior does this improve?**
The append of a native source event (`appendNativeSourceEvent` in
`server/src/services/heartbeat-run-events.ts`). It ran a `retained` CTE
that counted every event of the source on every append.

**Subsystem affected**
Server run event ingest (`heartbeat_run_events`, `heartbeat_runs`).

**Current behavior**
On a run that already holds 50,000 events, one in-order append takes
about 10–13 ms (p50) on a local PostgreSQL. On a run with 400 events it
takes about 3.3 ms.

**Proposed behavior**
The append reads the stored cursor in the run lock statement that it
already sends. An in-order event moves the cursor and does not count
events. On the same 50,000-event run an append takes about 1.7–3.3 ms
(p50).

**Reason and benefit**
Text deltas are about 85% of the server's statements in the load model.
Their cost must not grow with run length.

**Breaking changes**
None. The new column is nullable. The previous release ignores it.

## What Changed

- Add the nullable column `heartbeat_runs.native_source_cursors`
(`jsonb`, `{"<source instance id>": <seq>}`). Migration `0479` is `ADD
COLUMN` only and classifies as `online_compatible`.
- The run lock read of a native append also reads the cursor of its
source.
- When the event sequence is at most cursor + 1, the append statement
has no `retained` CTE. It moves the cursor in the UPDATE that reserves
the sequence, and the receipt uses the cursor.
- Every other case (no cursor, a gap, or a cursor behind the stored
events) runs the earlier recount. In the same UPDATE it stores the
cursor when the source has no gap, and removes it when the source has a
gap. So the event that fills a gap recounts.
- The cursor is a lower bound. A writer that does not keep it (the
previous release during a deploy overlap, or
`NativeRunCoordinatorStore`) can only make it lag. The next out-of-order
event then recounts and corrects it.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts
src/__tests__/heartbeat-run-event-sequencing.test.ts` passes (19 tests).
- Red/green: before the change, 3 tests failed: the two new tests and
the updated lock-read statement test. After the change, all pass.
- New: "keeps the source cursor on the run so an in-order append does
not count the run's events" checks that the append statement has no
`count(` and that a replay returns the cursor.
- New: "recounts a source whose run has no cursor or a cursor behind its
stored events" covers rows saved before the change, rows from a writer
that does not keep the cursor, a gap, and the event that fills the gap.
- Focused related tests: `paperclip-control-plane-port.test.ts`,
`native-observer-replay`, `native-event-boundary-golden`,
`chat-native-progress-work`, `live-recovery-work`,
`native-interaction-bridge`.
- `pnpm check:module-boundaries` passes. `pnpm verify` returned exit 2
(broader CI coverage required) because of local benchmark scripts that
are not in this pull request. Required CI supplies the broad coverage.
- Append cost on a run with many events (local embedded PostgreSQL,
1,000 in-order appends, busy machine, load average about 20–40):

  | Events already in run | Before p50 / p90 | After p50 / p90 |
  | --- | --- | --- |
  | 50,000 (run 1) | 10.4 / 15.8 ms | 1.7 / 3.7 ms |
  | 50,000 (run 2) | 12.9 / 21.6 ms | 3.3 / 5.9 ms |
  | 400 | 3.3 / 6.2 ms | 3.7 / 7.7 ms (noise) |

- Statements per event do not change: BEGIN, run lock read, one append
statement, COMMIT (4). The append statement no longer scans the source's
events.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`): the machine was too busy for a stable result.
Deltas only: before 20,000 of 20,000 done at 3,999 statements/s. After:
13,796 and 14,817 of 20,000 in two runs. All sources: before 6,346 of
20,000 deltas done. After: 20,000 and 2,083 in two runs. A 20-second
window gives only 400 events per run, so it cannot show this change. The
long-run figures above are the evidence.

## Risks

- Low. The column is nullable and the previous release does not read it.
Runs that are in flight during a deploy have no cursor. Their first
append recounts once and stores the cursor.
- A stale cursor can only make the acknowledgement lag, never pass the
stored events. Event rows are deleted only with their company or agent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:08 -06:00
DottaandClaude Opus 5.5 0d832a02d3 perf(ci): shorten the general server and chat test lanes (#21393)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the `PR` CI workflow before it can merge
> - The `General tests (server (N/12))` and `General tests (chat (N/3))`
jobs finish last, so they set the wall-clock time of CI
> - Before this change, server shards took 9.5 to 11.7 minutes and chat
shards took 6.4 to 9.6 minutes; the target is 7 minutes or less
> - Measurement shows that the chat lane was unbalanced and that the
server lane is limited by CPU on the 4-vCPU runners
> - This pull request balances the chat lane by recorded test time, runs
three chat partitions per job, and takes general server weights from PR
runs
> - The benefit is a chat lane under 7 minutes and measured evidence for
the shard count that the server lane needs

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The PR CI general server and chat test jobs. They are the slowest jobs
of every pull request.

**Subsystem affected**

CI test scheduling: `scripts/run-vitest-stable.mjs`,
`scripts/test-line-shard.mjs` and
`scripts/ci/refresh-test-durations.mjs`.

**Current behavior**

Baseline: green `PR` runs
[43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012)
and
[43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316),
all on m7i-flex.xlarge (4 vCPU, 16 GB).

- Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step,
effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports
`tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums
1,410 to 1,765 s of file time. The longest file is 303 s
(`deferred-execution-receipt-work.test.ts`), so the floor is total work
divided by workers, not one file.
- Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each
job ran two partitions of `chat-channels.integration.test.ts`. The
partitions were balanced by collected-case count, but one `it.each` line
can run many cases. One of six partitions recorded 532 s of test time
against a 377 s mean.

**Proposed behavior**

- Chat partitions keep each source line whole and balance by recorded
time per collected entry.
- Each chat job runs three partitions in place of two.
- General server weights come from the server jobs of merged PR runs,
which run files with the same concurrency as the lane they place.

**Reason and benefit**

The slowest job sets the CI wall-clock time for every pull request. The
chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes.

## What Changed

- `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional
recorded durations. With durations, it balances source lines by recorded
time; entries without a record take the median. Without durations, the
behavior is unchanged.
- `scripts/chat-shard-durations.json` (new): recorded time per collected
chat entry, from the CI profile records of PR run
[43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171).
926 of 928 collected entries are measured; the other two are the bogus
collection entries that the launcher already drops.
- `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that
manifest from downloaded `ci-profile-*` artifacts. Each executed case
maps to the collected `it`/`it.each` entry that registers it. A case
without an entry fails the refresh.
- `scripts/run-vitest-stable.mjs`: a sharded chat job runs three
partitions in place of two, and the chat launcher passes the recorded
durations. A chat job collects the suite once with `vitest list` in
place of once per partition, and the per-partition filter checks run at
the same time. Before, each partition ran two blocking collections of
the 86,000-line file one after another. The server worker count stays at
three.
- `scripts/ci/refresh-test-durations.mjs` and
`scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general
server weights from the two newest merged-PR runs. Each run must have
the full server shard count from `.github/ci-task-matrix.json` and cover
at least 90% of the lane. Serialized weights still come from master
Cloud readiness runs.
- `scripts/general-server-shard-durations.json`: refreshed with the new
source (PR runs 43439145752 and 43439279537).
- Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`,
`scripts/ci/test-selection-pr-timings.test.mjs` and
`server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted
partition, the entry mapping, the manifest format, the PR-run collection
rules and the three-partition chat launcher.
- `doc/DEVELOPING.md`: describes the three chat partitions, the chat
weights and how to refresh them, and the new source of the server
weights.

## Verification

Experiments ran in this PR's own CI with a temporary resource sampler
(removed before review).

| Setting (run) | Server step vs. same shard's baseline | CPU busy | Max
memory used | Result |
|---|---|---|---|---|
| 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) |
62-73% | 6.8 GB | 1 flaky retry |
| 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean |
| 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard
failure: `decision-models.test.ts` timed out at 15 s twice |
| 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB
| 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at
30 s on m5a |

The server lane is CPU bound: at four workers the CPU is 82-89% busy and
wall time improves by only about 10%. Five workers fail tests. Memory is
not the limit (no swap, no memory pressure, at least 6 GB free). Node's
compile cache and Postgres `fsync=off` showed no attributable gain
(import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to
0.2-0.7% of CPU time), so neither is in this PR.

Final configuration (server: 3 workers with refreshed weights; chat: 3
weighted partitions per job), job minutes / test-step seconds /
effective concurrency:

| Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503
attempt 1 | B: attempt 2 | C: attempt 3 |
|---|---|---|---|---|---|
| server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 /
2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 |
| server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85
| 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 |
| server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81
| 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 |
| server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 /
2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 |
| server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85
| 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 |
| server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69
| 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 |
| server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 /
2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 |
| server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83
| 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 |
| server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68
| 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 |
| server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84
| 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 |
| server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 /
2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 |
| server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 /
2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 |
| chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 |
5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 |
| chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 |
6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 |
| chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 |
5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 |

Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B /
C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server
worker setting is unchanged, so its run-to-run spread is mostly runner
type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min).

Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five
server shards on m5a.xlarge summed 2,545-3,037 s of file time against
about 1,700 s elsewhere; three reached the 20-minute job limit and two
failed tests on 15 s timeouts. The server lane settings in this PR are
the same as master's, so this is a fleet finding: m5a.xlarge cannot run
a server shard inside the job limit (follow-up for the fleet
configuration, outside this PR).

Flakiness of the final setting: the nine chat jobs in A, B and C had no
failure and no retried test. Server shards in A and B had no failure and
no retried test. Before the last commit (single chat collection), three
runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server
retries there (2 in one run, 1 in another) were on the unchanged
three-worker server setting.

Local checks (correctness only):

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs`: 68 passed.
- `pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/vitest-chat-shards.test.ts`: passed.
- `pnpm verify` exits 2 (global CI input changed), so the required CI
checks supply the broad coverage.

## Risks

- Three chat partitions per job use more memory: the peak was 11.4 GB of
16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63%
busy.
- Stale chat weights only change balance, never which tests run. Shard
membership is still checked with `vitest list` before each run.
- The weekly refresh now downloads general server logs from merged-PR
runs. Runs that selected a subset of files are skipped before download
because their shard count is smaller.
- The server lane is still above 7 minutes. See the remaining floor
below.

**Remaining floor and shard-count recommendation (for a follow-up under
`.github/`)**

A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs
give at most about 2.8 effective concurrency, so the test step is about
560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test
step of about 360 s, so about 1,000 s of file time per shard. The
current 19,200 s of general server file time therefore needs about 19-20
shards of 4 vCPUs at 3 workers. More workers on the same runners do not
help, because the CPU is already the limit.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
GitHub CLI) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:03 -06:00
DottaandClaude Fable 5.2 0199c963e2 feat(apps): ship the redesigned sign-in step for a pasted MCP server (#21469)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Apps lets an operator connect a remote MCP server by pasting its
address ("Connect your own MCP server")
> - Step 2 of that flow put every sign-in choice in an "Advanced
authentication" box with wrapped pill buttons and a hint paragraph
> - The box hid the choice behind a vague label, and its shape did not
match the access line ("Connects for everyone… Change") right below it
> - The owner approved a redesign: a stated sign-in default with a
Change link, and five options with their fields directly below each
option
> - This pull request wires that redesign into the real screen, removes
the old step, and updates copy that named the removed box
> - The benefit is a screen that reads as two stated answers and one
action, with the same default path (paste the address, press Check link)

## Linked Issues or Issue Description

No public issue exists. Enhancement:

**What existing behavior does this improve?**
The sign-in section of the "Connect your own MCP server" step in Apps.

**Subsystem affected**
UI: Apps connection setup (`ui/src/features/connections`). Small copy
changes in the server OAuth client registration message and docs.

**Current behavior**
Step 2 shows an "Advanced authentication" collapsible with five wrapped
pill buttons and a hint paragraph below them. The fields for the chosen
mode appear further down. Guidance banners tell the operator to look
"under Advanced authentication". The page also shows the "Unverified
server" badge twice (header and body). The stepper reads "Pick app · Add
your key", even when the server needs no key.

**Proposed behavior**
Step 2 states the sign-in choice in one line ("Paperclip asks the server
how to sign in.") with a Change link. Change shows five radio cards: Let
Paperclip check, Key or token, Browser sign-in, Custom headers, No
sign-in needed. The fields for the selected option are directly below
it. Done collapses the list to a one-line summary (for example "Signs in
with 1 custom header."). The list stays open while the chosen option
still needs a value. Client ID and secret are behind "Use your own
client ID" unless they have values or the server asked for them. The
stepper reads "Pick app · Connect".

**Reason and benefit**
The choice is visible as a stated default in the same shape as the
access line. Each option keeps its fields next to it. No copy points at
a box that does not exist.

**Breaking changes**
None. The five auth modes and their request payloads do not change.

**Additional context**
Related open PRs change server behavior for the same flow. They are not
duplicates. Refs #15192, Refs #15450.

## What Changed

- `ui/src/features/connections/GenericMcpConnectStep.tsx`: new step
component (same props as the old `LinkConnectStep`). It uses stable
field ids `generic-mcp-key`, `generic-mcp-client-id`,
`generic-mcp-client-secret` and a `data-testid` on the sign-in
Change/Done link.
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: render
`GenericMcpConnectStep`; delete `LinkConnectStep`,
`GENERIC_AUTH_MODE_OPTIONS`, `SegmentedOption` and unused imports;
change the pasted-link stepper label from "Add your key" to "Connect"
(`STEP_LABELS` is used only by the pasted-link flow; curated apps keep
their own `["Add your key"]`).
- `ui/src/pages/apps/generic-mcp-connect.ts`: `oauth_challenge` and
`oauth_manual_client_required` guidance now say "below" instead of
"under Advanced authentication".
- `server/src/services/tool-access/oauth/client-registration.ts`: the
manual-client message no longer names the removed box.
- `doc/connections/GENERIC-REMOTE-MCP.md`: describe the sign-in line,
Change, and the five options.
- Comments in `packages/shared` and `server/src/services/tool-access.ts`
reworded. `tests/runner-e2e/source-sdk-public-inputs.json` regenerated
for the two shared files (blob hashes only).
- `ui/storybook/stories/connection-setup/byo-mcp-sign-in.stories.tsx`:
14 stories for the shipped component; annotation is now `current` /
`production`.
- Tests: `AppsConnect.test.tsx` generic MCP block uses the sign-in
toggle and radio options; new tests for the collapsed summary, the
open-while-empty-key rule, and a single "Unverified server" label. New
`GenericMcpConnectStep.test.ts` unit-tests `genericMcpAuthSummary`.

Before: "Advanced authentication" box with pill buttons and a hint;
fields below the pills; two "Unverified server" badges; stepper "Pick
app · Add your key".
After: one sign-in line with Change; five radio cards with fields under
the selected card; one "Unverified server" badge in the header; stepper
"Pick app · Connect". Storybook: `byo-mcp-sign-in--choosing`,
`byo-mcp-sign-in--server-asked-for-key`.

## Verification

- `cd ui && pnpm exec vitest run src/pages/apps/AppsConnect.test.tsx
src/pages/apps/generic-mcp-connect.test.ts
src/features/connections/GenericMcpConnectStep.test.ts
src/features/connections/ConnectionSetupFlow.architecture.test.ts
src/features/connections/ConnectionSetupFlow.oauth.test.ts`: 5 files,
300 tests pass.
- `cd ui && node ../scripts/run-tsc.mjs -b`: pass.
- `pnpm -s typecheck` in `packages/shared` and `server`: pass.
- `pnpm check:token-gates`: all gates clean.
- `node scripts/storybook-catalog.mjs`: pass (generated indexes not
committed).
- `pnpm verify`: exit 2 (broader verification required, because
`doc/connections/GENERIC-REMOTE-MCP.md` is a global CI input). Required
CI supplies that coverage. The focused files above were rerun on the
final revision after the rebase.
- Headless Playwright against the Storybook dev server: the Default,
Choosing and Server-asked-for-key stories render the new step, the new
guidance copy, and the "Pick app · Connect" stepper.

## Risks

- Low risk. UI change on one screen; request payloads
(`genericConnectPayload`) do not change.
- Tests that click the first collapsed "Change" button now skip the
sign-in toggle, because the access line also has a "Change" button.
- The server message for `oauth_manual_client_required` changes text. No
code matches on the text; the client uses the error code.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:35:49 -06:00
DottaandClaude Opus 5.5 e9d7895648 fix(recovery): start an overdue saved-work recovery instead of looping on no pending path (#21478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run stops before its sandbox files are saved, Paperclip holds
the task and recovers the files first. A recovery queue does this work.
> - The queue runs two attempts at a time for the whole instance. A
remote attempt can take many minutes.
> - On staging, 29 recoveries waited with 0 attempts. The oldest waited
since 15:01Z. One task waited 3 h 33 min for its single attempt.
> - During that time the task said "Recovering saved work…", but the
assigned-work invariant saw no pending path. The sweep sent a
continuation wake every few minutes. The hold refused each wake, and the
activity log showed "could not start yet".
> - This pull request makes the queued recovery the task's pending path
for a bound. After the bound, the sweep starts the recovery itself
instead of sending a wake that the hold refuses.
> - The benefit: a held task gets its files recovered and continues
without a person, and the sweep stops its refused-wake loop.

## Linked Issues or Issue Description

**What happened?**
A scheduled task ran a legacy retry on a Daytona sandbox. Setup failed
with `setup_failed` ("remote_workspace_operation_timed_out: capture
(configured timeout 300000ms)"). Paperclip held the task for saved-work
recovery (`legacy_execution_requires_reconciliation`, resolved, replay
blocked, system recovery `scheduled`, 0 attempts). After 45 minutes the
recovery had not started. The task showed "Recovering saved work…" with
no Retry. The assigned-work sweep logged `issue.continued_automatically`
with outcome `not_started` and violation `no_pending_path` at 17:58,
18:02, 18:10 and 18:25 UTC.

**Expected behavior**
The recovery proceeds by itself and the task continues. If the recovery
cannot finish, the task shows a Retry that works. "Recovering…" never
shows with nothing behind it.

**Steps to reproduce**
1. A legacy run with a retained remote workspace fails during setup, and
its saved-work hold is adopted by system recovery.
2. The recovery queue does not reach the hold (it has two slots, and
they are busy).
3. The assigned-work sweep checks the task 10 or more minutes later. It
finds `no_pending_path` and sends a wake. The hold refuses the wake.
This repeats until the continuation cap.

The regression test
`server/src/services/saved-work-recovery-overdue.test.ts` reproduces
this state on real rows.

**Paperclip version or commit**
`master` at `c6301c92ff` (staging, 2026-11-13).

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with a Daytona sandbox on the
legacy runtime.

## What Changed

- `assigned-work-invariant.ts`: new pending intent
`saved_work_recovery`. A saved-work recovery that the queue still owes
counts as the task's pending path while an attempt runs, while its next
attempt is scheduled, or for `SAVED_WORK_RECOVERY_START_BOUND_MS` (10
minutes) after it became due.
- `assigned-work-invariant.ts`: new violation
`saved_work_recovery_overdue`, used when that bound passes. It comes
before the `no_pending_path`, `retry_offered` and blocked checks, so the
sweep never sends a wake that the hold refuses.
- `assigned-work-sweep.ts`: the sweep loads the fact with the same
eligibility predicate the recovery worker uses. For the overdue
violation it calls `startSavedWorkRecovery` and records no continuation
activity.
- `workspace-recovery.ts`: `startOverdueWorkspaceRecovery` starts one
attempt outside the queue's two slots, with at most two such attempts
per process. It skips runs that are already in flight and uses the same
ownership lock and attempt bookkeeping as the queue. The queue skips
runs that this path started. `recoverRunAction` moved to module scope.
Its "Retained workspace remains pending" warning now logs the error.
- `heartbeat.ts` and `recovery/service.ts`: wire the dependency.
- The sweep returns the end of the bound as the task's re-check
deadline, so the overdue start does not depend on another notification.

## Verification

- New `server/src/services/saved-work-recovery-overdue.test.ts` (real
Postgres, recovery fixture with a mocked sandbox):
- Overdue hold: the sweep starts the recovery and does not call
`enqueueWakeup`. No continuation activity is written. The system
recovery reaches `complete`. The next sweep records a fresh
continuation, and `deliverReconciledExecutions` starts a successor run.
- Hold within the bound: verdict `ok / pending_intent /
saved_work_recovery`. No wake is sent and no attempt starts.
- A second start for the same run returns `in_flight`. Exactly one
attempt is recorded, and a completed hold returns `not_eligible`.
- Red on the old invariant and sweep, with `no_pending_path` and
`enqueue_refused`, the same as staging. Green with this change.
- `server/src/services/recovery/assigned-work-invariant.test.ts`: 54
pass, with new cases for the bound, a running or scheduled attempt, a
stale attempt, and a pending question.
- `server/src/__tests__/assigned-work-invariant.test.ts`: 31 pass.
- `server/src/services/workspace-recovery-passive-fairness.test.ts`:
passes.
- `tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- A sweep-started attempt can run beside the queue's two attempts. That
adds up to two more concurrent recovery attempts per process, each with
its own database connection while it holds the ownership lock. Each
attempt for a run still takes that run's advisory lock, so two servers
cannot run the same recovery at once.
- The 10-minute bound is new. A queue that is slow but working now gets
help after 10 minutes. It does not get the hours it took on staging.
- Out of scope, reported for follow-up: why the queue was so slow on
staging (two slots for the whole instance, attempts that take minutes),
and why the setup capture hit its 300 s timeout. Server logs were not
available from this session.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser reads of the staging API, local test
runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:26:26 -06:00
d018a9d1cd fix(queue): retain chat bindings on resolved card continuations (#20825)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task can receive a card response while its agent runs.
> - The queue saves this response for the next turn.
> - Chat authorization requires the saved inbound message binding.
> - Queue promotion removed this binding when a prior turn received the
message.
> - This pull request retains live bindings for resolved card responses.
> - The next turn can check current chat access and continue the
approved work.

## Linked Issues or Issue Description

**What happened?**

An accepted plan card queued behind a chat run could start with a chat
source and no message IDs. Reviewed chat authorization then failed with
context_mismatch before provider work started.

**Expected behavior**

A resolved card must retain its live chat binding through queue
promotion. The new run must check current access. A consumed
message-only wake must not start another turn.

**Steps to reproduce**

1. Start a chat run with an inbound message.
2. Queue a card response with that message binding.
3. Complete the first run and release its queue.
4. Inspect the promoted context and check chat authorization.

Related work: #20689 cancels consumed message-only wakes. #19156
protects a successor checkout. This change retains the binding for an
independent card response.

## What Changed

- Retain delivered message bindings for resolved chat card responses
during queue promotion.
- Keep deleted, foreign-task, and self-authored message filters and
current chat authorization.
- Add a persisted queue regression and Slack/Discord
promotion-to-authorization checks, including access revocation.
- Document the queue behavior.

## Verification

- Before the fix, the persisted queue test failed because the promoted
context had no wakeCommentIds. Three existing tests passed.
- The first focused run passed 108 tests. The added Slack and Discord
checks passed after promotion and refused revoked access.
- Final head a38bcb9bc5: 110/110 focused
tests pass. Low-memory server typecheck passes. pnpm verify exits 2 and
requires broad hosted CI. CI typecheck, build, all
E2E/general/serialized tests and Runner checks pass except the Runner
support suite and its aggregate gate. The support suite fails on
inherited stale SDK lock pins (source-sdk-public-type,
source-sdk-evidence and source-sdk-public-inputs); existing PR #20623
owns that repair and has passing CI. No retry is appropriate for this
deterministic failure. Greptile reviewed this exact head at 5/5 with no
actionable findings; there are no inline or unresolved review threads.
- The hosted failed task independently resumed. No source task replay or
deployment was performed. Fixed-release hosted provider acceptance
remains unverified.
- Preflight installed dependencies but could not build Rust binaries
because Cargo is absent. The first server typecheck ended without a
diagnostic; the low-memory retry passed.

## Risks

An overly broad exception could deliver consumed input twice. The
exception applies only to resolved card responses with a chat source.
Message-only wakes still cancel. Message existence, task scope, author
filters, execution ownership, and current chat access still apply.

## Model Used

OpenAI GPT-6 Codex agent with reasoning, repository tools, and code
execution. The runtime did not expose a more specific model build ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:12:02 -06:00
Devin Foley b4861d5e37 fix(diagnostics): report Daytona acquisition state for runs retried as provider outages (#21476)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before an agent starts on a sandbox environment, the host gets an
environment lease. On Daytona, this creates a sandbox.
> - When the create does not finish in time, the Daytona plugin records
the failed phase, the elapsed time, the budget, and the Daytona sandbox
state. #15736 and #15946 added these fields to the run failure report,
so that operators can see where the acquisition stopped.
> - #20857 changed the error code of these failures from `setup_failed`
to `environment_provider_unavailable`, so that the host retries them.
> - The run failure diagnostics read the acquisition fields only when
the error code is `setup_failed`. Thus every Daytona create timeout now
reaches the error report without its phase or provider state.
> - On staging, more than 270 Daytona lease timeouts occurred in three
days, and none of them shows the provider state.
> - This pull request also reads the acquisition fields when the error
code is `environment_provider_unavailable`.
> - The benefit is that the error report shows again whether the sandbox
was stuck in `creating`, in a snapshot state, or absent. This evidence
is necessary to find the cause of the timeouts.

## Linked Issues or Issue Description

Refs #20857, Refs #20828, Refs #15946, Refs #15736

**What happened?**
A Daytona lease acquisition timeout is now classified as
`environment_provider_unavailable` (#20857). The run failure report for
this failure does not contain `environmentAcquisitionPhase`,
`environmentAcquisitionElapsedMs`, `environmentAcquisitionBudgetMs` or
`environmentAcquisitionProviderState`.

**What did you expect?**
The report contains the acquisition phase and the provider state, as it
did for `setup_failed` runs before #20857.

**Steps to reproduce**
1. Let a Daytona sandbox create pass its acquisition deadline.
2. The plugin returns "Daytona lease acquisition timed out; allocation
cleanup is pending" with an acquisition diagnostic.
3. The host fails the run with `environment_provider_unavailable`.
4. Look at the `run_execution` context of the error report. The
acquisition fields are not there.

## What Changed

- `collectRunFailureDiagnostics` reads the private acquisition
diagnostic for a setup failure with the error code `setup_failed` or
`environment_provider_unavailable`. The other conditions do not change:
the phase must be `setup`, and the diagnostic must belong to the same
run and company.
- A new test builds the real wrapper chain (lease failure, then provider
failure, then the RPC error that holds the diagnostic). It checks that
the fields are reported for the failed run, and that they are not
reported for another run or for the execute phase.

## Verification

- `cd server && npx vitest run
src/services/__tests__/run-failure-diagnostics-provider-unavailable.test.ts`
fails on master (the report has only `failurePhase`). It passes with
this change.
- `cd server && npx vitest run
src/services/__tests__/run-failure-diagnostics-provider-unavailable.test.ts
src/services/__tests__/run-failure-diagnostics.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts`: 3 files, 139
tests pass.

## Risks

Low risk. The change adds four bounded scalar fields to an error report.
These fields already exist for `setup_failed` runs. The change does not
alter retries, the error grouping, or the run result.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in
an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-11-13 12:04:25 -08:00
Devin FoleyandClaude Opus 5.5 89ab7e51ff fix(recovery): retain a failed run's sandbox in one lease write (#21474)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute in remote sandboxes. Each run holds an
environment lease row that points to its environment, task, workspace
and company
> - When a run fails before it copies its files back, Paperclip keeps
the sandbox for a file repair. It writes a stop-and-retain intent and a
repair-hold policy on the lease
> - The recorder wrote the same lease row two times in one transaction.
Postgres runs the foreign-key checks again on a row that the current
transaction already wrote, and these checks take `FOR KEY SHARE` locks
on the referenced rows
> - Other runs lock the shared environment row `FOR UPDATE` when they
acquire a lease. Then the second write waits, meets the 1 second lock
timeout, and the run fails with "Database query failed (55P03)"
> - This pull request writes the stop intent and the repair hold in one
update
> - The benefit is that a busy shared environment no longer fails runs,
and the file repair hold is recorded

## Linked Issues or Issue Description

**What happened?**
On staging, native OpenCode runs failed with `adapter_failed` and the
error "Database query failed (55P03)". The failed statement was `update
"environment_leases" set "lease_policy" = 'retain_on_failure',
"metadata" = "metadata" || {workspaceRestoreRecovery...} where ...
"status" = 'pending_cleanup'` in
`preserveLegacyWorkspaceRestoreSources`. The cause was `canceling
statement due to lock timeout`. The statement does not wait on the lease
row, because the transaction already holds it `FOR UPDATE`. It waits on
the foreign-key check `SELECT 1 FROM ONLY "public"."environments" x
WHERE "id" = $1 FOR KEY SHARE OF x`. Postgres runs this check because
the transaction wrote the same row one statement before, in
`prepareSandboxStopAndRetain`.

**What did you expect?**
The run records its workspace restore failure and keeps its sandbox for
repair, also when another run holds the shared environment row.

**Steps to reproduce**
1. Start a run with an active, ephemeral sandbox lease.
2. In another session, lock the lease's `environments` row with `SELECT
... FOR UPDATE` and keep the transaction open.
3. Record a workspace restore failure for the run
(`terminalizeLegacyExecution`, legacy or native
`recordRestoreFailureOnly`).
4. Before this change, the second lease write fails after 1 second with
55P03. The new test
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`
shows this for legacy and native runs.

No public issue exists. I searched open and recent pull requests for
this lock timeout and found no duplicate. #21071 bounds a different lock
wait (workspace source locks in run preparation) and does not overlap.

## What Changed

- `server/src/services/sandbox-stop-and-retain.ts`:
`prepareSandboxStopAndRetain` accepts an optional `retention` argument
(`leasePolicy` and extra `metadata`). The stop-and-retain update sets
these values in the same statement. Other callers do not change.
- `server/src/services/legacy-workspace-restore-recovery.ts`: the
active-source path passes the repair hold (`retain_on_failure` and
`workspaceRestoreRecovery` with `originalLeasePolicy`) to that update.
It no longer does a second update on the same row.
-
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`:
a regression test for legacy and native runs. Another session holds the
environment row `FOR UPDATE` while the run records its restore failure.
The test checks the run's recovery evidence and the lease policy,
status, stop intent and repair metadata.

## Verification

- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-lock-contention.test.ts`: both
cases fail on `master` with `canceling statement due to lock timeout` on
the `environments` `FOR KEY SHARE` check. Both cases pass with this
change.
- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-recovery.test.ts`: 73 existing
tests pass.
- Related suites pass (10 files, 485 tests):
`heartbeat-observed-workspace-preservation`,
`heartbeat-native-remote-failure-retention`, `live-recovery-work`,
`workspace-recovery-stop-proof`, `native-failed-workspace-recovery`,
`native-cancelled-workspace-preservation`,
`workspace-recovery-admission`, `native-workspace-reconstruction`,
`environment-retained-orphan-sandbox`, `abandoned-workspace-files-hold`.
- I did not complete a local server typecheck. The process ran out of
memory in the 10 GB sandbox. CI runs the typecheck on this head.

## Risks

- Low risk. The lease ends with the same status, policy and metadata as
before. The values are written in one statement instead of two. The
update keeps its ownership guards (status, provider, provider lease ID
and exact metadata).
- Other code paths can still write a lease row two times in one
transaction (for example the shutdown settlement path, which sets a
lease back to `active` before it records the restore failure). That path
handles 55P03 as a lock timeout and retries. This pull request does not
change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, code execution, embedded Postgres tests, and
GitHub tools).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:58:00 -08:00
DottaandClaude Opus 5.5 89f486cdab fix(heartbeat): run one dispatch pass per agent, back off refused claims, and enforce the run limit in the claim (#21433)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. Each trigger (a
run finishes, a wake arrives, a sweep runs) asks for one dispatch pass
of that agent
> - An agent could run more runs than its `maxConcurrentRuns`, dispatch
passes could stack up without a limit, and every pass offered the same
unclaimable runs again
> - On hosted staging (e23549cd9c), one agent with limit 20 had 20–35
runs `running` and 170–270 queued. The logs showed 375 "offered a free
slot but could not be claimed" lines and 117 "claim was rejected" lines
in 10 minutes
> - This pull request runs one pass per agent at a time, folds triggers
into one follow-up pass, backs off refused claims, enforces the limit in
the claim transaction, and reads only the columns the pass needs
> - The benefit is that an agent never runs past its limit (also across
two servers during a deploy), and a burst of triggers costs one or two
passes, not one pass per trigger

## Linked Issues or Issue Description

Refs #21391, #21242, #21234, #21277, #21324, #21349

**What happened?**

On hosted staging (Core e23549cd9c), one agent ("Codie",
`maxConcurrentRuns` 20) had 170–270 queued runs and 20–35 runs
`running`, so it was over its limit. The logs showed a refused claim
offered again on every pass. This pull request is about correctness and
load. It is not the cause of the staging out-of-memory crashes: a
separate leak (`sessionToolAuthorityEpochs` in
`native-session-executor.ts`, about 3 MB per finished native run)
explains those, and another change fixes it.

Live again at 17:02Z on staging (Core 04f12fd2be, which includes
#21391): the same agent had 52 runs `running` against
`maxConcurrentRuns` 40. All 52 started between 16:55Z and 17:02Z, right
after a deploy released the backlog, with 112 runs still queued. So
overlapping passes exceed the limit with claimable runs, not only with
unclaimable ones.

**Why did it happen?**

1. `withAgentStartLock` chained each trigger behind the previous one and
measured its 30 s stale limit from when the predecessor was chained, not
from when it started. When triggers arrived faster than one per pass,
every trigger after about 30 s of backlog started its own pass next to
the others.
2. Each pass counted free slots by itself, outside any lock that other
passes or another server see. Overlapping passes and two servers during
a deploy swap each claimed up to the limit.
3. Each pass read every queued run with `select *`, including the large
`contextSnapshot`, and offered every refused run again with no backoff.

A local reproduction of the staging shape by another session (250
unclaimable queued runs with 8 KB snapshots, one trigger per second, on
e23549cd9c and again on master e7e28ba241): passes that run at the same
time went 1 → 12 → 39 → 79 → 139 at t = 30/45/75/116/178 s, only 4
passes finished in 178 s, and post-GC heap grew by about 2.8 MB per
stacked pass (519 MB → 1,061 MB at 190 passes, mostly the parsed
`heartbeat_runs` rows and the dispatcher facts each pass holds). 242 s
after the triggers stopped, 216 passes were still outstanding.

**Expected behavior**

At most one dispatch pass per agent runs at a time. Triggers that arrive
during a pass cause one follow-up pass. The `running` count never goes
above `maxConcurrentRuns`. A refused run is offered again after a
bounded wait, and no queued run is stranded.

**Steps to reproduce**

1. Give an agent a limit of 4, 12 aged queued runs that the chat-control
gate refuses, and 12 claimable runs.
2. Start a pass. While its first claim waits, send 40 triggers, each
after more than 30 s of the old lock's clock.
3. Count the passes, the refused offers, and the highest `running`
count.

## What Changed

- `server/src/services/agent-start-lock.ts`: replaces the stale-bypass
lock with `dispatchForAgent`. One pass runs per agent. Triggers during a
pass merge into one follow-up pass that all their callers share. A
trigger from inside the agent's own pass does not wait on itself. A pass
has a 60 s deadline on a monotonic clock, measured from when the pass
started (not from when a waiter was chained). At the deadline the pass
stops claiming (an abort signal) and the follow-up may start. A trigger
that arrives after the deadline also ends the pass, so a stalled timer
cannot keep an agent blocked. Post-pass effects run after the agent is
free, as before.
- `server/src/services/heartbeat/run-slot-admission.ts` (new):
`assertAgentRunSlotFree` counts the agent's `running` runs and compares
them with `maxConcurrentRunsSql()` inside the agent-scoped advisory lock
that every claim already takes (`withExternalAdmissionGuard`). It throws
`AgentRunSlotsFull` when the agent is full. The count leaves out the run
that the claimed run replaces (`retryOfRunId` or `previousRunId`), so a
successor that starts while its predecessor finishes is not refused.
- `server/src/services/heartbeat/queue.ts`:
- Every claim path (ordinary, queued comment, native review through
`native-review-dispatch.ts`) checks the slot inside its claim
transaction. A full agent leaves the run queued, and the pass stops.
- The pass reads only the columns the dispatcher needs, with `issueId`,
`wakeReason` and `previousRunId` taken from `contextSnapshot` in SQL. It
reads the full row only for the run it is about to claim.
- A refused claim (declined or 4xx) of a run that has waited more than
an hour (the existing aging limit) is not offered again for 5 s, then
10, 20, 40, and at most 60 s. A person's run of that age waits 2 s. A
run queued for less than an hour is never backed off, so a successor or
retry that waits for its predecessor starts as soon as the predecessor
ends. A backoff also ends when the run's task changes (status,
`updatedAt` or execution run) or when a run of that task ends in this
process. A backed-off run does not take the oldest-run share. The
queued-run sweep (`resumeQueuedRuns`, `resumeQueuedRunsAfterDrain`,
which the queued-run redrive calls) offers refused runs again regardless
of the backoff.
- A pass makes at most 24 claim attempts. When it reaches that limit it
asks for one follow-up pass that skips every run the chain already
tried, so the follow-up reaches runs further down the line. The chain
ends when a pass does not reach the limit, so refused runs cannot cause
a continuous chain of passes. Only this automatic follow-up carries the
tried list. A wake, a completion or another fresh trigger that merges
into it resets the list, so a run whose blocker cleared during the chain
is offered again.
- Admission, suppression and cutoff are read once per pass, not once per
trigger.
- Seam for a skip before the claim: in `runQueuedDispatchPass`
(`claimQueuedRunsForAgent`), right after the skip line (`if
(triedRunIds.has(candidate.id) || (!retryRefused &&
claimBackedOff(candidate.id, new Date(), taskStamp))) continue;`),
before the claim-budget check, `claimAttempts += 1` and the full-row
read. A `continue` there spends no claim attempt, reads no row, does not
enter the claim backoff or the aged-decline note, and does not call
`dispatcher.claimed`, so the run keeps its place and its oldest-run
share. The dispatcher exposes `dispatcher.personOrigin(run)` (the
person-origin fact it already derives) and `dispatcher.personRun(run)`.
The candidate rows keep `id`, `companyId`, `agentId`, `status`,
`invocationSource`, `retryOfRunId`, `wakeupRequestId`,
`scheduledRetryAt`, `scheduledRetryAttempt`, `scheduledRetryReason`,
`createdAt`, `startedAt`, and
`contextSnapshot.issueId/wakeReason/previousRunId`.
- Each pass is registered with the execution drain
(`drainActiveRunExecutions`) until it ends or reaches its deadline.
- Tests: new `server/src/services/agent-start-lock.test.ts` and
`server/src/__tests__/queued-run-dispatch-stacking.test.ts`.
`server/src/__tests__/heartbeat-start-lock.test.ts` is removed: it
asserted the 30 s stale bypass that this pull request removes.
`workspace-recovery-preparation-dispatch.test.ts` holds and counts
passes through the new function.

The chat-control fallback lookup from #21391 (`issueId = X OR
nativeIssueId = X OR contextSnapshot->>'issueId' = X`) was measured on
embedded Postgres with 200,000 runs in one company. Postgres uses a
BitmapOr of three existing indexes
(`heartbeat_runs_company_issue_created_idx`,
`heartbeat_runs_company_native_issue_contract_id_uq`,
`heartbeat_runs_company_ctx_issue_created_idx`), not a sequential scan:
0.33–0.45 ms per call for a recent previous run, a 30-day-old previous
run, and no previous run. It is not changed.

## Verification

Red/green, `server/src/__tests__/queued-run-dispatch-stacking.test.ts`
(embedded Postgres, real heartbeat queue, limit 4, 12 refused aged runs,
12 claimable runs, 41 triggers):

| Measure | master | this pull request |
|---|---|---|
| Passes | 42 | 2 |
| Peak passes at the same time | 40 | 1 |
| Peak claims in flight | 9 | 1 |
| Refused-claim offers per trigger | 12 | 0.29 (12 in total) |
| Peak `running` (limit 4) | 8 | 4 |
| Passes outstanding after the burst | — | 0 |

- "waits a bounded time before offering a declined/rejected run again":
on master 10 triggers offer the refused run 10 times. With the fix they
offer it once, and once the backoff passes it is offered again. The
rejected case uses the 422 "Unable to resolve responsible user" path
from the staging reproduction.
- "starts a run that becomes ready during a chain of passes when a fresh
trigger arrives": red at 3d48c4bed2, green now.
- "stops chaining passes once every refused run was tried": 60 refused
runs and one trigger cause exactly 3 passes, then no more.
- "starts runnable work queued behind more refused person runs than one
pass tries": 30 refused person runs ahead of 3 runnable runs, with free
slots. The runnable runs start after one trigger, in at most 3 passes.
At 68e64a2de9 the same 24 person runs were offered on every pass and the
runnable runs never started.
- `server/src/services/agent-start-lock.test.ts`: one pass for a burst
of 50 triggers plus one follow-up, no second pass after 45 s, a hung
pass stopped at its deadline without overlap, no self-wait, settle after
release, and errors passed to the caller.
- `node server/scripts/typecheck-layers.mjs`: pass.
- Vitest (embedded Postgres): `heartbeat/queue.test.ts` (102),
`recovery-wake-herd.test.ts` (35),
`heartbeat-queued-run-claim-isolation.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`issue-reassignment-queued-run.test.ts`,
`queued-comment-consumption.test.ts`,
`issue-queued-comment-queue.test.ts`, `chat-run-milestone-work.test.ts`.

## Risks

- Refusals of runs queued for less than an hour are not backed off. A
backoff there delayed successors that wait for in-process predecessor
ownership, which can end without a task change (two CI tests failed with
it). Per trigger, such a run costs at most one claim attempt.
- A refused run that has waited more than an hour now waits up to 60 s
(a person's run 2 s) before an event-driven pass offers it again, unless
its task changes. The sweep and the redrive ignore the backoff.
- A successor may start while the run it replaces is still `running`.
The limit check leaves that one predecessor out of the count.
- Claims of an agent that is at its limit now fail inside the
transaction. A run that `executeRun` claims directly (for example a
native resume) stays queued when the agent is full, and the next pass
starts it.
- A pass that passes its 60 s deadline can still finish one in-flight
claim next to the follow-up pass. The limit check in the transaction
keeps the `running` count within the limit.
- No schema or migration change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:55:38 -06:00
aa4a06653c fix(native): name the recorded turn failure when a native run fails (#21235)
Failed native runs now report the runner's recorded turn-failure code instead of the generic unclassified fallback, and the Sentry run-failure fingerprint includes runner:<code>.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:22:55 -08:00
DottaandPaperclip 08341ea134 fix(queue): let board users discard agent-authored queued messages (#21458)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task composer holds messages that an agent has not consumed yet.
> - A board operator needs to remove an agent-authored pending message
before it starts more work.
> - The queue currently grants discard only to the human author, so
these messages have a disabled trash button and the API rejects the
board user.
> - This change derives the board discard permission in the queue
snapshot and uses that same permission under the existing issue lock.
> - The route checks task access before it changes the queue. The
benefit is a working discard action with the existing company, privacy,
revision, and dispatch guards.

## Linked Issues or Issue Description

Fixes #15464.

Supersedes #15465 with an independent implementation and red/green
regression evidence. Thanks to @MMTE for identifying the gap and
proposing a fix. Related: #18958 addresses deletion of comments that
have already been delivered; this change covers pending queue entries.

## What Changed

- Queue responses grant board users discard permission for
agent-authored messages. Editing and other human authors' messages
retain their existing permissions.
- The discard use case honors the actor-specific permission rebuilt
under the queue lock. Author agents keep their existing cancellation
path.
- The board discard route checks private-task access before deletion.
The denial for a non-author agent now points to the board action instead
of saying that no board action exists.
- Tests cover queue states, cancellation of an empty continuation,
persistence, queue linkage, audit attribution, private-task denial, and
cross-company denial. The API reference describes the supported action.

## Verification

- Red before the fix: the two snapshot regressions returned `canDiscard:
false`; the discard regression threw
`QueuedCommentMutationForbiddenError`. The route regressions also caught
the disabled permission, missing private-task guard, and stale agent
denial text.
- Green after the fix: 30 queue snapshot and use-case tests; all five
focused route regressions with real disposable Postgres; six Postgres
adapter tests; 42 existing queue UI tests.
- Server typecheck: `cd server && node scripts/typecheck-layers.mjs`
passes. `node scripts/check-module-boundaries.mjs` and `git diff
--check` pass.
- The [complete PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/43484735280)
passes on head `164200602828e23f01a8b209ff954821c5e14e9a`. [Greptile
reports
5/5](https://github.com/paperclipai/paperclip/pull/21458#issuecomment-6861824451)
on that same head with no open review threads.
- Local scope: the 83 focused snapshot, use-case, route, adapter, and UI
checks pass. The existing queue UI suite has 42 tests. The new route
cases use real disposable Postgres and verify the resulting database
state after reload.
- Additional local full-suite attempt: 174 tests passed and 13
approval-related tests failed with cleanup timeouts, Postgres deadlocks,
and duplicate-prefix errors after cleanup failed. The first failure,
`queues request_confirmation resolution while its legacy source run is
still running`, also fails on unchanged base code with the same
30-second `afterEach` and `afterAll` timeouts. This broader local run is
not counted as a pass. No test timeout or assertion was weakened. CI
supplies the broad passing result for the committed head.
- `node scripts/local-verify.mjs --explain` returned exit 2 because the
API documentation is a global CI input; all of its required CI lanes
passed. The initial adapter attempt could not initialize local Postgres
under the sandbox; the same six checks passed with local-process access.
The initial cross-company fixture was corrected to use a distinct
company prefix.

## Risks

- This grants board operators a new action on agent-authored pending
messages. The route must pass company membership and private-task access
checks; the mutation retains its transaction, revision, and dispatch
protections.
- No schema or migration is required. The UI uses its existing discard
control and error handling.
- The staging task was inspected to confirm the cause. No staging data
was deleted, and a deployed UI retest is not claimed by these local and
CI checks.

## Model Used

- OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use,
and code execution. The exact runtime model ID and context limit are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — the focused regression,
adapter, and UI checks listed above; the additional broad local cleanup
failure is documented separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 13:03:46 -06:00
commitperclip[bot] 8b7b2a0701 fix: Recognize Git mount-boundary absence diagnostics in workspace checks (#21048)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

> [!WARNING]
> This change crosses a trust boundary. See Risks.

## Thinking Path

- Issue #20525 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: Both callers recognized only
the ordinary Git absence diagnostic. Preserve the earlier exact boundary
matcher and all existing exit, signal, environment, and filesystem
checks.
- This pull request: Recognize Git mount-boundary absence diagnostics in
workspace checks

## What Changed

- `server/src/services/git-mount-boundary-message.ts`: Retain the shared
complete-diagnostic matcher, which rejects partial and appended output.
- `server/src/services/managed-repository-checkout.ts`: Retain boundary
recognition without changing ordinary-diagnostic behavior or fallback
guards.
- `server/src/services/heartbeat/workspaces.ts`: Retain boundary
recognition and ancestor metadata inspection.
- `server/src/__tests__/git-mount-boundary.test.ts`: Add root mount,
CRLF, and embedded-line-break cases through both callers. Confirmed six
failures with boundary recognition disabled; restored implementation
passes this suite, both neighboring suites, and typecheck.

## Verification

- Reproducing test(s), added in this PR:
`server/src/__tests__/git-mount-boundary.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- The repository's compile command passes with the fix applied.
- 10 existing test file(s) near the change passed.
- The full test suite passes except for the 16 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.
- It crosses a trust boundary. Check each crossing against what the
issue asked for, and against who asked:
- validation (server/src/services/heartbeat/workspaces.ts:860): Broadens
the accepted Git absence diagnostics to include the mount-boundary
message when classifying a checkout failure as an expected
non-repository result.
- validation (server/src/services/managed-repository-checkout.ts:265):
Allows the mount-boundary diagnostic to bypass a previously thrown Git
error when plainTaskDirectory and the existing exit, signal, and
environment conditions hold.

## Model Used

- opus (Anthropic Claude, via Claude Code), writing and checking the
change in an isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #20525

#20525 also mentions, without asking this fix for it:
- Non-Git directories in close readiness. (The report explicitly
identifies merged #19553 as handling 'a different caller'; this patch
does not modify close readiness.)
- Initial managed-clone readiness. (The report says '#2977 concerns
initial managed-clone readiness, rather than this diagnostic matcher.')
- CI jobs 131047332629 and 131040975860 fail; fix the change only if it
causes the failures. (This earlier-round conditional comment supplies no
demonstrated matcher defect. .github/workflows/pr-trusted.yml:1801-1804
asserts aggregate shard status, while
server/src/services/tool-access/catalog/health.ts:222 reports MCP health
failures; neither changed here. Historical job causation cannot be)

A search of paperclipai/paperclip's open and closed pull requests found
no other pull request that names #20525.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=20525 base=156453fe7c67
report=513d758246cb2ac5 -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:59:22 -06:00
commitperclip[bot] 6f2ea3b4d5 fix: Complete run cancellation when lifecycle event append fails (#20215)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

> [!WARNING]
> This change crosses a trust boundary. See Risks.

## Thinking Path

- Issue #20116 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: Cancellation awaited the
lifecycle event append after cancelling the run and wakeup. An append
failure rejected the stop before issue release, agent finalization, and
queue advancement.
- This pull request: Complete run cancellation when lifecycle event
append fails

## What Changed

- `server/src/services/heartbeat/run-control.ts`: Log cancellation event
append failures and continue essential cleanup, matching lease-release
terminalization.
- `server/src/services/heartbeat/run-control-event-failure.test.ts`:
Cover queued and running cancellation with an injected append failure
and the real issue-release adapter. Confirmed regression failures before
the fix; regression tests, neighboring tests, and final typecheck pass.

## Verification

- Reproducing test(s), added in this PR:
`server/src/services/heartbeat/run-control-event-failure.test.ts`,
`server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts`,
`server/src/services/heartbeat/run-control.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- 3 independent attempts wrote 3 reproductions; this fix passes every
one of them.
- The repository's compile command passes with the fix applied.
- 9 existing test file(s) near the change passed.
- The full test suite passes except for the 16 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.
- It crosses a trust boundary. Check each crossing against what the
issue asked for, and against who asked:
- logging (server/src/services/heartbeat/run-control.ts:1665): Logs the
event-append error and run ID to the server log when cancellation event
recording fails.

## Model Used

- codex:gpt-6.1-sol, opus, sonnet (Anthropic Claude, via Claude Code;
OpenAI, via the Codex CLI), writing and checking the change in an
isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #20116

The reviewer found the base already handles:
- A closed request socket produces a misleading PATCH 200 log that hides
the failure. (server/src/middleware/logger.ts:89–91 detects aborted or
unfinished responses and formats 'request aborted'; lines 150 and
157–158 log these at warn level with that message instead of the default
status-based success message.)

A search of paperclipai/paperclip's open and closed pull requests found
no other pull request that names #20116.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=20116 base=ca34bed25797
report=e2609f9803f7cc0c -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:58:29 -06:00
commitperclip[bot] a6608bd735 fix: Preserve plugin UI UUID validation and test trailing line terminators (#20058)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

## Thinking Path

- Issue #17332 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: Plugin keys reached a
UUID-column query, and Drizzle's nested PostgreSQL error bypassed the
original error handler. The existing UUID guard fixes this; its
non-multiline JavaScript regex also rejects trailing newlines.
- This pull request: Preserve plugin UI UUID validation and test
trailing line terminators

## What Changed

- `server/src/routes/plugin-ui-static.ts`: Retain the existing guard,
PostgreSQL UUID forms, and key fallback.
- `server/src/__tests__/plugin-ui-static.test.ts`: Add route tests for
four trailing line terminators and accepted PostgreSQL UUID forms.
Confirmed the original lookup fails; all 58 focused tests and typecheck
pass with the guard.

## Verification

- Reproducing test(s), added in this PR:
`server/src/__tests__/plugin-ui-static.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- 3 independent attempts wrote 3 reproductions; this fix passes every
one of them.
- The repository's compile command passes with the fix applied.
- 2 existing test file(s) near the change passed.
- The full test suite passes except for the 16 file(s) that already fail
on the base commit.
- Revised in 1 round(s) of automated code review.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.

## Model Used

- codex:gpt-6.1-sol, opus, sonnet (Anthropic Claude, via Claude Code;
OpenAI, via the Codex CLI), writing and checking the change in an
isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #17332

Other pull requests that name #17332:
- #17433 (open): fix(plugins): serve a plugin UI bundle addressed by
plugin key
- #17335 (open): fix(server): serve plugin UI assets requested by plugin
key

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=17332 base=ca34bed25797
report=b2e95a170c3be729 -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:58:11 -06:00
commitperclip[bot] 0eeee6ae96 fix: Defer chat milestone result reads and fix regression test transaction typing (#19875)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

## Thinking Path

- Issue #16728 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: PostgreSQL could evaluate
both result_json predicates before publication filtering, detoasting
already-published runs. The regression test also passed transaction
handles to helpers typed for full database handles.
- This pull request: Defer chat milestone result reads and fix
regression test transaction typing

## What Changed

- `server/src/services/chat-run-publications.ts`: Retain
publication-dependent guards around both result JSON predicates in the
shared milestone query.
- `server/src/__tests__/chat-run-milestone-toast.test.ts`: Use the
neighboring tests' transaction cast convention. Confirmed both cases
fail without the guards and pass with them; all seven selected milestone
test files and typecheck passed.

## Verification

- Reproducing test(s), added in this PR:
`server/src/__tests__/chat-run-milestone-candidates.test.ts`,
`server/src/__tests__/chat-run-milestone-toast.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- The repository's compile command passes with the fix applied.
- 8 existing test file(s) near the change failed, and fail on the base
too, so they cannot show whether the change breaks them.
- The full test suite passes except for the 15 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.

## Model Used

- opus (Anthropic Claude, via Claude Code), writing and checking the
change in an isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #16728

The reviewer found the base already handles:
- Milestone projection still runs on a one-second cadence.
(server/src/services/chat-run-milestone-work.ts:12 describes and
implements discovery on startup/committed changes with retries for
failures or unfinished batches; server/src/app.ts:1269 registers this
work owner. This code exists independently of the proposed predicate
change.)

Other pull requests that name #16728:
- #18686 (open): perf(runs): read result_json once per run in run lists
and index agent runs by created_at

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=16728 base=d1ee25d0f722
report=5924b3e8dd6d2fe4 -->

---------

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:57:45 -06:00
DottaandPaperclip 5859e7cfa6 perf(apps): avoid repeated writes during permission saves (#21033)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Apps give agents access to tools through saved profiles and approval
rules.
> - A save replaces the full set of action entries and access bindings.
> - It also updates each Ask first rule and each retired rule in a
separate query.
> - Adding one agent can therefore cause hundreds of database round
trips.
> - This pull request keeps unchanged rows and batches the required
changes.
> - The benefit is faster saves with the same action permissions.

## Linked Issues or Issue Description

**What happened?**

An app permission save repeats database work for actions and agents that
did not change. With 100 actions and 500 retired rules, one agent
addition makes 636 queries. Of these, 599 update approval policies.
Related boundary coverage: Refs #19973. This change keeps that issue's
Off and agent-selection protections.

**Expected behavior**

Adding an agent applies the saved Allowed, Ask first, and Off choices.
The save changes only the rows that need a change. Independent approval
rules remain in force.

**Steps to reproduce**

1. Create an app with 100 active actions and 99 Ask first rules.
2. Add 500 disabled app approval rules.
3. Save access for three agents. Then add a fourth agent.
4. Count database queries during the save.

**Paperclip version or commit**

Baseline: `51d77648bc62`.

**Deployment mode**

Built from source. The reproduction uses disposable local PostgreSQL.

## What Changed

- Keep action entries and access bindings when their saved values match.
- Insert new bindings in one query. Remove only bindings and entries
that no longer apply.
- Read managed approval rules for the selected app. Skip unchanged rules
and rules that are already disabled.
- Batch new or changed Ask first rules and disable stale enabled rules
in one query.
- Keep existing policy conditions, priority, author, and creation time
when a rule changes.
- Add real PostgreSQL regression tests and a repeatable benchmark.
Document independent approval rules.

## Verification

- Red: the original regression made 2,995 policy writes across five
saves and failed the zero-write check.
- Green: 636 focused service and runtime tests passed after the rebase.
CI supplies the broad checks required by `pnpm verify` (exit 2).
- Benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-app-permission-save.ts --baseline-ref=51d77648bc62
--output=/tmp/app-permission-save.json`.
- Local benchmark: two warmups and six samples in alternating order.
Restore three agents outside each timed sample, then add the fourth
agent. Median save time fell from 458.51 ms to 38.05 ms (12.0 times
faster). Query count fell from 636 to 32. Unchanged policy writes fell
from 599 to zero.
- Compare all saved profile, entry, binding, and policy values after
mapping fixture IDs and excluding generated row IDs and write times.
Check Allowed and Ask first decisions for the added agent. Tests also
cover Off, an unassigned agent, additive access, removal, independent
policies, company boundaries, and re-enabled rules.
- CI passes on `069e2c76252a19ec3c11a061de0506eccb0463d1`: [PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/43162600023).
This includes typecheck, build, server tests, and all eight browser
shards.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/21033#issuecomment-6815608829)
gives the final head 5/5. Its one finding is resolved. The separate
Paperclip Review check is still queued. Its review task stopped after a
Runner failure.
- Hosted request timing and the specific reported approval prompt are
not verified. No live permissions changed.

## Risks

- The save now retains unchanged row IDs and timestamps. Consumers must
use permission values to determine access.
- Existing transaction locks and permission checks remain in place. The
runtime concurrency tests exercise these paths.
- The benchmark measures a local service with synthetic data. It does
not establish hosted endpoint latency.
- No migration or startup change is required.

## Model Used

OpenAI Codex, GPT-6. Exact provider model ID is not exposed in this
session. Used reasoning, tool calls, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:57:03 -06:00
DottaandPaperclip 6ad4529c5a fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.

## Linked Issues or Issue Description

**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.

**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.

**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.

Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.

## What Changed

- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.

## Verification

- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit c8123cc35d.
- The database case saves two interrupted claims, reads their persisted
state, checks a different Runner is refused, and publishes verified
admission for the original Runner.
- Negative cases cover missing proof, unknown state, a changed receipt,
a different source run, a different Runner, and a different lease.
Existing company and claim fences also pass.
- `pnpm verify` returned exit 2 because the execution guide requires
broader CI. Required CI must cover the final commit.
- Local server typecheck was killed with SIGKILL, including its
single-thread retry. This is an environment gap. Required CI must
provide typecheck and build results.
- Repository preflight installed dependencies and built shared packages.
It could not finish Runner preparation because cargo is absent.
- A live OpenCode restart workflow and recovery of the original task are
not verified. No task replay or deployment was performed.

## Risks

- The added profile field carries historical evidence. It does not mark
the current claim verified. Its use is limited to an unchanged receipt
with the normal identity checks.
- Existing pending or unknown incidents without this evidence cannot
gain proof from this fix. They keep their normal recovery path.
- No provider attempt limit, approval gate, workspace hold, or company
boundary changes.

## Model Used

OpenAI GPT-6, through Codex, with code search, tool use, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:57 -06:00
DottaandPaperclip b0447f3c76 fix: restrict chat task sponsors to self or admins (#21218)
## Thinking Path

> - Paperclip manages AI agents and their authority to do work.
> - External chat channels can create tasks for those agents.
> - A task sponsor is the person responsible for that work.
> - GitHub bot settings accepted another member as a sponsor from any
connection manager.
> - Connection management must not grant authority to act as another
person.
> - This change checks each sponsor on the server and limits the picker
choices.
> - The benefit is that ordinary members cannot assign work to another
person's identity.

## Linked Issues or Issue Description

**What happened?**

A connection manager could save another member as a GitHub guest sponsor
or automatic-task responsible member. The server checked membership but
did not check authority to select that person.

**Expected behavior**

An ordinary manager can select only themselves. A company owner, company
admin, or instance admin can select another active non-viewer member of
the same company. The picker shows member avatars.

**Steps to reproduce**

1. Sign in as an ordinary member with connection-management permission.
2. Open a GitHub bot's Access settings.
3. Set another member as a guest sponsor or responsible member.
4. Save. The old server accepts the request. This change returns 403.

**Paperclip version or commit**

Base: `9a9cd20d91edc93dba4e2295e42f0f1f9fcd4862`.

**Deployment mode**

Built from source. The checks use local PostgreSQL and an assembled
authenticated API server.

Related work: #17737 adds delegated connection configuration. #16163
separates author selection from guest access. This fix preserves both
contracts. A search found no duplicate sponsor-permission fix.

## What Changed

- Check every submitted sponsor in the configuration transaction. Lock
the actor's membership and any required admin role through the save.
- Require self-selection unless the actor has a current admin role.
Check all target members in the same company.
- Keep delegated connection checks, authority notifications, tool
updates, and audit attribution.
- Use avatar pickers for guest sponsorship and automatic-task
responsibility. Default new guests to the signed-in member.
- Test service and HTTP denials, admin selection, role removal, invalid
targets, saved settings, and picker choices.
- Test forged sponsor changes with real agent API keys and run
credentials. Update the existing access-map policy.

## Verification

- 67 sponsor, HTTP, picker, and settings tests pass. The picker test
includes 600 members and bounds member lookups.
- 93 existing GitHub management, event, policy, and tool tests pass.
- The assembled connection-management fixture passes. Self-save
succeeds. Both credential types receive 403 for forged guest and
responsible-member sponsors. The saved revision stays unchanged.
- UI typecheck and production build pass. Token gates, map checks,
query-error policy, migration checks, and whitespace checks pass.
- Two broad chat-test shards pass: 720 cases. The local full-suite run
was then stopped; the complete local suite has no final verdict.
Current-head CI remains the full verification path.
- Full local typecheck and build reach Rust but cannot finish because
Cargo is absent.
- CI found the explicit retry override in the access query. Removed it;
all 45 affected UI tests pass and the CI policy gate passed on the next
commit.
- Historical head `4146c89f9345efd38a69739d17e3194a7cf91980` received
Greptile 5/5 and passed full CI:
https://github.com/paperclipai/paperclip/actions/runs/43330028028 .
Those results apply to that earlier head.
- Current head `890df0322f382bb8ccc0fd3d59b25ba1e0299c58` merges master
without conflicts. All eight changed files match the earlier reviewed
fix. The 67 focused sponsor, HTTP, picker, and settings tests pass
again. The production-bearer fixture also passes for both agent-key and
run-token requests. UI typecheck, token gates, map checks, query-error
policy, and whitespace checks pass.
- Greptile gives current head `890df0322f382bb8ccc0fd3d59b25ba1e0299c58`
5/5 with no open review threads. All checks for this head are complete
with no failures. The PR workflow passed:
https://github.com/paperclipai/paperclip/actions/runs/43363400247 . This
includes build, typecheck, general tests, serialized server suites,
Runner checks, policy checks, and all eight E2E shards. The Snyk status
also passed.
- `pnpm verify --explain` returns exit 2 because the access-map change
selects broad CI. The focused local checks pass. Broad checks are
assigned to the CI run above.
- The first broad checks inspected upstream migrations before the merge
commit existed and stopped at migration classification. After the merge
commit, the migration checks pass with zero new migrations.
- Earlier saved-session broad checks stopped because Cargo was absent or
memory was exhausted. These gaps are retained here. No live
external-provider delivery or staging deployment is claimed.

The configuration service owns sponsor authorization because both board
and delegated API requests use it. No provider protocol or execution
contract changes.

## Risks

- An ordinary manager cannot save a configuration that names another
sponsor, even when that sponsor is unchanged. They must select
themselves or have an admin save it.
- Membership and admin rows stay locked through the configuration write.
A concurrent role change can delay the save.
- Existing configurations are not rewritten. Each new save must pass the
current checks.
- Generic channel and AgentMail creation derive the sponsor from the
authenticated user. Their strict input schemas do not accept sponsor
overrides.

## Model Used

OpenAI Codex, based on GPT-6. The exact runtime model ID and
context-window size are not exposed in this session. Used reasoning,
repository inspection, tool calls, code execution, and tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this fix does not
duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue with the bug-template field labels
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My published branch name describes the change and contains no
internal task details
- [x] I have run the focused tests locally and they pass
- [x] I have added relevant tests
- [x] I have updated the relevant access policy documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green for the current head
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:13 -06:00
DottaandPaperclip 08be8be414 fix(recovery): report exact retained sandbox resumes as resumed (#21259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A controller restart must keep the same remote execution owner.
> - A retained sandbox lease stores the result of its first acquisition.
> - That result can say replacement after the same allocation resumes.
> - The Runner then rejects a valid resume as a changed sandbox.
> - This pull request reports the verified operation as resumed.
> - The saved acquisition history and identity checks stay intact.

## Linked Issues or Issue Description

Refs #18934. That change makes the retained sandbox ready before
preparation. This change corrects the acquisition receipt after that
resume.

**What happened?**
A remote native run resumed the exact retained sandbox after a
controller restart. The lease still carried a historical replacement
receipt. Runner verification rejected it with
`native_remote_recovery_lease_mismatch`, stage `runner_verification`,
field `replacement_sandbox`.

**What did you expect to happen?**
Report the verified exact-allocation operation as resumed. Preserve the
original saved history. Reject a changed, absent or expired allocation.

**Steps to reproduce**
Save an active lease with historical replacement metadata. Resume that
exact provider allocation through `acquireForRun`. Resolve its execution
target. Before the fix, the target still reports replacement and Runner
verification refuses it.

**Deployment mode**
Native Runner on a remote sandbox. The reported provider was Daytona.

## What Changed

- Return a separate operation receipt after the exact retained
allocation passes the existing resume checks.
- Pass the receipt through heartbeat preparation and apply it only to
the execution target.
- Keep the saved lease metadata and workspace unchanged.
- Test historical created, replacement and resumed receipts. Test
ordinary and remote-authority realization through the real execution
target resolver. Check every saved metadata write.
- Document the difference between saved acquisition history and the
current resume operation.

## Verification

- Head: `74d03e8ddadd1274891bea99a10455795c500557`.
- Red: three historical receipt cases fail without the resume receipt.
- Review regression red: realization writes resumed over the historical
replacement receipt. The saved metadata loses the previous allocation
and cause.
- Green: all 69 orchestrator tests pass. Ordinary and remote-authority
realization preserve history in every metadata write and return resumed
in the target.
- Green: 135 focused orchestrator and remote controller adoption cases
pass. Identity and changed-allocation refusals remain effective.
- Server typecheck passes, including its TypeScript build dependencies
and test layers.
- `git diff --check` passes.
- `pnpm verify` exits 2 because the execution-semantics document needs
broad CI. Required CI supplies that coverage.
- Linux x64, Node 25.9.0. Provider boundaries are simulated.
- The first post-merge attempt could not collect tests because the
compiled plugin SDK lacked the new taskLease schema. Shared and plugin
SDK builds passed, then the unchanged selection passed.
- Full preflight stopped because Cargo is absent. Dependency
installation completed. The later server TypeScript build and typecheck
passed.
- Live retained Runner reattachment is not verified. This workspace has
no provider credentials. No rollout or original-task recovery by this
patch is claimed.
- Required current-head CI and automated review remain pending.

## Risks

A wrong receipt could hide an actual replacement. The projection occurs
only after the existing company, task, run, agent, workspace, provider
allocation, expiry and directory checks pass. The provider must resume
the exact allocation. Runner process and identity verification still
runs. No replacement is acquired and no saved acquisition history is
changed. The heartbeat change only carries the typed receipt between the
owning orchestrator stages. Callers without a receipt keep their
existing behavior.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution and tool use. The exact backend snapshot and context window
are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:55 -06:00
DottaandPaperclip 2b1d0fc825 fix(connectors): keep result classification alive during run progress (#20763)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - App connectors screen returned content before they give it to an
agent.
> - The model screen checks run identity and reserves its budget in one
transaction.
> - That transaction took a repeatable-read snapshot before it locked
the run.
> - A run event could change the row first and cause all four attempts
to fail.
> - This pull request uses READ COMMITTED for this admission under the
existing locks.
> - The benefit is that connector results can pass screening while run
progress continues.

## Linked Issues or Issue Description

Refs #20385 and #20484. These changes fixed other run accounting paths.
They did not change classifier admission.

**What happened?**

A connector read can return the accounting contention error before the
model screen starts. A real gateway test reproduces SQLSTATE `40001` at
`prompt-injection-prevention.ts` when it reads `heartbeat_runs FOR
SHARE`. Each attempt commits a run event after the transaction starts
its snapshot.

**Expected behavior**

Return screened connector content and save its classifier charge while
run events continue. Refuse paid classification if the run stops, its
identity changes, or access or budget checks fail.

**Steps to reproduce**

Run this test with the production source from `9574065a28`:

`cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts -t 'returns a screened
connector result'`

The test fails after four `40001` errors. It passes with this change.

## What Changed

- Use READ COMMITTED only for prompt-injection admission. Keep company,
project, configuration, agent, run and secret share locks.
- Keep classifier settlement at repeatable read. Keep provider calls
outside transaction retries.
- Test screened results, stored charges and settled holds under run
progress. Test four concurrent reads and refusal after a run stops or
changes identity.
- Document the transaction boundary.

### Hypothesis and thesis for a fix

**Hypothesis:** the classifier admission snapshot becomes stale before
its run share lock. The gateway regression confirms this cause in
current source. The historical connector error alone does not identify
its SQL statement.

**Thesis for a fix:** read current run state under the existing locks at
READ COMMITTED. This removes stale snapshots caused by run events. The
change belongs to classifier admission because every screened connector
result uses this service.

Investigation source: [Braintrust
pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/d5369c32-4c0f-43a1-af94-a01fef0a697b).
Access is required to read the traces.

**Proof:** baseline `9574065a28` failed with `40001` after four
attempts. The same single-read regression passed after the change. The
full focused suites then passed, including four concurrent reads, stored
charges and refusal controls.

## Verification

- Real PostgreSQL gateway regression: baseline red in 28.07 seconds;
candidate green in 23.16 seconds.
- `cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts
src/__tests__/prompt-injection-prevention.test.ts`: 54 tests passed.
After sync with master `1ac8b6a711`, all 54 passed again in 34.46
seconds.
- The tests use the real gateway, content guards, classifier service and
database. The MCP server and paid model are simulated. This is not a
live deployment test.
- Existing controls cover company boundaries, budgets, credentials,
configuration changes, interruption and receipt recovery.
- Staging health reported `16024ffa2`. Its classifier and accounting
source matches the reproduced baseline and includes #20385 and #20484.
The revision at the earlier failure times remains unknown.
- `pnpm verify` selected 2,512 of 3,303 test files because its
dependency policy retained 2,510 files conservatively. The broad local
run was stopped after the selection became clear. It is not a pass. The
focused 54-test result above is the local proof; required CI supplies
broad coverage. An earlier selection was also invalidated by a
concurrent commit.
- Full preflight could not finish because this host has no Cargo. Pinned
dependencies and shared/plugin SDK builds completed. The focused tests
passed without Rust.
- [Greptile reviewed this head at
5/5](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790238737).
No inline thread is open. [Paperclip completed its assessment at 5/5
with no
findings](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790312753).
[Required CI passed for head
`643bbe51faa70dcb17f372b9e85e3cbbcb724b9d`](https://github.com/paperclipai/paperclip/actions/runs/42981473514),
including typecheck, build, 12 server test shards, nine serialized
suites, eight E2E shards and Runner checks. Snyk and Socket passed. The
full inventory contains 58 completed check runs; the neutral Paperclip
check is a completed 5/5 assessment, not an incomplete review.

## Risks

- READ COMMITTED uses a new snapshot for each statement. Existing
admission locks still protect the run status, identity, configuration
and secret checks. Budget checks and reservation writes still commit
together.
- Concurrent independent admissions retain existing budget behavior.
This change adds no retry of paid provider work.
- No schema, API or saved receipt format changes.
- Historical failing SQL is unavailable. The source regression proves
this failure path; it does not prove that every historical example has
the same cause.
- The change is unmerged and not deployed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment model ID and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:37 -06:00
DottaandClaude Opus 5.5 cfa959add7 refactor(slack): correct stale status comment and add production acceptance checklist (#20840)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Slack chat connector lets people talk to their agents in Slack
> - The connector is close to production, but its live evidence is out
of date
> - The September live results use old revisions, and the October setup
run did not test follow-up, reauthorization, or removal
> - Cloud managed setup has never run against real Slack, and its docs
say not to enable it until it has
> - This pull request adds one checklist for a fresh live pass on
current master, and corrects one stale code comment
> - The benefit is a clear finish line for Slack production readiness on
self-hosted and Cloud

## Linked Issues or Issue Description

**Issue type**
Missing documentation.

**Where is the issue?**
`doc/connections/SLACK-AUTOMATIC-SETUP.md`,
`doc/plans/chat-adapters/2026-09-05-slack-live-qualification-result.md`,
`doc/testing/slack-cloud-qualification.md`, and
`server/src/services/chat-channels.ts`.

**What's wrong?**
The Slack live qualification is spread over three documents. Their
evidence uses older revisions. No document lists what a production pass
must cover for Cloud managed setup. A code comment says the manifest
does not request `assistant:write`, but the manifest does request it.

**Suggested fix**
Add one dated acceptance checklist with pass conditions per row. Correct
the comment.

## What Changed

- Added `doc/plans/2026-11-11-slack-production-acceptance.md`. It has
sections for self-hosted automatic setup, manual and existing apps,
conversation behavior, Cloud managed setup on staging, and distribution
questions (Marketplace-only workspaces, non-Marketplace rate limits,
scope justification, zero-copy).
- Corrected the comment in `server/src/services/chat-channels.ts`. It
now says that Slack shows the working state through
`agents.sessions.setStatus` and the lifecycle reply. No code behavior
changed.

## Verification

- Documentation and a comment only. No runtime behavior changed.
- I checked the doc links and the referenced files against current
master.

## Risks

- Low risk. The change is documentation and one comment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
web search.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:54:50 -06:00
DottaandClaude Opus 5.5 c6b497d831 perf(auth): authenticate a run's agent JWT in three statements (#21434)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each running agent calls the Paperclip API with a signed run token
(an agent JWT)
> - The server authenticates each of these requests before the route
handler starts
> - This authentication sent 13 statements to the database for each
request, and none of them were cached
> - One server process starts to struggle at about 50 concurrent agents,
and the target is 1,000
> - This pull request cuts the work before the route handler to 3
statements, with no transaction and no row locks in the usual case
> - The benefit is less database work and less event-loop work for each
agent request, with the same security checks on each request

## Linked Issues or Issue Description

Refs #16577 (trimmed saved run JSON from agent authentication)
Refs #17931 (scoped run identity reads to the needs of each operation)

**What existing behavior does this improve?**
The authentication of an API request that an agent makes with its run
JWT. Before the route handler starts, `actorMiddleware` did 13 database
statements in sequence: a board key lookup, an agent key lookup, a
`select *` of the agent, the run read, a 7-statement locked identity
capture (BEGIN, run task read, task row lock, run row lock and wide row
read, pending steering read, context read, COMMIT), and two reads for
the responsible user.

**Subsystem affected**
Server: `server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts` and
`server/src/services/run-identity.ts`.

**Current behavior**
Each agent request costs 13 statements. The identity capture is a
transaction that holds a pooled connection and takes `FOR NO KEY UPDATE`
locks on the task row and the run row. These are the same rows that
event appends and steering lock, so agent requests wait behind them.

**Proposed behavior**
Each agent request costs 3 statements and no transaction: the agent
status columns, one run statement, and one membership statement. A run
with a pending steering context still uses the locked capture, so
steering acceptance does not change.

**Reason and benefit**
Item 1c of the thousand-concurrent-agents plan. The load model at 50
agents (API requests only) shows the database statements for this path
go from 165.7/s to 30.1/s, transactions from 15.1/s to 0, and event-loop
utilization from 0.74 to 0.25.

**Breaking changes**
None. The actor shape that routes get does not change. No route handler
changes. No schema change.

## What Changed

- **Skip key lookups for a verified run JWT.** The middleware verifies
the JWT first (HMAC only, no database). Every stored board key and agent
key is `pcp_` followed by letters, `_` and hex (`createBoardApiToken`,
`createToken`). A verified JWT has three parts with dots. So its SHA-256
cannot equal a stored key hash, and both lookups always returned no row
for it. A token that does not verify still tries the board key, then the
agent key, in the same order as before.
- **Read only the agent columns that authentication checks.**
`companyId`, `status` and `lifecycleHolds` replace `select *`. The agent
configuration JSON stays in the database. The agent-key path uses the
same columns.
- **Read revocation and identity in one statement.**
`loadAgentRunAuthentication` now also returns the run task ids, a
`pendingIdentityContext` flag (`exists` on a pending context of this
run), and the active context `id` and `responsibleUserId` (only when
that context belongs to this run). The new `resolveRequestRunIdentity`
in `run-identity.ts` uses these columns:
- The run is not running, or it has no active context: it keeps the run
row values, as before.
- A pending context exists: it calls `captureRunIdentity(...,
"request_authentication")`, as before. This keeps the locked
accept-or-refuse step and the 409 for an unacknowledged steer.
- Otherwise: it checks the run task ids with the same rule as
`lockIdentityTask` (now `assertRunTaskIdentity`), and returns the
context from the statement.
- **Reuse the run row for the legacy responsible user.** The legacy
lookup had the same filter as the run read. The middleware now uses the
`responsibleUserId` from that read, with the same non-UUID guard.
- **One statement for memberships.** The user existence check is an
`exists` condition in the membership statement, in place of a second
parallel statement.
- No in-process memo and no time-based cache. Every request reads the
current run row.

### Why the request-authentication read does not need the locks or the
transaction

The capture locks are released at COMMIT, before the route handler
starts. So they do not protect anything that the route does. Inside
authentication, they have two effects:

1. **They protect the acceptance write when a pending context exists.**
This PR keeps this case on the locked capture.
2. **They make the read wait for a steering transaction that is in
progress.** In PostgreSQL READ COMMITTED, one statement reads one
snapshot. So the run status, the cancellation fields, the active
context, its responsible user and the pending flag are all from the same
committed state. Each interleaving with a steering writer gives a result
that a serial order also gives:

| Writer state when the statement runs | What the statement sees |
Result | Same as a serial order? |
| --- | --- | --- | --- |
| Reservation (`reserveSteeredIdentity`) not committed | Old active
context, no pending | Old identity, 200 | Yes: request before the
reservation. The old identity stays active until the steer is accepted.
|
| Reservation committed | Pending flag | Locked capture: 409, or accept
if acknowledged | Same code path as before |
| Acceptance in progress (pending committed) | Pending flag | Locked
capture waits for the acceptance, then returns the new identity | Same
code path as before |
| Acceptance committed | New active context, no pending | New identity |
Yes: request after the acceptance |
| Cancel or supersede committed | `cancelled`,
`executionCancellation.state = requested`, or
`freshContinuationSupersededAt` | 403 on a write | Yes, on the very next
request |

`activate()` writes `activeIdentityContextId` and `responsibleUserId` in
one update, and the statement joins the context by id and run id. So a
mix of an old context with a new user is not possible. The tests show
the first and third rows on real PostgreSQL.

### Notes for reviewers (moved from code comments)

- The two correlated subqueries in `requestRunIdentityFields` name
`"heartbeat_runs"` and an alias for `run_identity_contexts` in raw SQL.
Drizzle writes the columns of a one-table select without the table name,
so a `${heartbeatRuns.id}` inside the subquery would resolve to the
inner table. The first test run showed this (the identity came back
empty), and the statement-count test catches it.
- The `exists` condition in `loadResponsibleUserMemberships` keeps the
old rule: a membership counts only while its user exists.
- The legacy responsible-user value comes from the run read, because
that read has the same filter (`id`, `company_id`, `agent_id`) as the
removed legacy lookup.

## Verification

- New `server/src/__tests__/agent-jwt-request-authentication.test.ts`
(embedded PostgreSQL, real `actorMiddleware`,
`meterDatabaseStatements`):
- Statement count: 3 statements, 0 transactions, 3 waits for a running
agent with an active identity and a responsible user. Before: 13
statements. The test also checks that fewer than 8 KiB come back, with a
120 KB agent configuration.
- Revocation: a write that passes is refused on the very next request
after `status = cancelled`, after a Stop request
(`executionCancellation.state = requested`), and after a
fresh-continuation supersede (`agent_run_superseded`).
- Identity change between two requests: the next request uses the new
context and the new responsible user.
- Uncommitted steering reservation: the request does not wait, and keeps
the committed identity. After commit, the next request gets 409. After
the acknowledgement, the next request gets the new identity.
- Acceptance in progress: the request waits for it (locked capture),
then returns the new identity.
- Wrong claims: an agent token signed for another company gets 401. A
token that names another agent's run gets no identity from that run. An
active context id of another run is not used.
  - Pending steering without an acknowledgement still gets 409.
- Red/green: on `origin/master` code, the statement-count test failed
with 13 statements, and the uncommitted-reservation test failed because
the request was blocked on the row lock. Both pass with this change.
- `agent-auth-identity-transfer.test.ts`: the heartbeat_runs read count
is now exactly 1 (it was a lower bound of 3).
- These pass: `agent-auth-middleware.test.ts`,
`agent-auth-identity-transfer.test.ts`,
`agent-run-authentication.test.ts`, `run-identity.test.ts`.
- `pnpm verify`: it selected 2,639 of 3,459 Vitest files, because
`auth.ts` is imported by most route tests. That is close to the full
suite, so I stopped it at 30 minutes on this busy machine (exit 124, no
failures before the stop). CI supplies the broad coverage. The focused
checks above pass on the final revision (41 tests).
- `pnpm check:module-boundaries`: passed. Server `tsc --noEmit`: passed.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from the plan
branch, not committed), `AGENTS=50 SECONDS=20`, one run at a time. The
machine had a load average of 25 to 33 from other work, so the timings
are noisy. The statement and transaction counts are not affected by the
noise.

| `SOURCES=api` | Before | After |
| --- | --- | --- |
| Statements/s (15 requests/s) | 165.7 | 30.1 |
| Statements per request (this model has no responsible user) | 11 | 2 |
| Transactions/s | 15.1 | 0 |
| Event-loop utilization | 0.741 | 0.252 |
| Event-loop delay p99 | 267.8 ms | 19.0 ms |
| API lateness p50 / p99 | 1104 / 5631 ms | 6.7 / 79 ms |

| All sources | Before | After |
| --- | --- | --- |
| API requests done | 300/300 | 300/300 |
| API lateness p50 / p99 | 291 / 1130 ms | 230 / 2681 ms |
| Delta appends done | 14977/20000 | 8260/20000 |

The all-sources run is limited by delta appends, which this PR does not
change. Its results move a lot between runs on this busy machine (the
delta count is lower after, with no change to that path), so it does not
show an effect of this PR in either direction.

## Risks

- The usual path no longer waits for a steering reservation that is in
progress. The request then uses the identity that is committed and
active. This is the identity that the run acts under until the steer is
accepted. The table above shows that each interleaving matches a serial
order.
- `loadAgentRunAuthentication` returns more columns (two small
correlated subqueries on `run_identity_contexts`, by primary key and by
the `(run_id, ...)` index). Other callers get extra fields and ignore
them.
- The JWT-first order depends on the stored key format. If a new key
format can contain three dot-separated base64url parts that verify as a
JWT, this order must be checked again.
- Low risk for routes: the actor object is the same, and no route
handler changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:38:01 -06:00
DottaandClaude Opus 5.5 16f00d62fb fix(connections): answer setup questions with a setup link in ranked results too (#21428)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21400 told the agent to answer a setup question ("What is the
easiest way to connect Notion here?") with a setup-page link, because
`connection_request` ends the turn before any answer text is shown
> - That rule covered only a single search match
> - In a paid eval run the agent searched "Notion integration connect
workspace", got ranked matches, raised the card and ended with no answer
> - This pull request adds the same rule to the ranked-matches
instruction
> - The benefit is that a setup question gets a written answer with a
working setup link, whichever search the agent runs

## Linked Issues or Issue Description

Refs #21400. This commit was pushed to #21400 after it merged, so it is
a follow-up.

**What happened?**
Paid run 43455387724, case `connection-setup.choose-provider`: the agent
called `connections_search` with a multi-word query, got the
ranked-matches instruction ("Use connection_request when human setup or
access is required"), called `connection_request`, and the turn ended as
"Waiting for Connect Notion" with no text reply.

**Expected behavior**
For a setup question, the agent answers in plain words and links
`/<prefix>/apps/connect?source=<app>`. For a data request, it raises the
setup card.

**Steps to reproduce**
1. Have no Notion connection.
2. Ask "What is the easiest way to connect Notion here?" and let the
agent search with a multi-word query.

## What Changed

- `server/src/services/connection-intents.ts`: the ranked-matches
instruction says to use `connection_request` for data or actions the
person asked for, and to answer a setup question with the app's
setup-page link instead. It also says not to create a connection or
draft unless the person asked for setup.
- Test: the ranked instruction includes the setup-page rule.

## Verification

- `cd server && npx vitest run
src/__tests__/connection-intents-service.test.ts -t "own-server"`.
- Paid run 43460401711 on a branch with this change:
`connection-setup.choose-provider` passed (answer with `[Connect
Notion](/COM/apps/connect?source=notion)`),
`connection-setup.notion-declined` passed.

## Risks

- About 60 more words in one tool result. No effect on agent boot or
time to first token.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:36:59 -06:00
DottaandClaude Opus 5.5 89e6575418 test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs get their tools from a session tool authority,
which holds the run's assigned app (MCP) tool catalog
> - #21415 releases a finished session's authority, so the server heap
no longer grows with each finished session
> - Two behaviors of that fix had no test: a released session fails
closed, and the release when a follow-up run retires its warm
predecessor and then fails before it admits its own authority
> - The memory saving was measured once, by hand, with no script in the
repository
> - This pull request adds the two tests and a benchmark script. It
changes no product code
> - The benefit is that a later change cannot silently bring back the
leak or let a released session reach its tools

## Linked Issues or Issue Description

Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate;
a reviewer comment there named the follow-up failure path).

**What is missing?**

#21415 tests that finished sessions return the authority map to its
starting size. No test checks that a released session refuses late tool
calls. No test checks the follow-up path, where a follow-up run retires
the previous warm session (so the release at forget time is skipped
while the follow-up owns the scope) and then fails before it admits its
own authority.

**Expected behavior**

A late tool call or completion feedback for a released session fails
with `native_session_tool_authority_unavailable`. The next run of the
same scope gets a new authority with only its own assignments. When a
follow-up fails after it retires its predecessor, the authority map is
back at its starting size after that run settles.

**Paperclip version or commit**

master `edb6945ff3` (after #21415, `85f4a91e70`).

## What Changed

- Test "refuses a released session's late tool call and completion
feedback and admits the next run with only its own assignments".
- Test "releases a retired warm predecessor's tool authority when the
follow-up fails before admitting its own". A follow-up of the same warm
scope with changed configuration retires the predecessor. Then its
assigned tool listing fails. The test asserts that the predecessor was
closed, that the follow-up failed with that error, and that
`sessionToolAuthorityCount()` is back at its starting value.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained
bytes of one assigned tool catalog per session, and schema copies per
`definitions()` call.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "released
native session tool authority|retired warm predecessor"`: 2 passed.
- Red check: with the release at settle removed from
`native-session-executor.ts`, both tests fail (`expected 1 to be +0`,
and the late tool call is not refused).
- Retained bytes per finished per-turn session were measured through
`executePaperclipNativeSession` with the real `createAssignedMcpTools`
(80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On
`85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794
bytes (test-harness noise), and the authority map stayed at its starting
size.
- `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes
2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per
assigned tool catalog. Each `definitions()` call makes 80 new schema
copies (about 296 KB), and none are kept.
- Full `native-session-executor.test.ts` locally, with a load average
near 78 from other work on the machine: 1,483 passed and 5 failed. All 5
failures were timeouts in tests this change does not touch (the slowest
took 163 s). CI gives the clean full-suite result.

## Risks

- Low risk. Tests and one benchmark script only. No product code
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:17:25 -06:00
DottaandClaude Opus 5.5 df1b12ad8e fix(runner): advertise flat tool schemas and stop repeated runner-side input rejections (#21398)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools. The runner shows each tool to
the model with an input schema and checks every call against that schema
before it runs
> - Some tool schemas had a root `oneOf`, `anyOf`, `allOf` or
`if`/`then`. `handoff_case` had no top-level `properties` at all
> - Some model hosts convert such a schema into an empty argument list.
The model then calls the tool with no arguments, and the runner rejects
the call as "Invalid tool input"
> - The repeated-failure guard from #21360 does not see these
rejections, because they happen in the runner before the call reaches
the control plane. The model sent the same rejected call again and again
> - This pull request gives every agent-facing tool a plain object
schema with named properties, checks the old cross-field rules with
named errors, and adds a runner-side guard for identical "Invalid tool
input" rejections
> - The benefit is that models on these hosts can see and send the
arguments, and a dropped argument ends in a clear correction or a stop,
not a long loop

## Linked Issues or Issue Description

**What happened?**
In a chat eval cohort with OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017`, the hosts Wafer and
DigitalOcean served most model calls. In the case handoff case,
`handoff_case` arrived with no arguments on all 10 calls across two
cases. The runner rejected each one with "must have required property
idempotencyKey, case, toAgentId, ...". The model's reasoning said that
the tool showed an empty property list. The model fell back to
`call_api`, but the reply waiter had already expired, so the case
failed. In the preserve-draft case, `request_human_input` was rejected 9
times in a row with the same root `then` error.

**Expected behavior**
Every tool that an agent sees has a plain object schema with named
properties, so every host can show its arguments. When the same call is
rejected twice with the same arguments, the model gets a correction that
lists the expected properties. On the third identical call, the model is
told to change the arguments or stop.

**Paperclip version or commit**
`master` at f7ddab8e94.

**Steps to reproduce**
1. Run an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, where OpenRouter routes
to Wafer or DigitalOcean.
2. Ask the agent to hand a case to another agent with `handoff_case`.
3. See `handoff_case` arrive with no arguments and get rejected on every
call.

## What Changed

- Add `providerToolSchemaShapeIssues`
(`packages/paperclip-runner/src/contracts/provider-tool-schema-shape.ts`).
It reports a root that is not `type: "object"`, has no `properties`, or
uses `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then`, `else`,
`dependent*` or `$ref`.
- Add inventory tests that run this check on every agent-facing tool:
the live semantic catalog, the semantic action catalog, the Codex tool
specs, `paperclip_finish` and `paperclip_block` (runner), and the
server-only tool definitions (server). Before this change, the runner
test listed 16 entries for 8 tools: `handoff_case`,
`request_human_input`, `read_case`, `update_case`, `create_case`,
`read_workflow`, `update_workflow` and `paperclip_finish`.
- Flatten these 8 schemas. `handoff_case` is now one object with `to`
(optional, `"done"`), and `case` is one object with optional `id` and
`create`. Descriptions on the properties state which fields each variant
needs.
- Add `toolInputVariantError`
(`packages/paperclip-runner/src/contracts/tool-input-variants.ts`). It
checks the former cross-field rules and returns an error that names what
is wrong and how to send the call correctly. For example: `Invalid tool
input for handoff_case: An agent handoff requires toAgentId, phase. To
finish an existing case instead, send to: "done" with
expectedHandoffVersion and expectedIssueStatusVersion.`
- The server runs this check first in
`PaperclipRunnerToolAuthority.execute` and returns `422`. The #21360
guard therefore counts these errors too. Assigned MCP tools and
connector tools are not checked. The runner semantic dispatchers run the
same check after schema validation. The runner MCP bridges run it for
`paperclip_finish`, which never reaches the control plane.
- Add `RepeatedToolInputRejectionGuard`
(`packages/paperclip-runner/src/drivers/repeated-tool-input-rejection.ts`)
and use it in the OpenCode MCP bridge and the runner tool bridge, where
runner-side validation happens. The key is the tool, the canonical
arguments and the error text, separated per MCP session. The second
identical rejection adds the arguments that arrived and the expected
required and optional properties. The third and later identical calls
say that the call was not accepted and that the model must change the
arguments or stop and report the blocker. State is in memory, per
bridge, and capped at 64 entries.
- Regenerate `semantic-tool-contracts.json`,
`semantic-action-catalog.json`, `protocol/manifest.json` and the seeded
eval fixture.
- Add `spec/fixtures/pre-flat-tool-input-schemas.json`: the 8 strict
schemas from the previous release, for the upgrade test.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/contracts
src/catalog src/protocol-actions src/semantic-tools
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/runner-tool-bridge.test.ts`: all pass.
- `cd server && npx vitest run
src/services/native-runtime/tool-input-variant-admission.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts`: 9
passed.
- Red/green:
- The runner inventory test fails on `master` and lists the 16 entries.
It passes after the change.
- The guard test replays the recorded empty `handoff_case` call 4 times
through the real OpenCode bridge. Without the guard, the second call has
no correction and the test fails. With the guard, the second call lists
`required idempotencyKey, case, instructions; optional to, toAgentId,
phase, ...`, and the third says the call was not accepted.
- Negative controls: changed arguments get a fresh error, and a valid
call still runs once.
- Upgrade safety: `tool-input-variants.test.ts` compiles the previous
release's strict schemas from the fixture. For a corpus of valid and
invalid inputs per tool (27 for `handoff_case`), it checks that "flat
schema accepts and `toolInputVariantError` is null" gives exactly the
same result as the old schema. Each tool has accepted and rejected
inputs.
- Generated outputs: `check:semantic-contracts`,
`check:capability-contract`, `check:protocol-coverage`,
`check:capability-inventory`, `check:protocol-manifest` and the replay
golden check pass.
- Rust tests that read the generated contract: see the CI `runner`
checks.
- Paid narrow run: see the comment below.

## Risks

- Previous-release story: tool schemas are not saved run identity. The
OpenCode bridge lists tools again when a session resumes. A durable Rust
run that is already attached keeps the tool catalog it saved, because
re-attaching the same run does not replace the catalog. Every input the
old strict schemas accepted is still accepted, and every input they
rejected is still rejected, so saved calls, receipts and replays carry
forward. No session is reset, and no profile version or provider
`commandDigest` changes. The pack `contractDigest` changes because
`generated/` changed, as it does for any tool contract change.
- A rejected variant now arrives later for the semantic tools: the
server returns `422` instead of the runner bridge rejecting it. The
error text names the problem.
- For `paperclip_finish` on Codex, the advertised schema no longer has
the root conditional. The Rust bridge accepts a `done` result with a
continuation object, and canonical result validation still rejects it,
as before.
- Latency: the advertised schemas of the 8 tools are 1,876 bytes smaller
(23,785 to 21,909), about 470 fewer prompt tokens when all are listed.
Compiling their validators in the bridge takes 164 ms instead of 214 ms
(median of 15). The guard adds one SHA-256 of the canonical arguments
for each rejected call only. Nothing is added before turn start.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:08:58 -06:00
DottaandClaude Opus 5.5 64ec273845 perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the trusted PR workflow, and the `Typecheck
+ Release Registry` job is one of its required checks
> - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner
(run 43344818387), against a target of 7 minutes or less for every job
> - The job runs six steps one after another, and the first step
type-checks the whole server program a second time, at about 13.5 GiB,
which makes the runner swap
> - Typecheck results are the check that authors most want back quickly,
so this job is on the critical path of most pull requests
> - This pull request removes the duplicate server type check and runs
the independent checks in parallel through one small script, inside the
same memory budget
> - The benefit is about 39% less wall time for the job's checks on the
same machine, with peak memory down from about 14 GiB (plus swap) to
about 8.5 GiB

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The `Typecheck + Release Registry` job in
`.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The
target for every job in the workflow is 7 minutes or less.

**Subsystem affected**

CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under
`scripts/` and `server/scripts/`.

**Current behavior**

After about 75 s of setup, the job runs six steps in sequence on a 4
vCPU runner, about 470 s in all (run 43344818387):
`typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s,
test projects 38 s, CLI guidance test 5 s, release registry tests 95 s.
Most of these steps use one thread. The `typecheck:build-gaps` step
builds the server with a full type check of the whole server program.
The server layer step then checks the same files again, and the `Build`
job checks them a third time. That full program peaks at about 13.5 GiB,
so the job peaks at about 14 GiB of 16 GiB and uses swap.

**Proposed behavior**

The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the
server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker
image does) and still checks the runtime assets. It runs the steps that
write shared files first, one at a time. Then it runs the remaining
checks in parallel, one per CPU, with the server declaration layers
first, because the test layers and test projects read them. The checks
that are not on that path run at a lower CPU priority.

**Reason and benefit**

On the same runner, the job's checks take 61% of the time and about 60%
of the memory (see Verification). The job should take about 6 minutes on
the fleet instead of 9.2.

**Breaking changes**

None. The same files are type-checked with the same options. Local
commands do not change. `pnpm run typecheck:build-gaps` still works as
before.

## What Changed

- `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It
writes shared outputs first, in sequence: links, plugin SDK, server
build without type checks, the runtime asset check, the sandbox provider
installs, and the Runner typecheck (it rebuilds the eval kernel, which
other checks read). Then it runs these checks in parallel: the server
declaration layers, each server test layer, each test project, the test
programs, the other build-gap workspaces, the CLI guidance test and the
release registry tests. A passing check prints its output in a collapsed
log group. A failing check prints its full output between `=== FAILED`
and `=== end of` lines, with an error annotation. The script ends with a
table of each check's start, duration and result, and exits non-zero if
any check fails.
- `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option
that checks one layer against the configs and declarations of an earlier
`--declarations-only` run, and an exported `listLayers`.
- `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and
`--skip-prepare` options, so the installs happen before the parallel
checks start.
- `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option,
so the script still finds the build-gap workspaces and still fails when
one has no `typecheck` script.
- `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry`
job runs the script instead of the six steps. No other job changes.
- Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to
`test:release-registry`), new cases in
`server/scripts/typecheck-layers.test.mjs` and
`scripts/typecheck-test-programs.test.mjs`, and the workflow wiring
assertions in `.github/scripts/tests/ci-plan.test.mjs` and
`scripts/__tests__/runner-release-crate-cache.test.mjs`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because
`package.json` changed.
- `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new
options.

## Verification

This PR's own CI ran the old job definition, because PR runs use the
workflow from master. So I measured the commands on a stand-in machine:
a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`,
now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon
6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs
[43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017),
[43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958),
[43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105),
[43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837)
and
[43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493).
Each step ran under `/usr/bin/time -v`, and a sampler read
`/proc/meminfo` every 0.25 s.

**Old steps, one at a time (3 runs, run 43445393958):**

| Step | Wall s | CPU s | Peak RSS of one process | System memory peak |
| --- | --- | --- | --- | --- |
| `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7
GiB |
| ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB |
13.7–14.7 GiB, swap up to 738 MiB |
| ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB |
3.9 GiB |
| Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB |
| Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB |
| Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB |
| CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB |
| Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB |

CPU seconds are about twice the wall seconds for the `tsc` steps even on
one checker thread. So each step uses about two of the four vCPUs.

**Old sequence and new lane on the same runner (4 runs, run
43450786493).** Each job copied one installed checkout, then ran both,
in alternating order:

| Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak |
New memory peak |
| --- | --- | --- | --- | --- | --- |
| 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB |
| 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB |
| 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB |
| 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB |

Hosted runners differ in speed by up to 1.7 times, so only the
same-runner ratio is meaningful. Both arms include a cold compile of the
Runner release crate (45–80 s) that the fleet restores from cache. The
fleet ran the old steps in 470 s; at the measured ratio, the new lane
should take about 290 s, and the job about 6 minutes with setup. The new
lane peaks at about 55% of memory, with no swap.

Local checks (on a loaded Mac, for correctness only):

- `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass.
- `node --test .github/scripts/tests/*.test.mjs` — 807 pass.
- `node --test scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass.
- `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/`
— the new cases pass.

## Risks

- This changes a required job that every PR depends on, and its own CI
cannot prove it. Please watch the first `Typecheck + Release Registry`
runs on master after merge: the wall time (expect about 6 minutes), the
summary table at the end of the step, and any `terminated by` message.
- A type error now shows in that check's `=== FAILED` block, not in a
step of its own. Passing checks are folded, so the failure is the only
expanded block.
- When the server declaration layers fail, the test layers and test
projects are reported as skipped. Before, the test layers still ran
after a source error. Fix the source errors first; the next run checks
the tests.
- The Honeycomb CI time board and any step-name queries see one step,
`Typecheck, CLI guidance safety and release registry tests`, instead of
six. The script prints the duration of each check.
- The release registry tests run beside the type checks at normal
priority. Their slowest test took 46–78 s here, against a 180 s timeout,
and 114 s on the slowest stand-in runner.
- Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB,
far below the old 14 GiB.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use (shell, GitHub CLI, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:01 -06:00
DottaandClaude Opus 5.5 b993702ae3 perf(ui): read the compact task index on the dashboard and refresh it once per window (#21413)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI keeps company lists fresh from live task activity
events
> - The dashboard, the activity feed and board chat read the full
company task list, and every burst of task activity refetches it
> - One full list request costs the server 60 statements and 61-90 MB of
allocation for 500 tasks, and the staging server is saturated by
allocation and garbage collection
> - This pull request moves those three views to the compact task index
under its own key, and task activity refreshes that key at most once per
30 seconds
> - The benefit is about one tenth of the server allocation per open
dashboard tab while agents work, with the same rows on screen

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The dashboard, the activity feed and board chat load the full company
task list. Every burst of task activity refetches that list.

**Subsystem affected**
Board UI: live updates, the dashboard, the activity feed and board chat.

**Current behavior**
The three views read the full company task list (`GET
/api/companies/:id/issues`, no view) with the key `["issues",
companyId]`. `LiveUpdatesProvider` invalidates that key on every task
activity event. Each open tab refetches the list at most every 5 seconds
while agents work. One request costs the server 60 statements and 61-90
MB of allocation for 500 tasks (local measurement). The staging server
allocates 57.6 MB per second and is saturated by garbage collection.

**Proposed behavior**
The views read only the fields they show, from the compact task index.
Task activity from other people refreshes the index at most once per
window. The viewer's own task changes refresh it at once.

**Reason and benefit**
Less server allocation and fewer statements per open tab, with the same
rows on screen.

**Breaking changes**
None. No API, schema or server change.

## What Changed

- The dashboard, the activity feed and board chat read `view=index`
under a new key `issues.listIndex(companyId)`.
- Task activity from other people refreshes that key at most once per 30
seconds (`TASK_INDEX_REFRESH_WINDOW_MS`). The viewer's own task activity
refreshes it at once. The first event after a quiet window refetches at
once. Later events in the window share one trailing refetch. Nothing
from the event goes into the cache. An unwatched index is marked stale
and refetches on the next mount. The user's own task changes still
invalidate the whole `["issues", companyId]` prefix immediately.
- The dashboard takes its ten recent tasks from the index, sorted by
updatedAt as before. It reads the status glyph fields for those ten
tasks from `issues/summaries`, because the index omits displayStatus and
blockerAttention.
- Toast and route lookups that read the full list cache also read the
index cache.
- `server/scripts/query-cost/` is the per-caller query cost harness used
for the measurements below. It records the caller of each query at
construction, the bytes allocated to build it, the rows it returns and
the statements sent.

## Verification

- `pnpm --filter @paperclipai/ui exec vitest run
src/context/LiveUpdatesProvider src/pages/Dashboard.test.ts`: 261 pass.
- Red/green: `LiveUpdatesProvider.task-index.test.ts` fails 6 of its
first 10 cases against the provider on master and passes with this
change. The tests cover a burst of events (4 index fetches in 90
seconds), the first event after a quiet window, a new task, a deleted
task, a hidden task, a status/title/priority change, an event for a task
outside the cache, an event for a task the viewer cannot read (nothing
from the event goes into the cache), the viewer's own activity
(refreshes at once), a task change made in this tab (refreshes at once
through the company list invalidation), the trailing refresh (the last
state shows when the window ends after activity stops), an unwatched
index, and the live-event batcher path.
- Other readers of `["issues", companyId]` (My Tasks, the task
properties picker, the command palette, task detail cache writes,
filtered and board-column lists) are not changed. The broad invalidation
excludes only the new index key, so they refresh as before. A hidden tab
keeps today's behavior: the live-event batcher still turns its refetches
into stale marks.
- Field parity (local, 1,500 tasks): the index returns the same 500 ids
as the full list, with 0 differences in id, identifier, title, status,
priority, createdAt, updatedAt and assigneeAgentId. It omits
displayStatus and blockerAttention on all 500 rows, so the recent rows
take those fields from summaries.
- Real browser (headless Chromium, Playwright) against a local server
from this branch, started with `test-drive --data-dir <scratch dir>`,
which sets `PAPERCLIP_HOME` and `PAPERCLIP_CONFIG` to that directory:
- Dashboard: the recent tasks and the "Tasks by Status" chart show the
new tasks. The page sent 0 full-list requests, 2 index requests and 1
summaries request.
- A task rename by this board user showed on the dashboard after 0.8
seconds, with 1 index refetch.
  - A second tab on the dashboard showed the same tasks.
- Board chat loaded with 0 full-list requests and 3 index requests. Its
activity feed showed the task names.
- Not checked in the browser: activity from another actor. An agent key
cannot edit a task without a heartbeat run context, so the window for
other people's activity is covered by the unit tests only.
- Nothing visual changed. The views keep their render code. Only their
data source changed.
- Cost per request (local, embedded PostgreSQL, Node 26, uninstrumented
pass, 5 requests each), measured:
`TSX_DISABLE_CACHE=1 UNITS=board ROUTES="issues list,issues list
view=index,issues/summaries (10 ids)" REPEAT=5 NODE_OPTIONS=--expose-gc
pnpm --filter @paperclipai/server exec tsx scripts/query-cost/drive.ts`

  | Request | Statements | Allocated |
  | --- | --- | --- |
  | full list | 60 | 61-90 MB |
  | view=index | 12 | 22-37 MB |
  | summaries, 10 ids | 23 | 2.8 MB |

  The ranges show the difference between runs.
- Estimate, not measured: per open dashboard tab with continuous task
activity, before about 13 MB/s (one full list per 5 seconds), after
about 1.3 MB/s (one index per 30 seconds and one 10-id summaries request
per 5 seconds). At the day-1 staging route rate (0.116 full lists per
second) that is about 6.5 MB/s less. The real rate of task activity
events on staging is not measured.

## Risks

- Freshness: the dashboard's task charts and recent-task order, the
activity feed's task names and board chat's task lookup can lag task
activity from other people by up to 30 seconds. The status glyphs of the
ten recent tasks stay on the 5-second lane. The viewer's own task
changes refresh at once. The window is one constant,
`TASK_INDEX_REFRESH_WINDOW_MS` in `LiveUpdatesProvider.tsx`, so a
different value is a one-line change.
- The index returns the same rows as the full list, so the charts count
the same tasks.
- The recent-task rows have no task description. The rows do not show
it.
- My Tasks, the task properties picker and the command palette still
read the full list. They are not in this change.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation describes this behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:50:24 -06:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
DottaandClaude Opus 5.5 426e8a35dc perf(runs): read live runs in a fixed number of statements (#21417)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board shows the runs that are live now. The sidebar and the
active agents panel read `GET /api/companies/:companyId/live-runs`
> - That route checked read access and built the output-silence summary
one run at a time, with 8 statements per run
> - On a hosted instance this route ran about 6 times per minute with an
average of 2.1 s of server time, and its statements and allocations add
load to a busy main thread
> - This pull request reads the access decisions and the silence state
for all runs of the request in one batch
> - The benefit is a fixed number of statements per request (15 instead
of 8 + 8 per run) and about one quarter of the allocation

## Linked Issues or Issue Description

No public issue exists. This is an enhancement.

**What existing behavior does this improve?**
The live-runs read on the board (`GET
/api/companies/:companyId/live-runs`).

**Subsystem affected**
Server: agent routes, run read authorization, active-run watchdog.

**Current behavior**
For each run, the route calls `canActorReadHeartbeatRun` (task
existence, instance-admin flag, membership, run policy, task privacy)
and `buildRunOutputSilence` (two watchdog decision reads and one
evaluation-task read). The instance-admin and membership reads repeat
the same answer for every run. Measured locally with a statement and
allocation harness:

| Live runs | Statements per request | Allocated per request |
| --- | --- | --- |
| 10 | 88 | 10.0 MB |
| 50 | 408 | 27.9 MB |
| 200 (route returns 50) | 408 | 28.6 MB |

**Proposed behavior**
The route reads every access decision in one batch and every silence
summary in one batch, so the statement count does not grow with the
number of runs.

**Reason and benefit**
Fewer statements and less allocation on a route that the board calls
often.

**Breaking changes**
None. The response is the same.

## What Changed

- `server/src/routes/agents.ts`: the live-runs handler decides reads
with the existing `getHeartbeatRunReadDecisions` (the run history list
already uses it) and builds silence summaries with one
`buildRunOutputSilences` call for the readable runs. Unreadable runs are
still redacted with `redactHeartbeatRunListRow`, and no silence state is
read for them.
- Active-run watchdog Postgres adapter: new `findLatestDecisions` and
`findOpenStaleRunEvaluations` read all runs of one company in one
statement each (`DISTINCT ON`). The single-run `findLatestDecision` and
`findOpenStaleRunEvaluation` now call the batch with one run, so both
paths use the same query.
- Watchdog application: new `createBuildRunOutputSilences` use case
groups runs by company. `createBuildRunOutputSilence` now calls it with
one run, so the single and batch summaries use one rule.
- `recoveryService` and `heartbeatService` expose
`buildRunOutputSilences` as pass-throughs.
- Tests: new `live-runs-batched-reads.test.ts`; updated watchdog
use-case tests and the live-run route unit test.

## Verification

- `cd server && npx vitest run
src/__tests__/live-runs-batched-reads.test.ts` (embedded Postgres):
- The batched read decision equals one `canActorReadHeartbeatRun` call
per run for 24 actor kinds (owner, admin, operator, viewer, member,
board key, cloud tenant, instance admin, local implicit board, board
without membership, agent keys, agent runs on behalf of users, low-trust
agents, agent of another company, unauthenticated). The runs mix open
tasks, private tasks readable by grant, creation or assignment, private
tasks nobody may read, private-project tasks, a hidden task,
company-scoped runs, a run whose task was deleted, runs of another
company, and runs of another company asked for as this company's.
- For every actor that passes the company gate, each live-runs row is
either the full row or exactly `redactHeartbeatRunListRow` of it (field
by field), as one check per run decides.
- Output silence per run equals the single-run reader for: no output
yet, active snooze plus a later continue, expired snooze, dismissed
false positive, recent output, open evaluation task, done evaluation
task, queued run, an evaluation task of another company, and a run that
has no rows at all.
- The statement count is the same with 19 and 49 live runs for 6 actor
kinds. Before the fix this test fails (for example 347 against 107
statements).
- `npx vitest run src/modules/active-run-watchdog
src/__tests__/heartbeat-active-run-output-watchdog.test.ts
src/__tests__/agent-live-run-routes.test.ts
src/__tests__/heartbeat-run-privacy-routes.test.ts`: pass.
- `npx tsc --noEmit -p server/tsconfig.json` and a test-tree typecheck
of the changed tests: pass. `pnpm check:module-boundaries`: pass.
- After the fix, measured with the same harness (embedded Postgres, 400
tasks, 40 agents, 5 requests each):

| Live runs | Statements per request | Allocated per request |
| --- | --- | --- |
| 10 | 15 | 6.4 MB |
| 50 | 15 | 7.3 MB |
| 200 (route returns 50) | 15 | 7.8 MB |

## Risks

- This changes an authorization read path. The route now uses the batch
helper that the run history list and the live events socket already use.
The test compares it with the per-run check for every actor kind in the
visibility fixture.
- If several open evaluation tasks point at the same run, the batch
picks one of them. The single-run read also picked an unordered one
before.
- No schema, migration, locking or transaction change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:10:30 -06:00
DottaandClaude Opus 5.5 85f4a91e70 fix(native): release a finished session's tool authority so the server heap does not grow per run (#21415)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a Runner session. The
server keeps a tool authority for each session scope, so the Runner can
call Paperclip tools and assigned app tools.
> - The server stored that authority in a map that lives as long as the
process. Only an aborted start removed an entry.
> - Each finished session kept its authority and the run's assigned
app-tool catalog. A large app catalog (hundreds of tools with schemas)
is several megabytes per run.
> - Under a busy queue with many short runs, the server heap grew until
the process ran out of memory. Each crash stopped every run that was in
progress.
> - This pull request releases a scope's authority when nothing can use
it again.
> - The benefit is that the server heap stays flat however many runs
finish, so the server does not crash and stop in-progress work.

## Linked Issues or Issue Description

**What happened?**
A hosted server ran out of JavaScript heap ("Ineffective mark-compacts
near heap limit") about 50 minutes after it started, and again after the
next start. The last GC line showed about 3.3 GB still live after a full
mark-compact. Each crash stopped every run in progress. The server
started about 300 native runs per hour, and most of them failed quickly.
Each run start loaded a large assigned app-tool catalog (the server
logged the "over the compile budget" warning with hundreds of tool ids
for each start).

**Expected behavior**
Memory that a run uses is released when the run's session ends. Heap
size does not grow with the number of finished runs.

**Steps to reproduce**
1. Run `executePaperclipNativeSession` with `useRunnerd: true` and a
`per_turn` session, for many distinct sessions, each with an assigned
MCP catalog.
2. Let each run succeed or fail.
3. Read `sessionToolAuthorityEpochs.size` and the post-GC heap. Before
this change, both grow by one entry and about 3 MB for each finished
run.

**Paperclip version or commit**
`master` (and the hosted build that crashed).

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`releaseIdleSessionToolAuthority(scope)`. It removes and revokes the
scope's tool-authority epoch when no execution, authority
initialization, settle, warm owner, or warm close holds the scope. It is
called:
- at the end of `executePaperclipNativeSession` (after the scope
reservation and the settle mark are released),
  - after a warm session's idle-timeout close,
- after a warm session's close in `closeIdleWarmNativeSessions` (restart
and sleep sweeps).
- A warm session that stays retained keeps its authority, so the next
run in that session still drains the previous run's accounting events as
before.
- `forgetWarmNativeSession` also calls it, so a warm session that a new
session identity supersedes (`retireSupersededWarmNativeSessions`)
releases its authority too (review finding).
- New `sessionToolAuthorityCount()` export, for the regression tests.

## Verification

Red/green regression test in
`server/src/services/native-runtime/native-session-executor.test.ts`:
"releases each finished session's tool authority and assigned tool
catalog (succeeded|failed)". It runs five runnerd `per_turn` sessions
with distinct session ids and an assigned MCP catalog, then checks that
the authority map is back at its starting size.
- Red (fix disabled): `expected 10 to be 5` (one entry stays for each
finished run).
- Green (fix): both cases pass.

Second red/green test, "releases a superseded warm session's tool
authority and keeps only the retained session's": two warm runnerd runs,
where the second has a new session identity and retires the first. Red
(no release in `forgetWarmNativeSession`): `expected 2 to be 1`. Green:
one entry for the retained warm session, and zero after the warm sweep
closes it.

Heap measurement, same harness, 200 runs with distinct sessions, each
with a 300-tool assigned catalog, provider failing fast, mock call
history cleared after each run, post-GC `heapUsed` delta:

| Finished runs | Entries before | Heap delta before | Entries after |
Heap delta after |
| --- | --- | --- | --- | --- |
| 50 | 50 | 150.5 MB | 0 | 16.2 MB |
| 100 | 100 | 300.6 MB | 0 | 16.8 MB |
| 150 | 150 | 452.8 MB | 0 | 17.3 MB |
| 200 | 200 | 605.3 MB | 0 | 18.0 MB |

Before: about 3.0 MB retained for each finished run, linear, never
released. After: flat.

Commands:
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t "releases
each finished session"`
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`
- `cd server && pnpm exec tsc --noEmit`

## Risks

- A late Runner callback for a session that already ended now gets "tool
authority unavailable" instead of reaching the old, revoked authority. A
finished session has no Runner left, so no supported path makes that
call. A warm session that stays retained keeps its authority.
- A same-run restart recovery in the same process starts with no
previous epoch for an ended session, so it does not drain a previous
run's accounting events. That only applies when the earlier session
already ended.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap measurement).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:00:09 -06:00
DottaandClaude Opus 5.5 24302a09ad fix(environments): stop warm sandboxes once their warm window ends (#21410)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes (for example Daytona) through
reusable environment leases
> - A warm run ends with its lease `retained` and its sandbox still
running, so the next turn on the task starts warm
> - After the warm window, only the provider's own idle auto-stop
stopped that sandbox. Paperclip never stopped it
> - On staging, idle Daytona sandboxes get provider activity about every
10 minutes. With a 12 minute auto-stop, they never stop and keep billing
> - This pull request makes Paperclip stop a warm sandbox itself when
its warm window ends and no live run holds it
> - The benefit is that idle sandboxes stop on time, while warm turns
inside the window stay warm

## Linked Issues or Issue Description

**What happened?**
Daytona sandboxes for ended warm runs stay in state `started` long after
their warm window. On 2026-11-13 the staging Daytona org had 111 started
sandboxes. One group of 23 idle sandboxes (auto-stop 12 min) got new
provider activity exactly 10 minutes after their last use, so Daytona
never auto-stopped them.

**Expected behavior**
Paperclip stops a sandbox when its warm window ends and no live run
holds it. Daytona auto-stop stays only a backstop.

**Steps to reproduce**
1. Run a warm `paperclip_runner` task on a reusable sandbox environment.
2. Let the run succeed. The lease becomes `retained` and the sandbox
keeps running.
3. Wait past the warm window. Nothing in Paperclip stops the sandbox; it
runs until the provider auto-stop fires, or forever if activity resets
the provider idle clock.

**Deployment mode**
Hosted (Paperclip Cloud staging) with the Daytona sandbox provider
plugin.

**Additional context**
Plugin Runner task leases (`driver: "plugin"`) are out of scope: their
release path does not use the stop-only contract, so this change leaves
them to their own cleanup.

## What Changed

- New `server/src/services/warm-sandbox-expiry.ts` owns warm sandbox
expiry:
- The warm window is the run's resolved runner idle timeout
(`resolveWarmRunnerIdleTimeoutMsForRun`). The window starts at the lease
release (`lastUsedAt`).
- Plugin Runner task leases (`driver: "plugin"`) are excluded. Their
cleanup releases through the plugin, not the verified stop-only
contract, so expiry could free the task's files.
- A lease saved before this change has no recorded window. It uses the
longest window a run can configure (24h), so it is never stopped early.
- A candidate is a `retained` `reuse_by_environment` lease whose run has
ended. It is held (never stopped, never a deadline) when another lease
holds the same sandbox, a run on the task is `running` (it may be
resuming this sandbox), or an operation is in progress on the lease
(stop intent, export resume, restore recovery, cleanup claim,
successor/supersede markers).
- Due sandboxes go through the existing stop-only contract
(`prepareSupersededWarmSandboxStop`). The cleanup sweep stops the exact
sandbox at the provider. The lease then ends `released` as stopped and
kept, which the next run on the task resumes.
- Acquisition (plugin and built-in sandbox paths) records the run's warm
window on the lease (`warmSandboxIdleTimeoutMs`).
- `nextEnvironmentCleanupAt` includes the earliest warm window end. The
environment cleanup worker sleeps until that deadline. It also wakes on
every lease release and runs at boot, so a sandbox left running across a
restart or deploy is reconciled. There is no interval scan.
- `recoveredOrphanFirstAttemptSql` treats a warm-window stop as due at
once, so the first stop does not wait for the 5 minute retry backoff.
- The pending-cleanup sweep hands due warm sandboxes to the stop-only
cleanup before it reads its rows, so one pass stops them.
- Only this instance's own lease rows are judged. Nothing lists provider
resources, so a sandbox of another company or environment is never
touched.

## Verification

- `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.test.ts`
(server): 9 passed.
- Inside the warm window: kept, and the worker deadline is exactly the
window end.
  - Past the window: stop intent recorded, first stop due now.
- Through the cleanup sweep: the provider stop is dispatched once; the
lease ends `released`, stopped and kept.
  - A live run's lease on the same sandbox: kept, and no deadline.
  - A run on the task is starting: kept.
  - The lease's own run is still running: kept.
- A plugin Runner task lease past its window: left to its own cleanup,
and no deadline.
- Orphan after a restart: a lease retained 35h ago without a recorded
window is stopped on the first (boot) pass; one retained 1h ago is kept
with a deadline at 24h.
- Red/green: with the wiring removed (sweep step, deadline,
first-attempt marker), 4 of the 8 tests fail (window deadline,
past-window stop, sweep stop, restart reconcile). With the change, all 8
pass.
- CI on this PR covers the broad suites for the final revision. A local
`pnpm verify` selected a very wide set (the change touches
`heartbeat/recovery.ts`) and hit 30 s timeouts in unrelated suites under
machine load, so I stopped it and rely on CI.
- After deploy, I check the staging Daytona org: started counts for
ended warm runs drop, and live or in-window sandboxes stay started.

## Risks

- A run that starts on the task in the moment between the deadline check
and the stop could resume a sandbox that is being stopped. A `running`
run on the task holds the sandbox, and the stop intent is a
compare-and-swap on the exact retained row, so this window is small. The
same race already exists for the retained-orphan settle.
- The first boot after deploy stops every retained warm sandbox that is
older than 24h (or past its recorded window). This is the intended
cleanup. It makes one stop call per lease, 25 leases per pass.
- The stop-only contract disables Daytona auto-delete on the stopped
sandbox (existing behavior of that contract). Daytona auto-archive still
moves it to cold storage. The retained-orphan settle destroys it when
the task no longer uses it.
- The deadline query reads `retained` leases without a dedicated index,
the same as the existing pending-cleanup deadline. It runs only when the
cleanup worker wakes.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, Daytona API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:47:43 -06:00
DottaandClaude Opus 5.5 e7e28ba241 feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns`
tasks at once. Other runs wait in the agent's queue with status `queued`
> - On hosted staging, one agent had all 20 slots full and 295 queued
runs. Some tasks waited 3 to 7 hours and never started
> - The UI showed these tasks as live or working: a pulsing "Live" badge
in the task header and in task rows, a spinning "Working … for 4 hours"
in the task chat, and "N live" counts that included every queued run
> - People could not tell that work was waiting, why it waited, or what
to change
> - This pull request makes a deep queue an honest, visible state:
"Waiting for Codie · 20 slots max · 41 of 295 in line · Configure"
> - The benefit is that operators see the real state and the one setting
that changes it

## Linked Issues or Issue Description

Refs #21391, #21349, #21242

**What happened?**

A queued run that never started looked the same as a running run. The
task header, the task rows in lists and the inbox, the task chat and the
sidebar counts all showed queued runs as live or working.

**Expected behavior**

A run that waits for one of its agent's concurrency slots shows that it
waits. It names the agent, the agent's limit and the run's place in
line, and it links to where the limit is set. The place in line is the
order in which the server starts the agent's queued runs.

**Why does it matter?**

With a deep queue, the UI said "working" for hours while nothing ran.
Operators could not see the cause or the fix.

## What Changed

- Server: a new module
`server/src/services/heartbeat/queued-run-line.ts` now holds the facts
that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The
dispatch path in `queue.ts` and the new "place in line" read use this
one builder, so the line the UI shows is the dispatch order. The logic
moved without change, including the recovery-share fix from #21391.
- Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit,
running count, position, queued count) and the label "Waiting for
<agent>" with cause `agent_at_capacity` to a queued, never-started run
when its agent has no free slot. It reads one batch of agents and
running counts. Then it builds one line per full agent (at most 8 agents
per read), using only the three context keys that ordering needs. No new
polling.
- Server: the place in line uses `queuedRunDispatchLine`, an O(n log n)
replay of the dispatcher. The plain replay rescans the queue after each
step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000,
600 ms at 10,000, all on the event loop). Dispatch itself is unchanged:
it stops after the free slots. Each agent's line is memoized for 5 s and
shared by concurrent reads. In a preview worktree, the line applies the
same creation-time cutoff as dispatch.
- Server: the task list summary adds `runningDescendantCount`
(descendants whose run started) next to `liveDescendantCount`.
`liveDescendantCount` still decides whether a blocked parent is covered.
The "n live below" chip uses the new count.
- Shared: the new `ExecutionAgentQueue` type and the optional field
`ExecutionProjection.agentQueue`.
- Server: `/companies/:companyId/live-runs` lists running runs before
queued runs, so a deep queue cannot push running work out of the 50-row
limit.
- UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20
slots max · 41 of 295 in line · Configure", with no spinner and no
elapsed timer. Configure opens the agent's Advanced view, where the run
policy (`maxConcurrentRuns`) is edited. The task chat (both renderers),
the run ledger and the Storybook story use it.
- UI: the task header and the task rows (task list, inbox, legacy views)
show a quiet "Waiting" instead of the pulsing "Live" when every live run
of the task has not started. The full sentence shows on hover. "n live
below" counts only started sub-tasks. A waiting sub-task still covers
its blocked parent.
- UI: the sidebar "N live" counts (Dashboard, agents) count only started
runs.
- UI: the task chat pill now names other server holds on a queued run
(for example "Waiting for maintenance") instead of "Working".
- Docs: DESIGN.md records the waiting-for-slot hold.
- Storybook: `runs-agent-queue-wait` stories (full line, task chat, task
row, one slot, mobile), with play checks.

## Verification

- Red/green server test:
`server/src/services/execution-projection-agent-queue.test.ts` (embedded
Postgres). Before this change, the projection has no `agentQueue`. Now
it checks the agent name, limit, running count, positions 1 to 4 in
dispatch order (a person's retry first, then priority, not creation
order), no wait while a slot is free, and isolation from other
companies' runs. It also checks that the positions equal the line that
the dispatch builder produces.
- `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`,
`recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and
`agent-live-run-routes` in `server/`.
- Runtime cost (rule `runtime-dependency-boundaries`): the assumption is
that the line stays cheap for a deep queue. `queued-run-line.test.ts`
checks that `queuedRunDispatchLine` equals the dispatcher's
claim-every-run order over 400 random queues (ages around the aging
boundary, recovery and person runs, blocked runs, deferred share runs,
all share counters). It also orders 10,000 runs in under 250 ms (the
quadratic replay takes about 600 ms). The DB test projects a 300-run
queue end to end with correct positions 1 to 300. Remaining limit: the
line is up to 5 s stale.
- Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts
`runningDescendantCount: 0` for a parent whose only live descendant is
queued and never started.
- UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep
"Live", no "running below" count for waiting sub-tasks) and
`TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the
Configure link, other holds named). The suites for the run ledger,
sidebar, sidebar agents, task detail, task columns and task chat also
pass.
- The Storybook stories render in headless Chromium and their play
checks pass.
- After deploy: open a queued task on staging and confirm the waiting
state.

## Risks

- Cost on read: a projection read that includes a queued run of a full
agent now reads that agent's queued runs (only small columns) and the
dispatch facts. This is at most one line per full agent every 5 s, for
at most 8 agents per read. Reads with no queued, never-started runs do
no more work.
- The place in line is the server's prediction of dispatch order. A
claim that is refused at start time (a gate, a blocker) can let the next
run start first. The position updates on the next refresh.
- The process-local "declined aged claim" hint that the dispatcher uses
is not visible to other servers, so the projection does not use it. This
can change only the aged-oldest share step.
- Configure opens the agent's Advanced view (the YAML run policy). The
agent's Runtime tab hides the run-policy fields today.
- Low risk otherwise: no schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless browser) and subagents for the UI survey.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:24:20 -06:00
DottaandClaude Opus 5.5 b0e99dfcfa fix(connections): show the setup card unless the person asked to set up the connection (#21400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents find and request app connections with `connections_search`
and `connection_request`
> - For one unconnected match, the search told the agent "The user asked
to connect" and said to configure the connection with `call_api`
> - That text was given for every search, also when the person only
asked a question or asked for app data
> - The agent then created a draft connection itself and sent an
absolute `http://127.0.0.1:<port>` link, and the person got no setup
card to accept or decline
> - This pull request gives the configure instruction only when the
person's own message asks to set the connection up, and makes the
agent's sign-in handoff link a relative app route
> - The benefit is that a question or a data request shows the setup
card, and setup links open on the host the person uses

## Linked Issues or Issue Description

**What happened?**
In chat eval cohort 12 (OpenCode with DeepSeek V4.1 Flash), the person
wrote "Read my Notion and tell me the latest on the launch".
`connections_search("Notion")` returned one result with the instruction
"The user asked to connect. If method.management.allowed, use
search_api/call_api ... to configure it". The agent called `POST
tools/apps/connect` twice and replied with
`http://127.0.0.1:<port>/COM/apps/connect?resume=...`. No setup card was
shown, so the person could not decline. The same happened for "What is
the easiest way to connect Notion here?".

**Expected behavior**
When the person asks about an app or needs its data, the agent shows the
setup card (`connection_request`) and answers in plain words. The agent
configures a connection itself only when the person asks for that setup.
Links the agent gives are relative app routes.

**Steps to reproduce**
1. Have no Notion connection.
2. In chat, write "Read my Notion and tell me the latest on the launch".
3. See that the agent creates a draft connection and links an absolute
URL, with no setup card.

## What Changed

- `server/src/services/connection-intents.ts`: for a single unconnected
tool match, read the latest human message on the task. Give the
configure instruction only when its first clause opens with a setup verb
and names the matched app ("Connect Notion, then ...", "Can you set up
Notion?"). "Connect Slack, then read my Notion" does not count for
Notion. Otherwise tell the agent not to create a connection or draft.
For a setup question, the agent answers in plain words and links the
catalog setup page `/<prefix>/apps/connect?source=<app>`, because
`connection_request` ends the turn before any answer text is shown. For
a data request, the agent calls `connection_request` to show the setup
card. A saved custom connection always uses the card. The check is
conservative: a miss never configures anything.
- `server/src/services/tool-access/oauth/authorization-start.ts`: the
agent handoff link (`authorizationUrl`, also
`requiredUserAction.setupPath`) is now a relative app route, not built
from the server's redirect origin.
- Tests: a red/green service test for the three messages, and the
handoff test now requires a relative link.

## Verification

- `cd server && npx vitest run
src/__tests__/connection-intents-service.test.ts -t "configure a single
match"`: passes for a data request, a setup question, a mixed-app
request and an explicit setup request. With the old
`connection-intents.ts` it fails (`expected 'The user asked to connect.
...' to contain 'connection_request with service notion'`).
- `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t "generic app
authentication"`: passes.
- A narrow paid eval run of `connection-setup.notion-declined` and
`connection-setup.choose-provider` follows on this branch.

## Risks

- A person who asks for setup with other words ("I'd like Notion hooked
in") now gets the setup card instead of agent-side configuration. The
card still completes the setup.
- One small indexed read of the latest human message per single-result
search. There is no change to agent boot or time to first token.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:02:36 -06:00
DottaandClaude Opus 5.5 e40a16e5c5 fix(heartbeat): let Stop take a task out of the agent's queue (#21407)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs a limited number of tasks at once. Other tasks wait
in line as queued runs.
> - An agent can have hundreds of waiting tasks. People need one click
to take a task out of line.
> - Stop already cancelled a queued run, but a task that was in progress
stayed in progress. It still looked busy.
> - The classic task thread also hid Stop until the run started.
> - This pull request makes Stop on a waiting task free the task and
show it as not started, in both thread views.
> - The benefit is that people can clear a long queue one task at a
time, and the agent moves on to its next task.

## Linked Issues or Issue Description

**What happened?**

An agent had many queued runs. A person pressed Stop on a task that
waited in line. The run was cancelled, but the task stayed "in
progress". In the classic thread, Stop did not show until the run
started.

**Expected behavior**

One Stop click takes the task out of line. The queued run and its wake
are cancelled. The task lock is released. The task does not look in
progress. No new run starts for that task. The agent's next waiting task
still starts. An activity entry records the change.

**Steps to reproduce**

1. Give an agent one run slot and keep it busy.
2. Assign two more in-progress tasks to the agent, so their runs wait in
line.
3. Press Stop on one waiting task.
4. See that the task still shows "in progress".

**Paperclip version or commit**

`master` at 30c718e04b.

**Agent adapter(s) involved**

Not adapter-specific (core bug).

## What Changed

- `POST /heartbeat-runs/:runId/cancel`: when the run was queued, the
route calls the new `returnStoppedWaitingTaskToTodo` after the cancel.
- New `server/src/services/heartbeat/stopped-waiting-task.ts`: one
guarded update moves an `in_progress` task to `todo` and logs
`issue.updated` (reason `stopped_while_waiting`). It changes the task
only if the run never started, the task is still assigned to that agent
and not to a person, the task lock is free or held by that run, and no
other live run exists for the task.
- The route keeps its company and run-read checks and still logs
`heartbeat.cancelled`. Recovery already stands down for an operator
stop, so no new run starts for the task.
- `IssueChatThread` (classic view): the composer shows Stop for a queued
run too, in the same place as a running task's Stop. The default
`TaskChatThread` already did this; a test now covers it. The waiting
banner is unchanged.

## Verification

- `server/src/__tests__/heartbeat-stop-queued-task.test.ts` (new,
embedded Postgres, real heartbeat): one busy slot, two waiting tasks.
Stop one waiting task. The run and wake are cancelled, the lock is
released, the task is `todo`, and an activity entry exists for the
in-progress case. After the slot frees, the next task runs. The stopped
task gets no new run, also after `resumeQueuedRuns` and
`reconcileStrandedAssignedIssues`. The in-progress case was red before
the fix.
- `server/src/__tests__/heartbeat-run-privacy-routes.test.ts`: a real
route test. Another user gets 404 and the task does not change. The
task's operator gets 200, the task becomes `todo`, and both activity
entries are recorded. Red without the route change, green with it. The
full file passes (30/30).
- The same route test sends a correlated Stop (`cancellationRequestId`)
to the waiting legacy run. The server returns 409 and the run and task
do not change. Correlated Stop is for native runs only.
`ui/src/api/heartbeats.test.ts` checks that task Stop sends the plain
request (`{}`).
- `ui/src/components/IssueChatThread.test.tsx` and
`TaskChatThread.test.tsx`: Stop shows for a queued run and calls the
stop action. The classic test was red before the fix.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/ui typecheck` pass.

## Risks

- Low risk. The status write is one guarded `UPDATE` and changes nothing
if any guard fails. Only the board Stop route calls it, and only for a
run that was queued.
- A task stopped while waiting shows `todo` instead of `in_progress`.
This is intended. A Stop on a running task does not change its status.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:58:54 -06:00
DottaandClaude Opus 5.5 b7774a01cf fix: name activity actors and explain stopped tasks in plain words (#21406)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People must never see internal jargon or raw ids in chat replies or
task comments
> - Chat eval cohort 12 found three sources: activity records show the
board user only as the raw id `local-board`, agents turn internal field
names such as `checkoutRunId` into words like "no checkout", and a
recovery notice says "the original assignee is not invokable"
> - The recovery notice appeared on 170 fixture tasks. It is not a loop:
each task got it once, because recovery blocked every in-progress task
of an agent that a person had paused
> - This pull request names activity actors, lists these internal terms
in the chat guidance, rewords the notice, and leaves in-progress work of
a manually paused agent alone
> - The benefit is plain replies and no false "blocked" tasks after a
person pauses an agent

## Linked Issues or Issue Description

**What happened?**
- `activity.audit-change`: the agent read `GET
/api/issues/{id}/activity` and wrote that the change was made by "the
board user identity (`local-board`)". Activity rows carry only
`actorType` and `actorId`.
- `activity.company-overview`: the agent wrote "no checkout" to explain
why no one is working on a task.
- `tasks.organize-open-work`: 170 of 501 fixture tasks got one system
comment each: "Paperclip cannot safely continue automatic recovery
because the original assignee is not invokable...". The owners were
paused by a person, and the stranded-work sweep blocked each in-progress
task. The existing rule already keeps `todo` work of a manually paused
agent, because blocking it fights the pause and strands the task after
resume.

**Expected behavior**
Replies name actors by name ("Board", a member's name, an agent's name)
and describe states in plain words. A person's pause does not block the
agent's in-progress work. A stopped-task notice says what happened and
what to do, in plain words.

**Steps to reproduce**
1. Change a task's priority as the local board user, then ask the chat
agent who changed it.
2. Pause an agent that owns in-progress tasks with no active run, and
let the stranded-work sweep run.

## What Changed

- `server/src/services/activity.ts`, `server/src/routes/activity.ts`,
`packages/shared/src/types/activity.ts`: company and task activity reads
add `actorName` (agent name, member name, "Board" for `local-board`,
"Paperclip" for the system). Raw ids stay in their existing fields.
- `server/src/services/agent-conversations.ts`: the existing "no raw IDs
or internal codes" rule now names checkout, native run and local-board
as internal terms, and tells the agent to name an actor by `actorName`.
- `server/src/services/recovery/service.ts`: the notice is now "<agent>
can't run right now, so Paperclip stopped this task instead of retrying
it. It is still assigned to them. To continue, resume or fix the agent,
or assign the task to someone else." The manual-pause skip now covers
`in_progress` as well as `todo` work with no run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 4 pass. With
the old recovery service the two new tests fail (`expected 'blocked' to
be 'in_progress'`, and the old notice text).
- `cd server && npx vitest run src/__tests__/activity-service.test.ts -t
"names each activity actor"`: passes (the method did not exist before).
- `cd server && npx vitest run
src/__tests__/agent-conversations.test.ts`: passes.
- `src/__tests__/activity-routes.test.ts` has three tests that time out
locally on unchanged master code too; CI runs them.
- Paid eval run of `activity.audit-change` and
`budgets-costs.spend-breakdown` on a branch with this change.

## Risks

- The chat prompt grows by about 40 words. This is a small part of the
existing directive, so time to first token does not change in a
measurable way.
- A budget- or error-paused agent still gets the notice, now in plain
words. Only a manual pause keeps in-progress work as is; the sweep picks
it up after resume.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:53:25 -06:00
DottaandClaude Opus 5.5 1529bca5c1 fix(chat): do not post a completion report that an earlier reply already gave (#21401)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a delegated task that finishes wakes the chat agent
with `chat_task_completed`, so it tells the person in the same
conversation
> - A turn that runs for another reason, such as the person's form
answer, can read the task when it is already done and report it
> - The completion turn then starts after it and reports the same news
again
> - The person gets two completion messages for one task
> - This pull request records when the chat agent reads its handed-off
task in the done state, and skips the completion turn when an earlier
reply already reported that completion
> - The benefit is one completion report per task, and no extra model
call for it

## Linked Issues or Issue Description

**What happened?**
In chat eval cohort 12, case `questions-and-approvals.duplicate-answer`:
the delegated draft task finished at 13:48:23 while the chat agent's
turn (woken by the form answer at 13:48:16) was running. That turn read
the task, saw it done and replied "Acorn is confirmed ... the draft is
finished" with a task link (13:48:50). The `chat_task_completed` turn
started at 13:48:53 and replied "The Acorn draft is done" (13:49:22).

**Expected behavior**
The person gets one completion report per finished task in the
conversation.

**Steps to reproduce**
1. In chat, delegate a short task.
2. While it runs, send a message that wakes the chat agent.
3. The task finishes during that turn, and the turn reads and reports
it.
4. See a second completion report from the `chat_task_completed` turn.

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`recordCompletedHandoffRead` appends a `chat_handoff.completed_read` run
event when the conversation agent reads its own handed-off task in the
done state (with the task's status version).
- `prepareChatCompletionTurn`: a delivery is marked delivered (with the
earlier reply as its response) when an earlier reply of the same agent
in the conversation was posted after completion, links the task, and
comes from a run that read the task at that exact done status. Both
conditions are structural (run event and task reference), not reply
wording. A reply that links the task without such a read (for example
"Morgan is working on it", from a read before completion) does not
count.
- `server/src/services/heartbeat.ts`: when every delivery of the
completion turn was already reported, the run ends as succeeded before
the conversation turn starts, with no model call.
- `server/src/routes/issues.ts`: task reads by an agent run call the
recorder (one primary-key read of the handoff, and an event only for the
chat agent's own done handoffs).

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts`: 53 pass, including two
new tests: the already-reported completion is delivered without a turn,
and a stale linking reply still gets the completion turn.
- Red: with the earlier-reply check disabled, "does not report a
completion again ..." fails (`expected false to be true`).
- A narrow paid eval run of `questions-and-approvals.duplicate-answer`
follows on this branch.

## Risks

- If the earlier reply linked the task but did not say it was finished,
the person does not get a second message. The read-at-done-status
condition makes this unlikely.
- The skipped completion run is a short succeeded run with no model
call. Boot and time to first token are unchanged for all other turns.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:49:22 -06:00
DottaandClaude Opus 5.5 5c5a2fcebe fix(dispatch): start the assigned-work sweep's continuations and count only system recovery for the recovery share (#21391)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. #21234, #21277
and #21349 order them: person work first, a recovery share, aging, and a
share for the longest-waiting run
> - On hosted staging, one agent had about 250 queued runs. For 90
minutes after #21349 deployed, every run that started was a retry or
continuation with `retryOfRunId`. The oldest queued runs, from 15 hours
earlier, never started
> - The dispatch order was not bypassed. The oldest-run share offered
those runs, but the claim refused them every time, without a log at info
level or a visible state
> - The refusal came from the chat-control recovery gate. It needs
`retryOfRunId` to trace an automatic continuation. The assigned-work
invariant sweep queues continuations that name their source only in
context, so the gate returned "unresolved", and an unresolved claim
stays queued forever
> - This pull request lets the gate check such a continuation against
the task's previous run, and makes the recovery share count only system
recovery starts
> - The benefit is that the sweep's continuations start (it already told
people "you don't need to press Retry"), and system recovery gets its
share even when a person's recovery restarts keep starting

## Linked Issues or Issue Description

Refs #21234, #21277, #21324, #21349, #21242

**What happened?**

On hosted staging (build e23549cd9c), one agent with `maxConcurrentRuns`
20 had about 250 queued runs. From 12:06Z to 13:30Z, all 30 runs that
started had `retryOfRunId` set. The oldest queued runs were automatic
continuations (`issue_continuation_needed`) created from 22:53Z the day
before. Each task's activity shows "Continued automatically after an
interruption ... you don't need to press Retry" from the assigned-work
invariant sweep, at the time the run was queued. None of these runs
started. Assignments created at 07:51Z and later did start, in creation
order. Those starts came through the oldest-run share after the share
passed over the refused continuations.

**Why did it happen?**

1. `assignedWorkSweep` queues the continuation with `previousRunId` in
its context and no `retryOfRunId`. Its wake is a system wake
(`requestedByActorId` null) with reason `issue_continuation_needed`.
2. At claim, `withChatControlRecoveryGate` calls
`readChatControlRecoveryStop`. For an automatic continuation without
`retryOfRunId`, it returned `unresolved`. At the claim stage,
`unresolved` returns null: the run stays queued and nothing changes. So
every dispatch pass offered these runs and every claim refused them.
Retries and continuations with `retryOfRunId` pass the gate, so they
were the only recovery work that started.
3. Separately, the recovery share (#21234) read the agent's recent
starts and counted every start with a recovery wake reason as a recovery
start. A person's `issue_recovery_action_restored` restart ranks as
person work in the dispatcher, but it still reset the share. So the
recovery share almost never fired while those restarts kept starting.

**Free slots.** The "14 of 20 running" figure came from a capped run
list. A full read at 14:03Z showed 22 runs `running` on that agent, so
there were no free slots. The agent was over its limit, not under it.
Overlapping dispatch passes after the 30 s stale agent start lock are a
likely cause (each pass re-tried every refused run, so passes were
slow). This pull request removes the refused runs that made passes slow.
#21242 changes the claim-then-execute timing.

**Expected behavior**

An automatic continuation that the system queued starts in its turn. A
chat close of the task's previous run still stops it. The recovery share
goes to system recovery after three starts that were not system
recovery.

**Steps to reproduce**

1. Give an agent one slot. Interrupt a run on an in-progress task so
that the assigned-work invariant finds a violation.
2. Run the assigned-work check. It queues an `issue_continuation_needed`
continuation.
3. Queue fresher recovery runs on other tasks and dispatch.
4. See that the continuation stays queued on every pass.

## What Changed

- `server/src/services/chat-control-recovery-stop.ts`: for an automatic
continuation that has no `retryOfRunId`, the gate now walks to the
task's latest earlier run (by `issueId`, `nativeIssueId` or the context
task id, in the same company) and checks that run. If the task has no
earlier run, the result is `clear`. A continuation with `retryOfRunId`
is unchanged. A chat close of the previous run still stops the
continuation.
- `server/src/services/heartbeat/queue.ts`: the recovery-share counter
reads which recent recovery-reason starts were person-originated, with
the same source-chain query that the dispatcher already uses
(`listPersonOriginSourceRunIds`, at most 4 ids). Only system recovery
starts reset the share. This matches the in-pass rule
(`isSystemRecoveryRun`).

## Verification

- Red/green, embedded Postgres, real heartbeat queue
(`server/src/__tests__/recovery-wake-herd.test.ts`):
- "starts the assigned-work sweep's continuation of a task whose run was
interrupted, ahead of fresher retries": on master the continuation stays
queued for 60 s of dispatch passes; with the fix it starts.
- "gives system recovery its share when the agent's recent starts were a
person's recovery restarts": on master a person run starts first; with
the fix the system recovery run starts.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: "checks an
automatic continuation that names no retry source against the task's
previous run" proves that a chat close of the previous run still stops
the continuation (`stopped`) and that it is `clear` when the close is
not published.
- `node server/scripts/typecheck-layers.mjs`: passes.
- Related suites: `recovery-wake-herd`, `queued-run-dispatch-order`,
`assigned-work-invariant`, `run-authority-read-projections`,
`chat-run-milestone-work`, `legacy-preparation-retry-hold`,
`explicit-native-continuation`, `saved-work-retry-eligibility`,
`native-failed-turn-continuation.integration`,
`heartbeat-process-recovery` pass, except:
- `heartbeat-process-recovery` "schedules bounded retries for failed
accepted interaction continuation wakes" is flaky on master too (1 of 3
runs failed on master source, same line).
- `recovery-wake-herd` "delivers the continuation after a restart..."
failed once with a hook timeout under parallel load and passes alone.
- `runtime-selection.test.ts` fails locally in a nested worktree because
a bundled skill file path does not exist there (environment, not this
change).
- After deploy: the queued `issue_continuation_needed` runs on the
affected staging agent should start within the oldest-run share, and no
run should stay queued longer than the aging bound plus a few starts.

## Risks

- Behavior change: an automatic continuation without `retryOfRunId` was
refused at claim and failed at dispatch with
`chat_control_recovery_proof_unresolved`. It now inherits the chat-close
check of the task's previous run instead. A chat close still stops it.
- Already-queued stuck continuations on hosted instances will start
after deploy. On the staging agent that is about 30 runs. They go
through the normal order and shares, so they do not take all slots at
once.
- One extra indexed read per gate check of this shape, and one small
recursive query per dispatch pass when recovery runs are queued. No
schema change. Mixed versions are safe.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, built-in browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:51:33 -06:00
DottaandClaude Opus 5.5 30c718e04b test(decisions): wait for both provider calls instead of a 1-second poll in the concurrent admission test (#21387)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Company decision models admit model calls against a budget, and the
server test suite checks this admission
> - The test "admits concurrent work from the same budget observation
and records both obligations" fails sometimes on loaded CI runners
> - Cloud readiness runs this test, and any Cloud readiness failure
stops the Core image from being published
> - On 2026-11-13 this test failed in Cloud readiness for two master
commits and stopped image publication
> - This pull request makes the test wait for the real condition instead
of a 1-second poll
> - The benefit is that a slow runner no longer fails the test, so the
release gate does not stop on a timing flake

## Linked Issues or Issue Description

No public issue exists. This is a test flake.

**What happened?**
The test failed in Cloud readiness "verify / General tests (server
(4/10))" on master. The failure was `AssertionError: expected "vi.fn()"
to be called 2 times, but got 1 times` at the second `vi.waitFor`. It
happened for master commit 11840f9904 (run 43435593630, attempt 3,
14:03Z) and for e3faf579a1 (run 43427997745, 13:00Z). The other 3661
tests in that run passed.

**Expected behavior**
The test passes on a slow runner. After the lock is released, both
decisions commit their reservations and call the provider. On a loaded
machine this can take more than 1 second.

**Steps to reproduce**
Run `server/src/__tests__/decision-models.test.ts` on a heavily loaded
runner. The second `vi.waitFor` uses the default 1000 ms timeout. The
line before it uses 3000 ms. If both provider calls do not start within
1 second of the unlock, the test fails.

**Paperclip version or commit**
master at 6917d0568d

## What Changed

- The test provider mock now counts the calls that start. It resolves a
promise when the second call starts.
- The test awaits that promise instead of polling
`toHaveBeenCalledTimes(2)` with a 1-second timeout. The test timeout
still applies if the calls never start.
- No product code changed.

## Verification

- `cd server && npx vitest run src/__tests__/decision-models.test.ts`:
37 passed locally.
- A red/green proof is not practical. The failure depends on runner
load, and the old code passes on an idle machine. The new code has no
timer between the unlock and the check, so it cannot time out early.

## Risks

- Low risk. This is a test-only change.
- If a regression stops the second provider call, the test now fails at
the test timeout instead of after 1 second. The failure is still
reported.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, GitHub CLI, AWS CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:38:43 -06:00
DottaandClaude Opus 5.5 944b1c64d7 feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use connected apps through Paperclip's MCP client in the tool
gateway
> - Some provider actions run for minutes: CI runs, deploys, batch jobs
> - Today such a call is lost if it outlives the tool timeout or if
Paperclip restarts or deploys mid-call, and a retry can start the work a
second time
> - Recovery must just work, and the target rate of runs that need a
manual retry is 0%
> - This pull request implements the MCP Tasks extension
(`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client
> - The benefit is that a long provider call becomes a durable task: it
survives timeouts, restarts, and deploys, and it is never started twice

## Linked Issues or Issue Description

**Problem**
A long provider tool call holds an HTTP request open. When it passes the
tool timeout, or when Paperclip restarts or deploys during the call, the
result is lost. The agent then often repeats the call, and the provider
does the work again.

**Proposed solution**
Implement the client side of the MCP Tasks extension. Declare it on
stateless tool calls. Store each returned task durably, wait on it
during the call, keep polling it in the background after the call ends,
and let the same call attach to the stored task instead of sending
`tools/call` again.

**Why it matters**
CI, deploy, and batch actions are where agents lose the most work today.
A durable task handle makes these actions safe across timeouts,
restarts, and deploys, without a person.

Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP
client, and MCP Skills work this builds on).

## What Changed

- **Off by default:** MCP Tasks are behind the new instance experimental
setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default
off. When it is off, Paperclip never declares the extension, nothing
attaches to or polls tasks, pending rows do not block idle sleep, and
behavior matches master. A test covers the off state.

- `mcp-http.ts`: the tasks extension is declared per request, only on
HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers
do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`.
- New `mcp-tasks.ts`: parses task results, bounds the server's timing
hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals
task IDs at rest (AES-256-GCM, bound to company and connection; a keyed
hash gives uniqueness).
- New company-scoped table `tool_mcp_tasks` (migration `0475`,
online-compatible). It links the task to the call, run, task,
connection, and grant. It stores no provider result.
- `tool-gateway.ts`:
- A returned task is recorded, then polled up to the normal tool
timeout. Its result goes through the same validation, redaction, and
recording as a direct result.
- Past the timeout, or when the caller disconnects, the agent gets
`provider_task_pending` ("call again with the same arguments"). The task
stays durable.
- A call with the same endpoint, tool, full arguments, grant, agent,
task, and actor attaches to the undelivered task. A retry with the same
idempotency key also attaches, and it can never start different work
under that key.
- Input: URL requests show the provider link (Paperclip never answers
them). Form requests become a question card; only the person's answer is
sent with `tasks/update`. Other requests stop with a clear "needs input"
error.
- Stopping the run sends `tasks/cancel` and marks the task cancelled. A
caller that only disconnects keeps the task.
- `failed` becomes a tool error with the provider's message. An unknown
task or a passed TTL becomes `provider_task_expired`. Every task stops
being tracked at its TTL, or after 7 days at most.
- Audit entries: `tool_gateway.task_started`, `task_input_sent`,
`task_cancelled`.
- New `mcp-task-work.ts`, registered at startup with the shared deadline
scheduler: it resumes polling pending tasks after a restart, cancels
tasks whose task closed or run was stopped, and wakes the assignee once
when a task settles after its run ended.
- Docs: `doc/connections/MCP-TASKS.md`, linked from
`doc/connections/README.md`.

Background work (per `rules/sleep-aware-background-work.mdc`): trigger
is a task row write or release (`mcp-tasks` queue) and one startup pass.
The deadline is the earliest `next_poll_at`. Cadence is the server's
`pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting
for a person. It is quiet when no row has a next poll. This is the
narrow external-source exception: providers give no wake Paperclip can
receive, and polling stops with the task.

## Verification

- `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake
2026-07-28 server that returns tasks only to a client that declares the
extension and checks `Mcp-Name`. They cover:
- polling to completion, with the result identical to a synchronous
call;
  - timeout, then an attaching call with no second `tools/call`;
  - a keyed retry, and a reused key with different arguments (refused);
- restart mid-task: a fresh gateway with no in-memory state polls the
row, wakes the agent once, and the recovered run gets the result once
with no second `tools/call`;
  - no wake while the run is still going;
- URL input (link, never answered), and form input (question card, then
`tasks/update` with the answer);
- cancellation by the run, by closing the task, and no cancellation when
only the caller leaves;
- failure, an unknown task, and TTL expiry (ends in under 3 s, no hang);
  - another agent never attaches;
- synchronous calls unchanged and older servers never see the extension.
- Red/green: with attach disabled, the timeout, keyed-retry, and restart
tests fail (3). With `tasks/cancel` disabled, both cancellation tests
fail (2). Before `tools/call` declared the extension, 13 end-to-end
tests failed. All pass with the change.
- Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`,
`mcp-skills`, `server-startup-feedback-export`, `tool-gateway`,
`tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests).
- `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db
check:migrations` passes (the migration is classified
online-compatible). Snapshots pruned to five.
- Agent boot and time-to-first-token: no change in the boot or
first-turn path. The only new per-call cost is one indexed lookup on
stateless tool calls.
- Not run: a live provider that returns tasks. No public MCP server that
returns tasks was available without new credentials.

## Risks

- Shipped off by default. Review findings that only affect the
setting-on path after this merge are tracked as follow-ups.

- Migration: a new table with indexes and foreign keys to `companies`,
`tool_invocations`, and `tool_connections`. Online-compatible.
- Behaviour change only for servers that return tasks, which they may do
only to clients that declare the extension. Older servers and stdio
servers are unchanged.
- A task that settles while nobody is waiting stays attachable for 1
hour, so a deliberate repeat of the same call within that hour collects
the earlier result instead of starting new work.
- The wake after a background task settles uses the generic task wake;
the agent must call the tool again to collect the result.
- Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling
and roots input requests; showing a long action's progress in the task
chat.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:32:06 -06:00
DottaandClaude Opus 5.5 5026093d0f feat(presentation): offer Move it forward now while a continuation is still waiting to start (#21385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and delivers it later as a new run. Since #21353 the task reads as
continuing (`working`, no Retry) while it waits
> - #21373 added "Move it forward now" (`retryAction: "move_forward"`).
It is offered only after the delivered continuation run waited 15
minutes in the agent's queue
> - Before delivery (`continuationDelivery: "pending"`) the task showed
no button. On hosted staging such continuations waited 30 to 110 minutes
behind a deep queue, and the person had no control
> - #21378 and #21379 made the click safe in that state: a
`move_forward` request with a recorded, undelivered continuation is
accepted and delivers that continuation as the one run, with its
recorded isolation
> - This pull request also offers "Move it forward now" while the
continuation waits for delivery, once it has waited 15 minutes since it
was recorded
> - The benefit is one action, with the same label and copy, for both
waiting states. The click never starts a second run

## Linked Issues or Issue Description

Refs #21353, #21373, #21378, #21379

**What happened?**

A task has a fresh continuation that is recorded but not delivered
(`continuationDelivery: "pending"`). It waits behind a deep agent queue
for 15 minutes or more. The task shows "continuing" with no action.

**Expected behavior**

After 15 minutes the task still reads as continuing (`working`,
responsibility `system`, never `failed`). It also offers "Move it
forward now". The click delivers the recorded continuation as the one
run.

**Steps to reproduce**

1. Record a fresh continuation for a task while its agent is busy. Do
not run the delivery pass.
2. Wait 15 minutes.
3. Read the task presentation. It shows `canRetry: false` and no
`retryAction`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: the
fresh-continuation read also loads the record status, schema,
`recordedAt` and reconciled run. A `continuing` state can now carry
`pendingSince` when the primary record is `resolved`,
`continuationDelivery` is `pending`, the schema is the
fresh-continuation schema, and the reconciled run is the observed run.
These are the same facts that `undeliveredContinuation` checks at click
time.
- The projection offers `canRetry: true, retryAction: "move_forward"`
(state `working`) when `pendingSince` is 15 minutes or older, no person
Retry for that run already waits, and the run's agent is the current
assignee. Under 15 minutes, nothing changes.
- No button when the system gave up delivery (`invalidated`; the normal
Retry rules apply), when the delivery is `not_requested`, when a person
already asked, or for a held restore (the existing execution-blocker
post-filter removes the offer).
- New integration test
`server/src/services/fresh-continuation-move-forward-offer.test.ts`. It
uses the real click path for the three recorded kinds
(`fresh_isolated_workspace`, `writer_ended`, native `remote_epoch`):
presentation, `requestTaskRetry` with the shown `observedRunId` and
`retryAction`, `resumeTaskRetry`, then the delivery pass.
- Updated guards in `task-interruption-automatic-path.test.ts` and
`recovery-wake-herd.test.ts` (#21353, #21373). Their assertions that a
pending continuation shows no Retry at 30, 45, 110 and 240 minutes now
expect the move-forward offer. The `working` / `system` / never-failed
assertions stay. New cases cover 14 minutes, `not_requested`, a waiting
person Retry and reassignment.
- `doc/execution-semantics.md`: describes the new offer and its races.
- No shared type, validator, UI or API client change. The action and its
UI copy ("Waiting for the agent to finish other work. Paperclip
continues this task by itself." / "Move it forward now") already exist
and do not depend on the state. No runner-e2e SDK public-input pin
covers the changed files.

## Verification

- Red: with the presentation change reverted, 16 tests fail (`expected {
intentId: null, …(7) } to match object { responsibility: 'system', …
retryAction: 'move_forward' }`).
- Green: `npx vitest run
src/services/fresh-continuation-move-forward-offer.test.ts`: 14 of 14
pass. For each kind:
- The click gives exactly one run. It is the recorded continuation
(`recoveryActionId`, `previousRunId`, `retryOfRunId`,
`forceFreshSession`). The fresh isolated workspace is kept
(`freshContinuationIsolation`, `readFreshWorkspaceReadiness`). The
record reads `delivered` with that run. Two later delivery passes and a
repeated resume create no second run, and the offer then ends.
  - The click and two concurrent delivery passes give one run.
- Delivered between the offer and the click: the click is refused
(`accepted: false`, no wake) and the only run is the delivered
continuation.
- The system gave up the delivery between the offer and the click:
`alreadyMovedOn: true`, no run.
- The task moved to another run between the offer and the click:
refused, the only run is that other run.
  - Under 15 minutes: no button.
- Guards: `task-interruption-automatic-path.test.ts` (37/37),
`recovery-wake-herd.test.ts`, `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`,
`fresh-continuation-retry-delivery.test.ts`,
`fresh-continuation-fallback.test.ts`, `task-retry.test.ts`: all pass.
Also the saved-work, workspace-recovery, native restart and
closed-session retry suites that read the presentation.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change is presentation only. It adds four selected JSON
fields to a read that already exists and one conditional module import.
The click path (`task-retry.ts`, `fresh-continuation-retry-fold.ts`) is
unchanged and already holds the task row lock. A record without
`recordedAt` or with an older schema gets no button, as before. People
now see a button on tasks that showed none after 15 minutes. The button
can only deliver the recorded continuation or do nothing.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:24:12 -06:00
DottaandClaude Opus 5.5 f7ddab8e94 fix(runner): give agent task and workspace reads the workspace page href (#21384)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People ask a chat agent where the files of a task are.
> - The answer must link the page of the workspace that holds the task's
files, when that page exists.
> - The agent reads the task with `get_task`. The task's
`currentExecutionWorkspace` had no `href`.
> - The chat directive tells the agent that a workspace without an
`href` has no page. So the agent said the page did not exist, although
isolated workspaces were on.
> - This pull request adds the workspace page `href` to the agent's task
and workspace reads, as `get_workspace` already does.
> - The benefit is that the agent links the real workspace page and does
not make a false claim.

## Linked Issues or Issue Description

**What happened?**
A person asked the CEO chat agent: "Where are the files for the website
task actually being worked on?" The agent named the task workspace and
linked the file. It did not link the workspace page. It also said "the
workspace has no separate page on this instance". The instance had
isolated workspaces on, so the page existed at
`/<prefix>/execution-workspaces/<id>`. The chat eval
`chat-features.runner-opencode.local.workspaces.locate-files` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) failed its
`bound-workspace-page-link` check in four cohorts and in one more paid
run for this reason.

**Expected behavior**
The agent reads the task, gets the workspace page link from the tool
result, and links that page in its reply.

**Steps to reproduce**

1. Turn on isolated workspaces (`enableIsolatedWorkspaces`).
2. Let an agent complete a projectless task, so the task has a bound
execution workspace.
3. In a different chat, ask the CEO where the files of that task are.
4. The agent calls `get_task` for the task.
`data.currentExecutionWorkspace` has no `href`. The agent reply names
the workspace and says it has no page.

**Additional context**
`get_workspace` and `list_workspaces` add the page `href` only for the
current task. The agent reads another task with `get_task` (`GET
/api/issues/{id}`) or with `call_api`. Those results did not get the
`href`. The run transcript shows the agent's reasoning: "The task API
includes currentExecutionWorkspace with no href", followed by the
directive rule "a workspace without an href has no page on this
instance".

## What Changed

- `server/src/services/task-workspace-tools.ts`: add
`withWorkspacePageHref` and `apiResultMayHaveWorkspace`. They add the
page `href` to the workspace in successful agent reads of `GET
/api/issues/{id}` (`currentExecutionWorkspace`), `GET
/api/issues/{id}/workspace` (`workspace`) and `GET
/api/execution-workspaces/{id}`. The function uses the same `href`
builder as `get_workspace`. It adds no `href` when the page is off, when
the read failed, or when the workspace is in a different company.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
apply the function to the results of the task bridge tools (`get_task`)
and `call_api`. The tool authority reads the page setting only for these
three operations.
- Tests: unit tests for the function, and an integration test through
the real tool authority and API. The integration test checks `get_task`
and `call_api` with workspace pages on (the `href` is present) and off
(no `href`).

## Verification

- Red/green: with the base tool authority, the new integration test
fails (`undefined` instead of `/<prefix>/execution-workspaces/<id>` for
both reads). With the fix, it passes. The unit tests fail on the base (4
failures) and pass with the fix.
- `cd server && npx vitest run src/__tests__/chat-project-tools.test.ts
src/__tests__/task-workspace-tools.test.ts
src/__tests__/agent-conversations.test.ts`: 110 passed.
- `cd server && npx tsc --noEmit -p .`: passes.
- Latency: the change runs only when the agent calls one of the three
reads. It does not touch run boot or time to first token. Local
`get_task` timing through the tool authority, 40 calls per sample, pages
on: base p50 51 ms and 74 ms; fix p50 49 ms and 54 ms. The difference is
in the noise.
- Paid eval: one narrow run of
`chat-features.runner-opencode.local.workspaces.locate-files` on this
branch:
https://github.com/paperclipai/paperclip/actions/runs/43436040405
(result in a comment below).

## Risks

- Low risk. The change only adds an `href` field to three agent read
results. It does not change REST responses for the board UI.
- The `href` appears only while isolated workspaces are on, because the
page route exists only then. This is the same rule `get_workspace` uses.
- An agent can now link a workspace page that it read through
`get_task`. The page itself still applies normal board access checks.

## Model Used

- Claude, Anthropic, model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:16:44 -06:00
DottaandClaude Opus 5.5 6917d0568d fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server starts queued agent runs. A legacy-runtime run gets a 60
s controller lease when it is claimed. Only the run's own execution
renews that lease.
> - The queued-run dispatch claimed all the runs of one agent in a loop.
It started their executions only after the last claim finished.
> - When the database is slow (for example, while a flood of recovery
runs fills the queue), one claim can take tens of seconds. The earlier
claims' leases then expire before their executions start. The run fails
with `controller_lease_lost` and `renewalCount: 0`.
> - Also, the lease watch waited for its first 10 s tick before it
renewed, even when the claim-time lease was almost spent.
> - This pull request starts each run as soon as it is claimed. It also
makes the lease watch renew immediately when the lease is older than one
renewal interval.
> - The benefit is that a slow claim batch no longer makes runs fail
before they do any work.

## Linked Issues or Issue Description

Refs #21127, #21042, #21222

**What happened?**
On staging, legacy-runtime runs failed with `controller_lease_lost`.
Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`,
and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease
watch started on a lease that had already expired, and no renewal was
ever sent. In one run, the claim was at 23:20:02 and the loss was at
23:23:11. Losses came in groups on the same agent at the same moment.
For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all
lost at 23:23:11.6.

**Why did it happen?**
`startNextQueuedRunForAgent` claimed every run of a batch before it
called `executeRun` for any of them. Each claim commits a 60 s lease,
but nothing renews it until the run's execution starts its lease watch.
A queue flood (fixed separately in #21222) made each claim slow, so a
batch took minutes. This bug predates today's changes. The queue flood
exposed it. The run-start hand-over (#21127) did not cause it: no
renewal was queued on the controller-lease connection.

**Expected behavior**
A claimed run starts its execution, and renews its lease, without
waiting for other claims. A lease watch that starts late renews at once.
A lease that lapsed and was revoked, reaped or taken over is still lost
and is never revived.

## What Changed

- `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent`
starts each run's execution right after its own claim, not after the
whole batch. The concurrency cap, the agent start lock and the
run-admission check before each claim are unchanged. If a later claim
throws, the runs claimed before it are already executing, not stranded.
After merging master, the loop walks master's queued-run dispatcher
(#21234: person-initiated runs first, paced recovery delivery): each
successful claim is reported to the dispatcher (`dispatcher.claimed`)
and then started at once, so master's order and pacing are unchanged.
Master's long-queued-run bookkeeping (#21349) is kept too: a claim that
returns nothing is still recorded as declined (`noteDeclinedAgedClaim`),
and a successful claim still clears that record before the run starts.
- `server/src/services/legacy-controller-lease.ts`:
`watchLegacyControllerLease` renews immediately when the claim-time
lease has less than 50 s left (older than one renewal interval). It uses
the same conditional renewal statement. That statement extends only a
lease that this boot still holds on a running row. The deadline is
unchanged.
- Tests: red/green tests for both changes. A test also proves that a
late watch on a revoked, reaped or lapsed lease still loses it and does
not change the row.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass.
- Red on master (source reverted, new tests kept): "starts each claimed
run before claiming the next" fails (`executeRun` not called while the
second claim is in progress). "renews at once when the watch starts on a
lease older than one renewal interval" fails (aborted at the claim-time
deadline). The three "starts late ... still loses it" cases fail on
master because master loses them by deadline and does not check
ownership.
- After deploy: `controller_lease_lost` with `renewalCount: 0` should
drop to about zero on staging, also with a full queue. A late watch now
shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a
silent expired start.

## Risks

- Low. The executions of a batch now start while later claims of the
same batch are still in progress. Execution does not claim runs and is
not awaited by the loop, so the cap and the lock are not affected.
- A late lease watch sends one extra renewal at start. A normal claim
starts its watch within milliseconds with about 60 s left, so it sends
no extra statement and adds no wait to a turn start.
- No schema change. Mixed versions are safe: the renewal statement is
unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:08:38 -06:00
DottaandClaude Opus 5.5 8af889f612 feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP Apps (#21119) show interactive widgets from MCP servers, such as
Circleback meeting cards, with the agent's reply in the task chat.
> - The first version deferred three parts of the MCP Apps spec. A
widget could not propose a chat message (`ui/message`). It could not
give the agent context (`ui/update-model-context`). It could not go full
screen.
> - Paperclip also ignored the `icons` that MCP servers declare on tools
(SEP-973).
> - The security review of #21119 showed what these features need from
day one: explicit consent, untrusted-data labels and strict icon
sanitizing.
> - This pull request adds the three host features and provider tool
icons, with those protections built in.
> - The benefit is that widgets can now work with the person and the
agent, not only display data.

## Linked Issues or Issue Description

Refs #21119

**Subsystem affected**
MCP Apps host (task chat widgets), tool catalog, run wake prompt.

**Problem or motivation**
The MCP Apps host refused `ui/message` (-32601), dropped
`ui/update-model-context` and kept every view inline. MCP tool `icons`
were not parsed or shown. So widgets could not ask the agent a
follow-up, the agent could not see what the person selected in a widget,
large widgets were cramped, and tool rows used only generic icons.

**Proposed solution**
Add consent-gated view messages, per-view app context delivered to the
task's next run as untrusted data, a full-screen dialog, and sanitized
provider icons. Details are under "What Changed".

**Alternatives considered**
- Send view messages with a new server route that creates the comment
itself. Rejected: comment creation has many rules (agent chat ownership,
paused conversations, queued delivery, wakeups, receipts). The widget's
Send button uses the same send path as the composer. A small route then
records which view the comment came from.
- Keep the iframe in place for full screen with CSS. Rejected: chat
containers can create containing blocks that break `position: fixed`. A
moved iframe always reloads, so the dialog hosts a new frame instead.

**Roadmap alignment**
Continues the MCP feature work after #21119, #21140 and #21219. It does
not touch the MCP Tasks work (gateway `tools/call` polling and
elicitation).

## What Changed

- **`ui/message`.** The host shows "<App> wants to send: …" with
**Send** and **Dismiss** under the widget (and inside the full-screen
dialog). Nothing is sent before the click. Send posts the text as the
viewer's own comment through the chat's normal send path
(`handleThreadAdd`). `POST
/api/issues/:issueId/mcp-app-views/:viewId/messages` then logs
`issue.mcp_app_message_sent`. It accepts only the caller's own comment
on the same task, matched by `clientRequestId`, from the last 10
minutes. One proposal per view can wait at a time. The view gets `{}`
after the send, or -32000 "Message sending denied" after Dismiss. Only
`user` text up to 8,000 characters is accepted.
- **`ui/update-model-context`.** `PUT
/api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the
latest context per view in `tool_mcp_app_views.model_context` (text
blocks and `structuredContent`, 16 KB at most). Updates are applied in
order and deduped in the browser and on the server. A change logs
`issue.mcp_app_context_updated`, and an empty update clears the context.
The task's next run gets the three most recently updated contexts as an
`## App context (untrusted)` wake-prompt section. The section quotes the
data as JSON and says it is third-party data, not instructions. The read
runs in parallel with the other run preparation and never fails the run.
CLI adapters and the native runner render it through the same
`renderPaperclipWakePrompt`.
- **Who may do this.** Messages and context need a signed-in person with
`issue:comment` on the task. Agents and API keys get 403. Other tasks
and companies get 404. `message` and `updateModelContext` are advertised
in `hostCapabilities` only where the viewer can send messages.
- **Full screen.** The host advertises `availableDisplayModes:
["inline","fullscreen"]` and handles `ui/request-display-mode`. It
honors the view's declared modes and answers with the resulting mode. It
sends `host-context-changed` with `displayMode` and the container (fixed
`height`/`width` in full screen), and again when that container resizes.
A new **Open full screen** control sits next to "Widget from …". Full
screen is a Paperclip dialog that closes with Exit, Escape or the view's
own request.
- **Provider tool icons.** `icons` on MCP tool descriptors are sanitized
into the new `tool_catalog_entries.icons` column, outside the version
hash, so a new icon never quarantines a reviewed tool. Only `https:`
URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A
`data:` icon is at most 32 KB, a URL at most 2,048 characters, with at
most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists
them keyed by gateway tool name, in one cached request per chat. Task
chat tool rows and the app's actions list show them through
`ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a
fixed size and never inlines markup. It checks the source again before
rendering and falls back to the taxonomy icon.
- **Schema.** Migration `0477` adds `tool_catalog_entries.icons`,
`tool_mcp_app_views.model_context` and `model_context_updated_at`, all
nullable.
- **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new
behavior, and the finished items are removed from "Limits and
follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a
message", "Full screen" and "Tool rows with provider icons". The fixture
view now uses `ui/message`, `ui/request-display-mode` and
`ui/update-model-context`.

## Verification

- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/openapi-routes.test.ts
packages/shared/src/mcp-icons.test.ts
packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These
tests cover:
- Context: stored, deduped, logged once, 413 when too large, cleared,
scoped to the task and company, refused for agents (403), and rendered
into the wake prompt as untrusted data.
- Messages: attributed only to the viewer's own recent comment on the
same task. A teammate's comment, an unknown request id or an old comment
gets 404. An agent gets 403.
- Icons: stored sanitized, and the version hash and status do not
change. Another company sees nothing.
- `cd ui && pnpm exec vitest run src/components/mcp-app/
src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69
pass. These tests cover:
- The host protocol for each new method: capabilities, consent wait, one
proposal at a time, invalid input, ordered and deduped context, the size
limit, and the display-mode rules.
  - The consent card (nothing sent without Send).
  - The full-screen dialog (one frame at a time).
  - Icon rendering (no-referrer `img`, no inline SVG, fallback).
- `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for
the server: clean.
- Red/green: the new route and host tests fail on `master`, where the
routes do not exist and the host answers -32601 or drops updates. The
icon descriptor test failed when the publication write was removed.
- Not run locally: the full suite, and
`packages/adapter-utils/src/server-utils.test.ts › runChildProcess ›
does not crash when the child closes stdin before a delayed write`. That
test times out on this Mac with or without this change, so it is a local
platform issue. CI covers both.

## Risks

- Migration risk is low: three nullable columns, no backfill.
- Every run on a task now does one extra indexed read, in parallel with
the other preparation. Errors are logged and the run continues without
app context.
- If the attribution request fails after the comment is sent, the
comment stays without the `mcp_app_message_sent` activity row. This is
documented.
- Switching between inline and full screen restarts the view, so state
inside the widget resets. This is documented as a limit.
- Icon names shortened with a hash by the gateway (very long names) do
not match, and those rows keep the taxonomy icon.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. One subagent (the same model) built the provider-icon
part.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:32:48 -06:00
DottaandClaude Fable 5.2 39ef43a73e fix(retry): move a recorded continuation forward before its delivery (#21379)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run cannot resume, the server records a continuation for the
task and delivers it later as a new run
> - #21378 made a person's Retry deliver a recorded, undelivered
continuation as the one run, with its workspace isolation
> - #21373 added the `move_forward` Retry action, which does nothing
when the observed run is not queued or running
> - Together, a `move_forward` request before delivery returned "already
moved on" and started no work
> - That state can last a long time when the agent has a deep queue, and
it is when a person wants to move the task forward
> - This pull request accepts the request when the task has a recorded,
undelivered continuation
> - The benefit is one action that moves waiting work forward in each
state, with one run and the recorded isolation

## Linked Issues or Issue Description

**What happened?**

A task has a continuation that is recorded but not delivered. The latest
run is the failed source run. A Retry request with `retryAction:
"move_forward"` returns `accepted: false, alreadyMovedOn: true`. No run
starts.

**Expected behavior**

The request is accepted. Wake admission delivers the recorded
continuation as the one run for the task. The run keeps the recorded
isolation. A later delivery pass creates no second run.

**Steps to reproduce**

1. Record a fresh continuation for a task and do not run the delivery
pass.
2. Call `requestTaskRetry` with `observedRunId` set to the source run
and `retryAction: "move_forward"`.
3. The receipt says `alreadyMovedOn: true`.

Refs #21373, #21378, #21348.

## What Changed

- `admitTaskRetry` in `server/src/services/task-retry.ts` does not
return `alreadyMovedOn` when `undeliveredContinuation` finds a recorded,
undelivered continuation for the task. The check runs under the task row
lock.
- A new test in `fresh-continuation-retry-delivery.test.ts` covers the
three record kinds: `fresh_isolated_workspace`, `writer_ended` and
native.
- The presentation code is unchanged. The interface does not offer the
action in this state yet.

## Verification

- The new test failed before the change for the three kinds (`expected {
requestId: null, … } to match object { accepted: true }`).
- `npx vitest run
src/services/fresh-continuation-retry-delivery.test.ts`: 15 of 15
passed.
- `npx vitest run src/services/task-retry.test.ts`: 117 of 117 passed.
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts -t move`: 2
of 2 passed. The case where the run ended and no continuation is pending
still returns `alreadyMovedOn`.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change adds one read under a lock that the function
already holds. The read runs only for a `move_forward` request when the
observed run is not queued or running. A request without a pending
continuation behaves as before.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 07:31:16 -06:00
DottaandClaude Opus 5.5 ed16a5b1a8 fix(recovery): do not continue from a latest run read while it was still finishing (#21365)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent's run ends without a live next step, a background
sweep finds the stranded task and queues a continuation
> - The sweep reads the task's latest run first, and checks for a live
execution path several queries later
> - A run that finishes between those two reads looks neither live nor
failed. The sweep fell through to its generic continuation
(`issue.continuation_recovery`) with the stale "running" snapshot
> - In a chat, that woke the agent about a second after its turn had
answered the person, and the agent posted a redundant "nothing new is
pending" reply
> - This pull request makes the sweep act only on a latest run it
observed as finished. A run it read while still active is left to its
own completion and to the next sweep
> - The benefit is that a chat turn that already answered is never woken
again by recovery, with no change to real recovery

## Linked Issues or Issue Description

**What happened?**
In a chat, the person asked a follow-up and the agent answered. The
answering run's success was written at 10:09:28.840. A continuation run
was queued at 10:09:28.841 with `wakeReason: issue_continuation_needed`,
`source: issue.continuation_recovery`, and `retryOfRunId` set to the run
that had just succeeded. The agent ran again and posted an extra reply.
The generic continuation source is only reachable when the sweep's
latest-run snapshot is neither succeeded nor live, so the snapshot was
read while the run was still `running`, and the run had finished by the
time of the live-path check.

**Steps to reproduce**
1. Have an assigned `in_progress` task whose latest run is `running`.
2. Start the stranded-work sweep. It reads the latest run as `running`.
3. Before the sweep's live-path check, let the run finish with
`succeeded` and release the task lock.
4. The sweep finds no live path and queues `issue.continuation_recovery`
for the finished run.

**Expected behavior**
The sweep queues nothing for a run it read while still active. Its own
completion settles it, and the next sweep decides from the committed
outcome.

## What Changed

- `server/src/services/recovery/service.ts`: after the live-path check,
the stranded sweep skips a non-review task whose latest-run snapshot was
not terminal. Review tasks keep their participant-run check, which
already does this.
- `server/src/services/recovery/service.ts`: an optional test hook,
`afterStrandedLatestRunRead`, runs after the sweep reads the latest run,
alongside the existing `beforeStaleLockClear` and
`beforeOrphanedRunTerminalWrite` hooks.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: a
regression test finishes the run between the two reads and expects no
continuation.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "still finishing"`
fails without the fix (one `issue.continuation_recovery` wake with
`retryOfRunId` set to the finished run) and passes with it.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts
src/__tests__/issue-recovery-actions.test.ts
src/__tests__/heartbeat-issue-liveness-escalation.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/durable-chat-wakeup.test.ts
src/__tests__/live-recovery-work.test.ts`: 790 passed. One unrelated
test timed out under parallel load and passes on its own (its latest run
is already terminal, so the new check does not apply).
- `cd server && npx tsc --noEmit` passes.

## Risks

- Behavior: for any task whose latest run is `queued`, `running` or
`scheduled_retry`, the live-path check already skipped it, so the only
changed outcome is the race above. A run stuck in `running` with no
owner is still settled by the existing orphaned-run reaper; the sweep
then sees its terminal status.
- Latency: the check is in memory on an already-read row. It adds no
query and nothing before a turn starts; recovery after a real stall is
at most one sweep later only inside the race window.
- Saved state: no persisted shape changes, so old saved runs and mid-run
deploys behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:28:44 -06:00
DottaandClaude Fable 5.2 3f030971dc fix(recovery): deliver a recorded continuation as the person's Retry run, keeping its isolation (#21378)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When Paperclip cannot resume a cut-off run exactly, the
fresh-continuation fallback records a continuation for the task. A
delivery pass later starts it as a run. Staging shows this pending state
can last 30 to 110 minutes for backlog items.
> - The fallback also decides how the next run is isolated from the old
writer. For `fresh_isolated_workspace`, the next run must prepare a new
workspace.
> - A person's Retry pressed while the continuation was recorded but not
yet delivered started its own run at once. That run did not carry the
recorded isolation, so it could use the shared directory next to a
writer that is not proven stopped. It also did not carry the
continuation's recovery context.
> - This pull request makes wake admission deliver the recorded
continuation as the person's run, under the task lock. It also folds a
saved Retry into a delivered continuation for every record kind.
> - The benefit is one run per interruption, whatever arrives first, and
the recorded isolation decision is never lost.

## Linked Issues or Issue Description

Refs #21348

**What happened?**
A person's Retry (`requestTaskRetry` + `resumeTaskRetry`), pressed while
a fresh continuation was recorded but not delivered, was admitted at
once as an ordinary `task_retry` run. That run had no
`freshContinuationIsolation` and no `recoveryActionId`. On a projectless
task the run used the shared directory that the fallback had decided to
isolate from. The later delivery pass marked the continuation as
superseded by that run. The reproduction showed exactly one run, so the
reported "two runs" did not happen in that order. The isolation was lost
for all three record kinds tested. For `writer_ended` and native
records, a Retry saved before delivery was not folded into the delivered
continuation, because the fold only covered `fresh_isolated_workspace`.

**Expected behavior**
With a recorded, undelivered continuation, the next run admitted for the
task is that continuation. It has the recorded isolation and the
person's Retry is recorded as answered by it.

**Steps to reproduce**
1. Record a fresh continuation (`ensureFreshContinuation`, delivery
`recorded`) for a projectless task whose legacy writer is on another
host.
2. Call `requestTaskRetry` and then `resumeTaskRetry`.
3. Before this change: one `task_retry` run without
`freshContinuationIsolation` or `recoveryActionId`, while the record is
still `pending`.

## What Changed

- `fresh-continuation-retry-fold.ts`:
- `undeliveredContinuation` reads the task's pending fresh-continuation
record under the task lock (`for update`). It returns nothing when a run
of the task was created after the record.
- `undeliveredContinuationContext` gives the run the continuation's
`recoveryActionId`, `retryOfRunId`, `forceFreshSession` and, for
`fresh_isolated_workspace`, `freshContinuationIsolation`.
- `deliverContinuationAsRun` marks the record `delivered` to the new run
in the same transaction, settles its coalesced holds, and folds other
waiting Retries.
- `foldWaitingRetriesIntoRecordedContinuation` folds a waiting Retry
into any delivered fresh continuation, not only an isolated one.
- `heartbeat/queue.ts` (wake admission, issue transaction): a
non-system, non-delivery wake with no execution blocker takes the
pending continuation. The run is inserted as before, then delivered as
the continuation. The fold after a delivery wake now applies to every
fresh-continuation kind. No change to ranking or to the claim loop.
- `execution-recovery-resolution.ts`: the delivery pass folds a saved
Retry for every fresh-continuation kind. The superseded-run fold keys on
`recoveryActionId`.
- The person's run keeps its `task_retry` wake reason and the user's
wake receipt, so the existing #21349 rules rank it as person work.
- Limits: the person-triggered delivery does not spend an automatic
attempt. It adds no `issue.continued_automatically` entry and no
scheduled transient retry, and it is not gated by the hourly cap or by
delivery-pass pacing. The record itself was already counted when it was
recorded. The agent's concurrency limit still applies at claim time.

## Verification

New file
`server/src/services/fresh-continuation-retry-delivery.test.ts`, with 12
tests. All cases cover `fresh_isolated_workspace`, `writer_ended` and
native (`remote_epoch`) records, through the real `requestTaskRetry` +
`resumeTaskRetry` path:
- A Retry during pending delivery gives exactly one run. That run is the
continuation, carries the recorded isolation (the workspace readiness
reads `fresh_isolated_workspace`), coalesces the Retry, and ranks as
person work. Later delivery passes add nothing.
- Delivery first, then a saved Retry: one run, and the Retry is
coalesced. A Retry on the delivered run joins it through the busy path.
- A Retry and two delivery passes race (concurrent transactions): one
run.
- After the continuation's own failure, a Retry gets its own run.
- The delivery pass is held by pacing (backlog source, agent queue
full), and the hourly cap and the shared attempt ledger are exhausted.
The person's Retry still starts the run.

Red before the fix: 8 of 12 failed. Green after: 12 of 12.

Local runs, one file at a time:
- `fresh-continuation-fallback.test.ts`: 71 of 72. The failure is "runs
a person's Retry pressed before delivery…", a deadlock in `afterEach`
truncate. It passes alone, and it fails the same way on unmodified
master under this machine's load.
- `__tests__/recovery-wake-herd.test.ts`: 29 passed
- `__tests__/execution-control-work.test.ts`: 27 passed
- `__tests__/issue-recovery-actions.test.ts`: 112 passed
- `workspace-recovery-fallback.test.ts`: 19 passed
- `heartbeat/queue.test.ts`: 101 passed
- `task-retry.test.ts`: 117 passed;
`legacy-preparation-retry-hold.test.ts`: 5 passed
- `server` `tsc --noEmit`: clean

## Risks

- A person's message, chat input or manual wake that arrives while a
continuation is pending also becomes that continuation, with a fresh
session and the recorded isolation. Before, such a wake ran without
isolation and superseded the record. This is the same rule for every
admission path, and it closes the shared-directory gap there too.
- A wake that takes the context and is then deferred by a later gate can
keep the isolation marker in its saved context. If the record is
delivered meanwhile, that later run prepares a new isolated workspace.
That is the safe direction.
- No schema change and no migration. Nothing is added before turn start
for healthy runs: the new read runs only at wake admission.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, file edit,
test runs). Model ID `claude-opus-5-5`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 07:13:49 -06:00
DottaandClaude Opus 5.5 5ba26600ef fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs agent turns through a Runner, often in a
remote sandbox, and must recover runs after a restart without help from
a person
> - When a Runner process is lost, a new Runner restores the
conversation. If the provider turn is gone, the Runner sends
`turn.failed` with `provider_turn_lost_on_restore` and `recoverable:
true`. The Runner sends this only after it proves that the old process
ended
> - The runtime can continue such a turn in the same conversation, but
only when the server sets `resumeInterruptedTurn`. The server set it
only for a local `resume_dead_runner` claim or a scheduled resume, and a
scheduled remote resume is refused. So on a remote Runner reattachment
the runtime did not continue the turn, and the run ended
> - This pull request sets `resumeInterruptedTurn` for a
`reattach_remote_runner` claim too
> - The benefit is that a remote run continues the lost turn in the same
conversation immediately, instead of failing and then restarting 3 to 6
minutes later

## Linked Issues or Issue Description

**What happened?**
On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3
hours ended with `native_provider_terminal_failed` and the message "The
runner restored the conversation after process loss, but the previous
turn is no longer active." The Runner marked each of these events
`recoverable: true`. Example: run 646e2efb. A new Runner process started
at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed
provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run
failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at
09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In
each case, a separate recovery path continued the task only 3 to 6
minutes later.

**Expected behavior**
When the Runner marks a lost turn as recoverable after a restore on a
remote reattachment, the run continues the turn in the same
conversation.

**Steps to reproduce**
1. Start a native run on a remote sandbox Runner.
2. Lose the Runner process while a turn is active, so that a new Runner
reattaches through a `reattach_remote_runner` claim.
3. The new Runner restores the conversation and reports
`provider_turn_lost_on_restore` with `recoverable: true`.
4. See that the run ends with `native_provider_terminal_failed` and does
not continue the turn.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with Codex on a Daytona remote
sandbox.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
`resumeInterruptedTurn` is now also true for a `reattach_remote_runner`
restart-recovery claim. The runtime still continues the turn only when
the Runner itself sends `turn.failed provider_turn_lost_on_restore` with
`recoverable: true` for the checkpointed active turn.
- `doc/execution-semantics.md`: one paragraph describes this behaviour,
and says that a lost turn that still ends the run keeps
`native_provider_terminal_failed`.
- Tests:
- `native-scheduled-resume-continuation.integration.test.ts`: "continues
a lost turn on a same-run remote Runner reattachment" (2 cases: a plain
reattachment and a `replaceUnansweredRunner` reattachment).
- New `native-recoverable-lost-turn.integration.test.ts`: a guard
through `executePaperclipNativeSession`. The runtime saves a
failed-terminal checkpoint and then raises the lost turn (recoverable
and not recoverable). The run ends as `native_provider_terminal_failed`,
and the hold cause stays one that the replacement lane continues.
- An earlier revision of this PR also mapped a recoverable lost turn to
`native_session_interrupted`. Review showed that this does not make the
run retryable, and the hold it writes is not picked up by any automatic
continuation. That hunk is removed.

## Verification

- Red without the product change: the two integration cases fail with
`passed` = `[false]` (expected `[true]`).
- Green with the change:
- `cd server && npx vitest run
src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts
src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts`
— 14 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
— passed (with the scheduled-resume file: 224 passed, 14 skipped,
pre-existing skips).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts` — 1469
passed, 7 skipped.
  - `cd server && npx tsc --noEmit` — exit 0.
- Staging evidence (hosted staging, build `d1d71a7755`, read-only
trace): 10 runs in 3 hours ended with `native_provider_terminal_failed`
although the Runner marked the lost turn `recoverable: true`. Run
646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49,
`turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run
failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed`
09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes
later through a separate recovery path.

## Risks

- This change has not run live on staging or on a Daytona sandbox yet.
The evidence is the staging trace and the local tests.
- This PR does not introduce or widen the gap where a recoverable lost
turn still ends the run. On master, that run already ends as
`native_provider_terminal_failed`, and the replacement lane continues it
after its decision (3 to 6 minutes on staging). This PR removes the
run-ending case whenever the Runner reports the turn recoverable for the
checkpointed active turn on a remote reattachment. Where the run still
ends (for example, when the continuation itself is lost), the behaviour
is unchanged, and the guard test
`native-recoverable-lost-turn.integration.test.ts` proves that it stays
`native_provider_terminal_failed`.
- The remaining case is tracked with the owner of the recovery path:
recovery evidence will accept `provider_turn_lost_on_restore` /
`recoverable` directly, after their ownership-deadline PR #21364. An
earlier hunk here tried to shorten that case. An integration test
through the real path showed that it would create a board-owned
ambiguous hold that no automatic pass continues, so the hunk is removed.
- This change does not fix why the turn is lost. In staging, a retry
replaced a live Runner. A separate fix for that is in progress.
- The runtime continues a turn only when the Runner sends the
recoverable lost-turn signal. The Runner sends it only after it proves
that the old process ended. So no provider turn runs twice.
- No Runner image change. No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:09:20 -06:00
DottaandClaude Opus 5.5 d9b42b50fd feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and continues the task by itself. Since #21353 the task reads as
continuing (`working`, no Retry) for as long as that continuation waits
> - That is truthful, but a person has no control while it waits. On
hosted staging one agent had 21 running and 209 queued runs, and
continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a
waiting continuation into the agent's queue within minutes, so most of
that wait is in the queue, behind other work
> - #21353 removed Retry from this state because a Retry started its own
run, which raced the continuation and could give the task two runs
> - A Retry that observes the queued continuation run cannot start a
run: admission sees a queued run and joins the Retry to it (`coalesced`,
`runId` = that run). Under #21234's ranking, a queued run that a person
joined starts ahead of system recovery and ordinary work
> - This pull request keeps the continuing state and, once the queued
continuation has waited 15 minutes, also offers Retry labelled "Move it
forward now"
> - The benefit is that a person can move a long-waiting task to the
front of its agent's queue, and the click can never create a second run

## Linked Issues or Issue Description

Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200

**What happened?**

On hosted staging, tasks with a recorded continuation waited 30 to 110
minutes behind a deep agent queue. The task showed "continuing" with no
action. The person could only wait.

**Expected behavior**

The task still never shows "failed". After a long wait the person can
move the waiting continuation forward. The click must not start a second
run.

## Which states a Retry joins (established before building)

I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its
tests, and drove the real click path (`requestTaskRetry` followed by
`resumeTaskRetry`, as the route does) in a probe against each record
state:

| Continuation state | What a Retry does today | Offer Retry? |
|---|---|---|
| Recorded, waiting for delivery (`continuationDelivery: "pending"`, no
run yet) | The Retry is admitted at once as its own `task_retry` run,
for every isolation (`fresh_isolated_workspace`, `writer_ended`,
native). The delivery pass then marks the record delivered (superseded).
#21348's fold never runs: it folds only a Retry that is still
`deferred_issue_execution` when the continuation run is created, and
only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred,
so the click races the delivery pass. | **No** (unchanged) |
| Delivered, continuation run `queued` | `requestTaskRetry` sees the
latest run is busy and records the Retry as `coalesced` into that run in
the same transaction. No run is created; `resumeTaskRetry` acts only on
deferred Retries. The run is now ranked as a person's (`joinedByUser` in
`queue.ts`). | **Yes, after 15 minutes** |
| Continuation run `running` | Same join, but the run already started,
so there is nothing to move forward. | No |
| Continuation ended | A Retry gets its own run (the existing failed-run
path). | Existing behaviour |

So the Retry offered here is joined by the existing busy-run admission
in `requestTaskRetry`, not by #21348's fold. Its source run is the
queued continuation itself (the run the presentation names as
`observedRunId`), not the run the continuation was created for.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: for a task
whose latest run is `queued`, created 15 minutes ago or more, and named
as `continuationRunId` by a delivered fresh-continuation record, the
projection returns `state: "working"`, `canRetry: true`, `retryAction:
"move_forward"`, responsibility `system`, `observedRunId` = that queued
run. It does not offer this when a person's wake already joined the run,
when the run's agent is not the assignee, or when a person-owned wait
applies (approval, reconnect, budget, pause and the others are checked
first). Two small reads run only for queued runs older than 15 minutes.
The existing restore-hold check that withdraws `canRetry` also withdraws
`retryAction`.
- `packages/shared/src/types/execution-projection.ts`: new optional
`retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It
only changes the label; `canRetry` still decides whether Retry is
offered.
- `ui/src/components/ExecutionBlockerNotice.tsx`: a `working`
presentation with `retryAction: "move_forward"` shows "Waiting for the
agent to finish other work. Paperclip continues this task by itself."
and a "Move it forward now" button that calls the same task Retry.
Token-only classes; `pnpm check:token-gates` is clean.
- One join-only action, `move_forward`, carried from the offer to
admission (contracts kept in sync):
- `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts
`retryAction: "move_forward"`. The route already passes the body to
`requestTaskRetry`.
- `server/src/services/task-retry.ts` (`admitTaskRetry`): right after
the stale check, still under the task row lock and before any other
Retry path, a `move_forward` Retry whose observed run is no longer
`queued` or `running` returns `{ accepted: false, alreadyMovedOn: true,
requestId: null }` and writes nothing. This block was cleared with the
owner of Retry admission. The owner will later extend it so that
"continuation recorded, not yet delivered" also counts as actionable,
after their pending-delivery change lands (about two lines on their
side). Until then this presentation never offers the button in that
state, so the early return cannot misfire from our UI.
- `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click
sends `retryAction: "move_forward"` when the offer has it.
`alreadyMovedOn` shows "The task already moved on, so nothing new was
started. It now shows its current state."
- What a click does at each timing: continuation `queued` → joined to
that run (moves it forward); `running` → joined (nothing more to do);
ended → "already moved on", nothing started.
- A joined click cannot become a run later. The busy path writes it as
`coalesced` with `runId` set to the continuation and `finishedAt` set
(`task-retry.ts:456-461`). Promotion after a run ends
(`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`;
`modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and
the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`)
and `resumeExecutionWaitComments` (`:876`) pick up only
`deferred_issue_execution` wakes.
- `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry
for this state. That row sends the failed run as the observed run, which
is no longer the latest run, so its Retry would be refused as stale.
- `doc/execution-semantics.md`: one paragraph on the offer, when it
ends, and why there is none before delivery.

## Verification

- Red (before the product change, new tests only), in `server/`: `pnpm
exec vitest run src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both
failures are the missing offer (`canRetry: false`, no `retryAction`) for
a continuation queued 15+ minutes.
- Green after merging master 8e41247b65, in `server/`: `pnpm exec vitest
run src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-automatic-path.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/task-interruption-query.test.ts`: 5 files, 279 tests pass.
UI: `useTaskRetry.test.tsx` and `ExecutionBlockerNotice.test.tsx`: 35
tests pass.
- New tests:
- (b, integration, `recovery-wake-herd.test.ts`) A continuation
delivered by the real delivery pass waits 16 minutes behind older
ordinary work. The task offers "move forward". The click goes through
the real `requestTaskRetry` and `resumeTaskRetry`. Result: the task has
exactly one run after the source run, the continuation; the Retry wake
is `coalesced` with `runId` = the continuation; the offer disappears;
the next delivery pass adds nothing; the real dispatcher
(`resumeQueuedRuns`) starts the continuation ahead of the older work;
after it ends, still only that one run.
- (Greptile P1, red then green) "starts nothing when the continuation
ended between showing Move it forward now and the click": the
continuation ends after the offer is read. Before the admission block
the click was accepted (`accepted: true`, a new deferred Retry that
becomes a turn). Now it returns `alreadyMovedOn`, the task has one run,
and there is no `task_retry` wake.
- "never turns a joined move-forward click into a new run after the
continuation it joined ends": the click joins the queued continuation.
The run is then cancelled through the real `cancelRun` release. Two
rounds of `resumeExecutionWaitComments`, `resumeTaskRetry`, the delivery
pass and `resumeQueuedRuns` follow. The joined wake stays `coalesced` on
the ended run. No run comes from the click, and the task's later runs
equal those of an identical task with no click (the system's own
`issue_assignment_recovery`).
- UI hook (`useTaskRetry.test.tsx`): the click sends `retryAction:
"move_forward"`, and the `alreadyMovedOn` outcome shows the plain
message.
- (a) Under 15 minutes: no Retry. Control in the herd file: without a
click the older work starts first.
- (b, presentation) Queued 15, 45 and 110 minutes: `working`, `canRetry:
true`, `retryAction: "move_forward"`.
- (c) No Retry while the continuation waits for delivery (16, 45, 110
minutes), when it is running, when a person already joined it, for a
queued run that is not a recorded continuation, and a paused agent still
asks the person to resume.
- UI (`ExecutionBlockerNotice.test.tsx`): plain wording with one "Move
it forward now" button that calls Retry; no button for a read-only user;
nothing shown for a plain `working` state.
- (d) The #21200, #21264, #21320 and #21353 guard tests pass unchanged,
including "follows the new run once a recorded continuation is
delivered" (a just-queued continuation shows no Retry).
- `node scripts/typecheck-layers.mjs` in `server/`, `tsc --noEmit` in
`packages/shared`, `pnpm run typecheck` in `ui/`: clean.
- Not related to this change:
`src/services/fresh-continuation-fallback.test.ts` has one failure
("starts the continuation while the unreachable old lease is active", a
deadlock in the `afterEach` truncate) that also fails on master
18fb6ee571 without this change.

## Risks

- Moving a waiting continuation forward lets it start before other
queued work of the same agent. That is the purpose, and it is the
ranking #21234 already gives any run a person joined.
- No Retry is offered while a continuation is still waiting for
delivery. The delivery pacing admits a waiting continuation into the
queue within minutes, and a fresh interruption at once (#21354), so the
offer appears after the continuation reaches the queue. For the recovery
owner (not changed here): offering Retry before delivery would need the
Retry admission to hold or deliver the pending continuation itself under
the task lock, instead of starting its own run beside it.
- Admission changes only for a request that carries `retryAction:
"move_forward"`. Every other Retry is unchanged. No schema, migration or
dispatch change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:03:23 -06:00
DottaandClaude Opus 5.5 eb54831d28 fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:53:38 -06:00