Commit Graph
3943 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 0904a65d82 fix(server): local board boundary identifies agent processes on Linux it cannot inspect (#21209)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the operator's own local processes get
implicit board access without credentials
> - #21181 added a local board boundary. It refuses implicit board
access to requests that come from an agent run's process tree
> - On Linux CI hosts, the boundary sometimes did not decide a loopback
request. It logged a warning and kept the earlier permissive behavior
for that request
> - The cause was not IPv6. IPv4 and IPv6 loopback lookups worked. The
gap was a connection that belonged to a same-user process that the
server cannot inspect
> - This pull request makes the Linux boundary decide those connections
from the process ancestry that the server can read, and refuse them when
that ancestry is not clearly the operator's
> - The benefit is that the local board boundary now holds on Linux for
IPv4 and IPv6 loopback, and the operator's browser and CLI keep working

## Linked Issues or Issue Description

Refs #21181

**What happened?**
On Linux CI hosts, the local board boundary sometimes could not
attribute a loopback request to a process. It then logged that it could
not identify the local process and kept the earlier permissive behavior
for that request. In one chat eval cohort, 37 of 191 server logs
contained this warning. The warning is logged once per server process.

**Expected behavior**
The boundary decides every uncredentialed loopback request on Linux. A
request from an agent run gets no implicit board access. Requests from
the operator's browser and CLI keep implicit board access.

**Actual behavior**
Some requests on Linux were not decided by the boundary, and they got
implicit board access.

## What Changed

- The Linux inspector now records which same-user processes it cannot
inspect. Before, it only recorded that one existed.
- The Linux TCP table lookup now also returns the socket's owner UID. It
covers `/proc/net/tcp` and `/proc/net/tcp6`, including IPv4-mapped
addresses.
- If no inspectable process owns the connection, the server reads the
TCP tables again. If the connection is gone, has changed, or belongs to
a different user, the request is refused.
- If the connection is still open and same-user, the server checks every
process it cannot inspect. Implicit board access stays only if each one
has a readable parent chain that does not belong to an agent run.
Otherwise the request is refused.
- The "cannot identify" fallback now applies only when the platform has
no inspector, or when Linux cannot read the TCP tables. The server keeps
logging a warning in that case.
- `doc/DEPLOYMENT-MODES.md` describes the new Linux behavior.

**How the operator is recognized**
- Browser and CLI processes are normal same-user processes. The server
reads their open sockets and environment, so their requests are
classified as before.
- A browser helper process that the server cannot inspect is still the
operator when its readable parent chain leads to the browser and has no
agent-run markers.
- The server's own requests and requests from the server's ancestors
stay the operator, as before.
- Connections owned by other users were refused before, and they are
still refused.

## Verification

- `pnpm --dir server exec vitest run
src/__tests__/local-agent-process-boundary.test.ts` passes on macOS (15
passed, 10 Linux-only skipped).
- The same file ran on Linux (Docker, `node:24-bookworm`, non-root user,
IPv6 loopback on). Result: 25 passed.
- Red check on Linux with the previous boundary and the new tests: the
agent-run cases for processes the server cannot inspect failed for both
`127.0.0.1` and `::1`. The operator cases passed.
- New Linux-only tests run real processes over `127.0.0.1` and `::1`:
  - an operator process keeps board access
  - an agent-run process is refused
  - an operator process the server cannot inspect keeps board access
  - an agent-run process the server cannot inspect is refused
- New platform-independent tests cover the classification rules and the
`/proc/net/tcp6` fixture parsing.
- A paid chat-feature run on Linux CI, with temporary diagnostic logging
on a separate branch, checks the server logs for the warning and for
refused operator requests. The results will be in a PR comment.

## Risks

- A rare operator tool that the server cannot inspect, and whose parent
chain the server cannot read, now loses implicit board access. Such a
tool can use a board API key.
- The extra TCP table read happens only when a connection has no
inspectable owner. Normal requests do no extra work.
- Platforms without process inspection keep the previous behavior.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, Docker on Linux) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:52:06 -06:00
DottaandClaude Opus 5.5 b51f1cb55f test(chat): size the long Board publication tests' timeout to their measured work (#21205)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-channel integration suite is one 86,000-line file. Its
cases share one database, and the chat services scan that whole database
> - The "Lossless long Board publications" cases publish
100,000-character messages in many durable parts. They use the file's
default 15 s test timeout
> - Run alone, a case takes about 0.6 s. Late in a full run of the file,
the same cases take 6–10 s locally and 1.4–15 s in CI, because earlier
cases leave rows that the services still scan
> - On #21041, "keeps an ambiguous slack text part blocked across
restart…" hit the 15 s limit twice (30 041 ms with its retry), and the
GitHub variant passed at 14 960 ms
> - This pull request gives this describe block a 45 s timeout, sized to
the measured work
> - The benefit is that the chat shard stops failing on unrelated PRs
when these cases run late in a shard

## Linked Issues or Issue Description

**What happened?**
`server/src/__tests__/chat-channels.integration.test.ts` > "Lossless
long Board publications" > "keeps an ambiguous slack text part blocked
across restart and retries only that part and its tail" failed with
`Test timed out in 15000ms` on PR #21041 (General tests chat 2/3), a PR
that did not change chat code. It passed on rerun.

**Expected behavior**
The case passes wherever the shard puts it.

**Steps to reproduce**
Run the whole file: `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts --reporter=json`. In my
local full run, "preserves all 100000 characters for slack new-comment
publication" took 9 505 ms and "keeps an ambiguous slack text part…"
took 5 856 ms. Run alone, they take 817 ms and 577 ms. In CI logs from
the last 48 hours, the ambiguous-part cases took between 1 408 ms and
over 15 000 ms.

**Paperclip version or commit**
master at 4e932fd178 and later.

**Deployment mode**
CI server tests (embedded PostgreSQL).

**Agent adapter(s) involved**
N/A, CI test.

## What Changed

- `server/src/__tests__/chat-channels.integration.test.ts`:
`describe("Lossless long Board publications", { timeout: 45_000 }, …)`.
Other heavy publication cases in this file already use 40 s.

## Verification

- With an injected 18 s wait in the ambiguous slack case, the case
passes on this branch. On master the same wait fails at 15 s.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"Lossless long Board publications"`: 16 of 16 pass.

## Risks

- Low risk. Test-only change. A real hang in these cases now fails after
45 s instead of 15 s.
- This does not fix the underlying slowdown: cases get about 10 times
slower late in this file because services scan rows that earlier cases
leave in the shared database. Splitting the file or giving groups their
own database is the structural fix. That is larger than this PR.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, GitHub CLI, Vitest).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:35:30 -06:00
DottaandClaude Fable 5.2 161ef8121e perf(recovery): prepare the blocker-stall waiter read once instead of writing its SQL for every stalled task (#21201)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a task stalls on a recovery action that needs the board,
recovery wakes the agents of the tasks that wait on it: its parent and
the tasks it blocks (`wakeWaitersOfStalledIssues` in
`server/src/services/recovery/blocker-stall-wakeups.ts`).
> - Each pass reads the waiters of every stalled task with one statement
for each stall. The statement has 18 parameters and two correlated
subqueries. Its SQL text is 0.8 KB, and it is the same for every stall
of the same shape.
> - The server wrote that text with the query builder for every stall on
every pass. On a hosted instance the server main thread was near one
full core and garbage collection used a quarter of it. A statistics
sample showed this statement at about 7.8 executions per second.
> - I measured it on a local PostgreSQL through the real client. The
query builder allocates 103 KB to write and run this statement. The same
statement from a prepared query allocates 15 KB.
> - This pull request writes the SQL text of the waiter read one time
for each client and shape. Each execution then only binds values.
> - The benefit is 76% less allocation and 5 times fewer garbage
collections for a pass over 20 stalled tasks. The statement, its
parameters and its rows do not change.

## Linked Issues or Issue Description

Related: #21195 (the same change for the native chat progress page read;
it added the `preparedStatements` helper that this pull request uses),
#21131 and #18965. No issue exists for this read.

**What existing behavior does this improve?**

The cost, in the server process, of the waiter read that the
blocker-stall wake pass runs for each stalled task.

**Subsystem affected**

Recovery: blocker-stall wakes
(`server/src/services/recovery/blocker-stall-wakeups.ts`).

**Current behavior**

`wakeWaitersOfStalledIssues` builds the waiter query with the Drizzle
query builder inside its loop, and Drizzle then writes the SQL text from
it, one time for each stall and page. Measured on the merge base, Node
26, local PostgreSQL 18, median of 5 passes of 300 calls. The fixture
has 20 stalled tasks, each with a parent and one dependent task that are
already woken, so a pass writes nothing:

| Call | Statements | Waiter SQL texts written | Allocated | Process CPU
| Collections per 1,000 calls (pause time) |
| --- | --- | --- | --- | --- | --- |
| Company pass over 20 stalls | 21 | 20 | 2,298.3 KB | 3,855 µs | 100
(163.7 ms) |
| Pass for one stalled task | 4 | 1 | 355.0 KB | 547 µs | 13.3 (14.3 ms)
|

At the sampled rate of 7.8 executions per second the read allocates
about 0.8 MB per second. This rate is one sample of one instance, taken
soon after a long outage, when recovery had a backlog. It is not a
steady-state figure.

**Proposed behavior**

The query moves into one builder function, `stallWaiterPageQuery`. The
module prepares it one time for each client and shape, with placeholders
for the values. The shapes are: the stalled task has a parent or not;
all waiters or one waiter; first page or later page. The loop executes
the prepared statement.

**Reason and benefit**

Same benchmark, this branch:

| Call | Statements | Waiter SQL texts written | Allocated | Process CPU
| Collections per 1,000 calls (pause time) |
| --- | --- | --- | --- | --- | --- |
| Company pass over 20 stalls | 21 | 0 | 547.3 KB | 1,474 µs | 20 (24.8
ms) |
| Pass for one stalled task | 4 | 0 | 266.9 KB | 496 µs | 10 (9.9 ms) |

One waiter read costs about 88 KB less. At the sampled rate this removes
about 0.7 MB per second of allocation, which is about 0.9% of the 77.7
MB per second that the instance allocated. It is a small part of the
total. The CPU numbers come from a machine with other load and are less
stable than the allocation numbers.

**Breaking changes**

None.

## What Changed

- `server/src/services/recovery/blocker-stall-wakeups.ts`: the waiter
query moves out of the loop into `stallWaiterPageQuery(db, binding)`.
`stallWaiterPageStatement` returns the prepared statement for the shape
and the values to bind. The loop executes it. The lines inside the two
SQL fragments keep their indentation, so the SQL text is byte-identical.
The pull request adds no comment. The one existing comment of this read
stays at the call site, where the code still makes that choice.
- `server/src/__tests__/issue-blocker-stall-waiter-statement.test.ts`
(new): statement equivalence, row equivalence on PostgreSQL, and the
red/green mechanism test.
- `server/src/__tests__/helpers/blocker-stall-fixture.ts` (new): the
fixture that the test and the benchmark share.
- `server/src/__tests__/prepared-statements-driver.test.ts` (new):
proves at the driver that a prepared read and the query builder ask
postgres.js for the same thing, with the driver's prepared-statement
setting unset, false and true. It covers this read and the native chat
progress page read of #21195.
- `server/scripts/benchmark-blocker-stall-waiters.ts` (new, no
comments): the benchmark above. It loads the module at `--baseline-ref`
next to the current module in one process. `sweep` is one company pass
over `STALLS` stalled tasks whose waiters are already woken: one stall
read, then one waiter read for each stall. `task` is the pass for one
stalled task. Before it measures, the script also runs a company with a
paused agent and 101 waiters on a stall with a parent and on a stall
without one, and one pass for a single waiter of each. This covers six
shapes: the first page, the later page and one waiter, each for a stall
with a parent and for a stall without one. The script fails unless both
versions send the same statements, byte for byte, with the same
parameters and return the same result. Passes alternate between the
versions with a full collection before each pass. CPU is the whole
process. Allocated bytes are V8's `total_allocated_bytes`. Optional
environment variables: `STALLS` (default 20), `ITERATIONS` (default 300
calls for each pass), `PASSES` (default 5 for each version).

Not changed: which statements run, their order, the other reads of the
pass (they are small and vary by waiter), the wake itself, the driver's
prepared-statement setting, and the schema. The read takes no lock and
runs on the root client outside a transaction. Its two lists (waiter
statuses and wake statuses) are module constants, so the statement has
no variable-length list parameter.

## Verification

- Statement equivalence (`sends the SQL text and the parameters that the
query builder writes, for every shape of the read`): for all eight
shapes the prepared SQL text equals the text that the builder writes
with the values, and the bound parameters are equal. The test also
proves that a stall without a parent binds `null` for the parent.
- Against the merge base at the driver: the benchmark compares every
statement that the merge base module and this branch send for six of the
eight waiter-read shapes. The texts are byte-identical and the
parameters are equal. The two shapes that it does not reach (one waiter
on a later page) cannot occur: a one-waiter read returns at most one
row, so it has no later page.
- Row equivalence on PostgreSQL (`returns the rows of the query builder:
same values, types and order, and no other company's rows`): two
companies. The fixture has a stall with a parent and six dependents, and
a stall without a parent. One dependent is already woken, one is done,
one is hidden and one has no assignee. One has a failed wake only, and
one has a wake row with the same key in the other company; both stay
waiters. The prepared statement and the builder return strictly equal
rows for all waiters, one waiter, a later page, and a stall read under
the other company's id (no rows). Null columns stay null.
- Mechanism, red then green (`writes the SQL text of the waiter read one
time and then only binds values`): a pass over three stalls writes no
waiter SQL text and sends one identical text three times. On the merge
base module the same test fails: the pass writes three texts.
- Driver behavior (`asks the driver for the same thing as the query
builder with DATABASE_PREPARED_STATEMENTS=...`, three cases): Drizzle's
`.prepare(name)` on postgres.js does not send the name to the database.
`PostgresJsSession.prepareQuery` drops it
(`drizzle-orm/postgres-js/session.js`, lines 87 to 100), and
`PostgresJsPreparedQuery.execute` calls `client.unsafe(text, params)`
(lines 33 and 43). An ordinary query reaches the same call through
`_prepare().execute()` (`pg-core/query-builders/select.js`, lines 795 to
799). The test runs both reads through the builder and through the
prepared statement on one connection for each setting. The arguments and
the options of the driver call are strictly equal. With the setting
unset or false, the server has no named statement after either path, so
a transaction-mode pooler sees no change. With the setting true, the
server has one named statement. Its name is the one that postgres.js
generates for the connection, and both paths use that same statement.
- Existing tests: `issue-blocker-stall-wakeups.test.ts` and
`legacy-workspace-rehome.test.ts` passed (30 tests) with the new file (3
tests). `prepared-statements-driver.test.ts` passed (3 tests).
- `pnpm --filter @paperclipai/server typecheck` passed. `pnpm
check:module-boundaries` passed.
- `pnpm verify` returns exit 2 (broader verification required, because
the change adds a file under `server/scripts`). I did not run the full
local suite. CI supplies that coverage.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-blocker-stall-waiter-statement.test.ts \
  src/__tests__/issue-blocker-stall-wakeups.test.ts src/__tests__/legacy-workspace-rehome.test.ts \
  src/__tests__/prepared-statements-driver.test.ts
TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-blocker-stall-waiters.ts \
  --baseline-ref=549816e6a0
```

Not verified: the effect on a hosted instance. The rate above is one
sample. The local numbers are from Node 26; the hosted instance runs
Node 24.

## Risks

- Low. The statement text and parameters are equal by test, the read
takes no lock, and it runs on the root client outside a transaction.
- A prepared statement keeps the text of the first build. The lists
inside the statement are module constants, so the text cannot go stale
in a running process.
- A test that mocks `db.select` to observe this read no longer sees it.
I found none for this read.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude
Code with tool use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:32:29 -06:00
DottaandClaude Opus 5.5 6ac7211cb9 fix(heartbeat): stop re-delivering a remote-stop follow-up that a succeeded run already read (#21203)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's follow-up comment must reach the agent once
> - #21175 made a succeeded run's mid-turn read of a comment (through
`get_task_history`) count as delivery for the queued-comment paths
> - The remote-Stop resume path decides delivery with
`undeliveredLegacyUserCommentIds`, which ignores those reads
> - So a follow-up that a succeeded run already read was sent again, and
the agent posted a second DONE on a finished task
> - This pull request applies the same rule on that path and adds a read
helper for a later change
> - The benefit is one answer per follow-up on every resume path, with
no dropped messages

## Linked Issues or Issue Description

Refs #21175

**What happened?**
A saved follow-up comment was resumed after a remote Stop (wake reason
`issue_commented`, idempotency key `remote-stop-comment:…`). A succeeded
run had already read this comment mid-turn. The resumed wake started a
new turn, and the agent posted a second DONE on a finished task. The
cause: `undeliveredLegacyUserCommentIds`
(explicit-native-continuation.ts) checks only wake input and accepted
first turns. It does not check the `task_history.read` records that
#21175 added.

**Expected behavior**
A follow-up that a succeeded run read at its current revision is
delivered. The remote-Stop resume path skips it. A read by a run that
did not succeed does not count, so that follow-up is still delivered
once.

**Steps to reproduce**
1. A native run reads the task history after a person posts a follow-up.
2. The run succeeds. The follow-up stays saved as a deferred wake.
3. The remote-Stop resume path runs for that wake.
4. Before this change, the follow-up is delivered again.

**Paperclip version or commit**
`master` at 549816e6a0.

## What Changed

- `server/src/services/queued-comment-delivery.ts`:
`historyReadDeliveredCommentIds` exposes the #21175 rule for a list of
comment IDs. `taskHistoryReadsForRun({ companyId, runId })` returns each
comment one run read, with the read time and the comment revision that
the read returned. It returns these whatever the run's outcome. It reads
only that run's events through `heartbeat_run_events_company_run_idx`.
No delivery decision uses it.
- `server/src/services/explicit-native-continuation.ts`:
`undeliveredLegacyUserCommentIds` also removes comments that a succeeded
run read at their current revision. Its two callers are the remote-Stop
resume path and the queued explicit-continuation check, and both now use
the #21175 rule.
- Deliberate asymmetry: a read by a run that did not succeed never
counts. A dropped person's message is worse than a duplicate answer.
- `doc/run-log-events.md`: documents both points.
- New test file
`server/src/services/queued-comment-history-read.test.ts`.

## Verification

- RED on `549816e6a0` (`npx vitest run
src/services/queued-comment-history-read.test.ts`): 2 failed, 1 passed.
  - The follow-up that a succeeded run read was still returned as owed.
  - `taskHistoryReadsForRun` did not exist.
- The case where a failed run read the follow-up (still owed) passed, as
intended.
- GREEN: 3/3 pass.
- Neighbouring suites (`first-turn-queued-input`,
`explicit-native-continuation`, `heartbeat-agent-directory-busy`,
`heartbeat-held-comment-wake`, `execution-control-work`): 517/517 pass.
Server `tsc` is clean.

## Risks

- Low risk. The function and its callers keep their contract. It returns
fewer comment IDs only when a succeeded run of the assignee read that
exact revision.
- Comments read through the REST API by non-native adapters are not
recorded, so their behavior does not change.
- No migration. The added lookup uses the time-bounded, index-backed
query from #21175.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:23:34 -06:00
DottaandClaude Opus 5.5 57aeda190b test(runtime): stop one slow plugin Runner eviction case from failing the cases after it (#21199)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner eviction integration test proves that a retained
Runner continues after the host evicts it
> - The test failed on many unrelated pull requests in the "Verify
Paperclip Runner (vitest native server)" lane
> - One slow case caused other cases to fail, and the PR retry could not
run
> - This pull request isolates each case from a case that timed out, and
gives the retry a fresh issue
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

**What happened?**

`server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts`
failed in CI on at least 5 unrelated branches in 48 hours, most recently
on #21149. The errors were `Test timed out in 60000ms` and `Timed out
waiting for automatic release A`. Each retry then failed at once with
`duplicate key value violates unique constraint
"issues_identifier_idx"`. The failed cases logged a Runner in
`connecting` with no wire frames.

Root cause: every case changed process-wide state. It set the default
TLS trust and `https.globalAgent` for its own certificate, and it
restored them in its own `finally` block. Vitest continues to the next
case when a case times out, but the timed-out function continues to run.
Its late `finally` then removed the TLS trust of the next case and
closed the warm sessions of the next case with
`closeIdleWarmNativeSessionsForRestart()`. The next case could not open
its Runner connection, waited for the 60s startup deadline, and also
timed out. That case then did the same to the case after it. In CI run
43266976387, one timeout caused 3 more failures. Locally, one timeout
caused 5 more failures.

Two more problems made this worse:
- The issue identifier came only from the case name. A retry of the same
case therefore failed on the unique identifier, so the PR retry never
ran the test again.
- Detached cases used a 5s idle timeout. The product limits a release to
`min(idleTimeoutMs, 30s)`, so these cases had 5s for the drain and the
park. Under load, one park took more than 4.8s, and the release expired.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make one case slower than 60s. For example, add a 62s sleep after
release A in `codex cached retired`.
2. Run `npx vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts
-t "codex with a (cached|detached) Core owner"` from `server/`.
3. Before this change, 3 of 4 cases fail: the slow case, and 2 cases
that it disturbs. With `--retry=1`, the slow case also fails on the
duplicate identifier.
4. After this change, only the slow case fails. With `--retry=1`, all 4
cases pass.

**Paperclip version or commit**
master at ef6d347f79, rebased on the current origin/master

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`. A
case no longer changes process-wide trust.
- Move the per-case teardown (close idle warm sessions, close the host
controls, close the TLS proxy) into an `afterEach`. It is registered
before the database fixture, so it runs before the database clients
close, also when a case times out.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the detached idle timeout from 5s to 10s, and wait 200ms past
it as before. This gives the release the time that it needs under load.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

All loops ran on a shared, loaded macOS host (load average 15 to 20 on
18 cores), plus 2 CPU busy loops. Each iteration ran the 12 cases that
match `with a (after inspection|missing state|reused lease) Core owner`.

- Before: 20 iterations, 2 failed (10%). In iteration 10, `claude reused
lease retired` timed out at 60s, and 5 later cases failed on
`readNativeSessionIdleWork() === "none"` because of the case that timed
out. In iteration 17, `claude after inspection retired` failed with
`Timed out waiting for automatic release A`.
- After: see the loop result comment on this PR.
- Injection test (one case sleeps 62s): before, 3 of 4 cases failed, and
with `--retry=1` the retry failed on the duplicate identifier. After,
only the slow case failed, and with `--retry=1` all 4 passed.
- Full file after the change: 40 of 40 passed in 320s locally (335s
before, because the file now creates one certificate instead of 40).
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The detached cases take about 5s more
each (12 cases, about 60s in all), which is partly offset by creating
one certificate per file. A case that hangs inside Core execution can
still leave a busy warm session in module state. This change does not
prevent that. It removes the causes of the cascade that this PR found.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:18:01 -06:00
DottaandClaude Opus 5.5 7f1cdb842d fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat eval catalog checks that agents in chat do the right thing
on saved product state.
> - In `routines.manage-other-owner`, the CEO cannot change Morgan's
routine. The valid outcome is to route the change to Morgan in one
linked task.
> - That valid path uses three runs: the CEO's turn, Morgan's run on the
task, and the CEO's automatic completion follow-up on the chat. The case
allowed only two runs, so the harness stopped valid attempts with
"exceeded its declared run bound".
> - In one of those attempts, the CEO also told Morgan to use UTC.
Nobody asked for UTC. The routine was on America/Chicago. The oracle
correctly failed that change.
> - This pull request admits the third run only in its declared role,
and it tells agents to brief only the change the person asked for.
> - The benefit is that the eval accepts the valid delegation path and
still fails extra work and unrequested setting changes.

## Linked Issues or Issue Description

**What happened?**

Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped
`routines.manage-other-owner` with "Scenario exceeded its declared run
bound". Both had three runs: the CEO turn, Morgan's run on the routed
task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the
CEO's brief also told Morgan to switch the routine to UTC, and
`same-owned-definition-and-triggers` failed.

**Expected behavior**

The delegation path with exactly one routed owner task passes with three
runs. The refusal path stays at two runs. An agent that relays a change
keeps every setting the person did not ask to change.

**Steps to reproduce**

Run `chat-features.runner-opencode.local.routines.manage-other-owner`
and let the CEO route the change to Morgan.

**Paperclip version or commit**

master at 7747487ee7.

## What Changed

- `tests/runner-e2e/chat-catalog-domains/integrator-routines.ts`: the
case bound is now 3 runs (minimum 1). A new `declared-run-roles` check
allows at most 2 runs, plus at most one run that is the CEO's completion
follow-up (`chatCompletionFollowUp` from `run-roles.ts`) on the source
chat for the single owner task with a bound Morgan run. The observation
now records the source chat id. Oracle version is now
`manage-other-owner.v7`.
- `tests/runner-e2e/chat-catalog-descriptors.json`: the static
descriptor matches the new bound and oracle version.
- `tests/runner-e2e/chat-catalog-domains/integrator-routines.test.ts`:
calibration tests. The delegation path with 3 runs passes. A 4th run
fails. A second follow-up fails. 3 runs without a delegated task fail. A
follow-up for a different task fails. The 2-run refusal path still
passes.
- `server/src/services/agent-conversations.ts`: one sentence in the chat
directive task guidance: "When you relay or delegate a change, state
only what the person asked to change, and keep every other setting, such
as a time zone, owner or schedule detail, as it is."
- `server/src/__tests__/agent-conversations.test.ts`: red/green test
that the directive and the built chat task prompt carry that sentence.
- No scenario yaml change, so the TestOps manifest does not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains/integrator`: 283 files, 3850 tests
pass.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `server`: `agent-conversations.test.ts` and
`board-access-denial.test.ts` pass (109 tests);
`codex-local-execute.test.ts` passes. The new directive test failed
before the directive change and passes after it.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs
792f8fb308 --check`: `changed: false`.
- `pnpm verify` exited 2 (descriptor JSON is a global CI input), so
required CI supplies broad coverage.
- Paid single-case run: Runner Full-Stack E2E 43313272302 on a throwaway
branch at this commit. Result is reported in a comment.

## Risks

- Low risk. The eval change only admits a third run in one declared
role. Extra or unbound runs still fail.
- The directive gains one sentence. It can make an agent more
conservative when it relays a change, which is the intent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:10:47 -06:00
DottaandClaude Fable 5.2 549816e6a0 perf(chat): prepare the native progress page read once instead of writing its SQL on every wake (#21195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run that came from a chat reports safe progress to that
chat. Each progress event wakes a queue, and each wake reads one page of
candidates (`nativeProgressPages` in
`server/src/services/chat-native-progress.ts`).
> - The page read joins six tables and has 44 parameters. Its SQL text
is 3.6 KB. The server wrote that text with the query builder on every
wake, and the text is the same for every wake of the same filter.
> - On a hosted instance the server main thread was near one full core
and garbage collection used a quarter of it. An allocation sample put
46% of all allocation under the query builder's SQL writing. The sample
does not name statements.
> - I measured single statements on a local PostgreSQL through the real
client. The query builder allocates about 20 KB for the smallest
statement, plus about 2 KB for each selected column and about 3 KB for
each predicate. This page read costs 296 KB and 0.34 ms of CPU to write.
The same statement from a prepared query costs 21 KB.
> - This pull request writes the SQL text of the page read one time for
each client and filter shape. Each execution then only binds values.
> - The benefit is 91% less allocation and 14 times fewer garbage
collections for one page read. The statement, its parameters and its
rows do not change.

## Linked Issues or Issue Description

Related: #21131 (fixed SQL text for the native run event append) and
#18965 (an earlier prepared read, in the GitHub review wait). No issue
exists for this read.

**What existing behavior does this improve?**

The cost, in the server process, of the candidate page read that every
native chat progress wake runs.

**Subsystem affected**

Chat channels: native run progress publication
(`server/src/services/chat-native-progress.ts`).

**Current behavior**

`nativeProgressPages` builds the complete query with the Drizzle query
builder for each page, and Drizzle then writes the SQL text from it.
Measured on the merge base, Node 26, local PostgreSQL 18, median of 5
passes of 2,000 calls:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 1 | 297.3 KB | 851 µs | 14.5 (30.7 ms) |
| One wake for a task (page read + one transaction) | 7 | 681.3 KB |
2,238 µs | 34 (76.6 ms) |

A statistics sample of one hosted instance on one morning showed this
statement at about 3.9 executions per second. At that rate the read
allocates about 1.1 MB per second for SQL text. This rate is one sample,
taken soon after a long outage. It is not a steady-state figure.

**Proposed behavior**

The query stays in one builder function, `nativeProgressPageQuery`. The
module prepares it one time for each client and each of the six filter
shapes (all tasks, one company, one task; first page or later page),
with placeholders for the values. Each page read executes the prepared
statement.

**Reason and benefit**

Same benchmark, this branch:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 0 | 26.8 KB | 219 µs | 1 (1.4 ms) |
| One wake for a task | 6 | 412.1 KB | 1,699 µs | 21 (55.5 ms) |

At the sampled rate this removes about 1.05 MB per second of allocation,
which is about 1.4% of the 77.7 MB per second that the instance
allocated. It is a small part of the total. The remaining six statements
of a wake run inside a transaction, where a prepared query gains
nothing, and this pull request does not change them.

**Breaking changes**

None.

## What Changed

- `server/src/services/chat-native-progress.ts`: the page query moves
into `nativeProgressPageQuery(db, binding)` without a change to its
text. `nativeProgressPageStatement` returns the prepared statement for
the filter shape and the values to bind. `nativeProgressPages` executes
it.
- `server/src/services/prepared-statements.ts` (new, 17 lines, no
comments):
- `preparedStatements(prepare)` keeps one prepared statement for each
client and shape. A prepared statement runs on the client that prepared
it, so the cache key is the client object. A transaction client gets its
own statement and gains nothing.
- `columnPlaceholder(name, column)` makes a placeholder whose value the
column encodes. A bare `sql.placeholder` skips the column's encoding, so
a `Date` would reach the driver unencoded. With this helper it reaches
the driver as the same ISO string as a value that the query builder
binds.
- `server/src/__tests__/chat-native-progress-statement.test.ts` (new):
statement equivalence, row equivalence on PostgreSQL, and the red/green
mechanism test.
- `server/src/__tests__/prepared-statements.test.ts` (new): the helper.
- `server/src/__tests__/chat-native-progress-work.test.ts` and
`chat-channels.integration.test.ts`: three tests watched or failed the
page read through `db.select`. A prepared statement does not call
`db.select`, so these tests now watch the driver (`db.$client.unsafe`).
The idle assertions are stricter: they now fail on any statement.
- `server/scripts/benchmark-native-progress-page.ts` (new, no comments):
the benchmark above. It loads the module at `--baseline-ref` next to the
current module in one process. `page` is `loadNativeChatProgressScopes`
for one task, which sends the page read and nothing else. `wake` is
`reconcileNativeChatProgress` for one task whose progress is already
published, so it writes nothing and every pass does the same work. The
script fails unless both versions send the same statements with the same
parameters and return the same result. Passes alternate between the
versions with a full collection before each pass. CPU is the whole
process (`process.cpuUsage`). Allocated bytes are V8's
`total_allocated_bytes`. A collection counts when it starts inside a
pass. Optional environment variables: `ITERATIONS` (default 2000 calls
for each pass) and `PASSES` (default 5 for each version).
`TSX_DISABLE_CACHE=1` keeps the tsx cache scan out of the first passes.

Not changed: which statements run, their order, the transaction of a
wake, its `FOR UPDATE SKIP LOCKED` reads, the driver's
prepared-statement setting, and the schema. The read has no
variable-length list parameter.

## Verification

- Statement equivalence (`sends the SQL text and the parameters that the
query builder writes, for every shape of the read`): for all six shapes,
and for an empty task id, the prepared SQL text equals the text the
builder writes with the values, and the bound parameters are equal. The
test also proves that the time filter is bound as the ISO string.
- One-time check against the merge base: I derived the builder of the
merge base from its source and compared 24 inputs (4 scopes, with and
without a cursor, 3 page sizes). The prepared SQL text is
byte-identical, and the parameters are strictly equal.
- Row equivalence on PostgreSQL (`returns the rows of the query builder:
same values, types and order, and no other company's rows`): two
companies, four candidate runs, and three runs that are not candidates.
The fixture has a null payload, a nested JSON payload, a run bound
through its task column only, and microsecond timestamps. The prepared
statement and the builder return strictly equal rows for every filter,
for pages of one row walked with the cursor, and for a task of another
company (no rows).
- Mechanism, red then green (`writes the SQL text of a read one time and
then only binds values`): three reads for different tasks write no SQL
text and send one identical text three times. On the merge base module
the same test fails with "expected sqlToQuery to not be called at all,
but actually been called 3 times".
- The benchmark asserts that both versions send the same statements with
the same parameters and return the same result, for the page read and
for a full wake.
- Existing tests: `chat-native-progress-work.test.ts` (19 passed),
`chat-channels.integration.test.ts` (1,954 of 1,963 passed in the
complete local run, including the 49 tests that match `native
progress|saved progress|safe native`. Two tests fail on this machine
because the worktree is inside another checkout: the bundled Slack skill
resolves to that checkout's skills directory, which has no
`slack/SKILL.md` (`ENOENT` in `connector-runtime.ts`). Seven later tests
failed in the same run from the state that those two left. The seven
pass when I run them by name. CI is the evidence for these nine),
`app-private-hostname-gate.test.ts` (passed).
- `pnpm --filter @paperclipai/server typecheck` passed. `pnpm
check:module-boundaries` passed.
- `pnpm verify` returned exit 2 (broader verification required, because
the change adds a file under `server/scripts`). I did not run the full
local suite. CI supplies that coverage.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-native-progress-statement.test.ts \
  src/__tests__/prepared-statements.test.ts src/__tests__/chat-native-progress-work.test.ts
TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-progress-page.ts \
  --baseline-ref=267983356c
```

Not verified: the effect on a hosted instance. The rate above is one
sample. The local numbers are from Node 26; the hosted instance runs
Node 24.

## Risks

- Low. The statement text and parameters are equal by test, and the read
runs on the root client outside a transaction.
- A prepared statement keeps the text of the first build. The lists
inside the statement (safe event types, conversation states, publication
states) are module constants, so the text cannot go stale in a running
process.
- The helper keys statements by client object. A caller that passes a
transaction client gets a statement for that transaction only. This is
correct, and it costs the same as before.
- Tests that mock `db.select` to observe this read no longer see it. I
found three and changed them to watch the driver.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude
Code with tool use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 15:56:28 -06:00
DottaandClaude Opus 5.5 7747487ee7 fix(budgets): retry contended accounting writes with jittered backoff (#21185)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget policies and usage records are written in serialized
accounting transactions for each company
> - When several writes for one company run at the same time, PostgreSQL
rejects all but one and Paperclip retries the others
> - The retries were few and short (4 attempts, 10 to 30 ms, the same
delay for every writer), so concurrent writers collided again and the
request failed with 503
> - The 503 message also said "this run's budget" for a policy write
that has no run
> - This pull request retries contended accounting writes with jittered
exponential backoff and gives the error a correct message
> - The benefit is that normal concurrent budget changes succeed, and a
real failure tells the user what happened

## Linked Issues or Issue Description

**What happened?**
Several budget policy writes for one company at the same time returned
503 after 4 serialization retries. The same retry exhaustion appeared
often in server logs of the chat feature evals. The message said
"Paperclip could not save this run's budget or usage records … Retry the
run" for a policy write.

**Expected behavior**
Concurrent budget policy writes succeed. If contention continues, the
error describes budget or usage changes, not a run.

**Steps to reproduce**
Send 9 budget policy upserts for one company at the same time. Before
this change, about half of them fail with 503.

**Paperclip version or commit**
master

## What Changed

- `withOperationAccountingTransaction` retries up to
`ACCOUNTING_TRANSACTION_ATTEMPTS` (8) times. The delay before each retry
is jittered exponential backoff (`accountingRetryDelayMs`): 10 ms ×
2^(attempt+1), capped at 400 ms, randomized between half and the full
value. The worst case total wait is about 1.4 s.
- `AccountingContentionError` uses a new generic message: "Paperclip
could not save these budget or usage changes because other updates to
the same company kept conflicting with them. Try again."
- A failed run keeps its run-specific `ACCOUNTING_CONTENTION_MESSAGE`
(run completion still uses it).
- The existing exhausted-retry test uses the new attempt constant.

## Verification

- `cd server && npx vitest run
src/__tests__/budget-policy-write-contention.test.ts`
- Red before the change: 5 of 9 concurrent policy writes failed with
`AccountingContentionError`, and the message mentioned a run.
- Green after: all writes succeed (5 of 5 repeated runs; also 3 of 3
runs with 17 concurrent writers).
- `run-accounting-contention`, `run-completion`,
`budget-control-publication`, `budget-evaluation-work` and
`heartbeat-native-predispatch-retry` tests.
- Paid runner E2E evidence for `goals.align-existing-work` is linked in
a comment.

## Risks

- Low. A write that keeps losing now waits up to about 1.4 s before it
fails, instead of about 60 ms. Only transactions that PostgreSQL rolled
back (40001 or 40P01) are retried, as before.
- Moving policy writes out of the accounting transaction was considered.
It is not done here, because the policy fence keeps budget evaluation
consistent with policy changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:33:36 -06:00
DottaandClaude Opus 5.5 de675d924a fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - The comment must reach the agent once: in the running turn, or in
one later turn
> - In a restart-chaos wave, a comment that the running turn already
read reopened the finished task, and the agent did the completion turn a
second time 15 minutes later
> - This pull request makes a read by the running turn count as
delivery, and holds the comment behind the running run so the run's own
release settles it
> - The benefit is no reopened done task, no repeated completion turn,
and an unread comment that is delivered when its run finishes, not after
a slot wait and a stale cancellation

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode,
native Runner), during a two-server overlap deploy (K15).
- 18:26:56 run A started on the old server. 18:26:55 traffic switched to
the new server.
- 18:27:03.953 a board user posted a follow-up comment. The new server
handled it. Run A was not in that process's `activeRunExecutions`, so
`filterZombieCoalesceTarget` treated it as a zombie. Admission returned
`proceed`, and the comment got its own queued run B (18:27:04.252). The
queued run is not part of the release drain.
- 18:33:32 run A called `get_task_history`, received the comment,
applied it, and finished the task (`done`, 18:34:20).
- Run B waited for an agent slot until 18:41:49 and was cancelled as
`continuation_task_ownership_changed`.
- `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its
"unread" test is only "a run started before the comment and finished
after it". `deliveredQueuedCommentIds` had no record of run A's read.
The release drain reopened the task (done to todo, 18:41:51), and run C
(18:49:22) posted a second "Task complete" summary. The harness grade
`done_comment_once` failed.

**Expected behavior**
- A comment that the running turn received is consumed. No queued work
outlives that run, and the task stays done.
- A comment that the running turn did not receive is delivered once,
through the normal comment-on-done-task path, when that run finishes.
- A stale continuation is never re-issued for a comment that a completed
run already read.

**Steps to reproduce**
1. Start a native run on a task.
2. Post a person's comment while another server instance executes the
run (or seed a queued comment run behind it).
3. Let the run read `get_task_history`, then finish the task.
4. Before this change, the queued comment run is cancelled as stale, the
task reopens, and the agent runs a second completion turn.

**Paperclip version or commit**
`master` at ef6d347f79.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: new run-log event
`task_history.read` and `recordTaskHistoryRead`.
`deliveredQueuedCommentIds` also counts a comment as delivered when a
succeeded run of the assignee on that task read the comment's current
revision (`historyReadDeliveredSql`). All existing consumers use this
one check: the release drain liveness check, the stale-continuation save
(`saveUndeliveredUserComments`), the ownership reconciler, and
first-turn queued input.
- `paperclip-runner-tool-authority.ts` and
`runner-semantic-authority.ts`: `get_task_history` records the returned
comment IDs and `updatedAt` values. The tool output does not change. A
failed record is logged and leaves the comment owed, so it is delivered
again.
- `server/src/modules/wake-queue/application/use-cases.ts`: a comment
follow-up for the same agent's `running` run is held as a deferred wake
even when this process does not track that run. Before, a run executed
by another server instance was treated as a zombie, and the comment got
its own queued run that bypassed the release drain. Coalescing into an
untracked run is still refused.
- `doc/run-log-events.md`: documents `task_history.read`.
- Regression tests in
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`.

Intended behaviour chosen: in t005 run A received the comment mid-turn
through its own `get_task_history` call (run events at 18:33:32 show the
comment ID in the tool result, and the result says it added the
follow-up line), so (a) applied. The fix makes that read the durable
proof of delivery. A comment that no run read still follows the existing
outbox: it is held behind the run and delivered once at the run's
release, reopening the done task as a person's comment does. Envelope
task history of another message's turn still does not count (unchanged
doctrine and test).

## Verification

- RED on master (`ef6d347f79`), `npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "mid-turn|another
server|held message"`: 3 failed, 1 passed.
- "consumes a message the closing run read mid-turn, so its stale queued
run reopens nothing": delivered 1, expected 0.
- "holds a message behind a run another server is executing instead of
queueing a run of its own": a queued run was created.
- "settles a held message as soon as its run closes the task (read
mid-turn: true)": delivered 1, expected 0.
- The unread variant (delivered once, no cancelled stale run) passes on
both, as intended.
- GREEN with this change: all 4 pass.
- Review follow-up (f43e79c3): a history read counts as delivery only
for a succeeded run on the read revision. Tests cover an edit after the
read (the stale queued run delivers the comment once) and reads by an
edited or failed run (the comment stays owed). Full file: 37/37 pass.
- Security-scan follow-up (239f0597): the delivery check probes each run
binding (`native_issue_id`, `context_snapshot->>'issueId'`) through its
own existing index with `offset 0`. It reads only the assignee's
succeeded runs of the task with `finished_at >= ` the comment's current
revision. It does not use `started_at`, because the t005 reader started
before the comment. It stops at the first matching `task_history.read`
event of each run through `heartbeat_run_events_company_run_idx`. No
migration is needed.
- Seeded plan (local PostgreSQL: 200 tasks, 4,000 runs, 1.2M run events,
one comment read by one run), `EXPLAIN (ANALYZE, BUFFERS)`:
- Before: 1.202 ms, 6,097 shared buffers. It scanned every event of all
21 runs of the task.
- After: 0.030 ms, 23 shared buffers. It reads one index-matched run and
one event. The full `deliveredQueuedCommentIds` call takes 1.55 ms.
- New case: a succeeded run's read is not delivery when the comment was
edited after that run finished. Full held-comment file: 38/38 pass.
- `muse-control.test.ts` compared the whole run row after a
`get_task_history` call. It now allows that read's own run-log append
(`nextEventSeq` +1, `updatedAt`), and every other column must be
unchanged. That CI failure (General tests server 2/12) came from this
PR. "e2e shard (5/8)" failed because the self-hosted runner lost
communication; that is not related to this PR.
- Runner and Muse suites (`dot-runner`, `paperclip-control-plane-port`,
`runner-prp-coordinator`, `historical-task-questions`,
`paperclip-runner-tool-authority`, `muse-native-follow-up`,
`muse-control`): 509/509 pass. Server `tsc` is clean.
- Neighbouring suites: `heartbeat-held-comment-wake`,
`src/modules/wake-queue/**`, `execution-continuation`,
`paperclip-runner-tool-authority`, `first-turn-queued-input`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`heartbeat-zombie-guard`: 555/556 passed in one loaded run. The 2
timeouts (one batching test, one reconciler `afterAll` hook) pass when
run alone (1/1, 44/44).
- `server` `tsc --noEmit`: clean.
- The restart-chaos harness already grades this class
(`done_comment_once` in staging-mix). No new cell is necessary.

## Risks

- Product choice: a comment counts as delivered when a succeeded run
read it through `get_task_history`, even if the model ignored it. This
is the same trust that applies to a comment delivered as wake input. The
alternative fails worse: it reopens a done task and repeats a completed
turn. An edit after the read, or a reader run that fails, leaves it
owed.
- Legacy (non-native) adapters that read comments through the REST API
do not write the record. Their behavior does not change.
- A comment follow-up behind a truly dead running run now waits as a
deferred wake until recovery releases that run. Before, its queued run
also could not start until the lock was released.
- Each `get_task_history` call appends one small run-log row. This is
not in the path before `turn.start`.
- Upgrade-safe: continuations queued by older builds still go through
the dispatch stale check, and that path now reads the same delivery
record.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:31:11 -06:00
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00
DottaandClaude Opus 5.5 8a90aa4731 fix(chat): name document revisions and link runs in replies, and record agent-started routine runs as manual (#21183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the chat directive tells
the agent how to write replies
> - #21081 added a rule that keeps raw IDs and internal codes out of
replies. It names only tasks, cases, approvals and people as records to
cite
> - After #21081, agents stopped saying "revision 4" and stopped linking
runs. In a full chat cohort, "revision N" mentions fell from 21 to 11,
and run links fell from 9 to 5
> - A person needs the revision number that Paperclip shows and a link
to the run that failed. The intended effect held: UUIDs in prose fell
from 24 to 7
> - Separately, an agent that started a routine on a person's chat
request sent `source: "api"`, so the run was not recorded as a manual
run
> - This pull request extends the same rule to name document revisions
by number and to link runs, and records every agent-started run on the
run-now route as manual
> - The benefit is replies that point to the records the person can
open, and truthful run sources

## Linked Issues or Issue Description

Refs #21081

**What happened?**

After #21081, a restore reply said "the version before the last edit"
and did not name the new revision. A failed-run reply did not link the
failed run. In a different case, the agent called `POST
/api/routines/{id}/run` with `source: "api"` after two 400 responses.
The run was recorded with source `api`, although a person asked for it
in chat.

**Expected behavior**

The reply names the revision as Paperclip shows it ("revision 4") and
links a run it discusses. A run that an agent starts on this route is a
manual run.

**Steps to reproduce**

Run the chat-features OpenCode cases
`documents-artifacts.restore-document`,
`runs-adapters.inspect-failed-run` and `routines.run-now` on master
9e910ff9ae, and read the replies and the routine run source.

**Paperclip version or commit**

`master` at 9e910ff9ae.

**Agent adapter(s) involved**

Not adapter-specific. Seen with the Paperclip runner and OpenCode
(`openrouter/deepseek/deepseek-v4.1-flash-1017`).

**Deployment mode**

The CI chat-eval runner (local server, embedded database).

## What Changed

- `AGENT_CHAT_DIRECTIVE` (`server/src/services/agent-conversations.ts`):
the "Write replies for the person" paragraph now also says to name a
document revision by its number (such as revision 4), and to link a run
it discusses, such as `[the failed
run](/<prefix>/agents/<agentId>/runs/<runId>)`. The ban on raw IDs in
prose does not change.
- `POST /api/routines/:id/run` (`server/src/routes/routines.ts`): for an
agent actor, the server ignores the `source` value in the body and
records `manual`. Board callers keep their `source` value.
- `routines.ts` service: for an agent manual run with an
`idempotencyKey`, the replay lookup also returns that same agent's
earlier `api` receipt (matched by its `routine.run_triggered` activity).
A retry across the upgrade does not start the work twice. A board
receipt is not matched.
- `docs/api/routines.md`: documents the agent rule for the run-now
source.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts` asserts
the revision-number and run-link sentences and that the raw-ID ban
stays. It failed before the directive change.
- Red/green: `server/src/__tests__/routines-routes.test.ts` sends
`source: "api"` as an agent and expects `manual`; a board caller keeps
`api`. It failed with `source: "api"` before the route change.
- Red/green: `server/src/__tests__/routines-service.test.ts` checks that
a board `api` receipt with the same key does not satisfy an agent's
manual request, and that the agent's own earlier `api` receipt is
returned.
- `server/src/__tests__/routines-e2e.test.ts` (agent routine-to-routine
provenance) still passes.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Paid OpenCode run on the five affected cases (with the companion eval
PR): see the comment below.

## Risks

- Low risk. The directive grows by two sentences. A model can still omit
a revision number or a run link.
- An agent can no longer record a run-now call as `api`. The documented
contract for this route is a manual run, and provenance uses the caller
run, not the source.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:11:52 -06:00
DottaandClaude Opus 5.5 f95bbe081e fix(auth): agent shells can only reach the local API with their own run credentials (#21181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the server gives board access to local
requests that carry no credentials, so the operator can use the app
without a login
> - Agent runs execute on the same machine, so their tools can also send
local requests
> - An agent run must act only with its own run credentials and
permissions
> - This pull request makes implicit board access belong only to the
operator's own local processes
> - The benefit is that agent shells and tools can only reach the API
with their own run credentials

## Linked Issues or Issue Description

**What happened?**
In `local_trusted` mode, a process inside an agent run could receive the
board identity for an API request that had no credentials. A permission
denial for the agent then did not stop the change.

**Expected behavior**
A request from inside an agent run uses the run's own credentials and
permissions. Implicit board access stays available to the operator's
browser and terminal tools.

**Deployment mode**
`local_trusted` (private, loopback). Authenticated deployments do not
use implicit board access and do not change.

**Additional context**
Found by the chat feature evals. The bundled board skill also described
implicit board access in words that an agent could follow.

## What Changed

- Add `server/src/middleware/local-agent-process-boundary.ts`. For a
request without credentials, it identifies the local process behind the
TCP connection (Linux: `/proc`; macOS: `netstat -anv` and `ps`). If that
process or one of its ancestors belongs to an agent run, the request
gets no implicit board access. An agent run is a descendant of
`paperclip-runnerd`, or a process that carries a run identity variable
(`PAPERCLIP_RUN_ID`, `PAPERCLIP_AGENT_ID`, `PAPERCLIP_AGENT_KEY_ID`,
`PAPERCLIP_RUNNER_INSTANCE_ID`) with a value that is different from the
server's own value.
- The walk stops at the server process and its ancestors. Requests from
the server process itself, and from an operator whose own development
server runs inside the same agent execution, keep board access.
- The result is cached for each connection. Only one process lookup runs
at a time. Connections that arrive during a lookup wait for the next
lookup, which handles all of them together (one `netstat` and two `ps`
calls on macOS, one `/proc` walk on Linux). All lookups are
asynchronous, so they do not block other work in the server.
- `actorMiddleware` and the live events WebSocket upgrade apply the
check in `local_trusted` mode. A refused request is unauthenticated
(`actor.type = "none"`), the same as an anonymous request in
authenticated mode.
- Platforms where the server cannot identify the process (for example
Windows) keep the previous behavior and log one warning.
- Rewrite the `paperclip-board` skill authentication section. It no
longer describes implicit board access to agents, and it says that a 401
or 403 inside an agent run is final.
- Document the behavior in `doc/DEPLOYMENT-MODES.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/local-agent-process-boundary.test.ts`. The test starts a
real `local_trusted` server and real child processes:
- an operator process without credentials gets board access (green
before and after);
  - an in-process request gets board access;
- a process with a run identity gets no board access (red before this
change, green after);
- an agent process that starts a tool with an empty environment gets no
board access (red before, green after);
- an operator inside the same execution as its own server keeps board
access.
- A burst test with a slow inspector: 12 parallel connections and 2
connections that close early. It checks that only one lookup is active
at a time, that the burst needs at most two lookups, that the event loop
stays responsive, and that later requests still work.
- Parser tests for the Linux TCP table and the macOS `netstat` format,
and a test that the runner daemon itself is not treated as an agent
process.
- Ran `agent-auth-middleware`, `agent-identity`, `live-events-ws`,
`live-events-toast`, `live-events-run-read` and
`issue-create-deduplication-routes` tests locally. `npx tsc --noEmit -p
server/tsconfig.json` passes.
- Paid runner E2E evidence for the affected chat cases is linked in a
comment.

## Risks

- Medium. The check runs once for each new local connection without
credentials. On macOS it starts `netstat` and `ps` (about 30 to 80 ms).
On Linux it reads `/proc`. Bearer requests do not run it.
- If a process can not be identified on a supported platform, the
request gets no implicit board access. On Linux, when a same-user
process hides its sockets, the check keeps the previous behavior so that
the operator's browser continues to work.
- This is a boundary for the documented path, not a sandbox. Agent runs
use the operator's OS account. A process that intentionally leaves its
run's process tree, or reads the instance files directly, is outside
this check. The broader option is a product decision: bind implicit
board access to a capability that only the operator's browser and CLI
hold, or run agents under an OS sandbox that blocks the server port.
This pull request does not change `local_trusted` semantics for the
operator.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:05:15 -06:00
Nicky LeachandPaperclip 2168f50741 Forward provider credentials from verified image catalogs (#21186)
## Thinking Path

> - Paperclip manages AI agents for work.
> - Application images can ship plugins that provide task environments.
> - Paperclip starts plugin workers with a restricted process
environment.
> - An image author can know which provider credentials an image-bundled
worker needs.
> - The image catalog already identifies selected bundles and verifies
their content.
> - This change adds a way for the image catalog to declare the provider
keys that a verified worker receives.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Image-bundled environment drivers can use the worker credential delivery
already available to built-in providers.

**Current behavior**

The host provides fixed credential forwarding for certain built-in
providers.

**Proposed behavior**

A verified image bundle can receive provider API keys declared on its
own catalog entry.

**Reason and benefit**

Image authors can scope deployment credentials to the verified worker
that needs them.

## What Changed

- Add an optional provider credential declaration to the image-owned
distribution catalog.
- Scope forwarding to the verified worker for the selected bundle.
- Publish distribution catalog support for image builders.

## Verification

- The new loader regression failed before the change because the old
catalog rejected the credential declaration.
- Ran six focused Vitest suites for bundled activation, package
identity, distribution catalog validation, worker environment, and
managed configuration. 166 tests passed and one existing test was
skipped.
- Ran `pnpm --filter @paperclipai/server typecheck`. It passed.
- Reviewed the diff for secret values and personal information. Test
keys are synthetic.
- A downstream image build fixture exercised the catalog reader, bundle
verification, and worker environment together. The supported build
passed only the declared provider key to the verified worker. The build
for an older target omitted the declaration.
- Local verification used embedded PostgreSQL and prebuilt fixtures.

## Risks

- Deploy images with catalog credential declarations only to Core
versions that support the field. Older hosts reject unknown catalog
fields.
- The image author controls which verified bundle receives each provider
API key.

## Model Used

- Implementation: GPT-6.1 Sol (`gpt-6.1-sol`), delegated through Codex.
Independent Codex review and verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in the PR
- [x] I have used public references
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 20:53:08 +00:00
DottaandClaude Opus 5.5 267983356c feat(mcp): speak stateless MCP 2026-07-28 to provider servers, keep older servers working (#21140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through Paperclip's MCP client. The client calls
each provider's remote (Streamable HTTP) or local (stdio) MCP server.
> - The client spoke only protocol 2025-06-18. It always did the
`initialize` handshake and it sent client capabilities only in that
handshake.
> - MCP 2026-07-28 removes the handshake and sessions. Every request
carries its version, capabilities and client identity in `_meta`.
Servers that only speak 2026-07-28 reject the old requests, and they
never see that Paperclip renders MCP Apps views.
> - Public servers already speak it: DeepWiki and Context7 answer
`server/discover` with 2026-07-28 today.
> - This pull request makes the client negotiate the version per server.
It speaks stateless MCP to 2026-07-28 servers and keeps the handshake
for 2025-11-25 and older servers.
> - The benefit is that new and old providers both work, and a tool call
to a stateless server is one HTTP request with no handshake.

## Linked Issues or Issue Description

Refs #21119 (MCP Apps host). This PR sends its MCP Apps capability to
stateless servers too.

**Subsystem affected**
Server: the MCP client in `server/src/services/mcp-http.ts`, catalog
discovery, the tool gateway (`tools/call`, context reads, stdio), and
the remote MCP OAuth flow.

**Problem or motivation**
The client used protocol 2025-06-18 only. MCP 2026-07-28 makes MCP
stateless: no `initialize`, no `Mcp-Session-Id`, per-request `_meta`,
`server/discover`, `resultType` with multi round-trip requests (MRTR),
cacheable lists, required `Mcp-Method`/`Mcp-Name` headers, and RFC 9207
`iss` validation. Servers that speak only 2026-07-28 cannot be used.
Paperclip advertised MCP Apps support only in `initialize`, so stateless
servers never saw it.

**Proposed solution**
Probe each server once with `server/discover`, pick the newest mutually
supported version, and save it on the connection. Send stateless
requests to modern servers and keep the handshake for legacy servers.
Tool calls use the saved version and never probe. The saved version is
checked again only when a request built from it is refused.

**Alternatives considered**
Inline negotiation (send each request in modern shape first and fall
back on `400`) adds a failed request to every call to a legacy session
server. Probing on every tool call adds a round trip before each tool
runs. Saving the negotiated version keeps tool calls at one request.

**Roadmap alignment**
MCP protocol currency for the Apps and connections substrate. It follows
the MCP Apps host work in #21119.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/changelog. Out
of scope (separate work): the Tasks extension, Skills over MCP,
`subscriptions/listen`, progress and cancellation, icons,
enterprise-managed authorization.

## What Changed

- `mcp-http.ts`: `negotiateMcpProtocolVersion` probes with
`server/discover`. A `DiscoverResult` or a modern error
(`-32020`/`-32021`/`-32022`) means a modern server. Anything else means
a legacy server, which gets `initialize`. A `401`/`403` keeps the
challenge for OAuth discovery. The era is remembered per connection and
endpoint.
- `mcpHttpRequestInit` builds every request. It always sends
`Mcp-Method` and `Mcp-Name` (Base64 sentinel when unsafe). For modern
versions it adds `_meta` (`protocolVersion`, `clientCapabilities` with
`extensions["io.modelcontextprotocol/ui"]` and URL elicitation,
`clientInfo`) and `Mcp-Param-*` headers from `x-mcp-header`.
- Legacy `initialize` now requests 2025-11-25 and advertises the MCP
Apps extension.
- Catalog refresh saves `config.mcpProtocolVersion`. It drops tools with
invalid `x-mcp-header` annotations. It reuses a fresh `tools/list`
result (`ttlMs`, `cacheScope`). It probes again when the saved version
is refused, and it negotiates again after a credential refresh.
- `tools/call`: a saved modern version is called directly (no probe, no
handshake). A modern rejection of a legacy-shaped call, or `-32022`,
gets one retry with a negotiated version. The server rejected the first
request before it ran, so the retry cannot repeat a side effect. A
remembered legacy server that refuses `initialize` is probed again.
- MRTR: `input_required` with URL requests becomes the existing provider
link flow (`provider_interaction_required`). The next identical call
sends `requestState` and `inputResponses`. Form, sampling and roots
requests stop with `provider_input_required` and a plain message. A
missing `resultType` means `complete`.
- Context reads (`resources/*`, `prompts/*`) use the saved version,
cache fresh results, and stop on `input_required`. The error code is
kept, so resource not found (`-32602` or `-32002`) is visible.
- stdio: probe with `server/discover` (2 s limit). Restart a server that
exits on the probe. Send `_meta` to modern servers.
- OAuth: refuse a callback without `iss` when the server advertises
`authorization_response_iss_parameter_supported`. Check `iss` before
acting on a provider error. Send `application_type: "native"` for
localhost callbacks and `web` otherwise. Issuer-bound clients and the
CIMD-first order already existed.
- Docs: `doc/connections/MCP-PROTOCOL-VERSIONS.md`, linked from
`doc/connections/README.md`.

## Verification

- New suite `server/src/__tests__/mcp-protocol-versions.test.ts` (19
tests) uses a fake 2026-07-28 server and a fake legacy session server.
It covers discovery, `_meta`, headers, version fallback, `-32022`,
`resultType`, MRTR link and retry, form stop, cache TTL, legacy flow,
and recovery when a saved legacy server moves to 2026-07-28. Embedded
Postgres runs the end-to-end cases.
- New OAuth tests in `generic-mcp-connection.test.ts`: `iss` required
when advertised, `iss` checked on provider errors, `native` for
localhost, and a stateless server listed again after a revoked token.
- Red/green: with the master service code, all 21 new protocol and OAuth
tests fail. With this change they pass. The token-refresh regression
test fails without its fix and passes with it.
- Existing suites: 12 touched files, 1088 tests pass. `tsc --noEmit`
passes for `server`. Updated expectations count the one-time
`server/discover` probe, and the Calendly DCR test now expects `native`
for its localhost callback.
- `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies the
broad coverage.
- Live check with the changed client, without credentials:
- DeepWiki (`mcp.deepwiki.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (3 tools).
- Context7 (`mcp.context7.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (2 tools).
- GitMCP (`gitmcp.io/docs`): discover 400 → `initialize` (2025-03-26) →
session → `tools/list` 200 (5 tools).
- Circleback (`circleback.ai/api/mcp`): discover 401 with its
`WWW-Authenticate` challenge kept for sign-in.

## Risks

- A legacy server that is not saved as session-required gets one extra
request (`server/discover`) the first time it is refreshed. Then the
version is saved. Saved session-required connections are not probed.
- DCR from a localhost callback now registers a `native` client. This is
what the spec asks for. Existing registrations are reused.
- An authorization server that advertises `iss` support but does not
send `iss` now fails sign-in with a clear message. The spec requires
this.
- MRTR answers are kept in memory for 30 minutes. After a restart the
server asks again, and the user sees the link again.
- No schema change. `config.mcpProtocolVersion` is a new key in the
existing JSON config.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, web fetch of the MCP spec), extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:34:26 -06:00
DottaandClaude Opus 5.5 82c6810107 fix(runtime): post the agent's blocker reply when a retried finalization blocks the task (#21169)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes an agent's run, applies the agent's
status decision to the task, and posts the agent's reply as a task
comment
> - When the first workspace copy-back fails, or the run reconnects, a
background retry commits the result instead of the live heartbeat
> - That retry path applies the "blocked" status but never posts the
agent's reply, so the task turns blocked with no visible reason
> - The retry path already shares one recovery step for replies, but
that step accepted only "done", "needs review" and conversation replies
> - This pull request lets the shared recovery step present a blocked
task's reply, so the retry path and the normal path converge on one
presentation step
> - The benefit is that a blocked task always tells the user why,
exactly once

## Linked Issues or Issue Description

**What happened?**

An agent reports that it is blocked. The task moves to "blocked". No
comment from the agent explains why. This happens when the run finishes
through a retried workspace copy-back or a reconnect loop instead of the
live heartbeat. The run succeeds and its result has the agent's blocker
summary, but the run has no presentation decision and the task has no
reply. On a staging company, 58 of the 150 most recent agent-blocked
tasks for one agent had no agent comment.

**Expected behavior**

Every finalization path that applies an agent's status decision also
presents the agent's reply. A blocked task always shows the agent's
explanation once. The status decision does not post a second copy.

**Steps to reproduce**

1. Run a native agent on a task. The agent reports `blocked` with a
summary and a task-wide blocker.
2. Make the first workspace copy-back fail
(`native_workspace_sync_out_failed`), so the heartbeat leaves
finalization to the retry.
3. Let the retry finish the copy-back and commit the result.
4. The task is blocked and the run succeeded, but the task has no agent
comment.

**Paperclip version or commit**

`master` at b8207c4ea9

**Deployment mode**

Hosted (staging cloud). The defect is in server finalization, so it
applies to all deployment modes that use the native runtime.

**Agent adapter(s) involved**

Native runtime (`paperclip_runner`), seen with the Codex-based agent.
The defect is adapter-independent.

## What Changed

- The response presentation recovery step
(`recoverCommittedNativeChatResponse`) now presents a non-conversation
task's reply when the agent reported `blocked` and the task is still
blocked.
- For a blocked reply, recovery uses the agent's final message when one
exists. Otherwise it uses the accepted result summary, the same as the
live completion path.
- A blocked reply on a channel task (an external chat run) uses the same
rule as live run completion. Recovery resolves the chat publication
authorization, prefers the final response, and posts with that
authorization. The worker does not refuse these runs, so it does not
record a false publication failure.
- The presentation work query now finds these runs, so the background
worker also presents them and the bounded publication retry applies to
them.
- A blocker reply is not presented when the task is no longer blocked or
when a later status decision was applied. This keeps old blockers from
posting stale comments.
- Presentation stays idempotent: the step records
`presentationDecision`, reuses a reply that the run already posted, and
never posts twice.
- Existing affected tasks need no separate backfill. The presentation
worker loads all pending candidates at startup, so each task that is
still blocked gets its saved reply once after deploy.

## Verification

- Red/green test:
`server/src/services/native-runtime/native-blocked-reply-recovery.test.ts`.
It completes a blocked native run, fails the first copy-back with
`native_workspace_sync_out_failed`, then commits the retry with
`finalizeNativeRun`.
- Red on `master`: the retry-path tests fail (no comment is posted, and
the worker finds no candidate). The channel task test also fails on the
first PR revision (5eb8cdab95).
- Green with this change: 4/4 pass. The tests check that the blocker
summary is posted once, that the final reply is preferred, that a
repeated retry and a repeated worker check do not double-post, and that
a stale blocker is not posted.
- `cd server && npx vitest run
src/services/native-runtime/native-blocked-reply-recovery.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts` passes (172 tests).
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- After deploy, tasks that are still blocked from earlier affected runs
get the agent's saved reply once. This is the intended backfill. Tasks
that moved on are skipped.
- The candidate query has one more branch with a `NOT EXISTS` on
`status_decisions`, keyed by the existing `(company_id, issue_id,
decision_version)` unique index. The cost is low.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser reads of staging API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:30:32 -06:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
Huanyi Xie 8bff629465 fix(environments): skip blank lease ID queries (#15200)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Runs can use environment leases.
> - A lease lookup sends its ID to a PostgreSQL UUID column.
> - An empty ID makes that query fail on every poll.
> - This pull request returns null before querying for blank IDs.
> - Valid lease IDs keep the existing lookup behavior.

## Linked Issues or Issue Description

Fixes: #15143

## What Changed

- Skip database access for empty or whitespace-only lease IDs.
- Cover empty, space, and newline IDs with a database-query spy.
- Keep an unknown valid UUID returning null. Existing lease tests cover
valid lookups.

## Verification

- The branch was rebased onto current master `252d9b50` to resolve the
test-file conflict.
- All 53 environment service tests passed on Node 24.19.0. The run used
cached dependencies and current database and shared-package source
aliases.
- The blank-ID cases assert that no database query occurs. The unknown
valid UUID still returns null. The existing suite covers stored leases.
- `git diff --check`: passed.
- A clean frozen install is blocked because upstream master adds two
OpenTelemetry packages without updating the lock file.
- Full typecheck and build were attempted on both unmodified master
`252d9b50` and this head in the same environment. Both fail with the
exact same runner `SessionRecord.cursor_prompt_usage` error (TS2339 at
`codex-runtime-adapter.ts:1572`). Full validation is pending CI on the
rebased commit.
- The earlier full test run had a Slack callback ordering failure that
also occurred on unmodified master.
- Greptile reviewed the previous head at 5/5 with no inline findings.

## Risks

Low risk. Only blank IDs return early. Non-blank IDs still use the
database. There is no schema or migration change.

## Model Used

OpenAI Codex, based on GPT-6, assisted with source review,
implementation and local test execution. The exact deployment model ID
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suite; full suite
pending above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user-facing configuration or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (new head checks are pending)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-12 13:49:09 -06:00
DottaandClaude Fable 5.2 a37f3056b6 perf(docker): keep dependencies and provider binaries in layers that survive a commit (#21167)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each merge to master builds the production Docker image for amd64
and arm64 and pushes it to GHCR
> - Each build pushes three layers again, about 3.5 GB compressed of a
4.7 GB image
> - A file-by-file comparison of two consecutive master images shows
that about 95% of those bytes do not change between commits
> - The unchanged bytes are in the same layers as small files that
change on every commit, so the registry sees new layers
> - This pull request moves the unchanged bytes into their own layers
> - The benefit is a smaller push for each master build and a smaller
pull for each deploy

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The production image build. It pushes about 3.5 GB of layers for each
architecture on each master commit. Each deploy pulls the same bytes
again.

**Subsystem affected**

Docker image build (`Dockerfile`, `production` and
`runner-provider-pack` stages).

**Current behavior**

I compared the amd64 images of two consecutive master commits (`2174f2d`
and `dfc510b`, 15 changed server source files). I hashed each file in
the three layers that changed.

| Layer | Compressed | Content that is identical between the two builds
|
| --- | --- | --- |
| `COPY --from=build /app /app` | 1,404 MB | 97.5%. The root
`node_modules` is 2.86 GB of 4.13 GB and has 0 changed files. |
| `COPY /provider-pack` | 1,008 MB | 100% of the 112 files of 1 MiB or
more (2.54 GB). Only 32 small files change (manifest, `.modules.yaml`,
`.bin` shims). |
| `COPY /provider-pack-transport/` | 1,059 MB | The archives differ only
in tar timestamps. This pull request does not change this layer. |

**Proposed behavior**

- Copy the root `node_modules` from the `deps` stage as its own layer.
Copy the remainder of `/app` from the `build` stage with
`--exclude=node_modules`.
- In the `runner-provider-pack` stage, move pack files of 1 MiB or more
to `/provider-pack-large` after the transport archive is made. Set their
mtimes to a fixed value.
- Copy `/provider-pack-large` with `COPY --link`, then copy the small
remainder of the pack to the same directory.

**Reason and benefit**

The dependency layer changes only when the lockfile changes. The large
provider-pack layer changes only when a provider version changes. The
expected push for a commit that changes source only goes from about 3.5
GB to about 1.5 GB for each architecture. The master build runs about
140 times a day on AWS runners, and this traffic is about 570 GB a day
of billed data transfer. Deploy pulls become smaller by the same amount.

**Breaking changes**

None. The file tree of the image is the same. Provider-pack files of 1
MiB or more now have an mtime of 1970-01-01.

## What Changed

- `Dockerfile`: the `production` stage copies `/app/node_modules` from
`deps` first, then `/app` from `build` without the root `node_modules`.
- `Dockerfile`: the `runner-provider-pack` stage splits large pack files
into `/provider-pack-large` with fixed mtimes. The `production` stage
copies that tree with `--link` before the remainder of the pack.
- `server/src/__tests__/docker-image-layer-reuse.test.ts`: a drift guard
for the copy order, the exclude, the `--link`, and the position of the
split after the archive step.

## Verification

- `cd server && npx vitest run
src/__tests__/docker-image-layer-reuse.test.ts
src/__tests__/docker-build-stamp.test.ts`: 9 tests pass. The 2 new tests
fail against the `Dockerfile` on master.
- `node scripts/check-docker-deps-stage.mjs`: PASS.
- `pnpm verify`: exit 2 (the `Dockerfile` is a global CI input).
Required CI supplies the broad coverage.
- A small Dockerfile with the same syntax version
(`docker/dockerfile:1.20`) and the same four `COPY` instructions, built
two times with a changed stamp file:
- `--exclude=node_modules` leaves out only the root `node_modules`. A
nested `node_modules` stays.
  - The dependency layer has the same digest in the two builds.
- The large-file layer has the same digest in the two builds only with
`--link` and fixed mtimes. Without one of them the digest changes.
- Hard links between large files stay hard links. The merged directory
contains each file one time.
- Evidence that the `build` stage writes nothing to the root
`node_modules`: in the published image, all 107,413 files there have an
mtime in the minute of the `deps` install. The app build ran 15 hours
later.

**Not verified:** a full build of the production image with this
`Dockerfile`. The local Docker disk was full, and no pre-merge workflow
builds this image. The first full build is the master build after merge.
The check after merge is to compare the layer digests of two consecutive
master images.

## Risks

- Medium until the first master build. If the image build fails, master
image builds stop until a revert. No broken image can be published,
because the build fails before the push.
- If a future build step writes to the root `node_modules` in the
`build` stage, the image will not contain that change. The comment in
the `Dockerfile` states this rule.
- A file that crosses the 1 MiB boundary moves between the two pack
layers. This only changes which layer is pushed.
- The first build after merge pushes all the new layers one time.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, AWS CLI, Docker, registry API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:38 -06:00
DottaandClaude Fable 5.2 eaef46b60b feat(ui): browser page-load tracing to Honeycomb for hosted instances (#21156)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI is the main surface. Its load speed is a product
requirement.
> - The team wants to track page-load times over time and give each
metric an owner and an SLO in Honeycomb.
> - Server traces show only the server part of a page load. They do not
show JavaScript start-up, lazy route code, rendering, or client-side
navigation.
> - Paperclip is also self-hosted by others. Browser tracing must stay
off there, and it must not slow any page.
> - This pull request adds real-user browser tracing for hosted
Paperclip Cloud instances only. It loads lazily and sends a closed set
of attributes.
> - The benefit is one `page.load` measurement per page (named by route
pattern), plus Web Vitals and API spans that join the server trace,
ready for Honeycomb queries and SLOs.

## Linked Issues or Issue Description

No public issue exists. Feature description:

**Subsystem affected**

Board UI (`ui/`), the authenticated session route
(`server/src/routes/auth.ts`), and the shared session schema.

**Problem or motivation**

We cannot measure how long the all-tasks page or a task page takes to
load for real users. Server traces stop at the API response. Most
navigation is client side, and the standard OpenTelemetry document-load
instrumentation does not cover it. Without a per-page browser
measurement, no agent can own a page-load SLO.

**Proposed solution**

The server sends a `browserTracing` config in the authenticated session
only when the instance is cloud managed and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. The browser then measures
every hard load and client navigation in one shared recorder tied to the
router's own route tree. After the page settles, it lazily loads an
OpenTelemetry chunk that exports `page.load`, `web_vital.*`, and
same-origin `/api/` fetch spans to Honeycomb. An allowlist removes every
other attribute before export.

**Alternatives considered**

`@honeycombio/opentelemetry-web` was the first choice. Its defaults add
the full page URL, query string, hash, referrer, and user agent to every
span and to the resource, and it bundles log and metric pipelines. We
would have to replace its exporter and remove most of its defaults.
Plain OpenTelemetry web packages plus `web-vitals` are smaller and send
only what we choose. Honeycomb needs only OTLP/HTTP and its ingest-key
header.

**Roadmap alignment**

This is internal hosted-instance observability. It changes no product
behavior and no self-hosted behavior.

## What Changed

- `server/src/browser-tracing-config.ts`: resolves the browser tracing
config. It returns `null` unless `isCloudManagedInstance()` is true and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. It accepts only a Honeycomb
ingest key (`hcaik_` prefix), requires `https` (except loopback), and
fails closed with one value-free warning on any invalid setting.
Optional: `PAPERCLIP_BROWSER_TRACES_INGEST_KEY`, `_SAMPLE_RATE` (default
1), `_SERVICE_NAME` (default `paperclip-web`), `_ENVIRONMENT` (default
`SENTRY_ENVIRONMENT`).
- `GET /api/auth/get-session` adds `browserTracing` (config or `null`),
the same delivery path as the front-end Sentry DSN. The shared
`authSessionSchema` has the new optional field
`browserTracingConfigSchema`.
- `ui/src/lib/browser-tracing/page-load-recorder.ts` and `controller.ts`
(main bundle, no OpenTelemetry imports): one `page.load` measurement per
hard load (from navigation start) and per client navigation (from the
history change). A measurement ends at the first painted frame where no
full-page loading state is mounted and either every page-declared
content signal is ready or no query without data is fetching. Redirects
continue the measurement. A new navigation interrupts it. 30 s times
out. The sample decision is made once per page session before any
download.
- `ui/src/components/PageLoadTracker.tsx`: `App` passes the same
`<Route>` tree it renders, so every page is reported by route pattern
(for example `/:companyPrefix/issues/:issueId`) with no per-page code.
- `usePageContentReady` hook and blocking gates: `PaperclipLoading`
(also the route `Suspense` fallback) blocks readiness. The task page
uses the existing `issue-detail:navigate→content-paint` point. The plain
all-tasks list signals when its first rows render.
- `ui/src/lib/browser-tracing/sdk.ts` (lazy chunk, loaded after the page
settles): OpenTelemetry web tracer, OTLP/HTTP exporter, fetch
instrumentation for this origin's `/api/` only with `traceparent` sent
only to this origin, `web-vitals` LCP/INP/CLS, and a context manager
that makes the open `page.load` span the parent of its API calls.
`span-sanitizer.ts` applies the closed attribute allowlist (no URLs,
ids, query strings, status messages, or event attributes). Uncaught
errors are not sent because Sentry already reports them.
- New `ui` dev dependencies, pinned exactly. Each is needed, and the
OpenTelemetry ones are the same versions the server already resolves:
- `@opentelemetry/api@1.9.1` (already in the workspace): tracer and
context API.
- `@opentelemetry/sdk-trace-web@2.10.0`: the browser tracer provider,
batch processor, and stack context manager.
- `@opentelemetry/instrumentation-fetch@0.221.0`: fetch spans and
`traceparent` injection.
- `@opentelemetry/resources@2.10.0` (already a server dependency): the
resource.
- `@opentelemetry/otlp-transformer@0.221.0` (already a server
dependency): OTLP JSON serialization for a 60-line `OtlpFetchExporter`.
This replaces `@opentelemetry/exporter-trace-otlp-http`, which is a
server optional peer: installing it in the workspace made the server's
exact-version gate test see it as present.
  - `web-vitals@6.2.3`: LCP, INP, and CLS.
All of them are in the lazy chunk only. The lockfile is not committed:
`pr-trusted` rejects lockfile edits in pull requests, CI installs with
`--no-frozen-lockfile` after the frozen attempt reports
`ERR_PNPM_OUTDATED_LOCKFILE`, and the lockfile bot refreshes it after
merge.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the three shared files (the session schema is part of the pinned public
SDK surface; the contract must live in `packages/shared`).
- Docs: new `doc/browser-performance-tracing.md` (enablement, config,
key handling, span and attribute list, page-load definitions, example
Honeycomb queries and an SLO SLI). Pointers in `doc/observability.md`
and the Honeycomb tracing skill.
- Two App routing tests mock `PageLoadTracker` because they render `App`
without a `QueryClientProvider`.

## Verification

- `pnpm verify` returns exit 2 (broader verification required) because
docs are global CI inputs. Required CI supplies the broad coverage.
- Focused local tests, all pass:
- `ui`: `vitest run src/lib/browser-tracing
src/components/PageLoadTracker.test.tsx
src/components/SentryGate.test.tsx src/api/auth.test.ts
src/lib/issue-detail-performance src/pages/IssueDetail src/pages/Issues
src/App src/components/CloudAccessGate.test.tsx
src/components/IssuesList`: 17 files, 484 tests.
- `server`: `browser-tracing-config.test.ts`, `auth-routes.test.ts`,
`sentry.test.ts`: 63 passed, 5 skipped.
  - `packages/shared`: `validators/access.test.ts`: 19 passed.
- New tests prove: the SDK module is never imported when the session
config is `null`, absent, or signed out, or when the session is not
sampled; route templates contain no ids, slugs, or tokens; a client
navigation produces a `client` measurement under
`/:companyPrefix/issues/:issueId`; `traceparent` goes only to this
origin's `/api/` and not to a third-party or non-API URL; exported spans
carry only allowlisted attributes. A mutation check (tracker forced on)
turned the "never imports" tests red.
- Typecheck: `ui` (`pnpm typecheck`), `server` (`pnpm --filter
@paperclipai/server typecheck`), and `packages/shared` pass.
- Real-browser run: a local dev server with a local OTLP collector and
headless Chromium. The self-hosted session returned `browserTracing:
null` with the endpoint set, and the page made 0 SDK and 0 collector
requests. With a cloud session config injected into the session
response, the collector received a hard `page.load` (route
`/:companyPrefix/issues`, `ready_signal=idle`), a client `page.load` to
`/:companyPrefix/issues/:issueId` (`ready_signal=content`), a back
navigation, LCP/INP/CLS spans, and `/api/` fetch spans parented to the
open `page.load` span. No task identifier, company id, task title, host,
or URL appeared in the exported payload. No third-party request carried
`traceparent`.
- Bundle size (`ui` production build, initial JavaScript closure of
`index.html`): before 3,813,986 B raw / 1,153,991 B gzip; after
3,820,743 B / 1,156,590 B (+6.8 KB raw, +2.6 KB gzip, +0.2%). The lazy
SDK chunk is 84 KB raw / 27 KB gzip and is not fetched unless tracing is
on and sampled.
- Review fixes (each red on the old code, green on the new): the SDK now
loads by a 35 s deadline even if the page never settles, so a timed-out
hard load is exported (`controller.test.ts`); an agent chat with no
saved task uses the idle rule instead of a content gate that never fires
(`usePageContentReady.test.tsx`); the sample rate is rounded to exactly
1/N so Honeycomb's `SampleRate` weight matches
(`browser-tracing-config.test.ts`). The real-browser run was repeated
after the exporter change with the same result.

## Risks

- Low risk for self-hosted installs: the server never sends a config, so
no tracing code beyond the small recorder runs, and the SDK chunk is
never fetched.
- On hosted instances the feature is off until an operator sets
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` and an ingest key. When on, the
controller wraps `history.pushState`/`replaceState` to time navigation
starts and restores them on sign-out.
- Every signed-in browser can read the ingest key, like the Sentry DSN.
Only Honeycomb ingest keys (send-only) are accepted. A collector
endpoint can keep the key server-side.
- Hard-load API calls happen before the SDK loads, so they have no fetch
spans. The hard-load `page.load` span is still recorded.
- The browser-to-server trace join needs server tracing. The hosted
image does not install the OpenTelemetry server peers today
(`CLOUD_BUNDLED_SERVER_DEPS` is `@sentry/node` only). Browser spans work
without it.
- `ui/src/pages/IssueDetail.tsx` was already over 1,000 lines. This
change adds 5 lines there for the task page content signal, next to the
existing content-paint measure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file edits, local browser automation). Reasoning effort: low.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:35 -06:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
DottaandClaude Opus 5.5 8ff45c0fa9 feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.

## Linked Issues or Issue Description

**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.

**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.

**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.

**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.

**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).

**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.

## What Changed

- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
  - App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
  - the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
  - warnings in the sign-in step;
  - `doc/connections/CIRCLEBACK.md`.
  
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.

## Verification

- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
  - `ui_meta` is stored;
  - an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
  - a call without a view records nothing;
  - the routes serve the view's HTML and CSP;
  - the view's tool call works and a model-only tool is refused;
  - another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
  - Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
  - Discovery stored the real view declarations:
    - `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
    - `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
  - A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
    - The 1 MB tool-result limit blocked the view document.
    - Views were hidden inside the folded tool history.

## Risks

- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:28:05 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Fable 5.2 a4a0668a5c perf(attention): count the sidebar summary without the reads that only a card shows (#21159)
GET /attention/summary returns counts. To get them it ran the feed with a
flag that skipped card bodies, and still read every blocked task through the
full task list, every review two more times (one time with its description),
the blocker and the blocked work of each card, and the spend of every budget
policy.

The summary keeps the collection path of the feed, so a card is counted by
the rule that shows it. It no longer reads what only a card shows:

- blocked work: the rows of the task list, in list order, with only the
  columns blocker attention reads (issueService.listBlockerAttentionRoots),
  classified by the function the full list calls
- reviews: the review rows carry the card's task condition, so the summary
  does not read each review again
- tasks of cards: without project and workspace
- budget alerts: the open incidents (budgetService.openIncidents), not the
  spend of every policy
- a fingerprint only for a card that has a dismissal; no queue membership
  and no "transferred by" read

The review classification reads the 20 columns it uses, not the whole task.

The request makes the same writes as before (decision queue seeds, retention
states) through unchanged code. The feed, the task index and the task list
return what they returned before.

One summary request at 3,000 tasks: 133 -> 78 statements, 2,718 -> 842 KB
fetched, 604 -> 144 KB of SQL text, 100 -> 45 MB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 12:39:21 -06:00
DottaandClaude Opus 5.5 b8207c4ea9 fix(runner): evict finished API snapshots instead of blocking reads at the company quota (#21133)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs read the Paperclip API through `call_api`. Large
responses are saved as "API snapshot" assets so the agent can read them
in pages.
> - Each company has a 20 GiB snapshot quota. Before a large read, the
server reserves 1 GiB of worst-case space against that quota.
> - Snapshot assets were never removed. Reservations from interrupted
captures (crash, restart, ambiguous upload) stayed at 1 GiB each until
"an operator reconciles" them. Nothing did that.
> - After a morning of high retry volume, the quota filled. Every large
read then failed with `api_response_company_storage_limit`, and agents
blocked their own tasks and asked an operator to free storage.
> - This pull request makes company admission reclaim space by itself:
it releases stale reservations of finished runs and evicts the oldest
snapshots of finished runs, oldest first, only as much as the new read
needs.
> - The benefit is that a read never fails because old snapshots filled
the quota, and no person must clear storage by hand.

## Linked Issues or Issue Description

**What happened?**

On staging, about 97 tasks of one agent had successful runs but ended in
`blocked`. In each run, `GET` calls through `call_api` for large records
(for example heartbeat runs and their events) failed with
`api_response_company_storage_limit`: "Company API snapshot storage
quota is exhausted. Read an existing asset or ask the operator to remove
old snapshots or increase the quota." The agent then reported a
task-wide blocker owned by "the instance storage operator". At least 15
tasks had a final comment such as "Blocked: company API snapshot storage
is exhausted". The error message itself told the agent that an operator
must act.

**Expected behavior**

A read must not fail because old snapshots filled the company quota. The
server must reclaim the space automatically. Only snapshots that active
runs still hold, or that users attached to issues, cases or logos, may
stay.

**Steps to reproduce**

1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB.
2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or
leave two 1 GiB unattached reservations from finished runs that are
older than 20 minutes.
3. In a running native run, call `call_api` for a response larger than
the inline limit.
4. Before this change, the call fails with
`api_response_company_storage_limit`. After this change, the call
succeeds and the oldest finished snapshot is evicted.

**Paperclip version or commit**

Staging deploy `704b3f4d63`. Fix is based on `origin/master` at
`8bdd53c11f`.

**Deployment mode**

Hosted (staging cloud), native runtime with Daytona sandboxes.

**Agent adapter(s) involved**

Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is
in the server-side `call_api` capture admission, so it affects every
native adapter.

## What Changed

- Added
`server/src/services/native-runtime/runner-api-capture-retention.ts`. It
measures company snapshot usage and reclaims space under the existing
per-company advisory lock.
- Reclaim step 1: release unattached reservations older than twice the
capture deadline (20 minutes) whose run is missing or not `queued`,
`running` or `scheduled_retry`.
- Reclaim step 2: evict snapshot assets oldest first until the new
reservation fits. A snapshot is eligible only when it is older than 1
hour, every run that captured it has finished (succeeded, interrupted,
failed, cancelled or timed out), and no issue attachment, case
attachment or company logo links to it. Each eviction writes an
`asset.evicted` activity-log row with the asset id and the owning run
id.
- A capture upload that finishes after its reservation was reclaimed is
rejected in the publish transaction. The same transaction records the
object for deletion, and the bounded cleanup removes it. The upload has
a 10-minute abort signal.
- Failed deletes are logged per company and counted with the
`runner_api_capture_delete_failed` metric. When cleanup keeps failing,
the read error says so in plain words.
- Eviction credits deletes that are already pending before it chooses
more snapshots. Eviction is durable. In the same transaction that
deletes an asset row, it inserts a reservation row with
`pending_delete_object_key`, so the bytes stay counted. Admission then
deletes the stored objects in pages, under a 30-second abort signal, and
removes each row only after its delete succeeds. A failed or interrupted
delete is retried at the next over-quota admission. No background poller
is added.
- Added the `pending_delete_object_key` column and an index on
`runner_api_response_reservations.asset_id` (migration 0472).
- `#reserveApiResponseCapture` now runs reclaim only when the quota
would be exceeded, so reads below the quota do no extra work. After
marking evictions it finishes the deletes and retries admission once.
- When no space can be reclaimed, the error message now names what holds
the space (runs still working, files attached to tasks, deletes in
progress). It no longer tells the agent to ask an operator.
- Replaced the test that asserted old snapshots block reads with tests
for eviction, multi-page eviction, durable delete retry, a bounded
stalled delete, stale-reservation release, and protection of active and
attached snapshots.

## Verification

- Red: the three new tests in `runner-api.integration.test.ts` fail on
`origin/master` (two get `api_response_company_storage_limit`, one gets
the old "ask the operator" message).
- Green: `cd server && npx vitest run
src/services/native-runtime/runner-api.integration.test.ts` passes 70/70
with the fix.
- The multi-page test failed first with
`api_response_company_storage_limit` because pending deletes ran in one
200-row batch. The paged delete loop fixed it.
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- An agent that reads a saved snapshot of a finished run after it was
evicted gets a not-found response. It can read the API again. Eviction
happens only under quota pressure, oldest first.
- A released stale reservation can leave an unknown storage object from
an ambiguous upload. These bytes were never linked to an asset, so they
were already unreachable. This is the same storage the old code kept
counted forever.
- Eviction runs inside the company capture lock. The candidate query is
bounded to pages of 200 rows and uses the new `asset_id` index.
- Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`)
and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It
is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and
safe for old and new servers during a rolling deploy.
- During a storage outage, an over-quota read still fails until deletes
succeed. The bytes stay counted, so the quota is never exceeded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:38 -06:00
DottaandClaude Opus 5.5 0f3a6a867d fix(recovery): continue open agent tasks left with nothing running, pending or waiting on a person (#21139)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents work on tasks through runs, and recovery code starts the next
run when a run stops
> - On staging, many open agent tasks show "running" or "blocked", but
no run, wake or retry is pending, and no person has to act
> - Staging deploys about every 40 minutes. Each deploy stops some runs,
and some tasks then stay idle until a person presses Retry
> - The stranded sweep has many lanes. Some lanes get "no retry" back
and skip the task with no record, and the sweep never looks at `blocked`
tasks
> - This pull request adds one invariant with one pure predicate, and
one event-driven sweep that repairs a violation with one ordinary
continuation wake
> - The benefit is that no open agent task stays with nothing running,
nothing pending and no one else's move

## Linked Issues or Issue Description

Refs #21107 and #21046 (the pattern for leaving a run of another live
server alone). Related open work: #21135 (fresh continuation from saved
work at wake admission), #21124 (sweep calls the bounded scheduler for
pre-provider legacy failures), #21127 (run-start hand-over).

**What happened?**
On a staging snapshot (2026-11-12, about 14:35Z), 1,030 open agent tasks
had no live run. The new predicate, run read-only against the staging
API, classifies 141 of them as violations: 40 offer a Retry only the
system can act on, 38 are blocked with no recorded reason and no hold,
14 are blocked by an automatic recovery outcome with no hold left, 5
have nothing pending, and 44 are blocked under an active recovery hold
(left to the hold's owner, see below). 25 more are blocked by recovery
for more than 7 days. The other tasks wait on a person or on a pending
intent: 542 chat threads, 122 agent-declared blocks, 102 credential
waits, 51 dependencies, 28 questions.

Nine Retry-started `in_progress` tasks show the main gap. A person
pressed Retry, a deploy stopped the run during setup
(`server_shutdown_interrupted`, no provider work), and the bounded
scheduler then refused with `continuation_user_authorization_missing`
("could not revalidate the original user continuation") on every sweep.
The sweep's escalation did not take effect either, so each task stayed
`in_progress` with a Retry button and nothing pending, for up to 10
hours.

**Expected behavior**
Each open task assigned to an agent has a live run, a durable pending
intent, or a named reason that is someone else's move. If it has none of
these, Paperclip continues the task automatically.

**Steps to reproduce**
1. Start a person's Retry on an `in_progress` task with a legacy agent.
Stop the server during setup, so the run ends `interrupted` with
`server_shutdown_interrupted`.
2. Let the bounded retry scheduler refuse to revalidate the person's
continuation.
3. Run the stranded sweep. Nothing is queued. The task stays
`in_progress` with nothing pending.
4. The same happens for a native predecessor
(`enqueueStrandedIssueRecovery` returns `null` for every native run) and
for every `blocked` task (the sweep reads only `todo`, `in_progress` and
`in_review`).

## What Changed

- `server/src/services/recovery/assigned-work-invariant.ts`: the
invariant as one pure predicate. Facts go in. The verdict is `ok` with
its clause (live run, pending intent, or one reason from a closed list
of waits), `settling` with a re-check time, or `violation` with its
reason. An unknown hold, block reason or person action is a violation.
It also holds the per-task continuation allowance: at most 6 per hour
(the server-restart lane's bound, the same cap as #21135) and, below the
cap, a spacing of 2, 4, 8 and then at most 15 minutes. At the cap the
next try waits until the oldest continuation leaves the hour, so a task
that keeps needing continuation is retried at about 6 per hour, never
left for hours.
- `server/src/services/recovery/assigned-work-sweep.ts`: an indexed
pre-filter, a batched fact loader, the repair and a read-only report.
- A running run counts as live unless the shared owner proof
(`owner-proof.ts`) proves its server gone. A held controller lease is
live, so a run of the other server in a deploy overlap is never touched.
A run whose owner is proven gone goes to heartbeat's existing
lost-process settle path. Native running runs are always treated as
live; the native settle watch judges their owners.
- Native active paths match the stranded sweep's check, including
#21108's exclusion of a native finalization that a person's Retry
replaced.
- A run that ended less than 90 seconds ago, or whose controller lease
another server still holds, is `settling`. The sweep looks again at that
deadline.
- A pending question counts as the person's move, unless the person
pressed Retry (or sent a message) after the question was asked.
- The repair takes the task's row lock, checks again that nothing is
pending, applies the allowance and writes a plain-language activity
entry. This entry is the claim, so two servers make one wake. For a
`blocked` task with no recovery hold, it also puts the task back to the
status the block recorded. Then it enqueues one ordinary
`issue_continuation_needed` wake. The wake has no retry link, so it does
not use the task's failure allowance. With #21135, wake admission turns
that wake into a fresh run from saved work when a hold would defer it.
- A `blocked` task under an active recovery hold is reported as a
violation but not repaired here: the hold's owner (#21135) re-continues
it, so the two never continue one task twice. When #21124 escalates once
on a permanent scheduler refusal, the task is blocked under a hold, so
#21135 continues it; when no hold is left, this sweep continues it.
Neither case leaves the task waiting on a person for a reason that is
not the person's.
- The claim re-runs the full predicate on a fresh read under the task's
row lock, so a question, approval, blocker, monitor, wake or run that
appeared after the first read stops the repair.
- A lifted block is restored to the status the blocking entry recorded
(`_previous.status`, `previousStatus` or `fromStatus`). A block that
interrupted a review goes back to `in_review` with no agent wake; the
review's own recovery wakes its participant. If admission refuses the
wake, the block is put back with its original block time.
- The settling window is keyed on three events only: the latest run
ending (a held lease can extend it, but never past 5 minutes after the
run ended), a never-run task's creation, and a blocker finishing.
Routine touches (comments, reads, labels) never postpone a repair. A run
cut by a deploy's shutdown is repaired about 90 seconds after it ended.
- Each process starts at most 10 repairs per minute, newest-updated
tasks first. The other tasks wait for a deadline.
- Each burst of checks writes one structured log line
(`assigned_work_invariant.pass`) with the counts by clause, the repairs
and the deferrals.
- `server/src/services/recovery/work.ts`,
`issue-recovery-notifications.ts`, `delivery-work-notifications.ts`: a
new `assigned-work-invariant` queue on the shared delivery-work
coordinator. The existing issue-recovery notifications (run state
changes, hold resolutions) and server start mark a company dirty. There
is no interval polling. Deadlines exist only for a real obligation
(settling grace, back-off, rate-limit deferral).
- `server/src/services/recovery/service.ts`, `heartbeat.ts`: wire the
sweep with the existing wake, live-execution and settle dependencies.
- `server/src/routes/agents.ts`, `server/src/routes/openapi.ts`,
`docs/api/agents.md`: `GET
/api/companies/:companyId/assigned-work-invariant` (board only,
company-scoped) and filtered by the caller's issue-privacy condition:
private tasks the caller cannot read are absent from every count and
sample returns the counts by clause, the violation count, the held
subset, stale blocks and up to 20 samples. Use it after a deploy to
check for 0 violations.

Why there are two sweeps: the stranded sweep decides how a known failure
resumes (bounded retries, disposition repair, review recovery,
escalation). The invariant decides only that something is pending, and
repairs with the ordinary continuation. It does not change the stranded
sweep's lanes or the recovery-refusal paths.

## Verification

- `server/src/services/recovery/assigned-work-invariant.test.ts` (45
tests): every clause, every wait reason, the unknown cases, settling,
the allowance and its long back-off.
- `server/src/__tests__/assigned-work-invariant.test.ts` (17 tests,
embedded Postgres): a refused or later-declined continuation stays a
violation and is retried under the cap; the real wake queue deferring
behind a recovery hold; scheduling stops when a task is done, cancelled
or reassigned to a person; running vs stopped or blocked session goals;
the deploy-interrupted Retry shape and the native reconciliation shape
(the stranded sweep does nothing, then exactly one wake); the nine
Retry-started tasks whose scheduler refused revalidation, including one
with a question asked before the Retry; a block after a legacy run's
retries ran out with no hold left (`controller_lease_lost`); a block
under an active hold left to its owner; two concurrent sweeps make one
wake; another live server's run is not touched; an owner-gone run is
settled; approval, question, dependency and declared-unblock tasks are
not touched; an unknown block reason is repaired; the per-task cap and
the per-process budget; a run that ended seconds ago; old-shaped rows (a
board-owned stranded action from an older release, native
`awaiting_runner_reattach` and `retryable_failure`, a running run saved
without its controller process, a block older than 7 days); query cost
at staging size.
- Query cost at staging size (800 open tasks, 14,000 runs over 7 days):
one suspect page 2 ms, one task check 165 ms (including the repair), a
full 800-task report 386 ms. The suspect anti-join uses the run status
index (no sequential scan on `heartbeat_runs`, 0.2 ms).
- Route test in `agent-live-run-routes.test.ts`: a board operator reads
its own company; another company and an agent get 403.
- Existing production-wiring suites pass with the new queue:
`issue-recovery-deadlines`, `issue-recovery-conversations`,
`live-recovery-work`, `issue-recovery-work-lifecycle`,
`recovery-deadline-acceptance`, `server-startup-feedback-export`,
`heartbeat-held-comment-wake`, `delivery-work-coordinator`.
- `node scripts/typecheck-layers.mjs` for the server source layers
passes.

## Risks

- On the first deploy, the sweep continues about 97 staging tasks over
about 10 minutes (10 per minute). Each continuation is an ordinary run.
- The stranded sweep can still move a task to `blocked` after its
bounded retries. The task is then continued by the hold's owner or by
this sweep, within the same cap and back-off. The task gets more
activity entries than before.
- On current master without #21135: when the wake reaches a task whose
last run left a recovery hold, admission defers it (a skipped wake, no
pending row). The activity entry then says the continuation did not
start, the task stays a violation, and the sweep tries again with
spacing (2, 4, 8, then 15 minutes at most; at most 6 per hour). It never
loops faster than the cap and never moves the task to blocked for a
person. A test with the real wake queue covers this. With #21135, the
same wake becomes a fresh run from saved work.
- The activity entry is written as an attempt and is published only
after admission: as queued, or as not started. A wake that the queue
declines later (for example behind the old run's cleanup) leaves the
task a violation, and it is retried under the cap.
- Chat-thread tasks are left out (the conversation clause), because
their next move is a person's message.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:14 -06:00
DottaandClaude Opus 5.5 90ae3c4337 fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run can be retired as superseded while its workspace export
fails. Its files stay unsaved, and the run stays the writer of the
task's workspace.
> - #21136 records those files for the automatic workspace recovery.
That record needs the sandbox's stop-and-retain receipt.
> - The native run projection worker checked for that receipt for one
hour after the run ended. After that, it dropped the run. Nothing woke
it when the receipt arrived later.
> - If the receipt came late or never, the run kept the task's workspace
fenced. Every queued run on the task waited and then failed setup.
Nobody owned the problem.
> - This pull request makes the worker process a late receipt, stop and
keep the sandbox itself when no receipt arrives, and never drop the run
silently.
> - The benefit is that the task's queue moves again by itself, and two
writers never use one workspace.

## Linked Issues or Issue Description

Refs #21136 (review thread on
`native-superseded-workspace-export.ts:84`)

**What happened?**
`preserveSupersededWorkspaceExport` returned `recheckAt: null` when the
sandbox stop receipt had not arrived within `finishedAt + 1h`. The
projection worker then deleted its entry. A later stop receipt
(`settleStopOnlyCleanup`) woke only the workspace recovery queue, not
the projection queue. `preserveNativeFailedWorkspace` requires
`hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run
stayed the dirty writer, and queued runs kept failing setup.

**Expected behavior**
A late receipt is processed at once. When no receipt arrives, Paperclip
stops and keeps the sandbox itself and waits for the provider's
confirmation. The workspace stays held while the sandbox may still run.
When the provider confirms that the sandbox is gone, the task continues
from its last save. When the stop stays unconfirmed for hours, the task
shows one plain notice, and the check continues.

**Steps to reproduce**
1. Retire a native run as superseded after its workspace export fails
(the #21136 shape).
2. Let the run's sandbox stop not settle (for example, the server that
owned the release is replaced).
3. Wait more than one hour after the run ended.
4. See that the projection worker drops the run, and that a later stop
receipt records nothing. The workspace stays held.

## What Changed

- `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also
notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company
and run). New `prepareUnconfirmedSourceSandboxStop` puts a
stop-and-retain intent on the source lease from its current status. It
skips a destroyed sandbox, a stop already in progress, and a cleanup in
flight.
- `native-superseded-workspace-export.ts`:
- Within one hour of the run ending, the worker checks every minute
(unchanged).
- After that hour, it requests the stop-and-retain itself. The existing
pending-cleanup sweep dispatches the stop and retries it until the
provider confirms. The worker checks again every 15 minutes, and the
receipt wakes it at once. The workspace stays held.
  - A sandbox that another lease holds is never stopped.
- When the provider confirms that the sandbox was destroyed, the files
cannot be saved. In one transaction (task, run, then workspace locks),
the worker sets aside only that run's workspace epoch, wakes queued-run
dispatch, and records `issue.previous_run_files_lost` on the task.
- When the stop is still unconfirmed six hours after the run ended (or
the sandbox is held by another lease), the task gets one
`issue.previous_run_files_waiting` activity entry. The worker keeps
checking. It never drops the run.
- A stop-only dispatch now settles when the provider confirms that the
exact sandbox is already gone. Before, the runtime accepted only a
`stopped` reply and threw on `destroyed`, so a gone sandbox kept the
stop pending forever.
- `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts
`stopped`, or `destroyed` for a stop-and-retain intent (a native
workspace export resume still needs `stopped`). `settleStopOnlyCleanup`
stores a `destroyed` receipt as `released` / `success` /
`workspace_source_unavailable`, removes the stop intent, never writes a
retain receipt, and wakes the projection and workspace recovery queues.
- `environment-runtime.ts`: the plugin `environmentStopLease` and
built-in `stopLease` checks use `acceptedStopOnlyReceipt`.
- `workspace-recovery.ts`: when its own stop settles as `destroyed`, the
source is unavailable (the existing alternative-strategy path) instead
of a binding failure.
- Daytona plugin: stop of a sandbox that Daytona reports as not found
returns `destroyed` (destroy already did this) instead of throwing. boxd
already returns `destroyed`. Other sandbox plugins have no stop-only
method, so nothing changes for them.
- New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`,
`pending_cleanup`). The stop-only cleanup ownership check and the
worker's "held by another lease" guard both use it, so a retained warm
sandbox is never stopped and the two checks cannot drift.
- `ui/src/lib/activity-format.ts`: plain labels for the two task
activity entries. They do not ask a person to fix anything.
- `doc/execution-semantics.md`: document the rule.
- Not changed: the shared owner prover still refuses to treat a stopped
instance on another host as proof that its child processes stopped.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts` (4
new tests, red/green):
- "checks again every minute while the run's own release may still stop
its sandbox": the recheck branch. Nothing is recorded or stopped, and
the workspace keeps its writer.
- "stops and keeps the sandbox itself when no stop receipt arrived, and
records the files once the late receipt lands": after the hour, the
lease gets a stop-only intent. A second pass changes nothing. The late
receipt wakes the projection queue. The next check records the hold, and
the system recovery frees the workspace. Fails on master (`expected {
…(19) } to match object { status: 'pending_cleanup' }`).
- "names the unconfirmed stop on the task once, and keeps checking, when
the provider never confirms it": the expiry branch. One notice, the
check continues, no hold, and the workspace stays held. Fails on master.
- "continues from the last save when the provider confirms the sandbox
is gone": epoch 1 is set aside, the workspace is free, the task says so,
and the result is not applied. Fails on master.
- Review-round tests (all 5 fail on the first head `a208bfa58f`, pass
now):
- `environment-runtime.test.ts` "settles a plugin stop whose provider
reports the sandbox gone (boxd, Daytona not found) through the cleanup
sweep": provider dispatch through `releaseRunLeases` and
`sweepPendingCleanupLeases`. The lease settles `released` with a
`destroyed` receipt and no retain receipt.
- `environment-runtime.test.ts` "settles a built-in stop whose provider
reports the sandbox gone, without claiming retained files": the built-in
`stopLease` path. No destroy call.
- `superseded-workspace-export-recovery.test.ts` "continues from the
last save when Paperclip's own stop finds the sandbox already gone": the
worker's stop request gets a `destroyed` reply, the projection queue
wakes, and the workspace is freed at epoch 2.
- `superseded-workspace-export-recovery.test.ts` "never stops a sandbox
a retained warm lease still holds, says so once and keeps checking": the
source lease is unchanged, a recheck is returned, and `sandbox_in_use`
is recorded.
- Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no
longer has as destroyed".
- `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421)
pass.
- Related suites pass: `native-run-projection-work`,
`retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`,
`native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`,
`environment-retained-orphan-sandbox`,
`workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398
tests), and `ui/src/lib/activity-format.test.ts`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The new behavior acts only on an ended superseded
run that failed in its workspace export, still holds a dirty workspace,
and has no retained-file hold.
- The worker now stops an ended run's sandbox one hour after the run
ended if no stop receipt arrived. The stop uses the existing stop-only
contract and keeps the sandbox. A sandbox held by another lease is never
stopped.
- A destroyed sandbox's unsaved files are not recoverable. The task
continues from its last save and says so.
- If a provider never confirms a stop, the workspace stays held (by
design: no second writer). The task shows one notice, and Paperclip
keeps retrying.
- Shared cleanup code now treats a confirmed `destroyed` stop-only reply
as settled. Every caller was checked: the cleanup sweep and run release
count it as settled; the workspace recovery treats the source as
unavailable; the fallback treats it as stopped (nothing can run there).
No caller reads it as retained files, because no retain receipt is
written. A native workspace export resume still requires `stopped`.
- Each extra delivery notification per settled stop-only cleanup is
cheap. The projection worker's scoped load finds nothing for runs that
do not match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:54:47 -06:00
DottaandClaude Opus 5.5 04082a8fd6 test(heartbeat): check preparation span parents only in the attempt trace (#21149)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service prepares each run in named phases and traces
each phase as an OpenTelemetry span under the execution attempt span
> - The test `heartbeat-preparation-gap-spans.test.ts` checks that each
named preparation span has the execution attempt span as its parent
> - The test failed in CI 5 times in 48 hours on unrelated PR branches,
always for `heartbeat.prepare.chat_retry_admission`
> - The cause is in the test: the native run trace also replays each
recorded preparation phase of 50 ms or longer as a historical span with
the same name, under `task.prepare` in a separate trace. When the
machine is slow, `chat_retry_admission` takes more than 50 ms, and the
test also checks the parent of the replayed copy
> - This pull request limits the span checks to the trace of the live
execution attempt
> - The benefit is that the test is stable and still proves that every
live preparation span is a child of the execution attempt

## Linked Issues or Issue Description

**What happened?**
`server/src/__tests__/heartbeat-preparation-gap-spans.test.ts` failed
intermittently in CI with `AssertionError:
heartbeat.prepare.chat_retry_admission: expected '<span id>' to be
'<attempt span id>'`. It failed 5 times in 48 hours on 5 unrelated PR
branches.

**Expected behavior**
The test passes on every run. It checks only the live preparation spans
of the execution attempt.

**Steps to reproduce**
Run the test 4 times in parallel on a loaded machine: `cd server && npx
vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts`. A
span dump of a failed run shows two
`heartbeat.prepare.chat_retry_admission` spans. One is the live span
under `task.execution_attempt` (correct). The other is the historical
replay under `task.prepare` in the native run trace, made because the
phase took 72 ms (the record limit is 50 ms).

**Paperclip version or commit**
master at f2e449f8f6.

## What Changed

- The test now finds the single `task.execution_attempt` span first.
Then it reads only the spans in that span's trace.
- The name checks and the parent checks use only those spans. The
`native.controller.*` spans are in the same trace, so their checks do
not change.
- No product code changes. The replay of recorded phases under
`task.prepare` is intended behavior (`drainRunPreparationPhases` in
`heartbeat/runtime-execution.ts` feeds the native run trace).

## Verification

- Before the change, 4 parallel loops of 5 runs each: 8 pass, 12 fail.
All 12 failures were the `chat_retry_admission` parent assertion.
- After the change, the same loops: 36 pass, 8 fail out of 44 runs. All
8 failures were `Timed out waiting for the run to settle` in one window
when the machine load average was about 34 from other work. There were 0
parent assertion failures. A second batch of 20 runs passed 20 of 20.
- A diagnostic dump of a failed run confirmed the replayed span is in a
different trace (`task.provider_session` > `task.prepare`) and has the
native trace schema attributes.

## Risks

- Low risk. This is a test-only change.
- The test no longer checks spans in other traces. The live preparation
spans are still checked for name and parent.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and code execution for the test
loops.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only change)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:45:02 -06:00
DottaandClaude Opus 5.5 3fb35fcf13 fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.

## Linked Issues or Issue Description

**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.

**Paperclip version or commit**
Staging at 1d153f62b6. The code is the same on master at 704b3f4d63.

**Deployment mode**
Hosted staging (paperclip.staging.paperclip.app), remote sandbox
execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapEvidence` returns the bootstrap
proof together with the runner instance that the checkpoint names.
`nativeRunnerCheckpointRemoteBootstrapProof` now wraps it and behaves as
before.
- `native-session-executor.ts`:
- State migration runs before this server registers its own controller
for the run. Under a `bootstrap_incomplete` claim for a remote sandbox,
it now reads the interrupted controller's checkpoint proof there
(`captureClaimTimeRemoteBootstrapProof`). A later read is not used,
because the new controller rewrites the checkpoint with its own
bootstrap commands.
- Host preparation: when this run's sandbox Runner is not suspended, it
runs the same-run restart proof. The proof accepts a Runner that never
connected (as before), or the claim-time proof for the exact runner
instance in the sandbox state. It then stops that exact Runner by its
identity marker and records the stop proof. A backup, a different run
id, another runner instance, or a failed stop still fails closed.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_connected_checkpoint_proven` case. The Runner is
`ready` and connected, and no stop proof exists yet. It fails before the
fix with `resumed_not_suspended` and passes after it.
`interrupted_bootstrap_unproven` still fails closed.
- `native-bootstrap-claim-time-proof.integration.test.ts` (embedded
Postgres, no mocks of the proof): an interrupted controller that had
connected registers its checkpoint, the claim-time proof is read, and
then a replacement controller registers its own checkpoint. A live read
after that returns `bootstrap_incomplete` even when the interrupted
controller had sent `turn.start`. The claim-time proof stays false for
that case and true for the bootstrap-only case, and it refuses another
runner instance.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` fails on the new case.
- Green: the same command passes (65 tests after merging master).
- `pnpm exec vitest run
server/src/services/native-runtime/native-bootstrap-claim-time-proof.integration.test.ts`:
2 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
server/src/services/native-runtime/native-restart-recovery.test.ts`:
1474 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- Merged `origin/master` (including #21134) without conflicts and reran
the tests above.

## Risks

- Low. The new path runs only under a `bootstrap_incomplete` claim for
the same run id, with no failover backup, and only after the Postgres
controller checkpoint proves that no provider turn was sent. It stops a
Runner only when the identity marker and process fingerprint match,
which is the same check the reattach path uses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21134 (the never-launched branch of the same helper), #21079
(stops a leftover Runner at launch time; this change acts earlier, in
host preparation).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:38:06 -06:00
DottaandClaude Opus 5.5 dfc510b887 fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud replaces the server process on each release. The old
server keeps serving until the new one passes health checks and takes
the traffic.
> - During that overlap the old server keeps starting new runs right up
to SIGTERM, although the new server can already start them.
> - SIGTERM then cuts off exactly those runs while they still prepare.
In one audited swap the old server started 30 runs after the new server
booted, and 23 of them ended `process_lost`.
> - Admission must never pause, so the fix must hand over who starts new
runs, not stop admission.
> - This pull request makes the new server record when it starts runs.
The old server then leaves new run starts to it, and starts them again
if the new server stops or dies.
> - The benefit is that a deploy no longer starts runs on a server that
is about to stop, and the company always has one server that starts
runs.

## Linked Issues or Issue Description

Refs #20852 (preserves preparing and launching turns that a deploy
interrupts; this PR reduces how many such turns the old server starts).
Related: #21046, #21107 (shared controller-instance owner proof), #20986
(queued-run dispatch signal).

**What existing behavior does this improve?**

The start of new runs while two servers overlap during a Cloud deploy.

**Subsystem affected**

Server run admission (`server/src/services/heartbeat/queue.ts`), the
controller instance table, and startup and shutdown in
`server/src/index.ts`.

**Current behavior**

The old server starts queued runs until it receives SIGTERM. Runs that
it starts in the last minutes of the overlap are interrupted while they
prepare. Nothing tells the old server that its replacement already
starts runs: `controller_instances.epoch` is written at process start,
and that happens minutes before the new server can start runs.

**Proposed behavior**

The new server records `dispatching_at` and a short
`dispatch_lease_expires_at` on its `controller_instances` row right
after its startup queued-run pass. The old server reads this through an
in-memory flag at run admission and leaves new run starts to the new
server. Runs it already started continue and are handed over at SIGTERM
as today. If the new server stops, drains or dies, the old server starts
runs again within seconds.

**Reason and benefit**

Fewer runs are cut off by each swap. There is never a moment without a
server that starts runs.

**Breaking changes**

None. The migration adds two nullable columns. Releases without this
code ignore the columns, and their rows never carry the marker, so a
mixed-version overlap behaves as today.

## What Changed

- `controller_instances`: new nullable columns `dispatching_at` and
`dispatch_lease_expires_at` (migration
`0471_controller_instances_dispatch_marker`, two `ADD COLUMN`
statements, no lock beyond the catalog update).
- New `server/src/services/run-start-handover.ts`. It is one worker on
the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`).
Its renewal and recheck are coordinator deadlines, and a drain change or
expiry wakes it.
- Replacement side: `begin()` records the intent and writes the marker
(it creates the boot's row if the row is missing). If the server drains
at that moment, the worker writes the marker when the drain ends. The
worker then publishes a `run-start-handover` hint and renews the 20 s
lease every 5 s. It renews only while an older boot that is not stopped
or fenced may still serve, and for at most 30 min. All marker writes run
in one serialized chain. `release()` at shutdown lands after any renewal
already in flight. A clear that fails is retried, and shutdown waits for
the release. While the server cannot dispatch (drain, quiesce, admission
closed), the worker clears the lease.
- Old-server side: `isYielding()` is an in-memory read. The hint
refreshes it, and so does one read that a dispatch pass waits for (at
most 1 s) when the last read is more than 2 s old. A read that gets no
answer within 1 s never keeps the server yielding. The server admits,
and the yield also ends when the replacement's last seen lease runs out
on the old server's own clock. Reads are a single flight on the
dedicated connection. While it yields, the server rechecks every 3 s,
calls `onResume` when the marker is gone, and sends a coalesced
queued-run-dispatch hint for skipped starts.
- It yields only to a newer epoch with a live marker. It never yields to
an older boot, to itself, or to a row without the marker, and it does
not yield without a row of its own.
- **Database invariant:** every statement of the hand-over runs in a
small transaction on the dedicated one-connection controller-lease pool
(`controllerLeaseDb`) with a 1 s statement timeout. That covers reads,
marker writes, the `run-start-handover` notification (commit-bound, in
the marker write's own transaction) and the skipped-start hint. None of
it touches the shared pool or registers tracked database work. The pool
is shared only with legacy controller-lease renewals, and never with the
liveness pool that native lease renewals use. A hand-over statement
waits at most one legacy batch (10 s), inside the 15 s between a marker
renewal and the marker's expiry. It delays a legacy renewal (60 s lease,
renewed every 10 s) by at most 1 s. I did not piggyback on the
controller-instance lease renewal, because that renewal runs only while
the server holds work and only every 30 s.
- **Import audit of `run-start-handover.ts`:**
  - `drizzle-orm` `sql`: builds statements, no I/O.
- `controllerLeaseDb`: the dedicated connection, used by every statement
here.
- `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs
`pg_notify` only on the transaction it is given and registers no intent,
so the coordinator's `reconcileDatabaseWork` has nothing of ours to
check.
  - `logger`: no I/O to the database.
  - `controller-process-identity`: in-memory boot id and host hash.
- `delivery-work-coordinator`: type only. The worker is registered for
lifecycle and deadlines; its `run` awaits only the serialized marker
chain, and reads are not awaited.
  - `DELIVERY_QUEUES`: constants.
  - `task-admission` drain subscriptions: in-memory events.
- `heartbeat.ts`: run admission is also closed while the server yields.
A `runStartHandover` service option (default: the process's registered
hand-over) lets one process host two assembled services in tests.
`queue.ts`: `startNextQueuedRunForAgent` and
`resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That
call returns `null` (nothing to await) unless the hand-over read is
stale.
- `index.ts`: creates the hand-over only when the instance lease is on,
the deployment notification channel exists, and the scheduler is
enabled. It calls `begin()` after the RunDispatch startup pass and wakes
that queue. It releases the hand-over when run admission closes at
shutdown, and stops it before the lease stops.
- `deployment-notifications.ts`: keeps the LISTEN connection open while
a hand-over involves this server, so skipped starts and lease changes
reach the other server at once.
- `doc/execution-semantics.md`: new section "Run starts during a deploy
overlap". It covers the marker, who yields, failure handling, what
counts as starting a new run, and what a post-deploy audit should show.

**Marker contract for other components (for example the deployer):** a
replacement starts runs when its `controller_instances` row has an
`epoch` greater than the old boot's, and `dispatching_at is not null and
dispatch_lease_expires_at > now() and stopped_at is null and fenced_at
is null`. `dispatching_at` stays set after the lease lapses, as the time
the boot began starting runs.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test
with two `heartbeatService`s, each with its own pool, listener,
coordinator and boot. A wake that the old server accepts stays queued
there. The replacement's real queued-run dispatch starts it, and claims
are observed per pool. After a replacement release, the old server
starts the run itself. Red: with the yield removed from
`isRunAdmissionClosed`, the old server claims the run.
- `cd server && npx vitest run
src/__tests__/run-start-handover.test.ts`: 19 tests pass against
embedded PostgreSQL. Each test uses two servers, each with its own pool,
boot id, LISTEN connection and delivery coordinator. The tests cover:
- the old server is ready and the new server starts; the new server sets
the marker; the old server yields; the queued run is claimed only by the
new server, and a skipped start wakes the new server's dispatch
- the old server learns from its own dispatch path without a
notification: a read older than 2 s is refreshed before the start
- the new server dies after it sets the marker: the old server resumes
and runs its queued-run pass
  - the new server shuts down: the old server resumes at once
- the new server drains: it releases the marker and takes it back
afterwards
- a replacement that starts up during a drain takes over when the drain
ends
  - a shutdown release during an in-flight renewal stays released
- a clear that fails at shutdown (statement timeout behind a row lock)
is retried by a later release
- an older server that outlives the lease TTL keeps yielding while the
replacement renews
- a yield that the old server cannot confirm within 1 s (its only
connection is busy) does not stop starts (red/green)
- a yield ends when the replacement's lease runs out, even while every
read waits for a connection (red/green)
- renewals blocked past the TTL hand starts back once, then the
replacement takes them again (bounded flapping)
- a clear that fails when the replacement starts draining is retried
well inside the lease (red/green)
- with both servers' shared pools blocked past the TTL, the marker is
written, renewed and released, and the old server yields and resumes,
all before the block ends (red when hand-over statements use the shared
pool)
  - an older-release row without the marker changes nothing
- a server never yields to an older boot or to itself, and does not
yield without a row of its own
  - a lone server writes once, never renews, and admits from memory
  - without a hand-over (self-hosted) there is nothing to await
- Red: on `origin/master` the test file cannot run, because the module
and the columns do not exist. The old server has no signal and keeps
starting the handed-over run.
- `cd server && npx vitest run
src/__tests__/deployment-notifications.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts
src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`:
all pass.
- `cd packages/db && pnpm check:migrations`: safety and compatibility
checks pass. `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify` asks for broad verification because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Run starts move from the old server to the new server during the
overlap. If the new server cannot dispatch, the old server resumes on
the lease clear (about 3 s) or the lease lapse (up to about 23 s after a
crash). If a hand-over read fails, the server keeps admitting.
- If the old server was not listening for notifications, it can start
runs for up to about 2 s after the marker (the age of its last read). On
a dispatching server, the pre-dispatch read adds one small bounded query
at most every 2 s.
- An older boot that has been idle for more than 30 min counts as gone.
The new server then does not renew, and the old server admits again as
today.
- The migration adds nullable columns only. The deployer can apply it
while the previous release serves.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:09:53 -06:00
DottaandClaude Fable 5.2 2174f2dfa5 perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed.

The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:45:44 -06:00
DottaandClaude Opus 5.5 f2e449f8f6 fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's Retry on a connected task (for example a GitHub pull
request review) must start work. #21080 made a Retry whose chat replay
is refused after the click continue from the recovered saved work
> - After #21080 was deployed, accepted Retries on a busy GitHub review
agent still did not start
> - Those tasks had a saved chat request that arrived while the failed
run was still running. Admission holds such a request as
`message_predates_stop`, so it can never start alone
> - #21080 refused to continue while any saved chat input existed, so
this held input blocked the continuation. For a second group, the latest
run was an admission that the recovery hold cancelled before dispatch,
and the recovered saved work belongs to the earlier failed run
> - This pull request lets the continuation adopt input that was held
because it predates the stop, and lets the recovered run that held a
cancelled admission count as the saved work
> - The benefit is that an accepted Retry starts work on these tasks too

## Linked Issues or Issue Description

Follow-up to #21080. No public issue exists. This is the bug
description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for
blocked tasks of a GitHub review agent (`paperclip_runner`), but no run
started. Each Retry receipt changed to `failed` about 5 seconds later
(on the 14:48 Retries, 5 seconds for one task and about 75 seconds for
two others), with no run and no `issue.retry_continued_from_saved_work`
activity. Every affected task had a saved chat request (a newer GitHub
review request) with an `issue.execution_waiting` record of reason
`message_predates_stop`. The request arrived while the failed run was
running, so admission holds it permanently. The interruption projection
kept pointing at that held request. In a second group of tasks, the
latest run was cancelled before dispatch with
`execution_reconciliation_required`, and the recovered saved work
belonged to the earlier failed run that held it.

Root cause: in `continueRefusedTaskChatRetry`
(`server/src/services/task-retry.ts`), any saved chat input from
`selectTaskRetryInput` stopped the continuation, so the receipt settled
`failed`. The held input can never start by itself, so nothing ran. The
recovered-work check also looked only at the latest run, so it missed
the recovered run that held a cancelled admission.

**Expected behavior**

An accepted Retry starts work. Saved input that admission holds because
it predates the stop joins the continuation, and the task continues from
the recovered saved work.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Let a native run fail with
`native_continuation_requires_reconciliation`, and recover its saved
work.
2. Save a chat request while that run is still running, so admission
holds it with `message_predates_stop`.
3. Click Retry while the chat identity lock is busy, so the source check
is deferred and then refused.
4. The route returns 202. The Retry receipt changes to `failed`, and no
run starts.

**Paperclip version or commit**

a65644d1e1 (the deployed #21080). Reproduced on ef2b53b675.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native runs, and legacy admissions cancelled before
dispatch). The fix is in core Retry code and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts` (`continueRefusedTaskChatRetry`):
- Saved chat input whose source comment was written no later than the
failed run's stop boundary (`userStopBoundaryAt`: the recorded user Stop
request, otherwise `finishedAt`) no longer stops the continuation. This
is the same boundary and timestamp that admission uses. Admission holds
that input as `message_predates_stop`. The continuation coalesces its
receipts into the Retry receipt (`coalescedIntoWakeupRequestId`), so the
task stops waiting on it. The receipt ids are recorded in the activity
details.
- Saved input created after the stop, and unadmitted chat input, still
settle the Retry as failed, as before.
- When the latest run is an admission cancelled before dispatch
(`execution_reconciliation_required`, never started), the recovered-work
check also accepts the run named by that run's
`executionWait.recoveryActionId`.
- `doc/execution-semantics.md`: both rules.
- Tests in `server/src/__tests__/chat-channels.integration.test.ts`: two
new variants of the #21080 regression test.
`person_with_input_before_stop` saves a chat request before the failed
run stops. `person_after_held_admission` adds a legacy admission that
the recovery hold cancelled. Both start one run that resumes the
recovered workspace (`resume_preserved_workspace`). The held-admission
fixture binds the task to the recovered workspace, as on the hosted
tasks. Refusal edges `input_after_stop` and `input_inside_user_stop`
settle the Retry failed with no run and leave the input deferred.

## Verification

- Red before the fix: both new variants fail. The Retry receipt is
`failed` (or stays held), and no run starts.
- Green after the fix: both variants pass, and the held input receipt is
coalesced into the Retry receipt.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"[Rr]etry"` with `src/services/task-retry.test.ts` passes 369/369 on the
final commit.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only.

## Risks

- Behavior change: input that arrived during a failed run no longer
starts its own turn through the chat retry. It is adopted into the
saved-work continuation, and the comment stays in the task thread.
Before this change, that input could never start at all.
- Input that arrived after the stop keeps its own admission path.
- No migration. Receipts that already failed stay failed. The next click
takes the new path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:36 -06:00
DottaandClaude Opus 5.5 d5bd38468d fix(runner): stop a leftover Runner on the same state folder before a relaunch (#21079)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. The Runner keeps its saved state in one folder per
session and listens on one port in the sandbox.
> - During an automatic upgrade, the old server can launch a Runner a
few seconds before it stops. That Runner stays alive in the sandbox and
keeps the state folder and the port.
> - The new server then restarts the run on the same sandbox. Each new
Runner fails at once with "Address already in use"
(`runner_ingress_bind_conflict`). Each launch also overwrites the
process marker that pointed at the old Runner, so the next check cannot
find the old Runner to stop it. The run restarted about 12 times in 19
minutes and then stopped, and the task stayed blocked.
> - Two Runner processes must never use the same saved state at the same
time.
> - This pull request makes each Runner launch stop any live Runner that
still names the same state folder, before the new Runner starts.
> - The benefit is that a restart after an upgrade gets the port and
continues the run, instead of looping on the bind conflict.

## Linked Issues or Issue Description

**What happened?**
On staging, a Runner launched by the old server just before an upgrade
kept the Runner port. Every relaunch by the new server exited within
seconds with "runner_ingress_bind_conflict: failed to bind listener:
Address already in use (os error 98)". The run looped for about 19
minutes and then failed with `native_runner_process_exited`. The task
was blocked and needed a person.

**Expected behavior**
The relaunch stops the leftover Runner of the same session, gets the
port, and continues the run.

**Steps to reproduce**
1. Launch a Runner for a run in a sandbox.
2. Stop the server before the Runner is adopted (an upgrade). The Runner
stays alive.
3. Let the new server restart the run's bootstrap. The new Runner cannot
bind the port, and the next restart cannot see the old Runner because
the marker now names the dead new process.

## What Changed

- `native-session-executor.ts`: the detached Runner launch script now
runs a short fence before it writes the process marker.
- It finds processes whose argv[0] is `paperclip-runnerd`, that are not
a `reap-lineage` command, and whose `--state-dir` is followed by this
exact folder. Runners on another folder, a folder named by a different
flag, and other programs are not touched.
- It records the stale Runner's lineage by PID **and start time**: its
descendants, and members of a process group or session that a recorded
process leads while that leader is alive (provider tools start their own
sessions). It signals a PID only while its start time still matches, so
a reused PID is never signalled, and it never signals a whole process
group.
- TERM, up to 5 s wait, then KILL, up to 2 s wait. Then it runs
`<runnerd> reap-lineage --state-dir … --runner-pid …` for the stopped
Runner, which stops the tools its lineage journal recorded that have
already left the process tree.
- It fails closed: a survivor (`runner_stale_state_holder_alive`) or a
reap that leaves a recorded process alive or gives no report
(`runner_stale_lineage_unproven`) makes the launch exit 75 **before**
the marker is written, so the existing marker is kept. The reason goes
to the process stderr log only when that path is a plain file (same rule
as the Runner stderr log). The whole stop is bounded at about 14 s,
under the 20 s marker wait.
- Two reap refusals let the launch continue with a
`runner_stale_lineage_partial` warning: `journal_unavailable` (the
Runner kept no journal; its visible lineage is already gone) and
`unattributed_orphan` (reported only after every recorded process is
gone, for orphans the reaper can neither claim nor signal) **only when
every orphan it names is proven to have another owner**: this launch's
own detached shell, a `paperclip-runnerd` serving a different
`--state-dir`, or that Runner's stderr trimmer. Any other orphan could
be an old tool that escaped between samples, so the launch fails closed.
- When `/proc` is not readable, or `grep` is missing, the fence does
nothing (the old behavior).
- `native-session-executor.test.ts`: Linux tests (a stale Runner whose
tool runs in its own session, plus four look-alikes that must survive;
fail-closed with the marker kept, with a plain and a symlinked log; reap
refusals for orphans owned elsewhere, an unknown orphan, and no journal;
a normal launch starts no extra node process) and a cross-platform check
that the embedded script parses and has no shell-special characters.

**When a launch happens vs. a reattach.** The launch is reached only
after the server chose not to adopt a Runner. A same-run reattachment
(`reattach_remote_runner`) first verifies the marker's Runner: if it is
alive it is adopted and nothing launches; a transient verification
failure (for example the read timeout that #21053 keeps the Runner alive
for) throws and never reaches a launch; only a Runner proven not running
falls through to a fresh launch. Warm recovery likewise throws instead
of launching when revalidation fails. Every launch overwrites the
marker, so after a launch no reattachment could adopt a leftover Runner
anyway, and its port would make the new Runner fail with
`runner_ingress_bind_conflict`. Stopping it therefore never removes a
Runner that the same run could still re-adopt.

## Verification

- Linux (Debian bookworm container, node 24): the Linux tests pass on
this head. The same tests against the previous head (8df6a9099e) fail:
it killed a `paperclip-runnerd reap-lineage --state-dir <same folder>`
command, and it launched although the stale lineage was not proven gone.
On master there is no fence, so the stale Runner stays alive.
- Boot time: a normal launch took 19–35 ms for the whole launch script
in the container with the fence; node is not started by the fence
(asserted by a test). With a stale holder, the stop and launch took
about 160 ms.
- macOS: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` passes
(1412 passed, 4 skipped: the Linux-only tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A launch now stops an older Runner on the same state folder. See "When
a launch happens vs. a reattach" above: this only happens when that
Runner is already impossible to adopt and would block the new one.
- A normal launch pays one `grep` over `/proc` plus reading argv[0] of
the processes that name the folder (its own subshell). No node start, no
extra round trip, no wait.
- If `paperclip-runnerd reap-lineage` cannot verify the stale lineage,
the launch fails closed instead of starting a second Runner next to old
tools.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:22 -06:00
DottaandClaude Fable 5.2 4e932fd178 perf(attention): index the agents and review paths once for every review of a request (#21132)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task in review needs a path to its next action: a reviewer, a
pending interaction or approval, a monitor, a run, a wake, an external
reply or a recovery. A review without a path is stalled, and the
attention feed tells a person about it.
> - The server decides this for each task in review
(`listIssueReviewAttentionMap` in `server/src/services/issues.ts`, with
the classification in
`server/src/services/recovery/issue-graph-liveness.ts`). The task list,
the blocked inbox and the attention feed call it. `GET
/attention/summary`, which the board sidebar reads on most pages, runs
it for every task in review of the company.
> - The classification did the company-wide work again for each task.
Each check of an agent copied the agent list, indexed it again and
walked the reporting chain. Each task scanned every run, wake,
interaction, approval, external wait and recovery of the request. Each
stalled review also filtered and sorted every agent again, with one such
check for each agent.
> - Thus the work grew with reviews × agents for covered reviews and
with stalled reviews × agents² for stalled reviews. It also allocated a
new list and a new index for each check, and the garbage collector of
the server paid for that.
> - On a hosted instance, an allocation sample of the server put 4.7% of
all allocated bytes under this function, and garbage collection used 25%
of the main thread. The two slowest board requests there are the
attention summary and the task list, which both call it.
> - This pull request builds one index for one classification: the
agents by id, the eligibility of each agent, the invokable agents of
each company in id order, and the paths of each task. Each task then
costs its own paths.
> - The benefit is that the review classification of a summary request
falls from 62 ms to 5 ms for 900 reviews and 60 agents, and from 123 ms
to 8 ms for 300 reviews and 180 agents. The request allocates 57% to 91%
fewer bytes. Every finding, every review path and every count is the
same.

## Linked Issues or Issue Description

Refs #20851 and #20858 (they batched the review-wait provenance check
and the task read decision of the same requests; this change is the next
cost on that path).
Refs #19194 and #13729 (open; they change the failed-run lookup of the
attention feed, which is a different statement and a different
mechanism; this pull request does not touch it).

**What existing behavior does this improve?**

The time and the memory that the server uses to decide the review
attention of the tasks in review, for `GET
/api/companies/:companyId/attention/summary`, `GET
/api/companies/:companyId/attention`, `GET
/api/companies/:companyId/issues` and the blocked inbox.

**Subsystem affected**

Server: task-graph liveness classification
(`server/src/services/recovery/issue-graph-liveness.ts`) and the
review-attention map in `server/src/services/issues.ts`.

**Current behavior**

- `classifyIssueReviewPaths(input, issue)` built `new
Map(input.agents…)` for each task and scanned six path lists of the
request for each task.
- `isInvokableAgent` copied the agent list for each check.
`getAgentOrgChainHealth` then indexed that copy again.
- `ownerCandidatesForRecoveryIssue` ran for each stalled review. It
filtered and sorted all agents, and it checked each agent again in two
more loops. Each check did the copy and the index above.
- `hasActiveExecutionPath` built a joined list of the runs and the wakes
for each blocker it looked at.
- `listIssueReviewAttentionMap` searched the review list for each open
recovery task (`reviewIds.includes`) and for each review
(`issues.find`).

Evidence from a hosted instance (sanitized):

- Server main thread at 93% of one core. Garbage collection: 25% of the
main thread (387 pauses in 32 s, the longest 224 ms).
- Allocation sample, 30 s: 77.7 MB/s in total.
`listIssueReviewAttentionMap` and its callees: 4.7% of all bytes
(`classifyIssueReviewPaths` 4.4%, `classifyIssueGraphLiveness` 2.3%,
`reviewFinding` 2.3%).
- `GET /attention/summary` took 16.3 s of server time and `GET /issues`
up to 15.3 s. Most of that time is waiting behind other work on the same
thread (see "Not verified").

**Proposed behavior**

One classification indexes its input one time. The cost of one task is
the number of its own paths. The cost of the agents is one eligibility
decision for each agent. The results do not change.

**Reason and benefit**

The summary request runs on most board pages. Its CPU time and its
allocation also delay every other request, because the server has one
main thread and one garbage collector.

**Breaking changes**

None. `classifyIssueGraphLiveness` and `classifyIssueReviewPaths` keep
their signatures and results. `packages/shared` does not change.

## What Changed

- `server/src/services/recovery/issue-graph-liveness.ts`
- New internal index for one input (`createIssueGraphLivenessIndex`):
agents by id, a memo of `getAgentWorkEligibility` for each agent record,
the invokable agents of each company in id order, and the paths of each
task for the six path lists. The path lists are indexed on first use and
keep the order of the input. The company check of each path stays.
- `classifyIssueGraphLiveness` builds the index one time and uses it for
the review paths, the waiting paths of blockers, the owner candidates
and the assignee eligibility of blockers.
- New export `createIssueReviewPathClassifier(input)`. It returns the
review paths of many tasks from one index.
`classifyIssueReviewPaths(input, issue)` stays for single tasks.
- The memo uses the agent record as its key, not the agent id. Thus an
input that lists one id two times gets the same answers as before.
- The eligibility decision itself is still `getAgentWorkEligibility` of
`@paperclipai/shared`, with the same agent list. The index calls it one
time for each agent, not one time for each check.
- `server/src/services/issues.ts` (`listIssueReviewAttentionMap` only):
one classifier for all reviews of the request, a set for the review ids,
and the review by position instead of a search. No query changes.
- Tests and the reproduction (see Verification):
- `server/src/__tests__/helpers/issue-graph-liveness-reference.ts`: the
classification as it was before this change. It is the oracle of the
equivalence tests.
- `server/src/__tests__/issue-graph-liveness-equivalence.test.ts`,
`server/src/__tests__/review-attention-classification-equivalence.test.ts`,
`server/src/__tests__/helpers/review-attention-fixture.ts`.
- `server/scripts/benchmark-review-attention.ts`: the reproduction
through the real routes.

## Verification

Reproduction and before/after measurement. The script starts a local
PostgreSQL, seeds a synthetic company, mounts the real `attentionRoutes`
and `issueRoutes` with a board user, and sends the two requests over
HTTP. With `--baseline-ref` it runs the pre-fix modules in a second
process against the same database. It fails unless both versions return
the same summary, the same full attention feed, the same task list and
the same review attention for every task in review.

```sh
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000,9000 --agents=60
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000 --agents=180
```

Fixture: one company with tasks of every status (10% in review, 10%
blocked), four of five tasks are children, a reporting tree of agents
with paused, terminated and pending agents. The reviews cover every kind
of path and three ways to stall (31% stalled). The blocked tasks wait
behind chains, cycles, cancelled, done, backlog and unassigned blockers.
Open recovery tasks (5% of the tasks) name reviews, blocked tasks and
done tasks. 6% of the reviews wait for a GitHub reply. A second company
with the same shapes is in the same database.

Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding.
Each version ran in a new process, three rounds in alternating order, 2
warm-up requests and 5 measured requests in each round. A CPU profile of
3 more requests gives the last two columns.

| Tasks / agents / reviews | Request | Version | Statements | Server ms,
median (min) | CPU ms, median (min) | Main thread busy ms, median (min)
| Allocated MB | Review classification ms | GC ms |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 3000 / 60 / 300 (94 stalled) | attention/summary | before | 69 | 121.4
(112.7) | 131.5 (114.4) | 80.2 (73.2) | 170.4 | 16.8 | 5.0 |
| 3000 / 60 / 300 (94 stalled) | attention/summary | after | 69 | 105.9
(94.7) | 113.3 (96.3) | 67.3 (58.6) | 73.3 | 2.4 | 3.4 |
| 9000 / 60 / 900 (283 stalled) | attention/summary | before | 85 |
280.9 (263.9) | 281.4 (253.4) | 203.7 (186.8) | 509.8 | 62.1 | 30.0 |
| 9000 / 60 / 900 (283 stalled) | attention/summary | after | 85 | 224.8
(213.4) | 220.1 (205.8) | 149.9 (142.3) | 219.1 | 4.8 | 6.6 |
| 3000 / 180 / 300 (94 stalled) | attention/summary | before | 69 |
241.2 (203.9) | 248.7 (227.4) | 197.1 (171.1) | 950.2 | 123.2 | 21.2 |
| 3000 / 180 / 300 (94 stalled) | attention/summary | after | 69 | 127.8
(108.5) | 128.5 (102.0) | 76.7 (64.8) | 87.3 | 7.8 | 5.1 |
| 3000 / 60 / 300 | task list (500 rows, 48 reviews) | before | 59 |
102.7 (96.7) | 92.4 (83.6) | 54.7 (48.9) | 73.2 | 2.6 | 2.4 |
| 3000 / 60 / 300 | task list (500 rows, 48 reviews) | after | 59 |
106.2 (91.2) | 89.9 (78.8) | 55.2 (44.8) | 63.4 | 0.9 | 1.6 |
| 9000 / 60 / 900 | task list (500 rows, 50 reviews) | before | 42 |
194.3 (186.0) | 77.5 (69.1) | 43.3 (38.4) | 57.3 | 1.0 | 1.8 |
| 9000 / 60 / 900 | task list (500 rows, 50 reviews) | after | 42 |
189.6 (182.3) | 72.1 (69.5) | 41.9 (39.4) | 57.8 | 1.2 | 2.5 |
| 3000 / 180 / 300 | task list (500 rows, 48 reviews) | before | 59 |
124.6 (95.7) | 110.2 (99.4) | 67.8 (58.1) | 165.4 | 15.4 | 3.8 |
| 3000 / 180 / 300 | task list (500 rows, 48 reviews) | after | 59 |
104.1 (91.9) | 90.7 (79.9) | 56.2 (47.1) | 78.2 | 2.9 | 5.3 |

How to read the table:

- "Review classification ms" is the time in
`listIssueReviewAttentionMap` and everything it calls on the main
thread, for one request, from the CPU profile. It is the direct measure
of this change. Before the fix it was the largest block of application
code in the summary handler: 22% of the busy time at 3000 tasks, 20% at
9000 tasks, and 61% with 180 agents. `getAgentOrgChainHealth` under
`isInvokableAgent` under `ownerCandidatesForRecoveryIssue` had most of
it. After the fix it is 4% to 9%.
- The scaling: three times the tasks gave 3.7 times the classification
time before and 2.0 times after. Three times the agents gave 7.3 times
before and 3.3 times after.
- "Allocated MB" is all bytes that the server process allocated for one
request (`total_allocated_bytes`), including the HTTP client of the
script.
- Other agents used the same machine during the measurement (load
average 8 to 16 on 18 CPUs), so the medians of server time have noise.
The profile columns and the allocation column are the stable ones.
- The task list changes little with 60 agents, because one page has
about 50 reviews. With 180 agents its classification falls from 15 ms to
3 ms and its allocation by half.
- Statement counts do not change: this change removes no statement.

Red/green regression test for the mechanism
(`server/src/__tests__/issue-graph-liveness-equivalence.test.ts`):

- `reads each agent and each path a number of times that does not grow
with the reviews`. The test counts the reads of the agent status and of
the path task id for 20 and for 400 stalled reviews with 40 agents and
360 paths. On the code before this change: `expected { agentStatus:
200800, pathIssueId: 144000 } to deeply equal { agentStatus: 10040,
pathIssueId: 7200 }`. After: 316 agent reads and 360 path reads for both
sizes (one read of each path). The test also runs the oracle and asserts
that its reads grow, so it shows what it catches.
- `classifies the review paths of a page from one index`. The same count
for `createIssueReviewPathClassifier`.

Equivalence tests (old code and new code on the same input):

- `returns the findings and review paths of the pre-fix classification
for random task graphs`. 400 seeded random graphs of two companies. Each
graph has reviews with and without paths, blocked chains and cycles,
terminal blockers, relations and paths that name the other company or a
missing task, reporting chains with cycles, missing managers and
managers of the other company, and ids that the input lists two times.
The test compares the findings of the graph and the review paths of
every task, for both `classifyIssueReviewPaths` and the new classifier.
It asserts that the graphs reach all six finding states and all nine
kinds of path, each at least 20 times.
- `returns the pre-fix findings for blocked chains, cycles and terminal
blockers`. One named graph with the expected state of each finding, so
the comparison cannot pass on two wrong answers.
-
`server/src/__tests__/review-attention-classification-equivalence.test.ts`,
on real PostgreSQL through the real routes. The test wraps the
classification module: every classification that a request runs is also
run by the oracle on the same input, and a difference fails the test.
- `gives every review the paths the fixture was built with, and the
pre-fix classification of every request`. `GET
/issues?status=in_review`, `GET /issues` and `GET
/issues?attention=blocked` (the blocked inbox, whose graph has the
chains, cycles and terminal blockers). Each of the 64 reviews has the
path kinds and the stalled reason that the fixture states for it,
including reviews that wait for an external reply and reviews that open
recovery tasks cover.
- `counts the reviews that need a person, leaves out the ones a person
dismissed, and reads one company`. `GET /attention/summary` and `GET
/attention`: the review count and the review cards are the reviews that
the fixture built to need a person. The second company has its own
counts and none of the first company's tasks. Three dismissed cards
leave the count and stay stalled in the task list.
- The benchmark above compares the complete responses of both versions
through the routes at each size (summary, full feed, task list, review
attention of every review).

Commands run locally:

- `pnpm --filter @paperclipai/server exec vitest run` for
`issue-graph-liveness-equivalence`,
`review-attention-classification-equivalence`, `issue-liveness`,
`recovery-classifiers`, `issue-review-attention`,
`issue-blocker-attention`, `blocked-inbox-review-wait-scope`,
`heartbeat-issue-liveness-escalation`, `attention-retention-work` and
`attention-service`: 210 tests passed. In that run the `afterAll` hook
of `attention-retention-work` and of `attention-service` (the stop of
the test database) timed out at 30 s while other agents loaded the
machine. No assertion failed. CI runs both files.
- `tsc --noEmit` for the server package: no errors. The new test files
and the script are outside the server `tsconfig`; a focused `tsc`
program for them reports no error in these files.
- `pnpm check:module-boundaries`, `node scripts/check-no-git-push.mjs`
and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`:
pass.
- The first CI run of this pull request failed `ci / Runner support
suite`. An earlier revision also let the eligibility functions of
`packages/shared` accept a prebuilt index, and the source of
`packages/shared` is pinned for the runner SDK source proof
(`tests/runner-e2e/source-sdk-public-inputs.json`). The fix does not
need that change, so the current revision leaves `packages/shared` as it
is on `master`: no pin moves, and the UI, the CLI and the runner use the
same eligibility code as before. `source-sdk-public-inputs.test.mjs`,
`source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` pass
locally on this revision (58 tests), and the drift guard reports no
stale pin.
- `pnpm verify` returns exit 2 (broader verification required) because
the change adds a file under `server/scripts/`. CI supplies the broad
coverage. The repo-wide typecheck, build and test suite were not run
locally (shared machine).

Not verified:

- The effect on the hosted instance. This pull request does not deploy.
The local requests take 0.1 s to 0.4 s, and the hosted requests took 15
s. Thus most of the hosted time is not this code: the hosted main thread
is saturated by other work, and each of the 69 to 85 statements of a
summary request waits for its turn on that thread. This change removes
CPU time and allocation from that thread. It does not remove a
statement.
- The number of agents and of stalled reviews of the hosted company. The
hosted allocation sample has most of the bytes in
`classifyIssueReviewPaths`, which is the cost for each review (reviews ×
agents). The local fixture has more stalled reviews, where the owner
candidates cost more.
- A remaining cost inside the classification: each eligibility decision
of `@paperclipai/shared` indexes the agent list again. One
classification now does this one time for each agent (agents² map
entries for one request, 3 ms with 180 agents), not one time for each
check. The helper can accept a prebuilt index in a later change; that
change moves the SDK source pins, so it needs its own review.
- A second, independent cost that this pull request does not change:
`GET /attention/summary` returns counts only, but it builds the full
feed. After this change, the largest application blocks in its profile
at 9000 tasks are the full task list of the 900 blocked tasks (15 ms on
the main thread, of which 9 ms decode description previews that the
summary never returns), the blocker attention of those tasks (9 ms), and
the text of its 85 statements. A later change can give the summary a
count-only path.

## Risks

- The attention feed decides what a person is told needs them. The risk
is a classification that differs from the previous one. Mitigation: the
rules did not move. Only the lookups changed (a map lookup instead of a
scan, a stored eligibility instead of a new one). The oracle tests
compare the findings and the paths for random graphs, for a named graph
and for every classification of the route tests. The benchmark compares
the full responses.
- The index is a snapshot of the input.
`createIssueReviewPathClassifier` documents that the input must not
change while the classifier is in use. The two callers build the input
and use it in the same synchronous block.
- When the input has no `now`, one classification now reads the clock
one time. Before, `classifyIssueGraphLiveness` read it one time for the
blocker checks and one more time for each review. All callers in the
server that classify many tasks pass `now` or run in one synchronous
call, so the values differ by less than the run time of the call.
- The oracle file is a copy of the previous classification (about 640
lines, test-only). A later change of a classification rule must change
the oracle in the same way. The header of the file says so.
- The repository rule in `CONTRIBUTING.md` says that a pull request has
no code comments. This pull request has comments where the code has a
constraint that the names do not show (the memo key, the snapshot rule,
the purpose of the oracle, the usage of the script), as the merged
changes around this code do.
- No migration. No query change. No change to company scope or to actor
permissions: the change is after the reads, on rows that the same
queries return.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests, CPU profiles).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:26:24 -06:00
DottaandClaude Opus 5.5 df89950eb6 fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result. Then a finalizer exports the
run's workspace files and applies the result to the task.
> - When a newer run already decided the task, the finalizer retires the
older result as superseded. Since #21106, this also ends the run.
> - If that retirement interrupts a failed workspace export, the run's
files stay unsaved. The run also stays the writer of the task's
workspace. Nothing records those files for recovery, so nothing ever
releases the workspace.
> - Every run queued on the task then waits behind a save that never
settles. After one hour, each queued run fails setup instead.
> - This pull request records those files for the existing automatic
workspace recovery. Recovery saves them from the retained sandbox, or,
when every bounded attempt fails, continues the task from its last save
with the files attached.
> - The benefit is that the task's queue moves again by itself, and the
run's unsaved work is saved or kept findable. The superseded result is
never applied.

## Linked Issues or Issue Description

Refs #21106

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. Its workspace export failed once with
`remote_workspace_archive_fs_eacces`. The finalizer then retired it as
`native_finalization_superseded`, because another run had already
decided the task. #21106 ended that run at 14:59:17Z (`failed`). The
task still could not move. The next queued run was cancelled with
`workspace_busy` ("ended, but its workspace save has not settled"). Its
retry re-armed every minute. Four more runs stayed queued. The task
showed no recovery action.

**Expected behavior**
The ended run's unsaved files are saved from its retained sandbox, and
the workspace is released. If they cannot be saved within the bounded
retries, the task continues from its last committed save, and the files
are attached to the task. The queued runs proceed in both cases. The
superseded result is never applied.

**Steps to reproduce**
1. Accept a native result for a run that writes a remote workspace.
2. Let the workspace export fail once, while a newer run has already
committed the task's decision.
3. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is `failed`.
4. See that the workspace still names the run as its dirty writer, and
that no retained-file recovery action exists.
5. Queue another run on the task. It defers with `workspace_busy`, and
after one hour it fails setup with "The workspace still has an
unfinished writer or save operation."

**Root cause**
`recordRetryableFailure` (`native-run-finalizer.ts`) retires a
superseded finalizer without preserving the export.
`preserveNativeFailedWorkspace` records a retained-file hold for an
accepted result only when the failure code is
`native_workspace_sync_out_retry_exhausted`. A
`native_finalization_superseded` coordinator never qualifies.
`readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the
holder until its save settles. Without a hold, no worker ever settles
it.

**Paperclip version or commit**
Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the
same behavior.

## What Changed

- `native-superseded-workspace-export.ts` (new): identifies a superseded
finalizer that failed in the workspace export. It selects ended runs
that still hold their workspace with unsaved files and no retained-file
hold. It records the hold through `preserveNativeFailedWorkspace`. It
checks again only while a just-ended run's sandbox stop receipt is still
arriving (at most one hour).
- `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace`
treats a superseded workspace export like an exhausted copy-back, with
the same proofs (run ended, no live controller, accepted result,
confirmed sandbox stop-and-retain, source still bound to the workspace).
It marks the hold `supersededWorkspaceExport`.
- `native-run-projection-work.ts`: the native run projection worker also
selects these runs. It loads them at startup and wakes when a finalizer
changes. Rows written by older releases are therefore settled after
deploy without a person.
- `workspace-recovery-fallback-discovery.ts` and
`workspace-recovery-fallback.ts`: a marked hold whose bounded re-export
is spent (at most 5 retries in 30 minutes) is eligible for the existing
continue-from-last-save fallback. That fallback sets aside only this
run's workspace epoch and keeps its location on the workspace record. It
attaches the readable files to the task unapplied. The task message says
Paperclip "could not save" the previous run's unsaved files.
- `preserved-result-finalization.ts`: a superseded result is never
resumed for application, even after its files are saved.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts`
(new). The first two tests fail on master and pass on this branch:
- "records the files for automatic recovery, saves them from the
retained sandbox and frees the workspace". It uses the staging row
shape. The projection worker records the hold, and a second pass changes
nothing. The workspace waits for that save. The system recovery saves
`saved.txt` into a committed snapshot. The workspace becomes free. The
finalizer, the result and the task's decision do not change, and the
result is not resumed. On master the worker selects nothing (`expected 0
to be 1`).
- "continues from the last save with the files attached when every
re-export fails". After an exhausted re-export budget, the hold is
resolved `restored`. The files are attached (`quarantine.state =
captured`). The workspace moves to epoch 2 with no writer, and the
abandoned epoch keeps its lease location. The message has no request for
a person to fix anything.
- "leaves a superseded result alone when it never failed in the
workspace export".
- Related suites pass: `native-run-projection-work`,
`native-finalization-recovery`, `workspace-recovery-fallback`,
`native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The change acts only on runs whose finalizer is
already terminal and superseded, that failed in the workspace export,
and that still own a dirty workspace. All existing preservation proofs
still apply.
- The saved files of a superseded run become the workspace's next saved
version. The run's result (the task decision) is still never applied.
- When the export can never succeed, the task continues from its last
save. Files that cannot be read are not attached. The workspace record
keeps the set-aside epoch's location. Saving the readable part of a
partly unreadable workspace is not in scope.
- Runs that already failed setup before this deploys are not restarted
by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:20:54 -06:00
Nicky LeachandPaperclip 782d6bead4 Configure managed plugin Runner lifecycle (#21065)
## Thinking Path

> - Paperclip runs agents in managed environments.
> - An agent can use an explicit environment or the instance default.
> - A lifecycle plugin must receive the driver that provisions the
agent.
> - A managed plugin environment also needs to select the Runner process
lifecycle.
> - This change gives lifecycle selection and provisioning the same
environment lookup.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Managed plugin environment provisioning and lifecycle configuration.

**Current behavior**
Lifecycle calls lack the selected plugin driver. Managed plugin
configuration cannot select warm Runner execution.

**Proposed behavior**
Resolve the environment once per lifecycle operation. Pass the driver
key for the plugin that serves it. Parse and persist the managed Runner
lifecycle mode.

The portable ownership and workspace placement changes are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066) and
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Share agent environment resolution between required plugin selection
and the lifecycle RPC.
- Pass `environmentDriverKey` to the plugin that serves the selected
environment.
- Allow managed plugin environments to set `warm` or `per_turn` and
persist the selected mode.
- Advertise `pluginEnvironmentRunnerLifecycleMode` in the release
capability manifest so senders can check support before including the
field.

## Verification

- Built shared types, the plugin SDK, and Runner. Server typecheck and
SDK source-pin check passed for the initial layer.
- The capability contract test failed before the manifest fix and passed
after it. Independent verification passed all 160 tests across managed
configuration, plugin environment persistence, lifecycle routing, and
environment reconciliation.
- Routing coverage includes the instance default, an explicit second
driver from the same plugin, and an explicit sandbox.

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862)
passed on `f07b192c21`. The first attempt timed out in an unchanged
workspace-recovery fixture and then collided with a background recovery
action. The exact test and all 44 tests in its suite passed locally; the
isolated shard retry passed without code changes.

## Risks

- Existing managed configuration can omit the lifecycle mode. Senders
must check the target release for `pluginEnvironmentRunnerLifecycleMode`
before sending the new field. Publishing this signal does not update
external deployment tools.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 08:18:04 -08:00
DottaandClaude Fable 5.2 1af987690d perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process serves every request and agent turn on one
thread. The same process saves each remote workspace as one encrypted
archive in object storage, and restores it before a turn
> - `server/src/services/remote-workspace-snapshot-storage.ts` hashed,
encrypted and decrypted every byte of those archives on that thread.
`server/src/storage/s3-multipart.ts` computed the Content-MD5 of each
uploaded part on that thread too
> - Hashes and ciphers are synchronous in Node. A CPU profile of a busy
hosted server showed this work on the main thread, which was at 93% of
one core
> - Each megabyte of a snapshot delays every request and turn that waits
behind it
> - This pull request moves those passes to a small, bounded set of
worker threads. The checks, their order, their failures and the stored
format do not change
> - The benefit is that a 1 GiB save uses about 0.7 s of the server's
thread instead of about 2.8 s, and an unrelated request during the save
is answered in 1.3 ms (p99) instead of 16.7 ms

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`workspace snapshot hash encrypt`, `event loop` and `archive io`.
Related merged work: #19900 gave these stream passes 1 MiB buffers,
#20554 removed a read-back from migration, #20722 took runner artifact
hashes off the event loop, #20709 records long event-loop blocks. No
open pull request covers this change.

**What happened?**

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core) showed:

- the two stream transforms in the snapshot save at 3.9% and 1.6%
inclusive. The first hashes the archive (SHA-256). The second hashes the
ciphertext (SHA-256);
- native hash `update` at 5.4% self time for the whole process. About
1.0 s of it was under those two transforms and 0.25 s was under
`fileRangeMd5`;
- the AES-256-GCM cipher of the same pipeline, and Buffer allocation
under the file read streams. Garbage collection was 25% of the thread.

What the code did with each byte, on the server's thread:

| Path | Pass | Digest or cipher | Purpose |
| --- | --- | --- | --- |
| Save | read archive | SHA-256 of plaintext | bound into the wrapped
key, checked on each restore |
| Save | same read | AES-256-GCM | confidentiality and authenticity |
| Save | same read | SHA-256 of stored bytes | recorded on the snapshot
row, checked on each restore and read-back |
| Save | read each part (S3) | MD5 | `Content-MD5`: the object store
rejects a part it did not receive intact |
| Save, read-back callers | read the kept file | SHA-256 of plaintext |
proof that the published file is the saved archive |
| Restore | download | SHA-256 of stored bytes | integrity of the object
against the snapshot row |
| Restore | read ciphertext | AES-256-GCM | authenticity |
| Restore | read plaintext | SHA-256 of plaintext | integrity against
the digest in the wrapped key |
| Restore | read plaintext again | none (copy) | publication beside the
destination |

**Expected behavior**

Digest and cipher passes over a snapshot do not run on the thread that
serves requests. Every check stays, with the same failure.

**Steps to reproduce**

1. On the parent commit, run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (copy the
script from this branch).
2. Read `upload.mainThreadCpuMs`, `upload.mainThreadBusyMs` and
`upload.unrelatedRequest`. The benchmark uses the real save and restore
functions, the real S3 client, and a local object store that checks each
part's `Content-MD5`. A separate process requests a trivial route from
the measured server and times each reply with its own clock.

**Paperclip version or commit**

`master` at 9e910ff9ae. The profile is from 1d153f62b6.

**Deployment mode**

All modes that save remote workspaces. The profile is from a hosted
server with S3 storage.

**Node.js version**

Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/storage/file-crypto-pool.ts` (new): runs file digests and
AES-256-GCM passes on worker threads.
- Bounds: at most 2 jobs run at once. Each worker has two reused 1 MiB
read buffers and one write in flight. A waiting job holds file names and
byte ranges, not file contents. Waiting jobs are served in order.
- Cancellation: a waiting job is cancelled at once. A running job stops
at its next chunk, closes its files, and then the caller sees the same
`AbortError` as before.
- Shutdown: an idle worker does not keep the process alive. A busy
worker does, until its job ends or its caller's signal aborts it.
- Key handling: the pool copies the data key for the worker, moves that
copy (it is not cloned), and the worker erases it. A copy that never
reaches a worker is erased on this thread. The caller keeps and erases
its own key as before.
- The worker program is sent as source text, as
`remote-workspace-snapshot-archive.ts` already does. It needs no extra
file in the build or the runtime bundle.
- Save (`uploadWorkspaceSnapshot`): the one read that digests the
archive, encrypts it and digests the stored bytes runs in a worker. The
identity check around that read (`archive_changed`) stays on this thread
and is unchanged.
- Part upload (`fileRangeMd5`): the `Content-MD5` of each part is
computed in a worker. The first round of parts is digested before any
part is sent, so the parts still leave together and a rate limit still
answers one burst (the existing throttle test covers this). Every digest
of that round has ended, also after a failure or a cancellation, before
the upload settles.
- Restore (`loadWorkspaceSnapshotArchive`): the download no longer
hashes on this thread. One worker digests the downloaded object while
another decrypts it into private scratch. The results are then checked
in the previous order: object size and digest, header, key, GCM tag.
- Restore, one pass less: the decrypted archive was read twice (digest,
then copy). It is now digested while it is copied beside the
destination, and linked to the destination only when the digest matches.
The digest covers exactly the published bytes.
- Read-back callers (`verifiedArchivePath`): the digest of the kept file
runs in a worker, through the same open descriptor. The file identity
checks are unchanged.
- `server/scripts/benchmark-workspace-snapshot-crypto.ts` (new): the
reproduction and benchmark.
- Docs: `doc/run-log-events.md` (what each `storage.snapshot.*` phase
now does) and `doc/DEVELOPING.md` (the worker threads and the
benchmark).

Module boundaries (`rules/module-boundaries.mdc`): the pool is in
`server/src/storage/` because both consumers are parts of the same
upload, and storage must not import from services. Its contract is file
paths, byte ranges, an algorithm, and AES-GCM parameters. The snapshot
format (magic bytes, additional data, key wrapping, the order of checks)
stays in the snapshot service. The part layout and `Content-MD5` stay in
the S3 module.

Not changed, and why:

- The stored format, the ledger, attempt journaling, atomic publication,
and `recoverWorkspaceSnapshotUpload` /
`recoverWorkspaceSnapshotArchive`.
- `verifyObject` (full read-back for a store that did not verify the
upload, and for a retried save) still hashes on this thread. Its bytes
arrive from the network on this thread. To move it, the object must
first be written to disk.
- Socket writes of part bodies stay on this thread. I measured 1 MiB
part-body read chunks in place of 64 KiB: main-thread time did not
change and the longest event-loop delay increased, so I did not keep it.
- `remote-workspace-transport.ts` also hashes an archive on this thread.
It is outside the snapshot storage path.

## Verification

Regression tests, red then green (`cd server && pnpm exec vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/storage/file-crypto-pool.test.ts`):

- Parent implementation with the tests of this branch: 3 tests fail.
- `keeps encrypting a save while the server thread is occupied by other
work`: `expected 4194323 to be 25166144`. The test blocks the thread
until the encrypted file is complete. On the parent the file stops at 4
MiB.
- `answers unrelated requests while a save is being encrypted, and
digests or encrypts none of it on the server thread`: `expected 8388627
to be less than 65536` (bytes hashed on the thread).
- `digests and decrypts none of a save or a restore on the server thread
when the store verified the upload`: `expected 33556772 to be less than
65536`.
- This branch: 58 of 58 and 6 of 6 pass.
- The tests use synchronization, not time. A named pipe keeps a save in
the encrypt step for as long as the test feeds it. Deadlines are 30 s
and matter only for a failure.
- Integrity tests that pass on both the parent implementation and this
branch, which shows that the failures are the same: a truncated object
(`object_hash_mismatch`), an oversized object (`object_size_mismatch`),
an archive that grows, shrinks or is rewritten in place after its
identity was read (`archive_changed`, before any upload), a ciphertext
that was changed with the recorded digest changed to match (GCM
failure), a plaintext that differs from the digest in the key
(`plaintext_hash_mismatch`, staged copy removed), and the order object,
header, key.
- Cancellation: during encryption the save rejects with `AbortError`,
and leaves no scratch file, no object and no recorded attempt. During
the store write it leaves no object.
- Part digests (`src/__tests__/storage-s3-multipart.test.ts`): `leaves
no part digest running or waiting when the source is cancelled|removed
during the first round`. Before the review fix in a354242865 it failed
with `{ running: 2, waiting: 0 }` at multipart cleanup.
- Pool tests: digests equal single-threaded digests for a file, a range
and a descriptor; encrypt and decrypt round trip; each damaged
ciphertext is reported; at most 2 jobs run and the others wait in order;
a waiting job and a running job are both cancelled.

Benchmark, same machine, 5 alternating pairs (parent implementation,
then this branch) for each size. Values are medians, with the range in
parentheses.

Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (and
`--mib=200`).

1 GiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 2,834 ms (2,717 to 3,007) | 742 ms (658 to 881)
| 1,602 ms (1,378 to 1,837) | 510 ms (470 to 589) |
| Main-thread busy time (event-loop utilization) | 2,938 ms | 858 ms |
1,673 ms | 597 ms |
| Total process CPU time | 6,735 ms | 4,762 ms | 5,564 ms | 3,306 ms |
| Wall time | 3,914 ms (3,656 to 4,253) | 2,580 ms (2,507 to 3,200) |
2,502 ms (1,756 to 2,922) | 1,799 ms (1,508 to 2,492) |
| Longest event-loop delay | 12.6 ms | 6.0 ms | 10.4 ms | 5.4 ms |
| Event-loop delay p99 | 9.2 ms | 2.0 ms | 2.1 ms | 1.8 ms |
| Unrelated request p99 (p50 is 0.2 to 0.3 ms in all runs) | 16.7 ms |
1.3 ms | 4.7 ms | 1.3 ms |
| Unrelated request, slowest | 18.5 ms | 4.5 ms | 10.0 ms | 3.5 ms |
| Peak resident memory | 495 MiB | 408 MiB | 524 MiB | 476 MiB |

200 MiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 571 ms | 181 ms | 378 ms | 141 ms |
| Main-thread busy time | 602 ms | 236 ms | 395 ms | 167 ms |
| Total process CPU time | 1,235 ms | 935 ms | 1,385 ms | 840 ms |
| Wall time | 842 ms | 679 ms | 538 ms | 382 ms |
| Longest event-loop delay | 12.4 ms | 4.0 ms | 7.3 ms | 2.9 ms |
| Unrelated request p99 | 16.5 ms | 2.0 ms | 1.9 ms | 1.9 ms |
| Peak resident memory | 437 MiB | 398 MiB | 510 MiB | 424 MiB |

How to read these numbers:

- The parent did not block the thread for seconds at one time. It worked
in 1 MiB slices, so the longest single delay was about 12 ms. The cost
was the sum: the thread was busy for about 75% of a 1 GiB save. That sum
is what the profile showed, and it is what decreases.
- A worker thread moves CPU time; it does not remove it. Total CPU time
also decreased here (about 29% for a save, 41% for a restore) because
the workers reuse two buffers in place of one new 1 MiB Buffer for each
chunk, and a restore reads the decrypted archive once in place of twice.
- Wall time decreased because the part digests no longer wait for the
thread that sends the parts, and a restore digests and decrypts at the
same time.
- What remains on the server's thread is network and file streaming:
socket writes of the parts, the download, and the database calls.
- Limits: the machine was shared with other work (1-minute load average
8 to 14 on 18 cores) and timings varied between runs, which is why the
pairs alternate. Earlier runs on a busier machine (load 15 to 28) showed
the same CPU-time ratios with wall times up to 4 times longer for both
implementations. The object store was on loopback without TLS.

Other checks:

- `cd server && pnpm exec vitest run src/storage
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-snapshot-reuse.test.ts
src/__tests__/remote-workspace-file-resources.test.ts` and the 8
`src/__tests__/storage-s3-*.test.ts` files: 249 tests in 15 files pass.
- `tsc --noEmit` over the changed TypeScript files: clean. `pnpm
check:module-boundaries`: pass.
- The pool, bundled with the options of
`server/scripts/bundle-runtime.mjs` and run with plain `node`: encrypt,
decrypt and digest round trip, and the process exits.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the hosted server after deploy, Linux x86 with Node 24
outside CI, a real object store with TLS, and archives larger than 1
GiB.

## Risks

- This is integrity and confidentiality code. The mitigation is that
each check and its failure is pinned by a test that passes on the parent
implementation and on this branch, and that the format is unchanged, so
objects saved before and after this change restore with either version.
- Order of work in a restore: the ciphertext is now decrypted into
private scratch while the object digest is computed, not after it. No
output is used before the digest, header, key and tag checks pass, and
the scratch directory is removed on every failure. A damaged object is
reported with the same error, but after the decrypt work.
- The staged copy beside the destination (`.workspace-snapshot-*.tmp`)
is now written before the plaintext digest is known, and after the GCM
tag is verified. It is removed on a mismatch. The destination itself is
linked only after the digest matches, as before.
- Resource trade-off: up to 2 more threads, each with about 2 MiB of
buffers and a small heap (64 MB limit), that stay for the life of the
process once used. In exchange the server's thread does about 70% less
work for each snapshot. With more than 2 snapshot passes at one time,
the others wait. A part digest can wait behind a long encryption.
- Run-log timing shifts: `storage.snapshot.digest` is now about 0 ms,
`storage.snapshot.verify_plaintext` includes the copy, and
`storage.snapshot.publish` is only the link. `doc/run-log-events.md`
describes this.
- A worker that stops unexpectedly fails its job with
`file_crypto_worker_exited`. The save or restore then fails as for any
other I/O error, and the next job starts a new worker.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:11:37 -06:00
DottaandClaude Opus 5.5 fb8b61bc3e fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A release swap stops the old server and a new server
recovers each run.
> - When the old server stops during a run's bootstrap, the new server
restarts that bootstrap on the same run (`bootstrap_incomplete`). The
claim is granted only when the Postgres controller checkpoint proves
that no provider turn was sent.
> - The old server can stop after it staged launch files (Codex home,
config) into the sandbox session folder, but before it launched the
Runner. Its detach fence stops it there.
> - The new server then finds no Runner state in the sandbox. A fresh
bootstrap must create the session folder itself, and the folder already
exists, so the run failed with `runner_harness_state_mismatch:
resumed_unavailable`. The failure is an ambiguous session, so the task
waits for a person.
> - This pull request lets host preparation prove that no Runner was
ever launched in that folder, move the folder aside, and restart the
bootstrap.
> - The benefit is that a release swap during this part of a bootstrap
no longer blocks the task. No provider work can be lost or replayed,
because no Runner ever ran.

## Linked Issues or Issue Description

Refs #21120 (same release-swap class: the Runner was launched and
connected before the old server stopped; that pull request stops it in
host preparation. This pull request covers the case where no Runner was
launched.)
Refs #21079 (launch-time fence for a stale Runner on the same state
folder.)

**What happened?**
On hosted staging, during a release swap on 2026-11-12, a Codex run on
Daytona failed with `native_session_interrupted` and
`runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_unavailable`. The old server was
in `harness_state.launch_material` when its shutdown drain detached the
run. It recorded no Runner process identity. The new server chose
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". Its host preparation found the sandbox resumed and no Runner
state. Then the claim of the session folder failed, because the old
server had already created that folder. The run needed a person.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a person. A provider turn is never replayed and two Runners never share
the run.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Stop the server after the bootstrap stages launch files into the
sandbox session folder and before it launches the Runner.
3. Let a new server recover the run with a `bootstrap_incomplete` claim.
The new server has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_unavailable`.

**Paperclip version or commit**
Staging release `704b3f4d63`. The code is the same on master at
`ef2b53b675`.

**Deployment mode**
Hosted staging, remote sandbox execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-session-executor.ts`: under a same-run `bootstrap_incomplete`
claim with no failover backup, when host preparation finds no Runner
state, a new check asks the sandbox in one command whether the session
folder exists and holds no Runner trace. A Runner trace is the live
`runner/runner-state.json`, the `runner/runner-process.identity` launch
marker (the launch script writes it before the Runner starts), an
`authority-epochs` archive, or any other `runner-state.json` or launch
marker anywhere under the folder. If the folder holds no Runner trace,
no Runner was launched there, so host preparation records the restart
proof (`absent_confirmed`). It then uses the existing path: move the
folder aside (it is kept for inspection), claim a new folder, and
bootstrap again. If the folder is missing, the existing fresh-claim path
runs unchanged. If any Runner trace exists, the run stays fail-closed as
before. An archived session is therefore never discarded.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes
restart the bootstrap. A launch marker, an archived authority epoch,
Runner state elsewhere in the folder, or a failover backup keeps the run
fail-closed with `resumed_unavailable`.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session" --maxWorkers=2` fails on
both `interrupted_bootstrap_unlaunched` restart cases with
`runner_harness_state_mismatch: resumed_unavailable`, which is the
staging error.
- Red: without the archived-state guard, the `_archived` and
`_nested_state` cases fail (the folder is moved aside). With the guard
they pass.
- Green: the same command passes (64 tests).
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
--maxWorkers=2`: full file, 1422 passed before the guard commit. The
focused 64 pass on the final revision.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- `git merge-tree` against #21120's head (`0d18f18928`) merges without
conflict. The new check runs only when no Runner state exists, so
#21120's new call site (it runs only when a Runner state exists) gets
the same results as before.

## Risks

- Low. The new path runs only in host preparation, under a
`bootstrap_incomplete` claim for the same run id, with no failover
backup. That claim is granted only after the previous controller's lease
has ended. With no Runner state or launch marker anywhere in the folder,
no Runner and no provider session existed there. The old folder is moved
aside, not deleted.
- A controller that is still launching at the same moment would have to
hold a lease that the claim already took over. The detach fence stops
such a launch, and #21079 adds a launch-time fence for a leftover Runner
on the same state folder.
- No added wait before a normal turn starts: the check runs only on this
failure path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser for read-only run evidence, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:06:55 -06:00
DottaandClaude Fable 5.2 5b698b68cf perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database
connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update,
insert, count/max read, COMMIT). The query builder built five of them
again for every event, and each event created two transaction objects.

A native source event now takes the run lock and then runs one statement
that finds a replayed event or reserves the sequence and inserts, and
reads the retained count and maximum for the acknowledgement. The lock
stays a statement of its own so the append reads a snapshot taken after
the lock was granted. The control-plane port passes its controller check
to the append instead of opening an outer transaction, so there is no
SAVEPOINT. The hot statements are fixed SQL text.

Events that wake a queue still register the wake before the write and
only when the event is new. Order, sequence numbers and durability do
not change: every event still commits before its append returns.

Local benchmark (server/scripts/benchmark-run-event-append.ts), one
native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated,
1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events.
One ordinary event: 105 -> 27 KB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:58:31 -06:00
DottaandClaude Opus 5.5 9e910ff9ae fix(chat): keep raw IDs and internal codes out of chat replies (#21081)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the agent replies in plain
text
> - In a full chat-eval cohort, 47 of 158 passing replies showed raw
UUIDs, error codes, field keys, tool names or server paths
> - Users must not see internal jargon or raw IDs, so these replies are
a product defect, even when the facts are correct
> - Agents copy these terms from API error messages and saved fields;
the chat directive did not tell them to translate
> - This pull request fixes the sources: the chat directive, and API
denial messages that put IDs and policy codes in the human message
> - The benefit is plain replies for every model, while machine codes
stay available in separate fields

## Linked Issues or Issue Description

**What happened?**

Chat replies quote internal names and raw IDs. Examples: run, handoff,
revision, interaction and trigger UUIDs in prose;
`native_session_retry_exhausted`, `board_access_required`,
`review_policy_denied`, `local-board`, `enableCases`; a raw user ID
inside the inbox denial message.

**Expected behavior**

Replies name records the way the person sees them (task identifier,
title or name, with a link) and explain codes in plain words. IDs stay
only in link targets, in commands the person must run, or when the
person asks for an ID.

**Steps to reproduce**

Run the chat-features OpenCode cohort (for example `cases.close-case`,
`routines.webhook-trigger`, `inbox-search.archive-without-grant`) and
read the agent replies.

**Paperclip version or commit**

`master` at f30005440b.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with the Paperclip runner and
OpenCode.

**Deployment mode**

Local dev server (embedded PGlite) and the CI chat-eval runner.

**Installation method**

Source checkout (`pnpm dev`).

## What Changed

- `AGENT_CHAT_DIRECTIVE`: add one rule. Name records by their
identifier, title or name and link them. Do not write raw IDs, internal
codes, field keys, tool names, process names or server paths, even in
inline code. The rule has exceptions for link targets, commands the
person runs, and IDs the person asks for. The rejection rule now asks
for the platform's reason "in plain words".
- Authorization denials (inbox management, membership, responsible user)
no longer put user or agent IDs in the explanation. The `reason` and
`code` values do not change.
- Review-policy denials explain the rule in plain words. The
`review_policy_denied` code and the `policy` and `allowedActor` fields
do not change.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts`
(directive rule, and it reaches the chat task prompt),
`server/src/__tests__/authorization-service.test.ts` (inbox denials
carry no IDs), `server/src/__tests__/issue-review-policy.test.ts` and
`server/src/services/issue-thread-interaction-resolution.test.ts` (plain
review messages). Each new assertion failed before the fix.
- `pnpm verify` and the server typecheck pass locally.
- A paid OpenCode run on 8 affected chat cases is in progress; results
will be added here.

## Risks

- Low risk. Only human-readable messages and prompt text change. No
client matches the old message text; codes and machine fields stay the
same.
- The directive grows by one paragraph. A model can still copy an ID;
the rule reduces this but cannot prevent it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:37:07 -06:00
DottaandClaude Fable 5.2 8bdd53c11f perf(server): start no git process for server info outside a checkout (#21122)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reports which commit it runs. `GET /api/health` and `GET
/api/liveness` return it, and the board UI and the hosting control plane
poll these routes
> - The commit comes from `getServerInfoSnapshot()` in
`server/src/server-info.ts`. The health route calls it on each request.
The snapshot keeps its Git metadata for 3 seconds, then reads it again
with `execFileSync("git", ["show", ...])`, which is a synchronous child
process
> - A container or packaged install has the `git` command but no
checkout. There `git show` can only fail, and the server then uses the
commit that the build supplied. The failed start still stops the event
loop for the length of a process start
> - A CPU profile of a busy hosted server showed this cost on the main
thread
> - This pull request looks for a checkout before it starts `git`.
Without a checkout the server takes the build-commit path directly
> - The benefit is that health polls start no child process on a server
that does not run from a checkout, and the responses stay the same

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`server-info git health`, `readGitInfo` and `getServerInfoSnapshot`.
Related merged work: #8753 added the 3 second refresh so that the debug
view follows a checkout, #10563 and #10566 added the build commit,
#18649 took the process start time off the Git probes. No open pull
request covers this change.

**What happened?**
`server/src/app.ts` creates the health routes without a `serverInfo`
option, so each `GET /api/health` request calls
`getServerInfoSnapshot()`. When the 3 second cache is expired, the call
starts `git show -s HEAD`. On an install without a checkout the command
fails each time, so the failure is paid again on each refresh. The rate
is one `git` start for each request when requests are 3 seconds or more
apart, and one start for each 3 seconds when requests are more frequent.
It is not one start for each request in a burst.

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core, process about 4 GB
resident) showed `getServerInfoSnapshot` → `readGitInfo` → the `git
show` command at 0.6% inclusive. That is about 190 ms in 32 seconds,
which agrees with one failed start for each 3 second refresh. The
profile showed only the `git show` command under `readGitInfo`, not the
status or branch commands that run after a successful `git show`. The
native child-process spawn was 0.9% self time in the full profile; other
callers also start processes.

**Expected behavior**
A server that does not run from a Git checkout starts no `git` process
for health or liveness requests. It reports the build commit, as before.
A server that runs from a checkout continues to read Git after each 3
second refresh, because the commit and the local changes of a checkout
can change while the process runs (#8753).

**Steps to reproduce**
1. Run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-health-server-info.ts --requests=200` on the parent
commit. The benchmark puts a counting `git` wrapper first on `PATH`. The
wrapper records the start and then runs the real `git`.
2. Read the row `packaged install: no checkout, build supplies the
commit`, pattern `polled every 5000 ms`. It shows 200 `git` execs for
200 health requests, and 1 exec when the routes are created.

**Paperclip version or commit**
`master` at ef2b53b675. The profile is from 1d153f62b6.

**Deployment mode**
All modes. The profile is from an `authenticated` hosted server that
runs the container image. The image installs `git`, and the build
context excludes `.git` (`.dockerignore`).

**Node.js version**
Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/server-info.ts`: the default `git show` command first
looks for a checkout. A checkout is a `.git` entry in the working
directory or in one of its parents, or a `GIT_DIR` value. Without one,
the command throws before it starts a process, and `readGitInfo` takes
its existing build-commit path. The same check applies when
`/api/liveness` reads the snapshot at route creation.
- The 3 second refresh, the checkout path, the injected-command path for
tests and `processStartedAt` are not changed.
- `server/src/__tests__/health-server-info-git-probe.test.ts` (new):
regression tests that count the `git` processes that the server starts.
The tests use real Git and real directories.
- `server/scripts/benchmark-health-server-info.ts` (new): the
reproduction and benchmark.
- No route file is changed. `server/src/routes/health.ts` continues to
read the snapshot on each request.

## Verification

Regression test, red then green:

- `cd server && pnpm exec vitest run
src/__tests__/health-server-info-git-probe.test.ts`
- Parent commit (with the test file of this branch): 1 of 4 tests fails.
`starts no git process when the server does not run from a checkout`
fails with `expected 6 to be +0`: one start when the routes are created
and one for each of 5 polls. The response assertions in that test pass
on the parent commit.
- This branch: 4 of 4 tests pass.
- The tests assert the number of `git` starts and the response content.
They do not assert time. The first test compares the full response text
of anonymous health, board health and liveness with the responses of a
server that has the snapshot injected. The other 3 tests pass on both
commits. They keep the checkout behavior: a new commit is visible after
the refresh, also from a subdirectory; a linked worktree (`.git` is a
file) is read; `GIT_DIR` is obeyed.

Benchmark, same machine, 200 health requests per row, 3 runs each
(ranges shown). Each row alternates anonymous and board requests against
routes that are created as `server/src/app.ts` creates them.

| Scenario | Request pattern | Before: `git` execs (route creation +
requests) | Before: ms per request | After: `git` execs | After: ms per
request |
| --- | --- | --- | --- | --- | --- |
| No checkout, build supplies the commit | polled each 5 s | 1 + 200 |
12.5 to 13.8 | 0 + 0 | 1.05 to 1.53 |
| No checkout, build supplies the commit | burst | 1 + 0 | 1.47 to 1.63
| 0 + 0 | 1.19 to 1.57 |
| Source checkout | polled each 5 s | 3 + 600 | 36.3 to 50.2 | 3 + 600 |
36.8 to 40.7 |
| Source checkout | burst | 3 + 0 | 1.38 to 1.40 | 3 + 0 | 1.03 to 1.45
|

- Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-health-server-info.ts --requests=200`
- The `polled` pattern moves the clock that the server reads forward by
5 seconds before each request. It does not wait for real time.
- The benchmark prints a SHA-256 prefix of the anonymous health body, of
`serverInfo.git` in the board health body and of the liveness body. The
values are the same before and after in all rows.
- The `git` call is synchronous, so the difference per request (about 11
to 12 ms locally, which includes the shell wrapper) is time that the
event loop is blocked.
- The source checkout rows do not change. This is the intended result.

Assumption and evidence (`rules/runtime-dependency-boundaries.mdc`): the
change assumes that real Git fails where the check finds no checkout,
and that the check finds each checkout that Git finds. The first
regression test runs real `git show` in the same directory and asserts
that it fails. The other tests run real Git from a repository
subdirectory, from a linked worktree and with `GIT_DIR`. The new check
itself is a small number of `existsSync` calls (one for each directory
level) on each 3 second refresh.

Other checks:

- `cd server && pnpm exec vitest run src/__tests__/server-info.test.ts
src/__tests__/health.test.ts src/__tests__/liveness.test.ts
src/__tests__/health-dev-server-token.test.ts
src/__tests__/health-workspace-readiness.test.ts
src/__tests__/cloud-warm-standby.test.ts
src/services/native-runtime/native-controller-identity.test.ts` and the
new test file: 58 tests in 8 files pass.
- `tsc --noEmit` over the three changed TypeScript files: clean.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the benchmark on Linux with a multi-gigabyte process,
and the hosted server after deploy. The local numbers come from a
process of about 700 MB, where a process start is cheaper than on the
profiled server. That the profiled server has no checkout is a
conclusion from the image build and from the profile; I did not read its
`serverInfo` response.

## Risks

- Low risk. The response bodies of `/api/health` and `/api/liveness` are
the same as before for an install without a checkout and for a checkout.
- Behavior shift: a server whose working directory is inside a bare
repository, with no `.git` entry and no `GIT_DIR`, reported the commit
of that repository before. It now reports the build commit, or no
commit. This layout is not a supported way to run the server.
- A server that runs from a source checkout continues to start up to 3
`git` processes on each 3 second refresh while health is polled. This
pull request does not change that path, because #8753 made it read the
checkout live on purpose.
- If a checkout is created around the working directory while the server
runs, the next refresh finds it, because the check runs on each refresh.

## Model Used

- Claude Fable 5.2 (Anthropic, model ID `claude-fable-5-2`), in Claude
Code with tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:28:07 -06:00
DottaandClaude Opus 5.5 ef2b53b675 fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - When the resume call times out, the run failed after 90 seconds with
`reusable_sandbox_resume_failed (cause=transport_timeout)`
> - The provider keeps starting the sandbox after the host call times
out, so the sandbox is usually ready a little later
> - The run gave up after one call, and the task then waited behind a
Retry button
> - This pull request keeps the same run retrying an idle sandbox whose
resume only timed out, up to the provider's start deadline
> - The benefit is that recovery works automatically and a sandbox with
unsaved work is never destroyed

## Linked Issues or Issue Description

Refs #20626, #20828, #21020

**What happened?**
Three Codex runs on Daytona failed before provider dispatch with:
`Reusable sandbox lease "<id>" could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.` Each run had `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. Each run took 100 to 122 seconds: one
resume call used the whole 90-second budget, so no in-run retry was
possible. The preserved leases were idle. Their last runs had ended
about two hours before, with a confirmed stop-and-retain. The workspaces
were not safe to replace, so the run failed strictly. The automatic
bootstrap retry was then refused ("The automatic retry could not
revalidate the original user continuation."), and all three tasks ended
`blocked` behind a Retry.

**Expected behavior**
A resume that only timed out, before any provider work, on a sandbox
that no live run holds, continues in the same run. The run keeps the
same sandbox and does not create a replacement or destroy the sandbox.

**Steps to reproduce**
1. Let a task's reusable Daytona sandbox stop and stay idle for some
hours (it auto-archives after 60 minutes).
2. Retry the task. The workspace has unsaved work, so replacement is not
safe.
3. The `environmentResumeLease` call takes longer than 90 seconds. The
host call times out (`RPC call "environmentResumeLease" timed out after
90000ms`), the 90-second budget is spent, and the run fails with
`reusable_sandbox_resume_failed`.

**Paperclip version or commit**
Staging at `849f109f` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`codex_local` (Codex) on a Daytona sandbox environment. The change is
adapter-independent.

**Root cause**
`server/src/services/environment-runtime.ts`, reusable resume loop: the
first `environmentResumeLease` call got the whole 90-second budget as
its RPC timeout. When that call timed out, the loop saw no time left and
threw. The 90-second budget assumes that the bounded bootstrap retry
resumes the same sandbox later. That retry is not always admitted. A
timeout is different from a refusal: the provider is still starting the
sandbox.

## What Changed

- `reusable-sandbox-resume-replacement.ts`: add
`reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no
other live run holds it). `reusableSandboxReplacementIsSafe` now uses
it, with the same rules as before. Add
`REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start
deadline plus the call margin).
- `environment-runtime.ts`: when the 90-second budget ends on a
`transport_timeout` and the preserved lease is idle, extend the deadline
one time, up to the provider start deadline (at most 330 seconds from
the first attempt). The run keeps retrying the same sandbox with
backoff. Each call still has at most 90 seconds. Refusals, a live owner,
and other failures keep the current behavior. The run never replaces or
destroys the sandbox because of this extension.
- `doc/execution-semantics.md`: document the idle-timeout extension.

## Verification

- Red before the fix, with the new tests in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(staging shape: released lease, finished owner run, workspace with
unsaved work, 300-second provider deadline, host RPC timeout): 2 tests
failed with the staging error `could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.`
- Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in
the same run with no create or destroy call. If every call times out,
the run stops at or before 330 seconds, makes more than one call, and
keeps the lease `released`. A sandbox that a live run still holds gets
one call and fails at 90 seconds, as before.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/services/remote-workspace-transport.test.ts`: 378 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 because a doc is a global CI input. Required CI
supplies the broad coverage.
- Not run: a live Daytona resume on staging. No later run attempted the
three preserved leases, so it is inferred, not observed, that the
sandboxes were ready after the host call timed out.

## Risks

- A run can now wait up to 330 seconds in preparation before its first
turn, but only when the resume timed out on an idle sandbox. Before this
change, that run failed and the task waited for a person.
- A provider that is down (not slow) now takes up to 330 seconds to fail
instead of 90 seconds. The failure is the same
`reusable_sandbox_resume_failed`, and the lease is still preserved.
- No persisted fields change. Leases saved by older releases follow the
same rule.
- The refused bootstrap retry admission for task Retry continuations is
a separate gap. This pull request does not change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:28:32 -06:00
DottaandClaude Opus 5.5 704b3f4d63 fix(server): a card accepted while its run settles keeps one continuation (#21094)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run ends while its task is in review, Paperclip checks that
the task still has a maintained review path. If it has none, Paperclip
queues one review-path recovery run
> - An approval card for a tool action sends its continuation through a
delivery receipt. The receipt waits until the source run has settled and
the action has finished
> - A person accepted a card while the source run was still settling.
The interaction was no longer pending, and the continuation wake did not
exist yet, so the review-path check found no path and queued a recovery
run
> - The real continuation then ran too, so the task got two follow-up
runs for one decision
> - This pull request counts an undelivered tool review continuation as
a maintained review path
> - The benefit is that a card accepted during settle produces exactly
one continuation run

## Linked Issues or Issue Description

**What happened?**
An agent posted a tool approval card and its run finished. The person
accepted the card 1.3 s later, while that run was still settling. The
review-path check saw no pending interaction and no queued wake, and it
queued an `issue_review_path_lost` recovery run. The accept continuation
was deferred behind that run and then started as a second follow-up.

**Expected behavior**
The accepted card's pending continuation counts as the task's review
path. Exactly one continuation runs.

**Steps to reproduce**
Have an agent in a task that is in review post a tool approval card.
Accept the card while the creating run settles and before the tool
action delivers its result. Before this change, a review-path recovery
run was queued in addition to the continuation.

## What Changed

- `server/src/services/issues.ts` (`listIssueReviewAttentionMap`): reads
tool review continuations for the review tasks in one statement. A
continuation counts as a queued review path when (a) its receipt is
undelivered, its card is resolved, and the requesting assignee can still
receive it (not paused, terminated or pending approval), or (b) its
delivered continuation wake, or that wake's run, is still active. The
path list leaves out a continuation whose wake or run is already listed,
so each continuation shows once. The path shows as "Queued tool action
review wake".
- `heartbeat-issue-liveness-escalation.test.ts`: a new embedded-Postgres
test runs a heartbeat for a task in review with an accepted card whose
tool action is still executing. It checks that no review-path recovery
wake is queued for that run. It then completes the action, delivers it,
and checks that exactly one continuation wake exists.

## Verification

- The new heartbeat test failed before the change (one
`issue_review_path_lost` wake) and passes after it.
- A new parameterized attention test covers an undelivered receipt, a
delivered continuation (listed and not listed in the wake list), a
pending card, a terminated assignee and a consumed continuation. The
terminated, pending-card and delivered cases were red against the
earlier revisions of this change.
- `pnpm vitest run
src/__tests__/heartbeat-issue-liveness-escalation.test.ts
src/__tests__/issue-review-attention.test.ts
src/services/recovery/review-path-recovery.test.ts
src/__tests__/attention-service.test.ts
src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts
src/__tests__/tool-gateway-service.test.ts`: 332 passed.
- Product evidence: a paid chat-eval run of
`questions-and-approvals.approve-tool-once` on a branch that combines
this change with three related fixes. The result is in a PR comment.

## Risks

- Every caller of review attention (review-path recovery, its
escalation, the attention feed and the approvals view) now sees a
pending tool review delivery as a covered path. That is the intended
state: the delivery service always sends one continuation for it, and
its periodic sweep retries a failed delivery.
- One more indexed query for each review-attention read. It reads only
undelivered receipts for the given tasks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:16:32 -06:00
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 a65644d1e1 fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks from a connected chat (for example GitHub pull request
reviews) show Retry when a run fails
> - A person's Retry on such a task first asks the chat retry owner to
repeat the exact failed chat request. #21038 made a refusal at the click
continue the task from saved work instead
> - The chat owner can also defer its source check (a busy identity,
endpoint, or agent row lock) and refuse it a few seconds later. Then the
route has already answered 202, and the task Retry receipt was marked
failed with no run
> - The task stayed blocked, and its execution blocker told the person
to send a message, while the interruption projection still offered Retry
> - This pull request makes that late refusal take the same board
continuation as a refusal at the click, and makes the blocker offer
Retry for recovered saved work
> - The benefit is that an accepted Retry always starts work, and the
two projections agree

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, about 20 blocked tasks of one busy GitHub review
agent (`paperclip_runner`, native remote runs) did not restart after
Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`,
a `chatActionId`, and `presentation.state: retry_scheduled`. No run was
created. The task Retry receipt changed to `failed` 5 to 10 seconds
later, and the task stayed blocked. The activity record of each click
(`chat.failed_run_retry_requested`) has `sourceAuthorizationPending:
true`, so the chat owner deferred its source check at the click. No chat
wake row was written, so the chat replay ended before admission. The
last run failed with `native_continuation_requires_reconciliation` /
`runner_remote_process_identity_unavailable`. Its saved work was
recovered (`workspaceRecovery.status: recovered`, remote lease destroyed
with a termination receipt). The projections disagreed: `interruption`
had `canRetry: true`, but `executionBlocker` had `canRetry: false` and
`nextAction: "Saved work recovered. Send a message to continue."`.

Root cause: when the chat retry action settles `failed`,
`releaseUnadmittedTaskRetryInput` in
`server/src/services/chat-channels/task-retry-input.ts` marked the
linked task Retry receipt `failed`. #21038 added a board-continuation
fallback only for a refusal inside the click transaction. A refusal
after a deferred source check (for example 8 lock-busy attempts on the
agent row, about 7 seconds) had no fallback. Separately,
`getExecutionBlocker` hid Retry for every chat-bound task (`canRetry &&
!chatBinding`), and `workspaceRecoveryDisplay` always said "Send a
message to continue" for recovered work.

**Expected behavior**

A Retry that is accepted starts work. If the exact chat replay is
refused after the click, the person's Retry continues the task from its
recovered saved work. The execution blocker and the interruption both
offer Retry, and neither tells the person to send a message to unblock a
failed run.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Make its latest native run
fail with `native_continuation_requires_reconciliation` and recover its
saved work.
2. Hold the chat identity advisory lock while you click Retry, so the
source check is deferred (`sourceAuthorizationPending: true`).
3. Hold a row lock on the agent while the chat retry processes, so the
deferred check fails 8 times.
4. The route returned 202. The chat action settles `failed`, the task
Retry receipt changes to `failed`, and no run starts.

**Paperclip version or commit**

`master` at a8580160 (the deployed revision). Reproduced on f30005440b.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native remote runs). The fix is in core Retry code
and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts`: new `settleRefusedTaskChatRetry`.
When a chat retry action settles `failed`, the linked task Retry receipt
becomes an ordinary board continuation if all of these are true: it is a
person's Retry (or an automatic Retry that the same person's click has
joined), its source run is still the latest run and its saved work is
recovered (system recovery `complete`, workspace recovery `recovered`),
the clicking person still has task access, the chat wake has no run, and
no saved or unadmitted chat input remains. Any other refusal still
settles the receipt as failed, so the existing source-edited, revoked,
and identity-expiry outcomes do not change. A paused agent does not fail
the receipt: it stays deferred and starts when dispatch allows. The
receipt drops `chatActionId`, stays `deferred_issue_execution`, notifies
the deferred-receipt worker, and records
`issue.retry_continued_from_saved_work`. Otherwise the receipt is failed
as before. `reconcileTaskChatRetryReceipts` uses the same check, so a
crash before settlement is also covered. Its failed write matches the
chat binding it read, so a stale pass cannot fail a receipt that
settlement already converted.
- The task Retry intent now records `initiatedBy` for an automatic
Retry, so an automatic Retry keeps the exact-replay requirement after
the click too.
- `server/src/services/chat-channels/task-retry-input.ts`:
`releaseUnadmittedTaskRetryInput` calls the task owner's settlement
instead of failing the receipt directly.
- `server/src/services/execution-blocker.ts` and
`workspace-recovery-receipt.ts`: for a failed or timed-out run whose
saved work is recovered, the blocker offers Retry also on a chat-bound
task. The text follows actual Retry eligibility: "Saved work recovered.
Retry continues from it." when Retry is offered, "Saved work recovered."
when it is not (for example cleanup quarantine), and "Send a message to
continue" only for a stopped run. Only the task blocker decides Retry.
The run-level projection (`execution-projection.ts`) has no task
context, so it says "Saved work recovered." and never claims Retry.
`execution-projection.ts` uses the same text.
- `ui/src/lib/activity-format.ts`: labels for the new activity.
- `doc/execution-semantics.md`: the late-refusal rule.
- Tests: a new real-service case in
`server/src/__tests__/chat-channels.integration.test.ts` that reproduces
the hosted state (remote saved workspace, destroyed lease receipt,
recovered recovery action, deferred source check, refused replay), for a
person's Retry and for an automatic Retry. A gate case per continuation
condition (unrecovered workspace and revoked access settle failed; a
paused agent stays deferred and runs after resume), a stale-reconcile
race case, and a joined-click case. Two existing text expectations are
updated.

## Verification

- Red before the fix: with only the source changes removed, the new test
fails on the projection (`canRetry: false`, "Send a message to
continue."). With only the Retry settlement removed, it fails with the
receipt `status: "failed"` and no run.
- Green after the fix: `npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "continues an
accepted GitHub task Retry from recovered saved work"` passes 2/2. The
person's Retry starts one queued run that resumes the preserved
workspace, and the automatic Retry still settles as failed.
- Focused suites pass: `task-retry`, `execution-projection`,
`execution-projection-recovery`, `workspace-recovery-retry-copyback`,
`chat-failed-run-retry-boundary`, `workspace-recovery-admission`,
`remote-workspace-recovery-receipt-fallback` (372/372), and other
recovery, blocker and continuation suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only. No POST was sent.

## Risks

- Behavior change: a person's Retry on a connected task that the chat
owner refuses after the click now starts a board continuation. It is the
same continuation #21038 already uses for a refusal at the click, with
the clicking person's authority, and dispatch keeps every normal gate.
- Receipts that already failed on staging stay failed. The next click on
those tasks takes the fixed path. No migration.
- The blocker now reports `canRetry: true` for chat-bound tasks only
when the failed run's saved work is verifiably recovered.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:57:42 -06:00
DottaandClaude Opus 5.5 cc6728e8de fix(recovery): continue upgrade-stopped Retry runs that the sweep refused or skipped (#21108)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Hosted servers are upgraded often. Each upgrade stops the old server
gracefully, and a run that is still preparing ends `interrupted`
(`server_shutdown_interrupted`) with bootstrap evidence that provider
work did not start.
> - #21078 let a run that a person started with Retry continue
automatically after such a stop. After it deployed, a read-only audit of
a staging server found four groups of Retry-started tasks that still did
not continue: some were refused on every sweep, some were never looked
at, and one Retry that a person pressed failed and was not retried.
> - Product rule: an upgrade never leaves a task that needs a manual
Retry, and a Retry a person presses must work. A refusal that cannot
succeed later must not repeat forever.
> - This pull request fixes each cause found, keeps every authorization
check of the continuation gate, and ends the refusal loop with one plain
escalation.
> - The benefit is that Retry-started tasks continue through upgrades,
including tasks earlier releases left waiting, with no manual step.

## Linked Issues or Issue Description

Refs #21078, #21046, #21019, #14987, #19384

**What happened?** On a hosted staging server after #21078 deployed
(read-only audit, run events and task activity):
1. A Retry-started task was refused every ~2 minutes ("The automatic
retry could not revalidate the original user continuation."). While
preparing, its Retry had switched to a fresh isolated workspace because
the saved files could not be restored (activity
`issue.fresh_workspace_selected` by the system, 40 minutes after the
Retry). That rewrote the Retry request's strategy from `continue` to
`fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts
only `continue`.
2. Two tasks whose Retry replaced a native session had no recovery event
for hours. The stranded-task sweep skipped them silently.
3. A task whose Retry selected a fresh workspace at admission
(`explicitUserContinuation.readiness`) was refused on purpose
("fresh-workspace Retries keep their own contracts"), yet that contract
had no path for a shutdown before provider work. The task stayed in
progress after every upgrade.
4. A person pressed Retry. The run failed before provider work with
`reusable_sandbox_resume_failed`, a transient failure that is already in
the bounded bootstrap retry lane. One second later the stranded-task
sweep escalated the task to `blocked`, which cleared the claim. Two
seconds after that, the run's own failure path scheduled its retry and
was refused because the claim no longer named the run.

**Expected behavior:** each of these continues automatically once, under
the person's original Retry. A refusal that cannot succeed later
escalates the task once with a working Retry.

**Root causes:**
- Cases 1 and 3: the continuation gate refused every fresh-workspace
Retry, whether the fresh workspace was chosen at admission or later in
preparation. That includes one that a shutdown stopped before it did
anything.
- Case 2: the sweep's generic skips run before the continuation. Two of
them can match only because of state that the replaced session left. An
unfinished session goal (`agent_task_sessions.goal_status`) is skipped
as "goal recovery owns it". A native finalization row is counted as a
live execution path by `hasActiveExecutionPath`. The person's Retry had
already started a fresh session, so neither owns the task. The audit
could not read the database, so which of the two applied on staging is
not proven; the tests cover both.
- Case 4: a race. The sweep does not wait while a terminal run's own
executor is still settling it, so it escalates before that executor
schedules the run's bounded retry.
- The refusal loop: after a refusal the sweep left the task
`in_progress` and tried again on the next pass.

**Paperclip version or commit:** master `482063aeca`.

## What Changed

- `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority`
renews a fresh-workspace Retry **only** when the parent run was stopped
by a graceful shutdown before provider work
(`isShutdownInterruptedPreparation`). The admitted `readiness` must name
the same Retry and source. A strategy that preparation switched to fresh
is rebuilt exactly as preparation records it: this Retry, its source,
and the source's leases. An automatic retry of the fresh attempt
inherits the selection from its own receipt. The successor's receipt
records the selection under `automaticRetry.readiness`. The successor's
`explicitUserContinuation` keeps its plain shape.
- `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads
that receipt for an automatic continuation, so the successor prepares
the same fresh attempt. A remote fresh workspace is keyed by the Retry,
so it is reused rather than created again. `freshReadinessSchema` is
exported and reused.
- `recovery/service.ts`:
- **Shutdown-stopped Retry preparations go first:** at the start of the
stranded sweep, every `in_progress` task whose latest run is a legacy
`server_shutdown_interrupted` preparation (`providerWorkStarted: false`)
carrying `explicitUserContinuation` is offered to the scheduler. It is
skipped when the agent is not invokable, the task has a live execution
path, a board-owned recovery action or a pause hold, or the run is still
settling. The unfinished-goal skip does not apply here, because the
person's Retry started a fresh session.
- `hasActiveExecutionPath` ignores a native finalization that a person's
Retry already replaced (`failureDetail.replacementDenied =
explicit_user_continuation` on a terminal run).
- The sweep skips a task while its latest terminal run is still being
settled: a live execution in this process, or another server's unexpired
controller lease. It checks again when that lease expires.
- A refused continuation (`continuation_user_authorization_missing`)
escalates the task once to `blocked` with the plain notice "Retry needed
… Press Retry to continue." Before, the task stayed in progress and was
refused again on the next pass.
- `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation
to the sweep.
- `doc/execution-semantics.md`: describes these rules.

**Failure classes admitted for a Retry-started run (case 4):** no new
class. The run's own bounded retry lanes are unchanged.
`reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`,
`plugin_worker_unavailable`, `environment_provider_unavailable`,
`repository_catalog_unavailable`, runner revision mismatch and the other
transient bootstrap failures were already retried before provider work.
The fix removes the race that took the claim away from that retry.

**Refusals that remain by design:**
- The task is reassigned, `done` or `cancelled`.
- Another run holds the claim, or newer or live work exists.
- The Retry actor, generation or run changed, or it is a chat or
queued-message Retry.
- The message was edited or deleted.
- The receipt is missing or revoked.
- The run had another interruption cause, or provider work started.
- A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its
one-run contract.
- A readiness that names another Retry or another source.

Each of these now escalates the task once instead of refusing on every
sweep.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` sources and green on this branch (checked by restoring
the master sources):
- "continues a fresh-workspace Retry that a shutdown stopped before
provider work":
- Runs twice: once with the fresh workspace selected at admission, once
with it selected at preparation.
- Two concurrent sweeps create one successor, which gets the claim, and
dispatch accepts it.
- `readFreshWorkspaceReadiness(successor)` returns the same fresh
attempt.
- A second upgrade stops the successor, and the next successor carries
the same attempt.
- "continues a shutdown-stopped Retry whose replaced session left state
behind": two cases, an unfinished session goal and a native finalization
that the Retry replaced.
- "escalates a refused shutdown continuation once instead of refusing it
on every sweep": two sweeps. The task ends `blocked` with one "Press
Retry to continue" notice and one refusal event.
- "leaves a failed Retry run to its own executor while that executor
still settles it":
- Production shape: `reusable_sandbox_resume_failed` and a live
controller lease from the server that ran the run.
    - The sweep leaves the task and its claim alone.
    - The run's bounded retry is then scheduled and dispatch accepts it.
- "keeps a fresh-workspace Retry's own contract when it does not match":
three refusal cases (other Retry, other source, failed rather than
stopped). Green on both.
- Existing suites pass:
- `task-retry`, `explicit-native-continuation`,
`heartbeat-retry-scheduling`, `heartbeat-process-recovery`,
`interrupted-preparation-hold-settle`,
`heartbeat-issue-liveness-escalation`, `issue-recovery-actions`,
`recovery-deadline-acceptance`, `legacy-continuation-authority`,
`workspace-recovery-admission`, `native-workspace-export-retry`,
`heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`,
`heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one
failure that a change in this branch fixed; rerun of the affected
suites: 560/560.
- The one failure came from the first version of the settling check,
which also waited for this process's own lease after a shutdown drain.
The check now uses this process's live executions and other processes'
leases only.
- `tsc --noEmit` for `server`.
- Required CI covers the full suite.

**After deploy, the next audit should show:**
- Tasks stopped by a shutdown while their Retry was preparing get one
`scheduled_retry` successor on the first sweep. This includes tasks
whose Retry selected a fresh workspace and tasks whose Retry replaced a
native session.
- They show no further "could not revalidate" events.
- A Retry-started run that fails transiently before provider work gets
its bounded retry, and the task is not escalated in between.
- A task whose continuation is truly refused is `blocked` once with
"Retry needed".

## Risks

- **Behavior change:** the sweep now offers shutdown-stopped Retry or
message preparations to the scheduler before its generic skips. All
authorization checks still run at admission, under the task and run row
locks. The scheduler reuses an existing successor, so two servers create
one continuation. The tests run two sweeps at once.
- **Settling wait:** a task whose latest terminal run still has another
server's live controller lease waits until that lease expires (at most
90 s) before the sweep acts. Normal turn starts are unchanged, and
nothing is added before a turn starts.
- **Escalation on refusal:** a refused continuation now moves the task
to `blocked` with a Retry notice, where before it stayed `in_progress`
with no path. Pressing Retry is a new, separately admitted request.
- **Rows from older releases:** no schema change and no backfill. A
shutdown-stopped fresh-workspace Retry is continued once. An older
successor receipt without `automaticRetry.readiness` behaves as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:41:25 -06:00
DottaandClaude Opus 5.5 849f109f62 fix(server): agent inbox reads now apply private-task visibility (#21092)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents read a person's Mine list with `GET
/api/agents/me/inbox/mine`, and their own queue with `GET
/api/agents/me/inbox-lite`
> - The company task list routes apply private-task visibility for the
reading actor
> - The two agent inbox routes should use the same visibility rules, so
that each route returns the same tasks for the same actor
> - This pull request makes agent inbox reads apply private-task
visibility
> - The benefit is one consistent visibility rule across task list
routes

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Agent inbox reads now apply private-task visibility.

**Subsystem affected**
Server API: agent inbox routes.

**Current behavior**
The agent inbox routes build their lists without the issue read
condition that the company task list routes use.

**Proposed behavior**
The agent inbox routes use the same issue read condition and per-row
read decisions as the company task list routes.

**Reason and benefit**
All task list routes apply one visibility rule.

## What Changed

- `server/src/routes/agents.ts`: `GET /api/agents/me/inbox/mine` adds
`issueReadSqlCondition` to its list query and keeps only the rows that
`access.decideReads` allows.
- `GET /api/agents/me/inbox-lite` adds the same read condition to its
list query.
- `inbox-mine-routes.test.ts`: a new embedded-Postgres test for the
agent Mine read with private and open tasks.
- `agent-permissions-routes.test.ts`: the mocked route test expects the
read condition and the read decisions.

## Verification

- `pnpm vitest run src/__tests__/inbox-mine-routes.test.ts`: 7 passed.
The new test was red before the route change and is green after it.
- `pnpm vitest run src/__tests__/agent-permissions-routes.test.ts
src/__tests__/openapi-routes.test.ts`: all passed.
- A chat eval run on a branch that combines this change with three other
fixes. The result is in a PR comment.

## Risks

- Low risk. The routes return only tasks that the reading agent can open
through the task routes.
- The Mine route adds one batched read decision for each request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:52:07 -06:00
DottaandClaude Opus 5.5 2a7fe6c585 fix(server): withhold a queued assignment run once its task moves to backlog (#21095)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Assigning a ready task to an agent queues an assignment run.
Paperclip does not queue one for a backlog task
> - An agent created a ready task, which queued an assignment run, and
then moved the task to backlog. The agent was busy, so the run stayed
queued
> - When the run started, the queued-run gate did not check for backlog,
and the run checked out and did the backlog task
> - This pull request makes the queued-run gate withhold a never-started
assignment run whose task is now in backlog
> - The benefit is that moving a task to backlog keeps it parked, as
when it is created in backlog

## Linked Issues or Issue Description

**What happened?**
An agent created a task without a status, so it was created ready with a
queued assignment run. The agent then moved the task to backlog. The
queued assignment run started two minutes later and completed the
backlog task.

**Expected behavior**
A queued assignment run that has not started does not execute a task
that has since moved to backlog. The run is cancelled with a clear
reason.

**Steps to reproduce**
Assign a ready task to a busy agent so that its assignment run waits in
the queue. Move the task to backlog. Let the agent become free. Before
this change, the assignment run started and checked out the task.

## What Changed

- `run-dispatch/domain/policy.ts` (`decideQueuedRunStaleness`): a queued
run whose wake reason is `issue_assigned` is stale when its task is in
`backlog`. The new error code is `issue_in_backlog`, with the reason
"Cancelled because the task moved to backlog before its assignment run
could start".
- `pre-dispatch-review-wait.ts`: lists `issue_in_backlog` as a
never-started dispatch cancellation, like the other queued-run gate
codes.
- `run-dispatch/adapters/postgres.test.ts`: a new test moves the task to
backlog after the run is queued and checks that `cancelStaleQueuedRun`
cancels it before it starts. A control test checks that a comment wake
on a backlog task still starts.

## Verification

- The new test failed before the change (`not_stale`) and passes after
it.
- `pnpm vitest run src/modules/run-dispatch/
src/services/pre-dispatch-review-wait`: 185 passed.
- Product evidence: a paid chat-eval run of
`questions-and-approvals.accept-one-proposal` on a branch that combines
this change with three related fixes. The result is in a PR comment.

## Risks

- Low risk. The rule applies only to assignment wakes. Comment,
interaction and recovery wakes on a backlog task keep their current
behavior.
- To start the task, a person or agent moves it out of backlog. That
wakes the assignee with `issue_status_changed`, as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:27 -06:00
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00
DottaandClaude Opus 5.5 6d8f59d494 fix(api): name the secret binding proposal as the way to give an agent a company secret (#21090)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents find product API routes through the runner API search,
which ranks operations by their OpenAPI summary and description
> - In the chat eval `secrets.configure-company-secret`, a person asked
the CEO to let Morgan use an existing deployment API key
> - The agent searched for agent configuration and environment variable
routes, tried `PATCH /api/agents/{id}` (403) and spent 3.5 minutes
before it used `POST /api/agents/me/secret-proposals`
> - The summary "Propose a company secret or agent secret binding" did
not match how agents describe the task
> - This pull request rewrites the summary and adds a description that
names the binding fields
> - The benefit is that the first search finds the supported route, so
the turn is short

## Linked Issues or Issue Description

**What happened?**
Runner API search did not rank `POST /api/agents/me/secret-proposals`
for the queries an agent uses to give another agent a company secret
("agent configuration update env secret", "agent environment variables
set secret ref"). The agent probed other routes for 3.5 minutes.

**Expected behavior**
Those searches return the secret binding proposal route in the first
results, and the result explains `secretId`, `configPath` and
`targetAgentId`.

**Steps to reproduce**
Call `searchRunnerApi({ query: "agent configuration update env secret",
limit: 3 })` on master. The proposal route is not in the results.

## What Changed

- `server/src/routes/openapi.ts`: new summary and description for `POST
/api/agents/me/secret-proposals`. The description states the
reporting-line target rule, the `env.<KEY>` config path, and that a
board member approves the binding.
- `server/src/__tests__/secret-binding-discovery.test.ts`: the four
queries return the route in their top three results. The test fails on
master (red) and passes with this change (green).

## Verification

- `npx vitest run` on 24 server guidance, discovery, OpenAPI and runner
API test files: 2305 passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid chat eval run: see the PR comment.

## Risks

- Low. Only OpenAPI text changes. Other search results can move down by
one place for queries that share these words.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:46:26 -06:00